Migrate document extraction to xberg 1.0#433
Draft
tobocop2 wants to merge 91 commits into
Draft
Conversation
tobocop2
changed the title
Migrate from kreuzberg 4.x to 5.x. Replace lilbee's in-process PDF rasterization, OCR orchestration, and per-page OCR cache with kreuzberg's extraction pipeline, exposing lilbee's vision model as a custom kreuzberg OCR backend. Single extraction path; per-page citations preserved. Fetch the vision mmproj on pull when the main GGUF is already installed (bb-7yd). Start the fleet reliably and accept kreuzberg's JSON-string OcrConfig callback (rc.35).
Derive supported formats from kreuzberg.list_supported_formats() (PDFs and images grouped, other formats keyed by extension) instead of a hand-maintained 38-entry map, so lilbee covers all ~108 formats kreuzberg extracts; source code still routes to the tree-sitter chunker. Add cfg.ocr_language (writable; env LILBEE_OCR_LANGUAGE), fixing image+tesseract OCR that kreuzberg 5.x errors on with an empty language list. Drop the dead max_concurrent_extractions knob and the OcrConfig str|dict shim. Regroup ingest types; add the AGENTS.md canonical-module-layout rule and a diff-scoped scatter trigger. Update the README format table; add coverage tests.
tobocop2
force-pushed
the
feat/kreuzberg-5
branch
2 times, most recently
from
June 25, 2026 01:56
1fcc3c4 to
a5bee1e
Compare
The dependency was renamed kreuzberg -> xberg (now v1.0.0rc1). Rename every reference across code, tests, docs, README, the marketing site, AGENTS.md and the build script; the internal type module is now xberg._xberg. Build the semantic chunking embedding config with the real EmbeddingModelType.preset() constructor (alef #147) instead of the bare-string shorthand. xberg's .pyi omits those per-variant constructors (alef-m07 / kreuzberg-7xq), so the one call carries a narrow type-ignore. API is otherwise compatible; the oracle differential shows identical extraction/OCR/chunking vs the kreuzberg-5.x baseline. The local wheel source is a placeholder until xberg publishes to an index.
…cation (bb-548) xberg ships the footnote/citation API (xberg-io/xberg#649) built for lilbee's wiki use case. Replace lilbee's own excerpt verification and unmarked-claim scanning with xberg.verify_excerpt / xberg.find_unmarked_claims, dropping the lowercasing _normalize, the _is_content_line helper, and the CITE_RE/INFERENCE_RE regex usage. Verification is now case-sensitive (xberg's contract) instead of case-insensitive; citations.py's check was already case-sensitive so only the wiki/citation.py path changes behavior. Domain wrappers (CitationRecord, render_citation_block, chunk page/line resolution) stay.
…48 Surface 1) Semantic chunking previously had xberg download its own ONNX preset embedder for topic-boundary detection. Expose the fleet's OpenAI-compatible embeddings endpoint via a new LLMProvider.embedding_endpoint() (FleetProvider returns the placed EMBED replica's /v1 base + bare model id; RoutingProvider delegates; SDK returns None), and build the semantic EmbeddingConfig with EmbeddingModelType.llm() pointed at it, falling back to the preset when no fleet endpoint exists. Verified end to end: a doc ingested with semantic chunking embeds through the fleet and is searchable. Also make lilbee's package __getattr__ resolve real submodules (e.g. lilbee.providers) instead of only Lilbee, so attribute-based access (mock.patch string targets) works without a prior import; unknown names still raise AttributeError.
tobocop2
changed the title
tobocop2
changed the title
Scanned-document OCR isn't English-only: Tesseract covers 100+ languages via
cfg.ocr_language (env LILBEE_OCR_LANGUAGE, e.g. eng+deu, verified on a German
scan) and vision models read many scripts. Note it in the README feature line and
supported-formats table, the marketing site's scans/OCR card, and the usage OCR
comparison. ('Ask in plain English' is unchanged: it describes phrasing a query,
not a content-language limit.)
…the +-delimited format The OCR language setting is a list, but the writable surfaces (lilbee set, the TUI settings screen, the HTTP and MCP config APIs) persist list values to config.toml newline-joined. The validator only split on '+' and ',', so a multi-language value like eng+deu was written as "eng\ndeu" and reloaded as the single bogus token ['eng\ndeu'], which Tesseract rejects. Split on newlines too, matching the crawl list-field validators. Document the format consistently across surfaces: '+' is the canonical join (Tesseract's own convention), commas also accepted. README, the OCR docs, the settings help text, and the field comment now say so, and the docs show the env, CLI, config, TUI, and HTTP/MCP entry points.
Annotate the two EmbeddingModelType.llm()/.preset() type-ignores with the sanctioned style-check: allow-smell tag. They cover xberg constructors that exist at runtime but are missing from the .pyi stub (alef #147), not owned-attribute smells, so the diff-scoped gate's default advice does not apply. Also reformat test_ingest.py to the locked ruff so format-check is clean.
…gin backend Replace the EmbeddingModelType.llm fleet-endpoint wiring with xberg's plugin embedding backend, the same mechanism the vision OCR backend already uses. xberg's semantic chunker now calls back into lilbee's embedder (provider.embed) to detect topic boundaries, so the model that vectorizes chunks for retrieval is the one that decides where they split -- in every case, not just when a fleet endpoint exists. xberg's bundled ONNX preset is no longer used. LilbeeEmbeddingBackend (name/initialize/dimensions/embed/shutdown) is registered in app.services.sync_embedding_backend, mirroring sync_vision_ocr_backend: it reads cfg.embedding_dim live and routes through provider.embed (so an embedding-model swap needs no re-registration), and re-binds whenever the provider is rebuilt. The retrieval embedding pipeline (Embedder, provider.embed, the store) is untouched. Drops the now-unused EmbeddingEndpoint dataclass, the embedding_endpoint protocol method and its three implementations, and the fleet client's base_url/model accessors (net -136 lines).
Point the local xberg source at the all-PRs build. It declares the EmbeddingModelType.plugin/llm/preset/custom constructors in the stub (the alef #147 gap), so the attr-defined ignore and its allow-smell tag are gone. One arg-type ignore remains: the public EmbeddingConfig still types model as the legacy str|int|LlmConfig alias rather than the EmbeddingModelType class its own .plugin() returns (upstream stub inconsistency; the constructor takes the class instance at runtime).
…kreuzberg-5 # Conflicts: # tests/test_fleet_provider.py
The merge pulled in the multi-GPU placement screen (#438 from feat/local-model-api), whose tests covered the worker error paths but not the _spec_from_editor guard branches, the unknown-button no-op, the clear single-flight guard, or the clear worker's error path (16 lines). Add error-path tests so the merged tree holds the 100% gate.
The all-PRs xberg wheel passes the OcrConfig to register_ocr_backend's process_image as a native object (alef typed trait callbacks), not the JSON string the trial wheel sent. The backend still json.loads-ed it, silently got nothing, and dropped the request token -- so per-page OCR timeout and progress ticks were lost on scanned-PDF vision OCR (a regression vs the old pdf_ocr path this branch replaced). Read vlm_prompt and backend_options off the object directly; verified the timeout and progress callback propagate again. Tests now pass a real xberg OcrConfig instead of a hand-built JSON string.
Replace the ELv2 text with the MIT License and switch every license identifier: pyproject (license = MIT), README badge + section, the marketing site, flake.nix (licenses.mit; drop the now-unneeded unfree allowance since MIT is free), the AUR PKGBUILDs + .SRCINFO, the flatpak metainfo, and the scoop bucket manifest.
Port lilbee's extraction layer to xberg's redesigned API (extract(ExtractInput) -> ExtractionResult.results[ExtractedDocument], async-only): new data/xberg_extract.py bridge, chunk.py/extract.py/vision_ocr_backend.py updated, process_image returns an ExtractedDocument. Text extraction + chunking verified working. BLOCKED on two xberg wheel defects (filed in ~/projects/kreuzberg): kreuzberg-6a2 (SIGBUS in the PDF OCR pipeline aborts the process) and kreuzberg-9n0 (OcrConfig.backend_options stub dict vs runtime str). OCR ingest + the test-suite migration + the full gate are deferred until the wheel is fixed and rebuilt.
…bsolete tests Complete the lilbee port to xberg's async extract()/ExtractedDocument API now that the rebuilt wheel fixes the OCR SIGBUS (kreuzberg-6a2) and accepts dict backend_options (kreuzberg-9n0): - vision_ocr_backend: process_image returns a native ExtractedDocument; backend_options read as a dict; drop the unused json import - xberg_extract: typed _input/_run helpers - services: register_ocr_backend arg-type ignore (xberg's OcrBackend Protocol lists file/document methods an image backend never needs; duck-typed at runtime) - tests: repoint extract mocks to the xberg_extract bridge, assert ExtractedDocument + dict backend_options, create files for read_bytes; drop tests for upstream-removed pdf_ocr and elastic ingest-pool (#447)
Bump the local-dev xberg pin from the rc1 cp312-only wheel to the rc2 cp310-abi3 wheel. abi3 resolves across Python 3.11/3.12/3.13, so the integration no longer pins a single interpreter. Validated against rc2: all xberg imports resolve, image OCR and PDF-OCR-with-layout run end-to-end (the OCR future Box::pin fix is present in rc2, no SIGBUS), lint/typecheck/format clean. An A/B full-suite run (rc1-abi3 vs rc2-abi3) produced identical results, so rc2 is a behavior-for-behavior drop-in.
…tests hermetic The developer's real ~/Library/Application Support/lilbee/config.toml was bleeding into the test suite two ways: 1. Import-time load. The cfg singleton is built at import, before conftest's autouse fixture set LILBEE_SKIP_TOML_CONFIG, so values like vision_model and memory_enabled came from the dev's machine. Tests asserting defaults (e.g. 'no vision model -> tesseract OCR backend') failed. Fix: set the skip flag at conftest module top, before cfg is imported, matching CI (which has no config.toml). 2. CLI/MCP overlay. overlay_persisted_settings() re-read config.toml directly and ignored LILBEE_SKIP_TOML_CONFIG, unlike the pydantic-settings source, so a CLI invocation re-applied disk values mid-command (memory_enabled flipped False->True during 'memory add'). This is also a real product inconsistency: the documented escape hatch worked on one config-read path but not the other. Fix: overlay_persisted_settings() returns early when the flag is set, giving parity across import-time load, CLI callback, and MCP server. Tests that specifically exercise the overlay-applies path (test_cli TestApplyOverrides, test_mcp TestInit, test_settings TestOverlayPersistedSettings) opt back in via a new overlay_reads_config_toml fixture, and a regression test covers the skip-flag no-op. Also reformats a pre-existing line-wrap drift in ingest/pipeline.py that was failing format-check. Full suite 100% coverage. bb-e7d.
…s-test mock leak
chunk_text() flaked nondeterministically under the parallel suite, returning
mock content ('Some extracted text.') instead of real output -- e.g.
test_content_before_first_heading asserting 'Preamble' in the result, or
test_long_text_produces_multiple_chunks getting a single chunk. Root cause:
extract_document() (sync) resolves aextract_document via module-global lookup,
so a mock.patch of lilbee.data.xberg_extract.aextract_document intercepts the
sync path too, and chunk_text() uses the sync path. The ingest/handler suites
patch that global at 50+ sites; under xdist a patch occasionally stays active
into an unrelated test on the same worker (confirmed by a captured traceback;
serial and small-xdist runs never leak). An autouse fixture now resets both
extraction globals to their pristine functions before every test, so an escaped
patch can't propagate. Verified: three consecutive full-suite runs clean after
the change. bb-ql1.
…kreuzberg-5 # Conflicts: # tests/test_tui_placement.py
Brings the latest local-model-api work onto the xberg (kreuzberg 5.x) branch: the MIT relicense (#443), the blast-radius audit fixes for fleet lifecycle, library scoping, role registry, and OpenAI params (#471), the fleet placement editor redesign (#469), the compat (pre-Haswell) packaging channels, upload-by-content ingest, and the release bump to 0.6.66b506. Conflict reconciliation kept the xberg-native OCR architecture as the direction while carrying local-model-api's improvements onto it: - extract.py: kept the single-pass xberg extractor (xberg OCRs scanned pages through the registered backend; no lilbee-side pdf_ocr/rasterize loop), and layered on local-model-api's active_config() library scoping and the to_ingest_thread offload helper. - providers/fleet/provider.py: kept the slim vision_ocr primitive (the pagination-owning pdf_ocr path is gone under xberg) plus local-model-api's chat_bounded total-deadline handling; dropped the now-unused concurrent.futures imports. - app/services.py: kept both the xberg OCR/embedding backend registration and local-model-api's per-call build_services/services_scope. Moved the backend registration into build_services so the library API's per-instance containers register their provider too, not just the singleton. - data/chunk.py: xberg 5.x ChunkingConfig fields (max_characters/overlap, plugin embedding backend) with active_config() scoping. - Removed data/ingest/ocr_cache.py and the old OCR-fallback tests; xberg owns page OCR and caching now. Full suite green at 100% coverage.
…t know them The evaluation handoff identified the remaining pinpoint gap: a docket-style reference lives in a document's text, and often nowhere in its filename, so filename-token resolution can never route it. When no filename resolves, a BM25 probe for the reference now routes to the single source that owns at least three quarters of the top hits; a reference scattered across documents (a number cited in related filings) stays topical, and several filenames legitimately carrying the token (a split document) remain ambiguous without probing.
… target The known-item route pulls a whole document, and the context budget trimmed it against the configured context target; the server can come up smaller (the fleet divides context across slots), so the routed document overflowed the real window and the request hard-failed with an HTTP 400. The budget now uses the provider's served per-slot context when known, still capped by an explicit num_ctx, falling back to the configured target when no engine is running. The topical path never exposed this because eight chunks cannot overflow any plausible window.
…xact typed counts Count questions over typed records (how many distinct X, how many X per Y) were declined because the store held no typed data: concepts are untyped strings and a general NER tag set has no notion of the identifier types a specific corpus carries. This adds the designed two-phase mode, default off. Phase 1 (lilbee entities induce, cheap): an LLM reads a stratified sample of the indexed chunks and proposes the corpus-specific type schema, written to a reviewable entity_schema.json; the artifact is the contract and is inspected before anything expensive runs. Phase 2 (lilbee entities backfill for the existing index, plus sync-time extraction behind the entity_extraction flag): each type is found by the cheapest extractor that serves it: compiled regex for identifier-shaped types, spaCy labels when the graph extra's model is available, an LLM only for the remainder, with extractor kinds degrading independently. Backfill reads chunk text from the store, so no re-ingest and no re-embedding. The entities table (entity, type, normalized_value, source, page, chunk_index, confidence) is additive: no existing schema changes, no migration, and stores without it read as nothing extracted. Source replacement deletes entity rows with the source's chunks. Values normalize for grouping (casefold, leading-zero-insensitive numerics). Aggregate routing answers the new shapes exactly and without a model: distinct counts scan per type, association counts group by chunk co-occurrence, and questions whose nouns don't resolve against the reviewed schema decline while naming which types are countable. Also fixes the overflow the private-corpus retest isolated: context budgeting now assumes dense-text tokenization (3 chars/token; 4 let a document whose true cost exceeded the window pass untrimmed), and a typed provider context-overflow triggers one refit-and-retry instead of failing the question.
…LI, drop a defensive getattr Blast-radius findings on the entity mode: the CLI duplicated the extractor's sample-size constant; the CLI commands doc had no entities rows; spaCy docs always carry .ents so the getattr defaulted nothing; and the streaming path's no-retry-on-overflow decision is now stated where it lives.
The first real-corpus validation found extraction far too sparse: induced regex patterns anchor with ^...$, which matches only a token that IS the identifier, while identifiers in OCR running text appear inline with adjacent punctuation and were never found. A fully-anchored pattern now full-matches each identifier-shaped token instead of the chunk (partial tokens still rejected; alternations of anchored branches fall through untouched), and the induction prompt forbids anchors outright, so both induced and hand-corrected schemas work as written. Separately, a question noun that reaches a type through a synonym was silently answered with a different quantity (counting tail numbers is not counting flights). Distinct-count answers now state which type they measured and that it may not be the asked-for quantity whenever the noun is not the type's own name.
Backfill appends (sync owns per-source replacement), so re-running it after a pattern fix double-counts every entity. The flag clears the table before extracting, which is the natural workflow when iterating on a schema against an existing index.
The evaluation campaign observed fully deterministic retrieval with expansion nominally enabled and could not tell whether expansion ran; the skip decision and variant count were silent. Both now log at info so a single run answers it.
…e hybrid path Graded relevance evaluation (LLM-judged precision@20, equal k, same judge and index across arms) showed the convex score fusion regressing raw hybrid precision about 20% against the old RRF ordering, at every blend weight, so the regression was structural rather than tunable. Two mechanisms, both reproduced in synthetic tests: - MMR newly ran on the hybrid path (it never did before this branch) and at the default lambda it diversifies away exactly what a lexical query's relevant set looks like: many mutually similar passages quoting the same identifiers. Removing it also exposed that the convex formula never actually guaranteed lexical-only survival; MMR had been rescuing the invariant test by accident. - Convex combination of normalized raw scores gives every dense neighbor a score floor (cosine similarities sit in a high narrow band) that crowds out lexically-certain rows the vector arm never saw. fuse_arms is now equal-weight reciprocal rank fusion expressed as a canonical [0,1] score (mean of per-arm (K+1)/(K+rank), K=60), and the arms stay exactly top_k deep: deep pools flood rank fusion with both-arm mediocre rows and bury single-arm certainty, which the invariant test demonstrates. fusion_alpha and fusion_overfetch_floor were knobs of the convex formula and are removed with it; both were born on this branch. Everything else the branch added stands: ANN recall recovery on the hybrid vector arm, canonical score downstream, dedup keeping both provenance fields, the far-row drop that spares BM25-supported rows, intent routing, and context budgeting.
Blast-radius sweep after the fusion reversal: the score-field comments, the pipeline flow diagram, the reranker section, and the tuning table still described convex score fusion and MMR on the hybrid path. Also corrects min_relevance_score guidance to the normalized rank scale (an arm's top hit scores 0.5, useful floors start near 0.4) and two drifted tuning-table rows (diversity default, expansion-skip semantics).
…nfigured With a cross-encoder configured, the fused ordering is only candidate generation, but the ask path retrieved just top_k rows (12 by default, up to 2x with expansion), so the reranker's configured candidate count (60) never materialized and the cross-encoder could only reorder a dozen rows. build_rag_context now retrieves max(top_k, rerank_candidates) when reranker_model is set. The deep pool is safe there precisely because the reranker re-scores it; without a reranker the pool stays at top_k, where the fused order is final and deep rank-fused pools measurably bury single-arm certainty.
A query naming one document got routed to that document only through build_rag_context, so retrieval-only surfaces (HTTP /api/search, MCP search) returned similarity neighbors of the question's wording, which for 'summarize survey_482.pdf' is mostly noise. The differential probe exposed this: its known-item arm drives bare search and measured 0.000 on both sides because neither ever routed. Searcher.search now consults the known-item route (gated on intent_routing, skipped under explicit mode: prefixes, which force a strategy). A resolved document returns its head in document order at full confidence, capped at the standard search return budget. The ask path is unchanged; its earlier route check still short-circuits before search runs.
test_enter_noop_outside_lilbee_app claimed to run without a TaskBarController, but its host class is a LilbeeApp subclass, so the controller was real, and with the installed-model scan patched to empty, Enter on a card submitted a genuine featured-model download. The worker thread outlived the test by a whole multi-gigabyte pull and starved whichever TUI test shared the xdist worker next, which is why a different test timed out each full-suite run while everything passed in isolation. The test now mocks start_download, asserts the submission it was silently making, and is renamed to say what it exercises. A new autouse conftest guard fails any test that leaves a live task-bar worker thread behind, so this class of leak fails at its owner instead of a random downstream victim. Two consecutive full-suite runs green after the fix; the flake previously hit three of four runs, and suite wall time dropped ~40% with no download saturating a worker.
The reranker always blended its scores with the retrieval fusion signal (position-aware, fusion-weighted up to 70% at the top), so measuring the reranker's own effect was impossible: a flat mean with heavy per-query reordering is indistinguishable between the reranker not separating relevant from plausible and the blend capping its upside. rerank_blend (default true) keeps the existing behavior; off applies the cross-encoder's ordering directly and stamps the normalized reranker score unblended. Also fixes a drifted default in the tuning docs (rerank candidates 60, not 20).
…out view transitions in tests A scheduled catalog refresh could fire while the screen was composing or being dismissed, query tab containers that don't exist, and crash the app with NoMatches; the catalog-quit test only ever passed by racing out before the refresh fired. Both refresh entrypoints now no-op when there is nothing mounted to paint into, placed before the row-cache update so a skipped refresh never marks itself done. The bracket-key navigation tests pressed through six screens on fixed pauses, but the view-switch guard deliberately drops a switch while the previous transition is in flight, so under a loaded xdist worker a keypress vanished and a random screen assertion failed. The cycling tests now wait for each transition, and the catalog-quit test waits on conditions instead of fixed pauses.
…ommand Typed entity extraction was a three-step operator workflow: run induce, review a JSON schema by hand, run backfill. That contradicts how every other lilbee feature behaves, so turning entity_extraction on is now the whole interaction. Sync induces the schema from the indexed chunks on first run, extracts across the index, and picks up new files at ingest; editing entity_schema.json is detected by digest and re-applied on the next sync. Full passes always clear prior rows first, so interrupted passes and schema edits never double-count. No chat model at induction time defers with a log line and retries next sync. The entities command and its tests are gone; the same induction, extraction, and storage code now runs from the pipeline. Entity status (types, extracted rows) surfaces on all four fronts: CLI status, the TUI status screen's storage panel, GET /api/status (the response model previously would have silently dropped the section), and the MCP status tool. Schema induction is pinned to deterministic non-thinking generation via a new validated 'think' chat option: llama-server maps it to chat_template_kwargs.enable_thinking, hosted-API translators drop it. End-to-end on a real scanned manual, a small thinking model burned its entire token budget looping inside <think> and emitted no JSON; think=false plus temperature 0 induced clean schemas on both test corpora (a PDF manual and twenty public-domain books, the latter extracting 337k entity rows across 7.5k chunks).
Quoted document names now require a matching quote pair detached from surrounding words, so contractions and possessives no longer pair into phantom references, and double-quoted names may contain apostrophes. Noun/type resolution moves to a shared noun_variants helper covering irregular and -ies/-es plurals, replacing two divergent naive pluralizers. Docstrings that still described the abandoned manual schema-review workflow now describe the automatic lifecycle.
… search
With nothing indexed, every question got the grounded refusal ('I couldn't
find anything in the indexed documents'), which implies a search ran and came
up empty. There is nothing to search, so ask, ask_raw, ask_stream, and the
server's SSE stream path now short-circuit to guidance pointing the user at
adding documents first. The check lives at the searcher choke point so the
TUI, CLI, HTTP, and MCP surfaces all answer consistently, and the copy stays
surface-neutral. The store is never searched and the model is never called on
the empty path.
ensure_entities and its digest helpers read the process-global cfg, so under the library API (Lilbee(config=...)) the sync-time lifecycle saw entity_extraction=False and silently skipped induction and extraction, and would have read its schema artifacts from the global data_dir. Read active_config() throughout, matching the rest of the ingest path.
The empty-library short-circuit left this helper's mocked searcher with library_empty unconfigured, so the truthy MagicMock diverted the stream before the embedding mismatch could surface.
…uality thresholds LILBEE_OCR_FORCE was meant to re-OCR every page, but on xberg rc25 it did nothing for born-digital PDFs. It relied on setting an impossible OcrQualityThresholds floor so every page would fail the text-quality gate and fall through to OCR. rc25 short-circuits any page that already has a clean text layer before that gate is consulted, so the floor never fired and forced runs produced byte-identical native extraction with the GPUs idle. Drive the force flag through ExtractionConfig.force_ocr instead, which overrides the text-layer short-circuit and OCRs every page. Scope it to the vision backend so it stays a GPU re-OCR lever; tesseract and OCR-disabled runs are unchanged and normal runs keep native-first extraction. Add a behavioral regression test that drives real xberg over a born-digital PDF through a counting backend: zero OCR calls unforced, one per page under force. This pins the xberg contract so a future bump cannot silently disable forced re-OCR the way rc25's short-circuit did.
The induced schema is machine state, so it now lives in a single-row _entity_schema table in LanceDB and travels with the index; the entity_schema.json artifact, the applied-digest marker, and the edit-detection machinery are gone. There is nothing to review, edit, or keep adjacent to the database, matching the fully-automatic direction: if induction quality needs improving, the fix belongs in induction. Induction now also collapses duplicate extractors (a small model can propose several names for one pattern, filling the table with identical row sets). A rebuilt index drops the schema with everything else and re-induces on the next sync.
# Conflicts: # tests/test_tui.py
The schema was induced once and frozen, so a library that later grew a new kind of document kept answering with the type set induced on day one: new values of known types flowed in at ingest, but no new type ever appeared. The schema row now records the document count it was induced from, and once the corpus has grown materially past that, sync re-induces from a fresh sample and unions in types the schema has never seen. A type is identified by what it extracts, not the name a model gave it, so a rename of a known pattern adds nothing and induction reuses the same key to collapse duplicate proposals. Re-induction that finds nothing new records the corpus size and skips the pass; one that adds a type re-extracts under the existing replace semantics, so counts never double.
# Conflicts: # src/lilbee/retrieval/query/formatting.py # src/lilbee/server/handlers/rag.py
The shared resolver inlined the same dims-match expression the helper already owned, and the chat stream's delegation left the helper with no callers.
The test waited for the sync flag and returned while the crawl worker's tail (browser teardown, slow on Windows) was still running, tripping the leaked-worker guard whose short grace is meant for mocked no-op targets. A test that runs a real task-bar worker owns the thread; join it before returning.
…erge-trial # Conflicts: # src/lilbee/core/config/model.py # src/lilbee/data/ingest/types.py # tests/conftest.py
…551) Tables in PDFs were only ever indexed as flattened text, so row/column meaning was lost to both BM25 and the embedder. With table_extraction on, xberg's table structure recognition runs during extraction and each table is indexed as its own chunk carrying the markdown serialization (new chunk_type 'table', covered by the 'raw' search scope), with long tables split header-repeating. With layout_detection on, page text is ordered by detected reading order and the running header/footer bands are stripped. Both flags are off by default and marked reindex-required; the xberg pin moves to 1.0.0rc29. bb-9t16 bb-orox
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
lilbee pinned
kreuzberg<5. The dependency moved to a major release that reworked the extraction/OCR API and was then renamed to xberg (now1.0.0rc22). lilbee needs to run on it, and the upgrade lets lilbee drop a lot of what the library now does natively.Solution
kreuzberg→xbergrename across code, docs, README, and the marketing site.xberg.list_supported_formats()instead of a hand-maintained 38-entry map; source code still routes to the tree-sitter chunker.cfg.ocr_language, default["eng"], envLILBEE_OCR_LANGUAGE), which also fixes image + tesseract OCR that errored on an empty language list.register_embedding_backend), and the semantic chunker selects it withEmbeddingModelType.plugin(...), so the model that vectorizes chunks for retrieval is the one that decides where they split.Also drops the dead
max_concurrent_extractionsknob and an OcrConfig string/dict shim, regroupsdata/ingest/types, and makes the package__getattr__resolve submodules.Full suite green at 100% coverage. Tesseract and vision (LightOnOCR) OCR, JPEG-2000 scans, and semantic-chunking-through-the-fleet are verified end to end, and a differential against the pre-migration build shows identical extraction/OCR/chunking.
xberg now ships to PyPI: the dependency is pinned
>=1.0.0rc22and resolves from the index like every other dependency, so the[tool.uv.sources]override that pointed at a locally built trial wheel is gone (still cp310-abi3, one wheel covering Python 3.11 through 3.13). The Intel Mac release cell drops its compat-index xberg pin too, since rc22 publishes an x86_64 macOS wheel. Upstream also fixed the Protocol mismatch: the publicOcrConfigis now the type theOcrBackendProtocol accepts, so a backend typed against the public API conforms and lilbee's integration carries no xberg# type: ignore. Two upstream limitations were re-verified as still present on rc22 and remain worked around: backend callbacks do not receive the caller's contextvars, so per-request OCR state travels as a token inbackend_options, andbackend_optionsreaches the backend as a JSON string rather than the dict the caller passed, which the config view handles.Along the way this also makes the macOS floor of each binary honest.
MACOSX_DEPLOYMENT_TARGETonly ever pinned the Nuitka launcher, but Nuitka bundles every dependency's dylib and dyld enforces each one'sminosindependently, so the real floor was whatever the newest bundled dylib demanded and nothing checked it. The 11.0 the binaries claimed was already untrue: uv resolves wheels for the build runner, so a macOS 15 image pulls numpy and scipy asmacosx_14_0and the binary could not launch below macOS 14. Each macOS asset now declares the floor it actually runs on, and a build step reads theminosof every bundled dylib and fails if any exceeds it, so a dependency that raises the floor breaks the build instead of the user's Mac. The Intel cells resolve wheels for the x86_64 target rather than the runner; numpy, scipy and pyarrow all publish older-built x86_64 wheels that run just as well on the macOS 15 image, which brings that floor down to 12.0 and reaches the Ivy Bridge and Haswell Macs that cap at macOS 12, the reason to ship an Intel build at all.Do not merge until a fixed xberg is released. Its published macOS wheels are built with
MACOSX_DEPLOYMENT_TARGET=15.0(the.socarriesminos 15.0), so no Mac below macOS 15 resolves a usable wheel and the macOS release cells will fail the floor check. That is deliberate: it blocks a binary that would refuse to launch rather than shipping one. Tracked upstream, and it is a one-line change to the publish workflow's deployment target.