fix(xberg-wasm): real per-line OCR geometry through the wasm bridge to web-ui#1253
Closed
jamon8888 wants to merge 238 commits into
Closed
fix(xberg-wasm): real per-line OCR geometry through the wasm bridge to web-ui#1253jamon8888 wants to merge 238 commits into
jamon8888 wants to merge 238 commits into
Conversation
added 30 commits
June 29, 2026 11:30
…on, and tests - Add PII detection (11 categories: email, phone, SSN, IBAN, credit card, IP, dates, postal) - Add redaction strategies: token_replace, mask, AES-256-GCM encrypted rehydration maps - Add structured output: redacted DOCX/PDF/text + category-breakdown report per file - Add stdio and SSE/HTTP MCP transports - Add RAG store singleton wrapping xberg-rag-node openSqlite - Fix extract.ts: camelCase ExtractionConfig fields, optional results array, Chunk types - Fix ingest.ts: Keyword.text field, results?? [], shared redaction modules - Fix cache.ts: null guards on MODELS array index access - Add 21 tests: redaction, encryption round-trip, module export checks - Add xberg-rag-node npm package: index.js loader, index.d.ts, package.json
…raction_stats, import_collection)
- rules/upstream-sync.md: branch model, conflict surface table, rebase procedure, how to regenerate CLAUDE.md after editing .ai-rulez/ - context/local-additions.md: full map of fork-specific paths (xberg-rag, xberg-rag-node, mcp-server), Windows build notes, file-level tour of mcp-server/src/
…rkflow - Add .ai-rulez/domains/rag/ with DOMAIN.md, rag-store and rag-embeddings rules - Add .ai-rulez/domains/mcp-server/ with DOMAIN.md, mcp-tool-patterns and pii-pipeline rules - Update .ai-rulez/context/crate-structure.md with xberg-rag, xberg-rag-node, mcp-server entries - Add task mcp:build/dev/test/lint/start and rag-node:build/rag-node:build:windows to Taskfile.yml - Add .github/workflows/ci-mcp.yaml triggering on mcp-server/** and crates/xberg-rag*/**
…, scrape_url tools Four new tools unlock xberg capabilities not previously exposed: - extract_entities: GLiNER ONNX or LLM-based NER with configurable categories - structured_extract: LLM JSON-schema extraction with strict mode - transcribe_audio: Whisper ONNX offline transcription for audio/video - scrape_url: URL scraping with headless browser fallback and crawl mode
…st_folder mergeNerEntities() converts GLiNER entities to PiiFinding format, deduplicating by span overlap and keeping the higher-confidence result. ingest_folder gains use_ner/ner_categories params that run ONNX NER during extraction and pipe entities through the merge before redaction. 7 new tests cover all merge paths.
…ToEnglish, fix NerEntity type
Four type bugs found by reading index.d.ts against the implementation:
- ChunkingConfig fields are max_chars/max_overlap not maxSize/overlap
- TranscriptionConfig has no translateToEnglish field
- NerEntity.label/score -> category/confidence matching actual Entity type
- NER_LABEL_TO_CATEGORY keys are lowercase ('person' not 'PERSON')
Adds hf_repo/hf_model_file/hf_tokenizer_file to NerConfig so callers without access to the private xberg-io/gliner-models catalog can point the ONNX backend at any public or private GLiNER export instead. Custom-repo downloads are not checksum-verified, unlike the pinned catalog. Wires the override through both NER dispatch sites (the NER post-processor and the redaction engine), the xberg-node bindings, and the MCP server's extract_entities/ingest_folder tools. Also adds the GLiNER2 ONNX backend implementation plan.
…feature branches - Rewrite all 228 xberg-io/actions references to jamon8888/actions - Remove github.repository == 'xberg-io/xberg' guard from ci-rust, ci-lint, ci-e2e, ci-docker, ci-mobile (blocked CI on the fork) - Extend pull_request.branches to [main, feature/**, fix/**] in ci-rust, ci-lint, ci-docs, ci-e2e so stacked PRs get checked
tobocop2
pushed a commit
that referenced
this pull request
Jul 18, 2026
…NNX Runtime Adds crates/xberg-gliner-candle, a Candle implementation of the GLiNER2 architecture (DeBERTa-v2 encoder, span representation, count-LSTM, schema-gather and scoring heads) with runtime PEFT LoRA adapter merge-at-load and a streaming safetensors loader that converts F32 to F16 from raw bytes to avoid materializing intermediate tensors. In xberg-gliner, the shared GLiNER2 pipeline pieces (v2 tokenizer, splitter, preprocess, decode) are added and the ORT dependency moves behind an ort-backend feature so the candle crate can reuse the v2 pipeline without linking ONNX Runtime; plain builds keep ort-bundled as the default, preserving prior behavior. A ner-candle feature wires the backend into text::ner::candle behind the existing NerBackend trait. ner-candle-wasm is defined but not yet part of wasm-target; the browser wiring lands separately. Carried over from PR #1253.
Member
|
@jamon8888 thank you for this work. The per-line OCR geometry design, the rehydration vault, and the Candle GLiNER2 backend are all genuinely good engineering, and the end-to-end verification you did (real PaddleOCR geometry, real browser model loads) gave us confidence in the designs. This is a meaningful contribution and we want it in xberg so thank you very much. What I didTo get this merged quickly, I carried your work over myself instead of asking for another round of changes, and I preserved your authorship. Your code lands in commits authored by you, and everything I changed on top is in my own commits, so you can see exactly what was adjusted and why. The OCR bridge: #1259Your OCR bridge and engine commit and your e2e coverage, with my review fixes on top. This is the core of what this PR set out to do: real per-line OCR geometry through the wasm bridge. Rehydration maps: #1261 (issue #1260)Your rehydration maps commit. This lands the core module first: encrypt and decrypt, find_subject, forget_subject. The engine and wasm integration are follow-ups described in the issue. GLiNER2 Candle NER: #1263 (issue #1262)Your Candle backend commit, including the v2 pipeline work in xberg-gliner. Native path first; the browser wiring is a follow-up described in the issue. The candle and safetensors fixes are now upstreamThe dependency bugs you found while getting the browser path working are real, and they are filed and fixed upstream now. I minimized reproducers from your vendored patches, verified them against stock 0.11, and opened both the issues and the fix PRs. For candle: huggingface/candle#3750 with fix huggingface/candle#3751. The root cause is debertav2 hardcoding F32 scalars (the attention score accumulator and the XSoftmax fill tensors), which breaks F16 forward. For safetensors: safetensors#817 with fix safetensors#818. Metadata validation spuriously overflows on 32-bit targets once a tensor passes 2^32 bits, which a DeBERTa-v2-scale embedding table does on its own. Once both merge and release, the browser NER path (#1268) unblocks. One verification result worth passing back: the candle-nn LayerNorm skip-fused patch in your fork turned out not to be load-bearing. With only the debertav2 fix applied, F16 forward passes on stock candle-nn 0.11, so you can drop that vendored patch. Friendly feedback on how we work hereFor future contributions, this is the shape that merges fast:
NextThis was the second review round where the feedback did not make it into the branch, so rather than a third round, let's pair through the follow-ups together. Closing this PR in favor of the three above. Please review them when you can; you know this code best, and your name is on it. |
tobocop2
pushed a commit
that referenced
this pull request
Jul 18, 2026
…NX Runtime Adds crates/xberg-gliner-candle, a Candle implementation of the GLiNER2 architecture (DeBERTa-v2 encoder, span representation, count-LSTM, schema-gather and scoring heads) with runtime PEFT LoRA adapter merge-at-load and a streaming safetensors loader that converts F32 to F16 from raw bytes to avoid materializing intermediate tensors. In xberg-gliner, the shared GLiNER2 pipeline pieces (v2 tokenizer, splitter, preprocess, decode) are added and the ORT dependency moves behind an ort-backend feature so the candle crate can reuse the v2 pipeline without linking ONNX Runtime; plain builds keep ort-bundled as the default, preserving prior behavior. A ner-candle feature wires the backend into text::ner::candle behind the existing NerBackend trait. ner-candle-wasm is defined but not yet part of wasm-target; the browser wiring lands separately. Carried over from PR #1253.
This was referenced Jul 18, 2026
tobocop2
pushed a commit
that referenced
this pull request
Jul 18, 2026
… split Adds a NER bridge mirroring the OCR bridge: an injected ner(text, categories) object resolves entities through the same timeout machinery, XbergEngine gains a ner() method, and JsNerBridge adapts the injected object to the NerBackend trait for core consumers. NerBackend's async_trait futures become ?Send on wasm32: a JS bridge's futures hold JsValues driven by wasm_bindgen_futures, which are not Send, and the target is single-threaded. Every other target keeps the plain Send contract unchanged. Carried over from PR #1253, trimmed to the injected path.
tobocop2
pushed a commit
that referenced
this pull request
Jul 18, 2026
Ports the NER bridge cases from PR #1253 to the wasm e2e vitest harness: entities round-trip through an injected stub, category forwarding (built-in and custom labels as plain strings), and the no-backend and malformed-injection error paths.
tobocop2
added a commit
that referenced
this pull request
Jul 18, 2026
redact_capturing_rehydration_map runs the same full redaction pass as redact and returns the token to original-text map. The capture point is the TokenCounter's existing dedup cache, which holds exactly the TokenReplace substitutions, so all six apply_strategy call sites are covered with no per-site changes; redact and the capturing variant share one body via redact_counted. Completes the engine half of the feature from PR #1253; the binding surface follows on the next canonical regeneration. Round-trip test included.
Member
|
@jamon8888 one more thing on the web UI: it should start as an issue, not a PR. It is a whole subsystem (Next.js app, Playwright suite, its own build), so the right first step is an issue describing what it is, what it depends on, and how it would be maintained here. We discuss it there, and once we agree on scope, you open the issue's PR. Thanks again for your awesome contributions |
tobocop2
pushed a commit
that referenced
this pull request
Jul 18, 2026
…gine handle Adds a hand-written JS bridge layer to the wasm crate: an injectable OCR backend whose results carry per-line text, confidence, and bounding boxes instead of a flat string, a promise-timeout helper for all bridge calls, and an XbergEngine handle that exposes extract() and ocr() to JS hosts. Carried over from PR #1253, trimmed to the OCR surface that compiles against current main. Co-authored-by: Candy Nguyen <cdy.nguyen@gmail.com>
tobocop2
added a commit
that referenced
this pull request
Jul 18, 2026
… OCR errors Review fixes from PR #1253: the timeout racer's cleanup chain now hangs off the raced promise with both handlers attached, so a rejecting bridge call can no longer surface as an unhandled rejection (fatal by default in Node); a failure to arm the race logs a console warning instead of silently dropping the timeout; the dead Tesseract fallback arms are gone along with the module doc claiming a fallback exists; and the no-backend error tells the caller what to inject instead of leaking crate internals.
tobocop2
pushed a commit
that referenced
this pull request
Jul 18, 2026
Ports the OCR bridge test cases from PR #1253 to the wasm e2e vitest harness so they run against the actually built package: per-line geometry round-trip, the missing-lines degrade path, language forwarding, and the no-backend / empty-input / malformed-injection error paths.
tobocop2
pushed a commit
that referenced
this pull request
Jul 18, 2026
… split Adds a NER bridge mirroring the OCR bridge: an injected ner(text, categories) object resolves entities through the same timeout machinery, XbergEngine gains a ner() method, and JsNerBridge adapts the injected object to the NerBackend trait for core consumers. NerBackend's async_trait futures become ?Send on wasm32: a JS bridge's futures hold JsValues driven by wasm_bindgen_futures, which are not Send, and the target is single-threaded. Every other target keeps the plain Send contract unchanged. Carried over from PR #1253, trimmed to the injected path. Co-authored-by: Candy Nguyen <cdy.nguyen@gmail.com>
tobocop2
pushed a commit
that referenced
this pull request
Jul 18, 2026
Ports the NER bridge cases from PR #1253 to the wasm e2e vitest harness: entities round-trip through an injected stub, category forwarding (built-in and custom labels as plain strings), and the no-backend and malformed-injection error paths.
Goldziher
pushed a commit
that referenced
this pull request
Jul 18, 2026
…rs, with per-person erasure (#1261) * feat(redaction): encrypted rehydration maps with find/forget-subject Adds a redaction-rehydrate feature gating text::redaction::rehydration: encrypt_map/decrypt_map (AES-256-GCM, scrypt-derived key, versioned XPII\x01 wire format, fresh salt and nonce per encryption), plus find_subject and forget_subject for searching and selectively deleting entries from a decrypted map. The map never touches disk inside xberg; persistence and passphrase handling stay with the caller. Includes a cross-implementation fixture asserting a map encrypted by the TypeScript implementation decrypts here. Carried over from PR #1253. * chore(redaction): keep rehydration off the binding surface for now alef(skip) on the rehydration types and functions until the engine/wasm integration lands the binding surface deliberately; regeneration stays byte-identical. * chore: record aes-gcm/scrypt in Cargo.lock * chore(redaction): run rehydration in CI's full-feature tests, add changelog entry Adds redaction-rehydrate to the full aggregate so the CI unit-test run (cargo test -p xberg --features full --all-targets) exercises the new module; without it the feature's tests never executed in CI. * fix(redaction): unicode-aware subject matching, zeroized key material find_subject/forget_subject now lowercase with Unicode semantics instead of ASCII-only, matching the TypeScript implementation's toLowerCase(); a GDPR erase query for a non-ASCII name previously matched in TS and missed here. New test pins it. The scrypt-derived key and the decrypted plaintext buffer are wrapped in Zeroizing so key material and PII do not linger in freed memory. Decrypt error message and comments reworded; the scrypt comment no longer points at a file that does not exist upstream. * feat(redaction): capture the rehydration map from the document pass redact_capturing_rehydration_map runs the same full redaction pass as redact and returns the token to original-text map. The capture point is the TokenCounter's existing dedup cache, which holds exactly the TokenReplace substitutions, so all six apply_strategy call sites are covered with no per-site changes; redact and the capturing variant share one body via redact_counted. Completes the engine half of the feature from PR #1253; the binding surface follows on the next canonical regeneration. Round-trip test included. --------- Co-authored-by: Jamin <rubio.jamin@gmail.com>
Goldziher
pushed a commit
that referenced
this pull request
Jul 19, 2026
…NX Runtime Adds crates/xberg-gliner-candle, a Candle implementation of the GLiNER2 architecture (DeBERTa-v2 encoder, span representation, count-LSTM, schema-gather and scoring heads) with runtime PEFT LoRA adapter merge-at-load and a streaming safetensors loader that converts F32 to F16 from raw bytes to avoid materializing intermediate tensors. In xberg-gliner, the shared GLiNER2 pipeline pieces (v2 tokenizer, splitter, preprocess, decode) are added and the ORT dependency moves behind an ort-backend feature so the candle crate can reuse the v2 pipeline without linking ONNX Runtime; plain builds keep ort-bundled as the default, preserving prior behavior. A ner-candle feature wires the backend into text::ner::candle behind the existing NerBackend trait. ner-candle-wasm is defined but not yet part of wasm-target; the browser wiring lands separately. Carried over from PR #1253.
Goldziher
pushed a commit
that referenced
this pull request
Jul 19, 2026
…gine handle Adds a hand-written JS bridge layer to the wasm crate: an injectable OCR backend whose results carry per-line text, confidence, and bounding boxes instead of a flat string, a promise-timeout helper for all bridge calls, and an XbergEngine handle that exposes extract() and ocr() to JS hosts. Carried over from PR #1253, trimmed to the OCR surface that compiles against current main. Co-authored-by: Candy Nguyen <cdy.nguyen@gmail.com>
Goldziher
pushed a commit
that referenced
this pull request
Jul 19, 2026
… OCR errors Review fixes from PR #1253: the timeout racer's cleanup chain now hangs off the raced promise with both handlers attached, so a rejecting bridge call can no longer surface as an unhandled rejection (fatal by default in Node); a failure to arm the race logs a console warning instead of silently dropping the timeout; the dead Tesseract fallback arms are gone along with the module doc claiming a fallback exists; and the no-backend error tells the caller what to inject instead of leaking crate internals.
Goldziher
pushed a commit
that referenced
this pull request
Jul 19, 2026
Ports the OCR bridge test cases from PR #1253 to the wasm e2e vitest harness so they run against the actually built package: per-line geometry round-trip, the missing-lines degrade path, language forwarding, and the no-backend / empty-input / malformed-injection error paths.
Goldziher
pushed a commit
that referenced
this pull request
Jul 19, 2026
… split Adds a NER bridge mirroring the OCR bridge: an injected ner(text, categories) object resolves entities through the same timeout machinery, XbergEngine gains a ner() method, and JsNerBridge adapts the injected object to the NerBackend trait for core consumers. NerBackend's async_trait futures become ?Send on wasm32: a JS bridge's futures hold JsValues driven by wasm_bindgen_futures, which are not Send, and the target is single-threaded. Every other target keeps the plain Send contract unchanged. Carried over from PR #1253, trimmed to the injected path. Co-authored-by: Candy Nguyen <cdy.nguyen@gmail.com>
Goldziher
pushed a commit
that referenced
this pull request
Jul 19, 2026
Ports the NER bridge cases from PR #1253 to the wasm e2e vitest harness: entities round-trip through an injected stub, category forwarding (built-in and custom labels as plain strings), and the no-backend and malformed-injection error paths.
tobocop2
added a commit
that referenced
this pull request
Jul 20, 2026
The XbergEngine bridge surface was only tested from JavaScript, against the built npm package. This adds #[wasm_bindgen_test] suites inside xberg-wasm covering the Rust side of the contract: constructor validation, OCR/NER dispatch through injected JS backends, wire-shape deserialization (line geometry, entity fields, category wire names), degraded shapes, this-binding preservation, and the bridge timeout. The generated manifest carries wasm-bindgen-test as a dev-dependency via alef's extra_dev_dependencies, declared in alef.toml; only the wasm manifest is regenerated here, since the rest of the tree already matches the pinned alef version. The suites live in the hand-written src/engine.rs module because the generated manifest builds only a cdylib, which tests/ integration tests cannot link. They run under Node via scripts/ci/wasm/run-crate-tests.sh: the test glue carries the same unresolvable env / wasi_snapshot_preview1 imports that fix-wasi-imports.mjs patches out of the published package, and the test runner has no post-build hook, so test-shims/ supplies those modules at require time through NODE_PATH. The wasm e2e job runs the script after the vitest suites, which keep covering the JS side. The suites derive from the wasm-bindgen tests contributed in #1253, ported to the merged engine surface. Closes #1269
Goldziher
pushed a commit
that referenced
this pull request
Jul 20, 2026
* test(wasm): run the engine's Rust tests inside the crate The XbergEngine bridge surface was only tested from JavaScript, against the built npm package. This adds #[wasm_bindgen_test] suites inside xberg-wasm covering the Rust side of the contract: constructor validation, OCR/NER dispatch through injected JS backends, wire-shape deserialization (line geometry, entity fields, category wire names), degraded shapes, this-binding preservation, and the bridge timeout. The generated manifest carries wasm-bindgen-test as a dev-dependency via alef's extra_dev_dependencies, declared in alef.toml; only the wasm manifest is regenerated here, since the rest of the tree already matches the pinned alef version. The suites live in the hand-written src/engine.rs module because the generated manifest builds only a cdylib, which tests/ integration tests cannot link. They run under Node via scripts/ci/wasm/run-crate-tests.sh: the test glue carries the same unresolvable env / wasi_snapshot_preview1 imports that fix-wasi-imports.mjs patches out of the published package, and the test runner has no post-build hook, so test-shims/ supplies those modules at require time through NODE_PATH. The wasm e2e job runs the script after the vitest suites, which keep covering the JS side. The suites derive from the wasm-bindgen tests contributed in #1253, ported to the merged engine surface. Closes #1269 * fix(ci): shield the wasm crate-test build from native RUSTFLAGS The e2e job exports RUSTFLAGS for the native builds (onnxruntime -L paths, -Wl,-rpath,$ORIGIN), and the env var overrides the [target.wasm32-unknown-unknown] rustflags in .cargo/config.toml entirely: rust-lld rejects the rpath argument and the wasm-required flags (getrandom_backend, bulk-memory, --allow-multiple-definition) are dropped. Unset it in run-crate-tests.sh so the config-file flags govern, matching the local environment where the suites pass. Verified by reproducing the CI env locally: the script fails before and passes after. * ci: trigger the e2e workflow on wasm CI script changes The wasm leg now runs scripts/ci/wasm/run-crate-tests.sh, but the workflow's paths filters did not cover scripts/, so a change to the runner script alone triggered no run and went untested. Add scripts/ci/wasm/** to both triggers, matching ci-rust.yaml's existing scripts/ci/rust/** entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
crates/xberg-wasm/src/bridge/ocr.rs) that was discarding real per-line confidence/bbox data from the injected JS OCR backend, keeping only the flat text —XbergEngine.ocr()now returns{text, lines: [{text, confidence, bbox?}]}.packages/xberg-web-ui/src/engine/engine.worker.ts'shandleOcr(no more newline-splitting a flat string / hardcoded confidence) andOcrLine/toParsedOcrOutput(adds an optional, caller-suppliedpagefield; groups by real page identity when present, defaults to page 1 otherwise).wasm-pack buildtoolchain end-to-end (was failing withAccès refuséon every invocation — root cause was a misconfigured$CARGOenv var, plus several downstream pre-existing bugs acrossxberg/xberg-rag/xberg-wasmthis surfaced once the build could actually run: stalealef-generated bindings,engine.rs/bridge/*.rsnever wired intolib.rs, and a Send/Sync cascade in the NER bridge). Full details in the commit messages.lot3/web-ui-advanced-viz(the web-ui OCR layout viewer this work builds on), since it hadn't landed on this branch's start point.Known issue — Lot 3 duplication
This branch's merge-base already predates
main's current tip (03ecbc9c0b, "Lot 3 advanced visualization... (#27)"), which appears to be the same Lot 3 content landed as a single squashed commit — while this branch mergedlot3/web-ui-advanced-vizwith its full, unsquashed commit history. The diff againstmainwill likely show this duplication. Flagging explicitly rather than attempting a local rebase/cleanup, so it's visible in review.Known follow-ups (not fixed here, intentionally out of scope)
pub mod bridge; pub mod engine; pub use engine::XbergEngine;addition tocrates/xberg-wasm/src/lib.rshas no durable home —lib.rsis fully regenerated byalef generate --lang wasm, and no mechanism was found to preserve hand-written module declarations across regeneration. Will be silently dropped by the next full regen unless someone gives it one.@xberg-io/xberg-wasmis still blocked: itspkg/nodejsoutput (wasm-packnodejstarget) doesrequire('fs').readFileSync(...)to load the.wasmbinary at module load time, which doesn't work in a browser. Needs a wasm-pack target change (web/bundler) for browser consumers — a real architectural decision affecting the Node.js/mcp-server side too, scoped out of this PR.OcrLine.pagestays unpopulated until that exists.crates/xberg-ffi/xberg-jni/xberg-node/xberg-php/xberg-pyare excluded from the root workspacememberslist (their source files are part of a large, pre-existing, unrelated interrupted cleanup that predates this branch) — needed to unblockcargo metadata/wasm-packforxberg-wasm.Test plan
packages/xberg-web-ui:pnpm --filter xberg-web-ui test:run— 47/47 passing (one pre-existing, unrelated unhandled-rejection artifact insync-client.test.ts, documented, not introduced by this PR)packages/xberg-web-ui:pnpm --filter xberg-web-ui typecheck— passes for the changed files; 3 pre-existing, unrelated errors remain inengine.worker.ts(confirmed viagit stashcomparison, not introduced by this PR)crates/xberg-wasm:cargo build -p xberg-wasm --lib --target wasm32-unknown-unknown— clean, zero errorscrates/xberg-wasm:wasm-pack build --target nodejs --out-dir pkg/nodejs— succeeds;XbergEngine.ocr()confirmed present in the generated.d.tscrates/xberg-wasm:wasm-pack test --headless --chrome -- --test hybrid_dispatch— not run (local Windows environment couldn't executewasm-pack test's Chrome-driven suite this session; the 3 relevant test cases were manually traced against the implementation instead — see commit2a1b0f1bc3)🤖 Generated with Claude Code
Update: real Candle GLiNER2 NER now works end-to-end (browser + mcp-server)
Follow-up work since the above was written, fixing the goal of getting the actual production app (not just the isolated test page) working with real models:
crates/xberg-gliner-candle): an F16/F32 dtype mismatch in the scores read-back, and an intermittent load-time OOM in the safetensors streaming loader (fixed by converting F32→F16 directly from raw bytes instead of materializing a full intermediate F32 tensor). Verified reliably across repeated fresh-browser-tab runs: realbge-m3embedder + real Candle GLiNER2 NER now load and correctly detect entities in the actual browser WASM UI.@xberg-io/xberg-wasmis still blocked" follow-up above is resolved (pkg/web/pkg/bundlertargets now build and are wired intonext.config.js).mcp-serverwas previously silently using the weaktransformers.jsbert-base-NERfallback instead of real GLiNER2 for itsingest()PII redaction step, becausexberg-wasm-runtime's Candle NER wiring refused to run outside a browser. Fixed (see latest commit): mcp-server now loads the real model, andengine.ingest()genuinely redacts PII end-to-end — confirmed by inspecting the actually-stored, redacted chunk text.crates/xberg-node(a separate native Node addon, unrelated to the WASM path above) has ~44 pre-existing compile errors from API drift against currentxberg— confirmed out of scope for this PR, needs its own dedicatedalefcodegen fix.Update: OCR and RAG retrieve verified end-to-end too
Closing out the remaining unchecked test-plan items from the top of this PR:
engine.ocr()in the actual running mcp-server engine (real PaddleOCR backend,ppu-paddle-ocr). Both lines came back correct with >99.8% confidence and accurate per-line bounding boxes — this is the same OCR bridge/binding this PR's core fix targets, confirmed genuinely working, not just compiling.engine.ingest()verification, also calledengine.query()against the collection afterward — real vector search correctly returned the ingested, PII-redacted chunk.crates/xberg-node/Cargo.lock, matching the existing convention for this crate's sibling standalone (non-workspace-member) crates.With this, OCR, NER, and RAG ingest/retrieve are all confirmed working end-to-end against real models — in the browser WASM UI and in mcp-server.
Update: full A-to-Z click-through verification, 4 more real bugs found and fixed
Deployed mcp-server's actual static-export UI (previously only a placeholder stub existed on disk, so this whole path had never been exercised) and drove it through a real browser: create folder → navigate into it → upload flow. This surfaced and fixed four genuine, previously-undetected bugs blocking every step past the first click:
candle-ner.tsbroke the browser build. The Node disk-cache addition from the previous update putnode:fs/node:pathimports at module scope;factory.ts's static import pulled that into the browser webpack bundle (UnhandledSchemeError: Reading from "node:fs"). Split the Node-only I/O intocandle-ner-node-cache.ts, dynamically imported behindtypeof window === "undefined"(same statically-foldable guardstore.tsalready uses forstore-node.js) so webpack's dead-code elimination prunes it from the browser bundle./ui/*behind the same?token=as the top-level page meant Next's hashed JS/CSS bundles, ORT's.wasm/.mjsruntime, and RSC payloads for client-side navigation all failed, since none of them carry the URL's query string. Only the top-level page and the actual data-mutating API routes (ingest/map/collection/admin) need the token now./folder/<name>or/document/<collection>/<id>request 404'd. Next's static export names a dynamic route's placeholder page as a flat<lastSegment>.htmlfile, never.../index.html—ui-route-resolver.tsandstatic-server.tsboth assumed the latter. Fixed the existence checks and added a$uri.htmlfallback.useParams()returns that file's baked-in param regardless of the real address bar URL on a hard navigation. Addedroute-params.tsto parse the real segments fromwindow.location.pathnamedirectly.Also fixed page.tsx to use the existing shared
resolveMcpBaseUrl()helper instead of a local duplicate, and confirmed viacurlthat the fix set is correct end-to-end: static assets 200, dynamic-route fallback 200, sensitive API routes still 401 without a valid token.