Repository navigation
Feat/search core - #2
Merged
Merged
Conversation
rust-core/ (crate rcore): C ABI (JSON in/out, rust_free_string, +crt-static), process-singleton store, Embedder trait + deterministic MockEmbedder, async index_segments worker with two-axis text_ready/vector_status state machine, configure/search(dense)/stats/reset dispatch, rcore_shutdown joins the worker. http-1c-dll: link rcore staticlib into the DLL via CMake; RustCore.h RAII wrapper. CI/release workflows: set up the Rust toolchain before the DLL build. .devtool/: kanban board tracking stages 0-5 (Stage 0+1 cards done). Verified locally: cargo test 27/27; full Release DLL build links rcore.lib into libhttp1cWin.dll (/MT <-> +crt-static match holds). Real fastembed/ort embedder and native tools/call routing are deferred to follow-up cards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task 1 (carried-forward fix): install a doc's text segments into the store synchronously at accept (vector=None, text_ready=true, vector_status=Building); the background worker now fills ONLY the vectors afterward, matched by (doc_id, segment_id). Re-ingest stays atomic — a superseded job's stale segment ids simply don't match and are skipped. Net: grep/keyword/get_segment see text the instant accept returns; only dense search waits for vectors. Task 2: new filter.rs with combinable, domain-agnostic meta filters — all (AND), any (OR), tags_all, tags_any — evaluated over a hit's effective meta (document meta merged with segment meta; segment overrides doc on key collision). Wired into both search and grep. Task 3: new grep.rs regex search over stored segment text using regex 1.12.3 (RE2, linear-time, no ReDoS) via RegexBuilder (case_insensitive, multi_line). CRLF->LF normalized; 1-based inclusive line numbers per segment; context_lines, max_matches (sets truncated) and max_per_doc honored. A broken pattern returns a structural bad_pattern error envelope, never a panic. Adds the regex dependency only. cargo test: 48 passed (27 prior + 21 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hema merge
Route search/get_segment/grep tool calls to the linked Rust core via
rcore_dispatch instead of forwarding to 1C, and merge the component-owned
JSON schemas for those tools into tools/list so the search subsystem is
self-contained.
- tools/call: before the ExtEvent forward, short-circuit when toolName is a
native tool. Serialize arguments, call rcore_dispatch (wrapped in RustString),
parse the {ok,result|error} envelope and build an MCP tool result
{content:[{type:text,text:...}], isError:ok==false}. Tool-level failures are
structural results, never transport/session errors. Wrapped in try/catch so a
bad payload degrades to an isError result rather than throwing across the
handler.
- tools/list: append 3 hardcoded {name,description,inputSchema} definitions
(search/get_segment/grep) to cachedToolsJson before pagination; existing
paginateJsonArray applies to the union.
- #include "RustCore.h".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 1 native-tool-routing + Stage 2 grep/meta-filters (and the text-sync-on-accept fix) integrated and verified: cargo test 48/48, full Release DLL build links the routing + grep + regex cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add two coupled dispatch methods to the Rust search core (rcore):
index_raw (async, like index_segments): on accept under the short write
lock it normalizes the text CRLF->LF, builds a full-document line offset
table, stores the full normalized text + offset table on the doc, chunks
the text into line-snapped segments (vector=None), and enqueues one embed
job. Returns {accepted, collection, doc_id, segment_count} immediately;
the worker fills only vectors off-thread. doc_id is auto-assigned and
returned in the ack when omitted; provided ids upsert atomically.
Chunker: greedy line-granular split by token budget (target ~300, hard
cap = max_seq_len) snapped to line boundaries with ~2-line overlap; a
single line over the cap becomes one oversized chunk (truncated for embed
only, returned whole by get_segment). Token counting is a documented
heuristic (max of whitespace-word count and chars/4) in one function, to
be swapped for the real tokenizer when the real embedder lands.
get_segment: O(1) line-range slice over the offset table; out-of-range
clamps and returns the actual {line_start, line_end}; respects max_lines;
works the instant after accept (text+offsets are synchronous). Atomic
index_segments records (no offset table) -> structural no_line_index
error; unknown doc_id -> not_found.
Extend Document with optional full_text + line_offsets (raw docs only);
index_segments behavior unchanged. Add NOT_FOUND / NO_LINE_INDEX codes.
Tests: chunk budget/line-snap/overlap, oversized single line, offset
table, get_segment slice/clamp/max_lines/no_line_index, CRLF norm,
get_segment immediately after accept, auto doc_id in ack, raw doc
greppable immediately and dense-searchable when ready. Also drain the
shared background worker in the test harness guards to remove a latent
cross-test race (a stale job flipping a same-named collection Ready
early); test-only, no production change. 67 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit HttpServerComponent shared mutable state under concurrent httplib workers, the new synchronous native search path (rcore_dispatch), and the 1C-thread native methods. Verdict: the native path introduces no new C++ data races. It is a stateless, synchronous pass-through to a Rust core that does its own RwLock-based locking; the native tools/call branch returns before touching pendingRequests/requestCounter, cachedToolsJson is read under toolsMutex and copied for the tools/list merge, rcore_dispatch is called with no C++ lock held, and the RustString RAII lifetime is correct (single free, Rust allocator). Documents two pre-existing benign config races (authToken, loggingEnabled/timeout) and latent risks for future stages. No C++ code changes, so no rebuild required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 2 index_raw + chunker + line offset table + get_segment (Rust, 67/67 cargo tests, integrated DLL build green) and the Stage 0 C++ thread-safety audit (no material races; THREAD-SAFETY.md) all verified. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hybrid search (mode: dense | keyword | hybrid, default dense):
- keyword: full lexical scan over stored segment text; scores by
distinct-query-term coverage + 0.1*occurrences (whole-token match).
Works while vector_status==Building (no vectors needed).
- hybrid: fuses dense + keyword rank lists with Reciprocal Rank Fusion
(k=60) keyed by segment_id, so a doc strong in only one channel still
surfaces (the SKU/ИНН case).
- min_score is applied per the active mode and documented as such:
dense=absolute cosine floor, keyword=match-score floor, hybrid=fused
RRF-score floor (relative, not a cosine).
- All modes respect collection, k, max_per_doc, include_text, meta
filters; dense/hybrid keep the partial flag for Building collections,
keyword is never partial.
Incremental delete (atomic, under one short write lock):
- delete_document(doc_id): removes the doc and all its segments; drops
the collection if it becomes empty; unknown doc_id -> {deleted:false}.
- delete_collection(collection): removes the whole collection; unknown
-> {deleted:false}. In-flight embeds for deleted docs/collections
become harmless no-ops via the existing stale-id / missing-collection
guards in apply_job.
16 new tests (keyword ranking, hybrid RRF fusion both channels,
keyword/hybrid honor filter+k+max_per_doc, keyword while Building,
delete removes from search+grep+get_segment, delete_collection clears,
deletes of unknown -> structural false). 83 tests pass (67 prior + 16).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Call rcore_shutdown() after the listener thread joins in doStopListen() (not in the destructor — the Rust singleton outlives one component), so the background worker is cancelled+joined before the DLL can unload. Board: hybrid-RRF, incremental delete, and rcore_shutdown-wiring done. Verified: cargo test 83/83, Release DLL build links rcore_shutdown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement the production FastEmbedder (multilingual-e5-small via fastembed/ort/tokenizers) as an optional, feature-gated path while keeping MockEmbedder as the default for the fast unit-test suite. - Cargo.toml: optional `fastembed = "=5.16.0"` + `anyhow` behind a non-default `fastembed` feature, so `cargo test` stays mock-only and never compiles ort. - src/fastembed_embedder.rs (cfg fastembed): FastEmbedder implementing Embedder. Two-instance design — separate `bulk` and `query` TextEmbedding models, each behind its own Mutex — so a long bulk reindex never blocks query latency (fastembed's embed is &mut self). Applies e5 "passage: "/"query: " prefixes, probes dim (384) at construction. Built-in (HF download/cache) and offline local-path constructors; embed failures degrade to zero vectors (no panics). - core.rs: Config gains `model`; configure() selects mock vs real and resets the index on the 64<->384 dim change. Real-model failure falls back to mock. - lib.rs: parse/echo `model`; gated integration test asserts ru/en contracts outrank an unrelated uk sentence and dim==384. cargo test (no features): 83 passed, fast, no ort. cargo test --features fastembed: 84 passed (real e5 cosine ru=0.86 en=0.90 cat=0.78). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Real fastembed FastEmbedder verified (cargo test --features fastembed 84/84; ru/uk ranking sane, dim 384). Records the §11.2 finding on the onnxruntime-bundling card: ort forces /MD + onnxruntime.dll must ship. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add opt-in DirectML acceleration to the real fastembed embedder, with
ort's automatic best-effort CPU fallback (no manual fallback code).
- Cargo.toml: name `ort` (=2.0.0-rc.12) as an optional dep and route the
`fastembed` feature through `fastembed/directml` + `ort/directml` so the
`ort::ep::DirectML` EP type is available. Additive to fastembed's
existing `ort/download-binaries`; mock build still compiles no ort.
- fastembed_embedder.rs: add `pub enum Device { Cpu, DirectML, Auto }`.
Cpu -> empty EP list (ort's default CPU); DirectML/Auto ->
vec![ort::ep::DirectML::default().build()] (best-effort, auto CPU
fallback). Thread `device` through new_builtin/new_local and both
loaders via with_execution_providers. Two-instance (bulk+query) design,
e5 prefixes, and all-zero graceful degradation kept intact.
- configure: accept `device: "cpu" | "dml" | "auto"`, default "auto"
(= DirectML with CPU fallback). Parsed, mapped to Device, echoed back.
Ignored by the mock build.
- Gated test now configures device:"auto" (DML registered) and still
embeds + ranks the contract query (dim 384); does not crash on a
GPU-less machine (ort falls back to CPU).
Verified: `cargo test` (no features) 83 pass, fast, no ort;
`cargo test --features fastembed` 84 pass (real e5-small ranks
ru=0.8612 en=0.8976 cat=0.7783, contracts outrank the cat).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 1C C++ component is /MT and cannot compile under /MD (char16_t streams hit MSVC C2491), but ort/onnxruntime requires /MD. So the real search core ships as a separate rcore.dll (cdylib, /MD, static onnxruntime) that the component loads at RUNTIME via LoadLibrary + GetProcAddress instead of link-time staticlib coupling. One libhttp1cWin.dll now serves both packages: * lite — rcore.dll absent -> search tools return "install RAG" * full — rcore.dll present -> real fastembed search - rust-core/Cargo.toml: crate-type staticlib+cdylib -> cdylib only; the component no longer links a staticlib. - RustCore.h: replace the extern "C" link-time decls with a lazy, thread-safe (std::call_once) loader (RCore) that locates rcore.dll NEXT TO this module (GetModuleHandleExW FROM_ADDRESS + GetModuleFileNameW, not cwd), resolves all 4 entry points, exposes available()/dispatch()/shutdown()/version(), and frees via the loaded rcore_free_string pointer. Robust no-op when rcore.dll is absent. - HttpServerComponent.cpp: dispatchNativeTool returns a structured rag_not_installed result when !RCore::available(); doStopListen calls RCore::shutdown(). tools/list surface stays uniform across lite/full. - CMakeLists.txt: drop the rcore staticlib link (lite is pure C++/MT, no cargo dependency); add option(RCORE_FASTEMBED) that builds the cdylib with the dynamic CRT and copies rcore.dll into bin/ for the full package. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Package the search subsystem as an optional full distribution alongside the existing lite component: - package-http1c-addin.sh: VARIANT=lite|full selector. full bundles rcore.dll (from bin/) + DirectML.dll (from C:\Windows\System32, override via DIRECTML_SRC) and lists them as <file> entries in MANIFEST.XML; fails clearly if either is missing. SKIP_TEMPLATE=1 emits only the ZIP without touching the committed (lite) Template.bin. lite remains the default, back-compat preserved. - release.yml: build both variants (full via -DRCORE_FASTEMBED=ON) and attach http1c-addin-lite-v<ver>.zip and http1c-addin-full-v<ver>.zip to the release; version-extraction logic preserved. - ci.yml: add a cached cargo build --features fastembed (dynamic-CRT RUSTFLAGS) step to catch ort/fastembed breakage without ballooning CI time. - README: document the search subsystem (rcore, search/grep/get_segment, dense/keyword/hybrid, index_segments/index_raw async ingest), lite vs full and runtime rcore.dll detection, GPU/CPU DirectML with CPU fallback and the DirectML.dll hard dependency, the /MD vs /MT rationale, and lite/full packaging. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- GPU (DirectML) execution with CPU fallback (device: cpu/dml/auto). - Two-version component: lite (libhttp1cWin.dll only, "install RAG") vs full (+ rcore.dll + DirectML.dll), via runtime LoadLibrary of rcore.dll. - Records the §11.2 cdylib resolution on the onnxruntime-bundling card. Verified: mock 83/83, fastembed+DML 84/84, lite build (no cargo) + full build + loader smoke (available() toggles, dispatch round-trips). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces an optional, runtime-loaded Rust search core (rcore.dll) that provides native semantic/keyword/hybrid retrieval plus regex (grep) and line-slicing (get_segment) to the 1C MCP component, including build/packaging and CI/release wiring for lite vs full distributions.
Changes:
- Added a Rust
rcorecdylib implementing a JSON-in/JSON-out ABI with async ingest, dense/keyword/hybrid search, grep, get_segment, and delete operations. - Added C++ runtime loading (
LoadLibrary/GetProcAddress) and native routing forsearch/grep/get_segment, plus tool schema merge intotools/list. - Added lite/full packaging, docs, and CI/release workflow updates to build and ship
rcore.dll(+ DirectML.dll in the full bundle).
Reviewed changes
Copilot reviewed 49 out of 51 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| rust-core/src/protocol.rs | Defines the stable JSON envelope and error-code constants for the FFI boundary. |
| rust-core/src/lib.rs | Implements the C ABI entry points, JSON dispatch, request parsing, and end-to-end tests. |
| rust-core/src/grep.rs | Adds RE2-style regex scanning over stored segment text with limits/context. |
| rust-core/src/filter.rs | Adds combinable metadata filtering used by search and grep. |
| rust-core/src/fastembed_embedder.rs | Implements the feature-gated fastembed-backed embedder (DirectML/CPU options). |
| rust-core/src/embed.rs | Defines the embedder trait + deterministic mock embedder and vector utilities. |
| rust-core/README.md | Documents the Rust core (currently out of sync with the new cdylib/runtime-load design). |
| rust-core/Cargo.toml | Defines rcore as a cdylib and adds feature-gated ML dependencies and build profiles. |
| rust-core/.gitignore | Ignores Rust build artifacts and fastembed cache directory. |
| rust-core/.cargo/config.toml | Sets MSVC target flags (static CRT default) for the Rust core build. |
| README.md | Documents the new search subsystem (tools, ingest behavior, lite vs full packaging, GPU/CPU). |
| http-1c-dll/THREAD-SAFETY.md | Adds a thread-safety audit focused on native search execution on httplib worker threads. |
| http-1c-dll/src/RustCore.h | Adds the runtime loader/RAII wrapper for rcore.dll and its ABI functions. |
| http-1c-dll/src/HttpServerComponent.cpp | Routes native tools to the Rust core and merges native tool schemas into tools/list. |
| http-1c-dll/CMakeLists.txt | Adds an optional build path to produce/copy rcore.dll when RCORE_FASTEMBED=ON. |
| build/package-http1c-addin.sh | Packages lite/full add-in bundles (full includes rcore.dll + DirectML.dll). |
| .github/workflows/release.yml | Builds and publishes both lite and full release artifacts, including Rust setup/cache. |
| .github/workflows/ci.yml | Adds Rust setup/cache, Rust core tests, and a build check for rcore.dll (fastembed). |
| .devtool/features/stage5-deferred-backlog-2026-06-08.md | Adds deferred roadmap items for later stages. |
| .devtool/features/stage4-products-adapter-2026-06-08.md | Adds Stage 4 adapter backlog item (products). |
| .devtool/features/stage4-clients-adapter-dedup-2026-06-08.md | Adds Stage 4 adapter backlog item (clients + dedup). |
| .devtool/features/stage3-vanessa-step-registry-2026-06-08.md | Adds Stage 3 investigation backlog item (Vanessa registry access). |
| .devtool/features/stage3-qa-adapter-segments-2026-06-08.md | Adds Stage 3 adapter backlog item (QA segments). |
| .devtool/features/stage3-gherkin1c-json-output-check-2026-06-08.md | Adds Stage 3 investigation backlog item (Gherkin1C JSON positions). |
| .devtool/features/stage3-find-step-usages-2026-06-08.md | Adds Stage 3 adapter backlog item (find_step_usages). |
| .devtool/features/stage2-concurrent-query-embedding-2026-06-08.md | Adds Stage 2 backlog item (concurrent query embedding). |
| .devtool/features/stage0-pin-crate-versions-2026-06-08.md | Adds Stage 0 task about pinning/validating Rust crate versions. |
| .devtool/features/stage0-onnxruntime-bundling-1c-2026-06-08.md | Adds Stage 0 investigation notes about onnxruntime/CRT constraints and bundling strategy. |
| .devtool/features/stage0-offline-fastembed-init-2026-06-08.md | Adds Stage 0 backlog item (offline model init/no network). |
| .devtool/features/stage0-model-delivery-int8-2026-06-08.md | Adds Stage 0 backlog item (model delivery + int8 choice). |
| .devtool/features/done/stage2-meta-filters-2026-06-08.md | Records Stage 2 meta-filters completion notes. |
| .devtool/features/done/stage2-index-raw-chunker-offsets-2026-06-08.md | Records Stage 2 index_raw/chunking/offset-table completion notes. |
| .devtool/features/done/stage2-incremental-upsert-delete-2026-06-08.md | Records Stage 2 incremental upsert/delete completion notes. |
| .devtool/features/done/stage2-hybrid-rrf-2026-06-08.md | Records Stage 2 hybrid (RRF) completion notes. |
| .devtool/features/done/stage2-grep-2026-06-08.md | Records Stage 2 grep completion notes. |
| .devtool/features/done/stage2-get-segment-2026-06-08.md | Records Stage 2 get_segment completion notes. |
| .devtool/features/done/stage1-wire-rcore-shutdown-2026-06-08.md | Records Stage 1 wiring of rcore_shutdown completion notes. |
| .devtool/features/done/stage1-stats-tool-2026-06-08.md | Records Stage 1 stats tool completion notes. |
| .devtool/features/done/stage1-search-dense-2026-06-08.md | Records Stage 1 dense search completion notes. |
| .devtool/features/done/stage1-rag-lite-full-packaging-2026-06-08.md | Records Stage 1 lite/full packaging design completion notes. |
| .devtool/features/done/stage1-native-tool-routing-2026-06-08.md | Records Stage 1 native tool routing completion notes. |
| .devtool/features/done/stage1-gpu-directml-fallback-2026-06-08.md | Records Stage 1 GPU DirectML fallback completion notes. |
| .devtool/features/done/stage1-flat-store-rwlock-2026-06-08.md | Records Stage 1 store/RwLock completion notes. |
| .devtool/features/done/stage1-embedding-shared-session-2026-06-08.md | Records Stage 1 embedding session completion notes. |
| .devtool/features/done/stage1-configure-model-load-2026-06-08.md | Records Stage 1 configure/model-load completion notes. |
| .devtool/features/done/stage1-async-ingest-worker-state-machine-2026-06-08.md | Records Stage 1 async ingest worker completion notes. |
| .devtool/features/done/stage0-ffi-skeleton-rust-staticlib-crt-2026-06-08.md | Records Stage 0 FFI skeleton completion notes. |
| .devtool/features/done/stage0-cpp-thread-safety-review-2026-06-08.md | Records Stage 0/1 C++ thread-safety review completion notes. |
| .devtool/features/done/stage0-confirm-embed-signature-concurrency-2026-06-08.md | Records Stage 0 fastembed signature/concurrency findings. |
- CI: remove Git for Windows' GNU link.exe (usr/bin), which shadows the MSVC linker under `shell: bash` and broke cargo/rustc link steps (the failing "Test Rust core" job). Both ci.yml and release.yml. - Native tool schemas advertise `filter` (was `meta_filters`) to match the Rust core's MetaFilter::parse — otherwise search/grep meta filtering was silently ignored for schema-following clients. - rcore_dispatch counts calls via a lock-free atomic instead of taking the index write lock on every call (no longer serializes concurrent readers). - RustCore.h: rename the reserved include guard __RUSTCORE_H__ → HTTP1C_RUSTCORE_H. - rust-core/README.md: rewritten for the cdylib/runtime-load design + the real Stage 1/2 method set (was stale Stage 0 staticlib skeleton). Addresses Copilot review comments on PR #2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds tests::fastembed_offline_local_init (#[cfg(feature = "fastembed")], gated on RCORE_TEST_MODEL_DIR so CI and the default suite are unaffected). It drives configure -> new_local against a pre-staged model dir, then embeds + dense-searches. Verified locally with all egress blocked (HF_HUB_OFFLINE=1 + dead HTTP(S) proxy): dim 384, correct ranking, no network - empirically closing the offline-init open question (TZ 11.4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moved to done (verification findings appended to each card): - stage0-onnxruntime-bundling-1c: onnxruntime static-linked into rcore.dll (no onnxruntime.dll), loaded next to the component, x64, Template.bin. - stage0-pin-crate-versions: all deps =x.y.z, ort=2.0.0-rc.12, Cargo.lock. - stage2-concurrent-query-embedding: two-instance bulk/query, no rayon over embedding (intra_threads noted inert in fastembed 5.16). - stage0-offline-fastembed-init: offline new_local recipe + OFFLINE-PASS smoke-test (network blocked). stage0-model-delivery-int8 -> review: sidecar-zip delivery plan + int8 identity confirmed; retrieval-delta measurement is the empirical remainder (needs a ru/uk gold set). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
C++: add a BSL-callable RagDispatch(method, payloadJson) to the component that forwards to the Rust core (RCore::dispatch) and returns the JSON envelope verbatim — the ingest/admin methods that are NOT MCP tools (configure, index_segments, index_raw, stats, reset, delete_*). On the lite component (rcore.dll absent) it returns a rag_not_installed envelope. search/grep/get_segment stay served by the component itself. Processor (http1c) form: a RAG demo — Configure / Index demo (this base's catalog+document metadata as segments) / Stats / Search (dense|keyword| hybrid) / Clear commands, async callbacks, a hit formatter, OnOpen defaults; new form attributes + commands + a RAG/Search group. Template.bin repackaged (lite) so the embedded component carries RagDispatch. Verified: lite C++ build links clean. For real search build the full variant (rcore.dll) or drop rcore.dll + DirectML.dll next to the component. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a /Cragselftest-gated self-test that drives the full RAG chain in real 1C: attach -> configure(offline e5, cpu, dim 384) -> index_segments -> poll stats until vector_status:ready -> hybrid search, on two real corpora (Vanessa steps from test/steps.json + IRP Gherkin scenario titles). Both phases return partial:false with relevant top hits. Key fix: attach the component IN-PLACE from the on-disk DLL in ExtCompT (BeginAttachingAddIn with the direct libhttp1cWin.dll path) so rcore.dll is loaded beside it and real search is enabled. Attaching from the template bundle only extracts the declared component file to a session- temp copy, leaving rcore.dll absent -> rag_not_installed. Also: Connect is now attach-only (no InstallAddIn modal), default port 8888 on open, and a BuildVersion() tag logged each run to prove the running .epf is fresh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…r, timings
Extends the headless RAG self-test and the interactive demo:
- Scenario-level segmentation of Gherkin .feature files (one scenario = one
segment, verbatim text + meta {type, feature, file}) — the right retrieval
granularity vs the token chunker, which shredded files into ~54k micro-chunks.
- Batched index_segments (500/doc) instead of one giant job: bounds peak memory
(a single 7.5k-segment batch padded ONNX tensors to ~10 GB) and lets the
worker report incremental progress.
- Progress indicator in the form: Status(message, %, explanation) native bar +
the RagOutput field; the Index button polls stats until vector_status=ready.
- Millisecond timings for configure / embed / query latency. Findings: search
is ~15 ms; embedding is the bottleneck, dominated by text length and layer
count. Dynamic int8 cut model memory/load but not CPU inference speed; a
6-layer MiniLM is ~2.7x faster than 12-layer e5-small.
- Offline model selected via model_path (new_local) so int8 / smaller models
drop in by swapping onnx/model.onnx; device "auto" = DirectML GPU + CPU
fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- stage0 model-delivery-int8 -> done: int8 e5-small staged offline and run in real 1C (dim 384, no network); int8 = memory/load win, not CPU-speed (6-layer MiniLM-L6 ~2.7x faster instead). Delivery = swap onnx/model.onnx at model_path. - stage3 gherkin1c-json-output-check -> done: the Gherkin1C parser DOES emit per-element .line numbers and distinguishes scenario/outline/background/tags/ examples (Cucumber-messages-like). Adapter can rely on it; BSL line-tracking fallback documented. - stage3 vanessa-step-registry -> done: full step catalog = in-memory ValueTable ТаблицаИзвестныхStepDefinition (phrase/description/type/param-hints), readable at runtime; ПодготовитьТаблицуДляВыгрузкиШагов emits the steps.json shape. Both blockers cleared -> stage3 qa-adapter-segments is unblocked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ontour
The processor now contains ZERO absolute paths. Environment paths (offline
model, output dir, ExtCompT DLL) arrive via the launch parameter
(ragselftest;model=…;out=…;extcompt=…), parsed by ParseLaunchConfig; result
and trace files resolve through SelfTestOutDir (launch dir or TempFilesDir).
Corpora are replaced by in-BSL pluggable stub data sources + adapters that
map them to index_segments payloads with rich metadata — ready to swap for
real Gherkin1C / step-registry / catalog feeds:
- QA step catalog: canonical phrase + embed_text(phrase|description|params).
- QA scenarios: verbatim text + line_start/line_end + meta{type,feature,tags}.
- find_step_usages: keyword reverse lookup step -> scenario with real params.
- Products: string array + tags metadata.
- Clients: dedup (trim+casefold) before indexing.
A generic asserting driver runs each adapter end-to-end (index -> poll-ready
-> query -> ASSERT) and reports ALL PASS / FAIL. Current run: ALL PASS (5/5),
incl. dedup count assert (7 raw -> 4 unique) and line addressing.
Closes stage-3/4 adapter cards: qa-adapter-segments, find-step-usages,
products-adapter, clients-adapter-dedup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…6/6 contour
- Products: meta {sku, article, category, brand, tags}; article/SKU in segment
text (keyword channel reads text only) so exact-id keyword retrieval works;
embed_text = name+brand+category for semantic intent. New contour case proves
keyword "ART-1003" -> DeLonghi.
- Clients: meta {inn, city, segment}; adapter-side dedup by exact INN (one
entity decision stays out of the core). Contour asserts 6 raw -> 4 unique.
Contour: ALL PASS (6/6).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… 6/6) - qa-adapter-segments, find-step-usages, products-adapter, clients-adapter-dedup -> done: implemented as pluggable BSL stub adapters with the exact metadata contracts (line_start/end, embed_text, sku/article, inn dedup), verified by the http1c asserting self-test (ALL PASS 6/6). Stub data sources swap in for real Gherkin1C / step-registry / catalog feeds. - stage5-deferred-backlog -> closed as the intentional post-MVP parking lot. Board: stages 0-4 complete; nothing open. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rewrite the search ranking path to defer Hit materialization to the ≤k survivors instead of building (and deep-cloning doc.meta + segment text) a full Hit for every scanned segment: - New Scored<'a> carries borrows during scoring/fusion — zero per-segment allocation; dense/keyword channels return unsorted Scored. - finalize() does a single bounded top-k via select_nth_unstable_by (no full sort when max_per_doc is None), then materializes Hits for survivors. - make_hit honors include_text: clones only the PREVIEW_CHARS prefix when the caller doesn't want full text (was always cloning the whole segment). - rrf_fuse operates on lightweight refs. - dot(): 8-lane chunked accumulator LLVM can auto-vectorize, replacing the serial zip().sum() dependency chain (closes the SIMD bullet of perf-ann). - apply_job moves the embedding out of the job (mem::take) instead of cloning it under the write lock. Closes perf-topk-heap-deferred-hit. 85/85 cargo test green (adds top-k ordering + include_text:false preview-cap tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Each Segment now caches a token multiset (kw_counts) built once at index time from its text, with the same tokenizer the query uses. keyword_score is now O(query_terms) hash lookups with no per-query re-tokenization or Vec<String> allocation. Populated in both ingest paths (accept_index, index_raw chunker). - grep's normalize_newlines returns Cow: a single \r scan lets LF-only text (every index_raw segment, plus any LF source) be borrowed with zero allocation instead of two String-allocating replace passes. The deeper zero-copy grep writer (borrowing &str in GrepHit, hoisting per-doc fields) is left as a documented follow-up: GrepHit owns its strings and clones per match (bounded by max_matches), and borrowing would put a lifetime on the public result type. Closes perf-grep-keyword-cache-tokens. 86/86 cargo test green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- README: four native tools (+list_collections example), session resurrection, configure surface (model whitelist, cache_dir, list_models), index_raw collection_description, effective hit meta, Vanessa Automation integration section. - TESTING: tests 20/23/30/31 updated (resurrection, four tools); new tests 37-44 (list_collections, resurrection across restart, collection_description, effective meta, prompt/tool skill parity, file collections, vanessa_kb, auto-reindex watcher). - vanessa-plugin/README: collections (vanessa_kb, synonyms, watcher), search-guide prompt + get_search_guide tool, 41 tools, hook notes. - rust-core/README: hit meta + collection_description on index_raw. - Stale "three native tools" sweep across docs and header comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… bat Copilot review follow-ups on PR #2: - metaFiltersSchema now advertises tags_all/tags_any (the Rust core always parsed them; additionalProperties:false made schema-driven clients drop the tag filters silently). - search tool schema gains the collections array (multi-collection scope, takes precedence over collection) for the same reason. - _build-dll-vcvars.bat: paths derive from the script location with VCVARSALL/CMAKE/NINJA env overrides instead of hard-coded absolute paths, and failed builds now actually exit non-zero (the errorlevel checks never fired - replaced with || exit /b). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot review round 2 on PR #2 - all three confirmed real: - authToken was written by ApplyConfig while HTTP workers compared it in validateAuth (security-critical race): now guarded by authTokenMutex, readers snapshot via authTokenCopy(). timeout had the same race in wait_for: now std::atomic<int> with explicit load/store. - tools/list now drops 1C-registered tools whose names collide with the native ones instead of advertising duplicates; native wins to stay consistent with tools/call routing (native is intercepted first). - MB2WCHAR no longer writes UTF-16 through a wchar_t* cast into the u16string buffer (strict-aliasing UB): the Win32 API fills a wstring and the result is widened element-wise. Embedded lite template refreshed from the rebuilt DLL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RegisterTools (and ApplyConfig tools_json) now fails with an explicit 1C error when a tool definition uses a name owned by the native search subsystem (search/grep/get_segment/list_collections): such a tool would be unreachable because tools/call routes native names to the Rust core before the 1C registry. The cache stays unchanged on rejection. The tools/list dedupe remains as defense-in-depth for caches written by older builds. Embedded lite template refreshed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
http1c_smoke.exe loads bin/libhttp1cWin.dll exactly like 1C
(GetClassObject + stubbed IMemoryManager/IAddInDefBaseEx capturing
AddError/ExternalEvent), drives it over real HTTP and asserts:
T1 load/init/GetProcessId T6 auth 401/200 + token-flip
T2 ApplyConfig + initialize concurrency smoke (no 5xx/hangs)
T3 tools/list union, names unique T7 timeout honored, ToolCall event
T4 reserved native name rejected fired, error within ~1s
loudly, cache unchanged T8 UTF-8/UTF-16 round-trip (RU)
T5 bogus session resurrected (200) T9 native list_collections callable
Wired into CI (lite build runs the harness; rcore absent there so T9
takes the rag_not_installed branch) and into build/run-tests.sh via a
separate build-smoke dir that never relinks a possibly-locked DLL.
54 checks green locally in ~2s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot round 3 follow-ups: - Session resurrection accepted arbitrary client-supplied ids into an unbounded map (memory DoS over time). Tracked sessions are now capped at 256; on overflow the least-recently-active session is evicted (both in createSession and on resurrection). Covered by smoke test T5b: a paced flood of 300 unknown ids is served while GetStatus active_sessions stays <= 256. - RustCore.h included <windows.h> unconditionally, breaking the UNIX build path CMake still supports: the loader is now #ifdef _WIN32 with inert lite stubs (available()=false, empty dispatch) elsewhere. Embedded lite template refreshed. Smoke harness: 59 checks green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot round-3 follow-ups, all five accepted after triage: - RustCore.h: replace std::call_once with mutex+atomic retry so rcore.dll installed NEXT to a running component (plugin "install from archive") is picked up on the next call - no 1C restart. Lock-free fast path once loaded (acquire/release on `ready`). - THREAD-SAFETY.md: verdict table + "pre-existing races" + latent-risk items now reflect the fixes this PR made (authTokenMutex, atomic timeout, rcore_shutdown wired into doStopListen). - CMakeLists: Cargo.lock + .cargo/config.toml added to RCORE_SOURCES so dependency/CRT-flag changes re-trigger the cargo build. - rag_not_installed message now names list_collections too (leftover from round 2). - smoke T10: temp-dir lite copy answers rag_not_installed naming all 4 native tools, then rcore.dll is dropped in and the SAME module flips to full without a restart (flip stage auto-skips on lite CI). 75 checks green; lite Template.bin + bundles repackaged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nfig Remaining round-2 Copilot findings, verified unfixed and now closed: - ApplyConfig + the Timeout property clamp non-positive timeouts to 1s (0/-N made every forwarded call "time out" instantly). Covered by new T7 checks reading the value back through the Timeout property. - get_segment schema: minimum:1 on line_start/line_end (the 1-based contract was prose-only; max_lines already had its minimum). - rust-core/.cargo/config.toml: the comment described the dead Stage-0 staticlib/(/MT) architecture and +crt-static contradicted the official cdylib recipe (ort needs /MD) - a naive `cargo build --features fastembed` produced a broken DLL. Flipped to -crt-static (the rustc default, same flag the build scripts/CI pass via RUSTFLAGS) with a truthful comment. cargo test: 95 green with the flipped flag. - harness: T6 deterministic auth checks now retry through 429 - T5b can legitimately leave the token bucket drained (timing-dependent flake). 79 smoke checks green x2; lite Template.bin + bundles repackaged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment on lines
+466
to
470
| // Non-Windows: a LOCAL converter (no `static`) — each call owns its own, so | ||
| // there is no shared mutable state to race on. | ||
| std::wstring_convert<std::codecvt_utf8_utf16<char16_t>, char16_t> cvt_utf8_utf16; | ||
| return cvt_utf8_utf16.to_bytes(reinterpret_cast<const char16_t*>(src.data()), | ||
| reinterpret_cast<const char16_t*>(src.data() + src.size())); |
Comment on lines
+681
to
+686
| std::u16string method16 = (std::u16string)method; | ||
| std::u16string payload16 = (std::u16string)payload; | ||
| std::string methodUtf8 = WCHAR2MB(std::basic_string_view<WCHAR_T>( | ||
| reinterpret_cast<const WCHAR_T*>(method16.data()), method16.size())); | ||
| std::string payloadUtf8 = WCHAR2MB(std::basic_string_view<WCHAR_T>( | ||
| reinterpret_cast<const WCHAR_T*>(payload16.data()), payload16.size())); |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live-MCP comparison table (substring search_for_steps_by_keywords vs native hybrid search), why substring fails (morphology, word order, no ranking) and what it costs an AI agent (dead-end zeros vs silently wrong steps), plus the follow-ups the comparison produced (segment meta, vanessa_kb, search_string schema fix, RRF score note). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional search subsystem (RAG) to the 1C MCP component: a Rust core that provides semantic (vector) + keyword + hybrid retrieval, regex (
grep), and line slicing (get_segment) as native MCP tools — served in-process, not forwarded to 1C.The core ships as a separate
rcore.dllthat the C++ component loads at runtime, so onelibhttp1cWin.dllserves two distributions:libhttp1cWin.dllonly (pure C++, no Rust)rag_not_installed+ rcore.dll + DirectML.dllWhy a separate runtime-loaded DLL
ort/onnxruntime require the dynamic CRT (
/MD), but the 1C C++ component can't compile under/MD(std::basic_stringstream<char16_t>→ MSVCC2491). So the search core is built as a/MDcdylib (rcore.dll, with a self-contained static onnxruntime) and loaded viaLoadLibrary/GetProcAddress(http-1c-dll/src/RustCore.h). The JSON-string FFI + Rust-side string ownership keep that DLL/CRT boundary safe. The lite component stays/MTand needs no Rust toolchain at all.What's in the Rust core (
rust-core/, cratercore)rcore_dispatch+ version/free/shutdown), panic-guarded.index_segments,index_raw+ token-budget chunker + line offset table): returns immediately, a background worker embeds; collections expose a two-axistext_ready/vector_statusstate polled viastats.RwLock; atomic upsert +delete_document/delete_collection.mode: dense | keyword | hybrid(RRF fusion),filter(combinableany/all/tags meta filters),min_score,max_per_doc.grep(RE2-style regex over stored text, CRLF-normalized, context lines) andget_segment(O(1) line slice via the offset table).MockEmbedder(tests) vs feature-gatedFastEmbedder(fastembed 5.16 + ort 2.0.0-rc.12, multilingual-e5-small, dim 384,query:/passage:prefixes; two model instances so a bulk reindex never blocks queries).configuredevice: cpu | dml | auto(defaultauto) registers DirectML with automatic CPU fallback.C++ component
Native routing of
search/grep/get_segmentinsidetools/call(handled byrcore.dll, not forwarded to 1C); their JSON schemas are merged intotools/listso the tool surface is uniform across lite/full.Verification
cargo test→ 83/83 (mock);cargo test --features fastembed→ 84/84 (real e5 ranks ru/en contract passages above an unrelated sentence, dim 384).rcore.dllimport); full buildslibhttp1cWin.dll+rcore.dll; runtime-loader smoke confirmsavailable()flips false→true anddispatchround-trips, freed via the loaded pointer.CI / release
ci.yml: Rust setup + cache, rust-core tests, lite C++ build + tests, and a--features fastembedbuild check.release.yml: builds and publishes bothhttp1c-addin-lite-v*.zipandhttp1c-addin-full-v*.zip(full bundlesrcore.dll+DirectML.dll).Notes / follow-ups
DirectML.dllis a hard dependency ofrcore.dll(ort's prebuilt onnxruntime bundles the DirectML provider); a future CPU-only onnxruntime build would drop it.new_local) is implemented but not yet exercised with a pre-staged model directory..devtool/features/(kanban board).🤖 Generated with Claude Code