Repository navigation
fix(server): warm BGE-M3 through patchable loaders - #545
Merged
Merged
Conversation
Resources.startup warmed BGE-M3 through _get_tokenizer and _get_model imported by name from server.query_encoder, which binds the original functions when resources.py is first imported. Tests that mock the model patch server.query_encoder, so the startup warm-up never saw the mock. Every test that boots the server loaded the real 2.3 GB model wherever it was cached, and failed wherever it was not. Startup now calls the loaders through the module, so one patch covers both the warm-up and request-time encoding. The trap was known and routed around rather than fixed: patch_bge_m3_model and one test patched server.resources as well, and three test docstrings described dual-module patching as a rule. Those now patch query_encoder only, and the docstrings say so. The cacheless-boot probe, which patched neither, now mocks the model too. A new test boots the app with recording stand-ins for the two loaders and expects both called. It fails on the old binding, where the real loaders run instead; with the model cached it would see no calls. Measured on Linux with no model available: the full suite goes from 214 failures to 123, all 91 fixed being tests that boot the server, and none new.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Resources.startupwarmed BGE-M3 through_get_tokenizerand_get_model, imported by name fromserver.query_encoder. That binds the original functions whenserver/resources.pyis first imported. Tests that mock the model (mocked_bge_m3) patchserver.query_encoder, so the startup warm-up never saw the mock. Every test that boots the server loaded the real 2.3 GB model wherever it was cached, and failed wherever it was not.mocked_bge_m3's docstring promises the suite "runs without GPU and finishes in milliseconds, not minutes". On a machine with the model cached, every startup test has been loading the real model instead.Change. Startup now calls
query_encoder._get_tokenizer/query_encoder._get_modelthrough the module, so one patch covers both the warm-up and request-time encoding.The trap was known and routed around rather than fixed:
tests/_corpus_helpers.patch_bge_m3_modelpatched both modules.tests/test_bootstrap_mode.pyand onetest_server_startup.pytest patchedserver.resourcesdirectly.All of those now patch
query_encoderonly, and the docstrings say so.test_a_cacheless_boot_says_cacheless_and_not_warmpatched neither module, and now mocks the model too.Verification
Run on Ubuntu (Linux) at
60331e6, built onmainate2357bd, with no BGE-M3 files available. The failure baseline below was measured onmainatd9ddaa8, before #539, #540 and #541 merged. No test those PRs added shows up as a failure here.TestModelWarmupIsPatchableboots the app with recording stand-ins for both loaders and expects["tokenizer", "model"]. On the old binding it fails here with the model-load error. With the model cached, it would see no calls at all.pytest, this branch alone: 123 failures and errors, against the baseline's 214, with none new. The 91 fixed are every test that boots the server:test_security.py,test_tools_all.py,test_snippet_contract.py,test_server_startup.py,test_server_metrics.py, theResourcesintegration tests intests/retrieval/, andtest_corpus_integrity_probes.py.pytestwith test: run chunker tests on a stand-in tokenizer #544 merged in as well: 17 left:test_formal_release_pin.py. Those tests need themfccontract package, which Remove release pins and the formal-contract tooling #547 removes along with the file.test_symlink_loop_is_rejected, which fails in this sandbox for a reason unrelated to the model.ruff check .is clean.make test. Startup tests there should also get noticeably faster, since they stop loading the real model. I could not measure that here.Notes
server/resources.py, plus its import. Startup behaviour is unchanged when nothing is patched.