Repository navigation
fix: eight small findings from the architecture review - #538
Merged
Merged
Conversation
cap_result_list re-serialized the whole payload after each row it dropped, so trimming n rows cost bytes quadratic in n, on the event loop, exactly when the payload was largest. A 200-row payload trimmed to nothing serialized ~6.4M characters for a 61K-character response. Serialize each row once and subtract its bytes as it is dropped. With the default separators a nested value serializes to the same bytes it occupies inside its parent, so the size stays exact: a row costs its own bytes plus the ", " before it. New tests compare the rows kept against a brute-force reference over random payloads with non-ASCII text and escapes, and bound the bytes serialized to a small multiple of the payload. The docstring drops milestone and finding ids. Closes #351.
The parse worker's success-path status write sat outside its try/except. If it raised (SQLite "database is locked" under a concurrent status poll is the likely case), the task ended exceptionally, the row stayed parse_status=running, and every later upload to that notebook was refused until a restart. The restart's orphan sweep then recorded a parse that had succeeded as failed. Retry the complete write twice with a short backoff. If it still fails, write a terminal failed status that says the parse finished but could not be recorded, so the notebook is unblocked. The failure path already guarded its own write; the success path now matches. Closes #367.
The gate looked for the parsed corpus at a working-directory-relative path. Started from anywhere but the repo root (a subdirectory, an IDE runner), it found nothing and skipped with "parsed corpus absent or too small (0 equations)" while the corpus was present, so the math-fidelity gate silently did not run. Resolve the path from the test file, as every other test module does, and name the full path in the skip message. The module's docstrings drop milestone ids. Closes #375.
The restore drill could never validate the citation graph, for two stacked reasons: - It looked only for a DIRECTORY named index/kuzu, but the pinned kuzu stores a database as a single file there. A restored graph was always reported absent, which the drill accepts, so the check passed without looking at the graph. - Once found, it queried MATCH (p:Paper). The schema's node table is papers, so the query raised and the drill would have called a healthy graph unreadable. Accept a file or a directory directly under index/, and count the papers table. A new test builds a graph with the production schema at the path ingest writes it, inserts two papers, and expects a count of 2; the unfixed drill returns None. Docstrings drop finding ids. Refs #302: the other half, checkpointing every WAL database before backup and not just notebooks.db, is not in this change.
ARXMCP_TRUST_REMOTE_CODE=1 is the documented way to opt in to trust_remote_code. server.model_loader reads it straight from the environment, but the startup scan only accepts ARXMCP_* names that are Config fields, so setting it made the server refuse to start. The scan now also accepts a small set of variables the server reads directly, each name imported from the module that reads it. A Config field was the other option, but a pydantic bool would accept "true" and "yes" while model_loader deliberately accepts only "1", and the ingest CLIs, which have no Config, read the variable too. A new test builds the app with the variable set; on the unfixed code it fails with the issue's error. The scan's docstring drops milestone and finding ids. Closes #290.
get_paper pulled every column of up to 10,000 chunk rows to compute a chunk count, a section count and two version strings: both 1024-float embedding columns and the full body text included, for a response that uses none of them. Select the three columns it reads. A new test runs the handler against a real LanceDB table, checks the summary, and checks that no embedding or body column was fetched; the unfixed handler fetches all of them. Comments in the handler drop issue and milestone ids. Closes #353.
The backup checkpointed only notebooks.db before snapshotting, but each notebook's documents.db and paper_metadata.db also run in WAL mode and sit inside the backed-up var/arxmcp/notebooks tree. A file-level copy taken while committed frames sit in a -wal restores stale or as "database disk image is malformed". The helper gains --backup-set <var/arxmcp>, which checkpoints notebooks.db plus every *.db under notebooks/ and prints one "<status> <path>" line each. It skips each notebook's cache/ retrieval.db, which the backup excludes. The wrapper loops over the lines: any degraded database adds its own -wal/-shm sidecars and marks the snapshot partial, as notebooks.db alone did before. New tests build real WAL databases with committed frames in the -wal and check that every backed-up one is truncated and the query cache is left alone; the unfixed helper rejects the flag. The global theorem_names.db is not in the backup set at all; that is #360. Refs #302: with the graph half in this branch, this completes it.
MinerU writes into a scratch directory that is reused for every parse of the same paper, under the same file stem. The output locator took whatever markdown it found there, so a re-parse that exited 0 without writing anything (an encrypted or undecodable PDF, say) returned the previous run's markdown, the renderer re-rendered the old book, and the parse was recorded complete. Note the wall-clock time before MinerU starts and ignore output files older than it, less a 2 s margin for coarse filesystem mtimes. A run that leaves only stale output now fails with a message saying so. New tests: over an hour-old earlier parse, a run that writes nothing raises, and a run that rewrites the same files returns the new content. Docstrings and error messages drop milestone and finding ids. Closes #368.
This was referenced Nov 15, 2026
main now carries its own fixes for the restore drill's citation-graph check and get_paper's column projection, each with a test. Keep main's code for both, drop this branch's duplicate test_handlers_paper.py, and keep this branch's shorter comments in get_paper.
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Eight small fixes from the architecture review, one commit each. Review commit by commit; each commit stands alone and reverts cleanly.
Two of the eight had already landed on
mainin #539 when it merged first: the restore drill's graph check (#302) and theget_paperprojection (#353). Mergingmaininto this branch (d6547f5) keptmain's code and tests for both. In the final diff,358a3a9and3c8d4danow change only comments.Closes #351. Closes #367. Closes #375. Closes #290. Closes #368. Also fixes the WAL half of #302, which #539 left open.
83b57fccap_result_listserializes each row once and subtracts its bytes as it drops rows. Before, it re-serialized the whole payload after every dropped row: O(n²) work on the event loop, exactly when the payload is largest. With default separators a nested value serializes to the same bytes it occupies in its parent, so the arithmetic is exact.758ab9fdatabase is locked), the row stayedrunningand blocked every later upload to the notebook until a restart, which then recorded the successful parse as failed. It now retries twice, then falls back to a terminalfailedthat says the parse finished but could not be recorded.6b446ee358a3a9index/kuzu, and thepaperstable) is onmainfrom #539.3677650ARXMCP_TRUST_REMOTE_CODE=1, the documented opt-in, made the server refuse to start. The startup scan now also accepts variables the server reads directly from the environment, with each name imported from the module that reads it.3c8d4daget_paper's comments. The three-column select is onmainfrom #539.7684e9fnotebooks.db, but each notebook'sdocuments.dbandpaper_metadata.dbalso run in WAL mode inside the backed-up tree. The helper gains--backup-set <var/arxmcp>, which checkpoints all of them and skips each notebook'scache/retrieval.db, which the backup excludes. The wrapper loops over the results, and any degraded database adds its own sidecars and marks the snapshot partial.ce72c89Verification
Every new test below fails on the unfixed code unless it says otherwise.
tests/test_cap_result_list.py, 8 tests.complete, and a persistent one ends atfailed, neverrunning.tests/. There is no test of the test.create_appwith the variable set. The unfixed code fails with the issue's error.main's tests from fix: six more findings from the architecture review #539. The branch's own versions were dropped in the merge.-wal. They check thatnotebooks.db,documents.dbandpaper_metadata.dbare truncated and reportedok, and thatcache/retrieval.dbis left alone. The unfixed helper rejects the flag.CHECKPOINT_DEGRADEDwas set.bash -npasses.Results on the Ubuntu workstation at
d6547f5, which is this branch merged withmainate2357bd:ruff check .is clean.pytest: 5583 passed, 144 skipped, 1 xfailed, 0 failed.-m requires_resticwithARXMCP_RUN_RESTIC_TESTS=1and restic 0.18.1 passes. It runs the backup wrapper end to end.Notes
Configfield: a pydanticboolwould accepttrue,yesandon, whilemodel_loaderaccepts only the literal"1", and the ingest CLIs read the variable too. The broader namespace problem is One ARXMCP_* namespace is shared between a strict extra-forbid server Config and loose ingest/test consumers, so eight documented environment variables hard-fail server startup #347..limit(10000)still capschunk_countat 10,000 for a paper with more chunks. That is unchanged here and should get its own issue.theorem_names.db(index/sqlite/) is not in the backup set at all, so it is not checkpointed. That is The absent-index fallback for find_lemma_by_name materializes the entire chunks table into memory on every call, and theorem_names.db is outside the backup set that would prevent reaching it #360.