Skip to content

fix: eight small findings from the architecture review - #538

Merged
chris-dare-dev merged 9 commits into
mainfrom
claude/bold-goldberg-igonyq
Nov 15, 2026
Merged

chris-dare-dev merged 9 commits into
mainfrom
claude/bold-goldberg-igonyq

Conversation

@chris-dare-dev

@chris-dare-dev chris-dare-dev commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

What

Eight small fixes from the architecture review, one commit each. Review commit by commit; each commit stands alone and reverts cleanly.

Two of the eight had already landed on main in #539 when it merged first: the restore drill's graph check (#302) and the get_paper projection (#353). Merging main into this branch (d6547f5) kept main's code and tests for both. In the final diff, 358a3a9 and 3c8d4da now change only comments.

Closes #351. Closes #367. Closes #375. Closes #290. Closes #368. Also fixes the WAL half of #302, which #539 left open.

Commit Issue Change
83b57fc #351 cap_result_list serializes each row once and subtracts its bytes as it drops rows. Before, it re-serialized the whole payload after every dropped row: O(n²) work on the event loop, exactly when the payload is largest. With default separators a nested value serializes to the same bytes it occupies in its parent, so the arithmetic is exact.
758ab9f #367 The parse worker's success-path status write was outside its try/except. If it raised (most likely SQLite database is locked), the row stayed running and blocked every later upload to the notebook until a restart, which then recorded the successful parse as failed. It now retries twice, then falls back to a terminal failed that says the parse finished but could not be recorded.
6b446ee #375 The equations-parity test resolved the corpus path against the working directory, so starting pytest from anywhere but the repo root skipped the math-fidelity gate with a wrong "corpus absent" reason. The path is now anchored to the test file.
358a3a9 #302 (graph) Now only drops issue ids from the drill's comments. The fix (a single-file graph at index/kuzu, and the papers table) is on main from #539.
3677650 #290 ARXMCP_TRUST_REMOTE_CODE=1, the documented opt-in, made the server refuse to start. The startup scan now also accepts variables the server reads directly from the environment, with each name imported from the module that reads it.
3c8d4da #353 Now only shortens get_paper's comments. The three-column select is on main from #539.
7684e9f #302 (WAL) The backup checkpointed only notebooks.db, but each notebook's documents.db and paper_metadata.db also run in WAL mode inside the backed-up tree. The helper gains --backup-set <var/arxmcp>, which checkpoints all of them and skips each notebook's cache/retrieval.db, which the backup excludes. The wrapper loops over the results, and any degraded database adds its own sidecars and marks the snapshot partial.
ce72c89 #368 MinerU's scratch directory and file stem are reused for every parse of a paper. So a re-parse that exited 0 without writing anything handed back the previous run's markdown, and the old book was re-rendered and recorded complete. Output files older than the run's start (less a 2 s margin for coarse mtimes) are now ignored.

Verification

Every new test below fails on the unfixed code unless it says otherwise.

Results on the Ubuntu workstation at d6547f5, which is this branch merged with main at e2357bd:

  • ruff check . is clean.
  • Full pytest: 5583 passed, 144 skipped, 1 xfailed, 0 failed.
  • -m requires_restic with ARXMCP_RUN_RESTIC_TESTS=1 and restic 0.18.1 passes. It runs the backup wrapper end to end.

Notes

cap_result_list re-serialized the whole payload after each row it
dropped, so trimming n rows cost bytes quadratic in n, on the event
loop, exactly when the payload was largest. A 200-row payload trimmed
to nothing serialized ~6.4M characters for a 61K-character response.

Serialize each row once and subtract its bytes as it is dropped. With
the default separators a nested value serializes to the same bytes it
occupies inside its parent, so the size stays exact: a row costs its
own bytes plus the ", " before it.

New tests compare the rows kept against a brute-force reference over
random payloads with non-ASCII text and escapes, and bound the bytes
serialized to a small multiple of the payload. The docstring drops
milestone and finding ids.

Closes #351.
The parse worker's success-path status write sat outside its
try/except. If it raised (SQLite "database is locked" under a
concurrent status poll is the likely case), the task ended
exceptionally, the row stayed parse_status=running, and every later
upload to that notebook was refused until a restart. The restart's
orphan sweep then recorded a parse that had succeeded as failed.

Retry the complete write twice with a short backoff. If it still
fails, write a terminal failed status that says the parse finished
but could not be recorded, so the notebook is unblocked. The failure
path already guarded its own write; the success path now matches.

Closes #367.
The gate looked for the parsed corpus at a working-directory-relative
path. Started from anywhere but the repo root (a subdirectory, an IDE
runner), it found nothing and skipped with "parsed corpus absent or
too small (0 equations)" while the corpus was present, so the
math-fidelity gate silently did not run. Resolve the path from the
test file, as every other test module does, and name the full path in
the skip message.

The module's docstrings drop milestone ids.

Closes #375.
The restore drill could never validate the citation graph, for two
stacked reasons:

- It looked only for a DIRECTORY named index/kuzu, but the pinned
  kuzu stores a database as a single file there. A restored graph
  was always reported absent, which the drill accepts, so the check
  passed without looking at the graph.
- Once found, it queried MATCH (p:Paper). The schema's node table is
  papers, so the query raised and the drill would have called a
  healthy graph unreadable.

Accept a file or a directory directly under index/, and count the
papers table. A new test builds a graph with the production schema at
the path ingest writes it, inserts two papers, and expects a count of
2; the unfixed drill returns None. Docstrings drop finding ids.

Refs #302: the other half, checkpointing every WAL database before
backup and not just notebooks.db, is not in this change.
@chris-dare-dev chris-dare-dev changed the title fix: linear result capping, parse status, parity gate path fix: result capping, parse status, parity gate, restore drill Nov 14, 2026
ARXMCP_TRUST_REMOTE_CODE=1 is the documented way to opt in to
trust_remote_code. server.model_loader reads it straight from the
environment, but the startup scan only accepts ARXMCP_* names that
are Config fields, so setting it made the server refuse to start.

The scan now also accepts a small set of variables the server reads
directly, each name imported from the module that reads it. A Config
field was the other option, but a pydantic bool would accept "true"
and "yes" while model_loader deliberately accepts only "1", and the
ingest CLIs, which have no Config, read the variable too.

A new test builds the app with the variable set; on the unfixed code
it fails with the issue's error. The scan's docstring drops milestone
and finding ids.

Closes #290.
@chris-dare-dev chris-dare-dev changed the title fix: result capping, parse status, parity gate, restore drill fix: five small findings from the architecture review Nov 15, 2026
get_paper pulled every column of up to 10,000 chunk rows to compute
a chunk count, a section count and two version strings: both
1024-float embedding columns and the full body text included, for a
response that uses none of them. Select the three columns it reads.

A new test runs the handler against a real LanceDB table, checks the
summary, and checks that no embedding or body column was fetched; the
unfixed handler fetches all of them. Comments in the handler drop
issue and milestone ids.

Closes #353.
The backup checkpointed only notebooks.db before snapshotting, but
each notebook's documents.db and paper_metadata.db also run in WAL
mode and sit inside the backed-up var/arxmcp/notebooks tree. A
file-level copy taken while committed frames sit in a -wal restores
stale or as "database disk image is malformed".

The helper gains --backup-set <var/arxmcp>, which checkpoints
notebooks.db plus every *.db under notebooks/ and prints one
"<status> <path>" line each. It skips each notebook's cache/
retrieval.db, which the backup excludes. The wrapper loops over the
lines: any degraded database adds its own -wal/-shm sidecars and marks
the snapshot partial, as notebooks.db alone did before.

New tests build real WAL databases with committed frames in the -wal
and check that every backed-up one is truncated and the query cache
is left alone; the unfixed helper rejects the flag. The global
theorem_names.db is not in the backup set at all; that is #360.

Refs #302: with the graph half in this branch, this completes it.
MinerU writes into a scratch directory that is reused for every parse
of the same paper, under the same file stem. The output locator took
whatever markdown it found there, so a re-parse that exited 0 without
writing anything (an encrypted or undecodable PDF, say) returned the
previous run's markdown, the renderer re-rendered the old book, and
the parse was recorded complete.

Note the wall-clock time before MinerU starts and ignore output files
older than it, less a 2 s margin for coarse filesystem mtimes. A run
that leaves only stale output now fails with a message saying so.

New tests: over an hour-old earlier parse, a run that writes nothing
raises, and a run that rewrites the same files returns the new
content. Docstrings and error messages drop milestone and finding ids.

Closes #368.
@chris-dare-dev chris-dare-dev changed the title fix: five small findings from the architecture review fix: eight small findings from the architecture review Nov 15, 2026
main now carries its own fixes for the restore drill's citation-graph
check and get_paper's column projection, each with a test. Keep main's
code for both, drop this branch's duplicate test_handlers_paper.py, and
keep this branch's shorter comments in get_paper.
@chris-dare-dev
chris-dare-dev merged commit 554c83b into main Nov 15, 2026
1 check passed
@chris-dare-dev
chris-dare-dev deleted the claude/bold-goldberg-igonyq branch November 15, 2026 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment