You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
get_paper silently caps chunk_count at 10,000 #542
handle_get_paper (server/handlers/paper.py, the chunks query, around line 169 on main at d9ddaa8) fetches a paper's rows with .limit(10000), then reports:
chunk_count = arrow.num_rows;
section_count from the same rows;
chunker_version and embedder_version from the first row.
A paper with more than 10,000 chunks reports exactly 10,000, and its section count covers only those rows. Nothing in the response says the numbers were truncated, which breaks the CLAUDE.md §4.9 rule that a degraded answer must never look complete.
10,000 chunks is not reachable by an ordinary arXiv paper, but textbook ingest puts a whole book under one paper_id (textbook:<slug>), and a long monograph or a multi-volume text could approach it.
Found while fixing #353 in #538. That PR changed which columns the query reads, not the limit.
Suggested remedy
Take the count from chunks_table.count_rows(filter) rather than from the rows fetched. Fetch the summary columns without a row cap, or derive section_count from a projection of section_path alone. If any cap stays, the response should say when it was hit rather than return a number that looks exact.
Add a test against a real LanceDB table with more rows than the cap (or with the cap patched down) that expects the true count. tests/test_handlers_paper.py from #538 has the fixture.
Activity