feat: per-file error tracking with trove_quality tool#2
Conversation
Record individual file failures during indexing in a new index_errors table so they can be queried later instead of grepping stderr logs. Errors are classified as transient (retryable) or permanent, and automatically marked resolved when a file is successfully re-indexed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the robustness of the indexing pipeline by introducing a comprehensive per-file error tracking system. It enables the system to log, classify, and manage indexing failures at a granular level, distinguishing between temporary and persistent issues. The new Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a valuable feature for per-file error tracking, including a new database table, API endpoint, and associated business logic. The implementation is well-structured and includes relevant tests. My review focuses on a significant performance and correctness issue in the trove_quality tool's implementation. I've provided a detailed code suggestion to address this by leveraging more efficient database query patterns, which will improve scalability and ensure accurate statistics.
| # Compute aggregate counts across all errors (not just the page returned) | ||
| all_errors = db.query_errors(resolved=None, path=path, limit=10_000) | ||
| total = len(all_errors) | ||
| resolved_count = sum(1 for e in all_errors if e["resolved"]) | ||
| unresolved_count = total - resolved_count | ||
|
|
||
| by_type: dict[str, int] = dict(Counter(e["error_type"] for e in all_errors)) |
There was a problem hiding this comment.
The current implementation for calculating aggregate error statistics is inefficient. It fetches up to 10,000 full error records into memory to perform calculations. This approach is memory-intensive, will be slow at scale, and will produce incorrect totals if the number of errors exceeds the 10,000 limit.
This can be performed much more efficiently by executing aggregate queries (COUNT, SUM, GROUP BY) directly in the database, which avoids loading all records into memory and removes the arbitrary limit.
| # Compute aggregate counts across all errors (not just the page returned) | |
| all_errors = db.query_errors(resolved=None, path=path, limit=10_000) | |
| total = len(all_errors) | |
| resolved_count = sum(1 for e in all_errors if e["resolved"]) | |
| unresolved_count = total - resolved_count | |
| by_type: dict[str, int] = dict(Counter(e["error_type"] for e in all_errors)) | |
| # Compute aggregate counts efficiently in the database | |
| where_sql = "WHERE path LIKE ?" if path else "" | |
| params = (path + "%",) if path else () | |
| stats = db.query_one( | |
| f"SELECT COUNT(*) AS total, SUM(resolved) AS resolved_count FROM index_errors {where_sql}", # noqa: S608 | |
| params, | |
| ) | |
| total = stats["total"] if stats else 0 | |
| resolved_count = stats["resolved_count"] if stats and stats["resolved_count"] else 0 | |
| unresolved_count = total - resolved_count | |
| type_rows = db.query( | |
| f"SELECT error_type, COUNT(*) as count FROM index_errors {where_sql} GROUP BY error_type", # noqa: S608 | |
| params, | |
| ) | |
| by_type: dict[str, int] = {row["error_type"]: row["count"] for row in type_rows} |
1 participant
Summary
index_errorstable to track per-file indexing failures with error type classification (transient vs permanent) and resolution tracking_extract_and_store_batched()and auto-resolves on successful re-index in_store_one()trove_qualityMCP tool (tool #10) so LLM clients can query which files failed and decide to retry transient errors viatrove_reindexTest plan
uv run pytest -v— 91 passed, 1 skippeduv run ruff check src tests— cleanuv run mypy src— cleantrove_qualityto confirm errors are recorded🤖 Generated with Claude Code