Skip to content

perf: optimize cache freshness, indexing, and search hot paths#13

Open
DevvGwardo wants to merge 3 commits into
MisterGuy420:mainfrom
DevvGwardo:perf/optimize-core-functions-20260302-175417
Open

perf: optimize cache freshness, indexing, and search hot paths#13
DevvGwardo wants to merge 3 commits into
MisterGuy420:mainfrom
DevvGwardo:perf/optimize-core-functions-20260302-175417

Conversation

@DevvGwardo

Copy link
Copy Markdown
Contributor

Summary

This PR improves runtime performance and cache correctness across core indexing/search paths.

✅ Changes made

  • peppy/cache.py

    • Replaced md5 cache keying with sha256.
    • Added optional ttl_seconds support to expire stale cache entries.
    • Added cache freshness signature validation based on indexed file stats (count, max mtime, total size).
    • Added safer UTF-8 reads/writes for cache files.
  • peppy/indexer.py

    • Optimized ignore-path checks with precomputed exact-vs-glob rule buckets.
    • Switched gitignore matching to stable relative POSIX paths.
    • Added adaptive default worker sizing when max_workers is not provided.
    • Precomputes and stores symbol_types and file_extensions in index metadata.
  • peppy/searcher.py

    • Added robust file-pattern matching against relative path, basename, and full path.
    • Reduced grep overhead with tighter matching flow and lower per-line allocations.
    • Builds context payload only when context_lines > 0.
    • Reuses precomputed index statistics when present.

Why

  • Reduce repeated compute in indexing/search hot paths.
  • Improve cache correctness so changed codebases don’t silently reuse stale index data.
  • Improve scaling behavior for larger repositories.

Notes

  • These are non-breaking internal performance/correctness improvements.
  • Existing API surface remains unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant