feat: TypeScript scale — make fmm work on 39k+ file codebases (#ALP-923)#104
Merged
feat: TypeScript scale — make fmm work on 39k+ file codebases (#ALP-923)#104
Conversation
- Add `indicatif = { version = "0.17", features = ["rayon"] }` to Cargo.toml
- Rewrite `sidecar::generate()` with four-phase progress reporting:
- Scan phase: spinner while walking directory tree
- Phase 2 (parse): progress bar with files/s and ETA via rayon ParallelProgressIterator
- Phase 3 (write): sequential progress bar per file written to DB
- Phase 4 (deps): spinner while rebuilding reverse dependency graph
- Add `--quiet` / `-q` flag to `fmm generate` — suppresses all progress,
prints only the final "Done ✓ N file(s) indexed in Xs" summary line
- Timing breakdown always printed (parse/write/deps/other) unless --quiet
- Progress bars suppressed for trivial runs (< 10 dirty files)
- Remove per-file `✓ file.ts` output (replaced by progress bars)
- "all up to date" path now reports total elapsed: "Found N files · all up to date (Xs)"
- Update all call sites: watch.rs, init.rs, and 28 test invocations (quiet=true in tests)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…les) Phase 1 — bulk staleness check (39k queries → 1): - Add db::writer::load_indexed_mtimes(): loads all (path, indexed_at) pairs in a single SELECT, returning a HashMap for in-memory comparison - Replace 39k individual is_file_up_to_date() calls with one bulk query - Parallelize mtime syscalls via rayon par_iter() (OS can pipeline stat() calls; measurable gain on M-series SSD with 12+ cores) Phase 2b — parallel JSON pre-serialization: - Add PreserializedRow struct, ExportRecord, MethodRecord - Add serialize_file_data(): does all serde_json::to_string work outside the single-threaded SQLite transaction — CPU-bound, rayon-safe - Add upsert_preserialized(): writes pre-serialized bytes, no JSON work - Insert parallel pre-serialization pass between Phase 2 (parse) and Phase 3 (write); 195k serde calls across 39k files now run in parallel Phase 3 uses upsert_preserialized() instead of upsert_file_data() so the transaction loop is pure SQLite I/O with zero serialization overhead. upsert_file_data() kept for backwards compatibility (tests, validate path). Expected impact on TypeScript repo (39k files, M-series Mac): - Phase 1: ~2s → <0.1s (single query vs 39k queries) - Phase 3: serialization cost moves to Phase 2b and runs in parallel All 57 tests pass; no data corruption (verified with existing test suite). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two correctness fixes in generate() output: - Add phase2b_elapsed to track parallel serialization time separately. Previously this cost was absorbed into "other", making the timing breakdown useless for diagnosing serialization performance on 39k+ file repos. Now shown as "serialize: Xs" between parse and write. - Use serialized_rows.len() instead of dirty_files.len() for the "Done N file(s) indexed" summary. dirty_files includes files that failed to parse or serialize; serialized_rows is the count actually written to the DB. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extends tree-sitter extraction to capture depth-1 declarations inside function bodies (both exported and non-exported top-level functions). ## What was extracted - Depth-1 function declarations → kind "nested-fn". Fully searchable. Naming: "createTypeChecker.getIndexType" (parent.child convention). - Depth-1 non-trivial prologue var/const/let declarations → kind "closure-state". Non-trivial = has a call expression, new expression, as_expression, or type annotation in its initializer. Trivial literals (false, 0, "") are skipped. Prologue = before the first nested function declaration in the body. - Depth > 1 declarations: NOT extracted (locals inside nested functions). ## Changes src/parser/mod.rs: - Added `kind: Option<String>` field to ExportEntry (skipped in serde for backward compat). - Added ExportEntry::nested_fn() and ExportEntry::closure_state() constructors. src/parser/builtin/typescript.rs: - Added extract_nested_symbols() — walks top-level function_declaration nodes (exported and bare), walks their body for depth-1 nested fns and prologue vars. - Added is_non_trivial_declarator() for prologue filter logic. - Called from parse_with_aliases() after class method extraction. - 7 new unit tests covering all acceptance criteria. src/manifest/mod.rs: - FileEntry gains nested_fns and closure_state HashMaps (#[serde(skip)]). - From<Metadata> routes ExportEntry by kind into the correct bucket. - add_file() clears nested_fns/closure_state from method_index on update. - remove_file() clears nested_fns/closure_state from method_index on remove. - All nested symbols (all kinds) go into method_index for fmm_search. src/search.rs: - Added step 2 (exact method_index match) and 2b (fuzzy method_index scan). - "silentNeverType" now finds "createTypeChecker.silentNeverType" via 2b. - Added seen_exports.insert() after fuzzy export hits to prevent duplicates. src/format/yaml_formatters.rs: - format_file_outline() refactored to handle nested_fn and closure_state sub-entries. - Nested functions always shown under parent. Closure-state shown only with include_private. - Header annotation updated (e.g. "52 lines, 3 nested functions, 2 closure-state"). - Sub-entries sorted by start line when mixed kinds are present. src/db/mod.rs: - SCHEMA_VERSION bumped to 2. - methods table gains `kind TEXT` column (NULL = class method). src/db/writer.rs: - MethodRecord gains kind field. Both upsert paths pass kind to the INSERT. src/db/reader.rs: - load_methods() reads kind column, routes into nested_fns/closure_state/methods. src/mcp/tests.rs + tests/cross_language_validation.rs: - Added ..Default::default() for new FileEntry fields in mcp tests. - Updated typescript_real_repo_internal_module to assert export_names().is_empty() instead of exports.is_empty() (nested symbols now extracted from non-exported fns). fixtures/typescript/mega_function.ts: - New fixture: createTypeChecker with prologue vars (non-trivial and trivial), depth-1 nested fns, depth-2 nested fn, and non-exported internalHelper. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…comment - Drop source_bytes parameter from is_non_trivial_declarator(): it was never used inside the function. The let _ = source_bytes suppression inside the inner loop was the only reference, added purely to silence the compiler warning. Call site updated accordingly. - Fix step 2 comment in bare_search(): the old comment claimed step 2 handles "silentNeverType" -> "createTypeChecker.silentNeverType" but that is step 2b (fuzzy contains). Step 2 is an exact lookup by the full dotted key, so the comment now reflects that. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes
fmm generateviable for large TypeScript codebases (39k+ files) through three coordinated improvements:indicatifprogress bars with phase timing,--quietflag for CImethod_index, searchable viafmm_searchKey design decisions
kind TEXTcolumn on methods table)nested-fnandclosure-statekinds added to ExportEntryTest plan
fmm generateon a large TypeScript repo shows progress bars and timingfmm generate --quietsuppresses progress, prints summary onlyfmm search "nested function name"finds symbols inside mega-functions🤖 Generated with Claude Code