Skip to content

Add codebase query over ephemeral file projections #52

Description

@x1f9-ricon

Idea

Add a codebase query surface that projects selected codebase files into a temporary queryable dataset, then runs read-only SQL against it.

This is the codebase analogue of the session-query prototype from the mass-analysis workbench: keep the filesystem as the source of truth, build an ephemeral relational projection on demand, answer a question, then throw the DB away.

Why

A lot of codebase-wide questions are awkward as bespoke scripts but natural as SQL-like queries:

  • Which repos have workflows but no mise run test?
  • Which repos configure [_.codebase].lint, and where is it enforced?
  • Where do || true / set +e / retry loops show up, grouped by repo and file type?
  • Which .mise/tasks call gh pr merge, notes commit, or git push?
  • Which README/AGENTS/docs mention a command that no longer exists?
  • Which repos have similar task names with divergent bodies?

The value is not replacing grep/lints. It is making ad hoc cross-repo questions cheap and repeatable.

Possible UX

# Safe help/schema; should not scan everything by accident.
codebase query

# Query selected roots/files through an ephemeral projection.
codebase query --root ~/agents/x1f9 --include '*/mise.toml' --include '*/.mise/tasks/**' --sql '
select repo, relpath, line_no, line_text
from lines
where line_text like "%|| true%"
order by repo, relpath, line_no;
'

# Query from an existing manifest.
codebase query --manifest artifacts/files/manifest.tsv --sql-file queries/mise-task-summary.sql --format jsonl

A --text / --allow-source-text boundary may be useful, similar to session analysis:

  • metadata-only: file paths, sizes, hashes, suffixes, line counts;
  • line-text: include bounded line text;
  • full-text: expose full selected file text, only when explicitly requested.

Possible ephemeral schema

Initial tables/views could be small and stdlib-friendly:

  • roots(label, root_path)
  • repos(repo, owner, name, root_path, git_head, default_branch?)
  • files(file_id, repo, root_path, relpath, suffix, bytes, lines, sha256, is_binary)
  • lines(file_id, repo, relpath, line_no, line_text) when text is enabled
  • matches(...) optional helper for regex/LIKE searches
  • toml_kv(...) maybe later, if we decide to parse TOML structurally
  • mise_tasks(...) maybe later, derived from .mise/tasks/**

SQLite in-memory is probably enough for a first pass. No MySQL/Postgres/durable index required.

Non-goals / constraints

  • Do not make a persistent DB the source of truth.
  • Do not scan the whole machine by default.
  • Do not include secrets, auth files, .git, dependencies, build artifacts, encrypted blobs, or raw logs unless explicitly included.
  • Keep source text exposure explicit and bounded.
  • Start with one runtime/backend; optimize or add caches only after the shape proves useful.

Acceptance sketch

  • codebase query with no SQL prints schema/help/examples.
  • A bounded query can project selected roots/manifests into an ephemeral SQLite DB.
  • Read-only SQL (select / with / pragma) can run over the projection.
  • Output formats include at least table/grid, TSV, JSON, and JSONL.
  • Tests cover:
    • no accidental unbounded scan;
    • include/exclude filtering;
    • binary/large-file skips;
    • text-disabled mode not exposing line/source text;
    • a sample query over fixture repos/tasks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions