Skip to content

Implement Stage 10 file and folder ingestion - #10

Merged
aarambh-darshan merged 1 commit into
mainfrom
stage-10-file-folder-ingestion
Jul 1, 2026
Merged

Implement Stage 10 file and folder ingestion#10
aarambh-darshan merged 1 commit into
mainfrom
stage-10-file-folder-ingestion

Conversation

@aarambh-darshan

Copy link
Copy Markdown
Member

Summary

Implements Stage 10 file and folder ingestion for Manas.

What Changed

  • Replaced the placeholder manas-ingest crate with a std-only ingestion pipeline.
  • Added IngestSource::{RawText, File, Folder} and TextChunk with per-chunk source metadata.
  • Added deterministic text normalization and bounded chunking with CHUNK_SIZE and CHUNK_OVERLAP.
  • Added parsers for .txt, .md, .rs, .toml, .json, and .csv.
  • Added recursive deterministic folder walking that skips unsupported files in folders.
  • Added explicit unsupported-format errors for direct file inputs.
  • Added Source display support for readable source metadata.
  • Added Trainer::learn_with_source so file source metadata is attached to learned neurons.
  • Wired manas teach <text|file|folder> [--recursive] through the CLI.
  • Updated teach reporting to aggregate multi-chunk and multi-fact ingestion.
  • Updated README, ROADMAP, CHANGELOG, and ARCHITECTURE for Stage 10 completion.

Tests

  • cargo test -p manas-ingest -p manas-cli -p manas-store
  • cargo test -p manas-learn anti_forgetting
  • cargo test --workspace
  • git diff --check

Notes

Stage 10 is now complete. Stage 11, importance scoring and promotion, is marked as next in the docs.

@aarambh-darshan
aarambh-darshan force-pushed the stage-10-file-folder-ingestion branch from 49f8564 to 272df78 Compare July 1, 2026 09:07
@aarambh-darshan
aarambh-darshan merged commit a1974b8 into main Jul 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant