Skip to content

[SSoC'26]Feat: document ingestion queue for large files#390

Merged
imDarshanGK merged 2 commits into
imDarshanGK:mainfrom
diksha78dev:fix/issue-291
Jun 12, 2026
Merged

[SSoC'26]Feat: document ingestion queue for large files#390
imDarshanGK merged 2 commits into
imDarshanGK:mainfrom
diksha78dev:fix/issue-291

Conversation

@diksha78dev

Copy link
Copy Markdown
Contributor

Closes #291.

Built a lightweight document ingestion queue for large files using FastAPI BackgroundTasks and SQLite.

  • Background Processing: Document embedding now runs asynchronously, preventing server timeouts on massive files.
  • Batched Indexing: Yields the GIL between batches to keep the main event loop responsive.
  • Real-time UI Tracking: Added status tracking (queued, processing, failed) and UI polling to display live chunk progress.
  • Crash Safety: Wrapped AI imports in error boundaries to cleanly mark documents as failed if the process crashes.

@vercel

vercel Bot commented Jun 11, 2026

Copy link
Copy Markdown

@diksha78dev is attempting to deploy a commit to the Darshan's projects Team on Vercel.

A member of the Team first needs to authorize it.

@imDarshanGK imDarshanGK added SSoC26 Part of Social Summer of Code 2026 Hard Larger or more advanced changes labels Jun 12, 2026
@imDarshanGK imDarshanGK merged commit 1534279 into imDarshanGK:main Jun 12, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Hard Larger or more advanced changes SSoC26 Part of Social Summer of Code 2026

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a document ingestion queue for large files

2 participants