Skip to content

docs: fix drift in Background Processing - #273

Draft
jack-arturo wants to merge 1 commit into
mainfrom
docs/audit-architecture-background-processing-20260730
Draft

docs: fix drift in Background Processing#273
jack-arturo wants to merge 1 commit into
mainfrom
docs/audit-architecture-background-processing-20260730

Conversation

@jack-arturo

Copy link
Copy Markdown
Member

Docs-accuracy audit of /docs/architecture/background-processing (never previously reviewed). 4 proven discrepancies fixed.

Claim Current state Fix Evidence
"/health exposes real-time statistics for all background systems", with rows for embedding.queue_depth, consolidation.last_runs, consolidation.next_runs health_data contains only status, falkordb, qdrant, memory_count, vector_count, sync_status, vector_dimensions, enrichment, timestamp, graph. There is no embedding or consolidation key at all. Scoped the sentence to the enrichment pipeline, dropped the three fabricated rows, added the real enrichment.status row, and pointed consolidation timings at GET /consolidate/status automem@8ff266e:automem/api/health.py#L86
enrichment.pending = "Memories not yet enriched in graph" pending is len(state.enrichment_pending) — the dedup set of enqueued jobs the worker has not yet claimed; it is discarded the moment a job goes inflight Reworded to "Enqueued but not yet picked up by the worker" automem@8ff266e:automem/api/health.py#L31
"Admin Endpoints — advanced monitoring available via admin token" over /enrichment/status, /enrichment/reprocess, /consolidate, /consolidate/status; /enrichment/status described as returning "sample pending IDs" Only enrichment_reprocess() calls require_admin_token(). /enrichment/status, /consolidate and /consolidate/status have no admin gate. /enrichment/status returns status, queue_size, pending, inflight, max_attempts, stats, classification — no ID samples. Retitled to "Monitoring Endpoints", stated that only /enrichment/reprocess needs X-Admin-Token, and corrected the /enrichment/status description automem@8ff266e:automem/api/enrichment.py#L38, automem@8ff266e:automem/api/enrichment.py#L25
Relationship-count LRU cache linked to consolidation.py#L152 @lru_cache(maxsize=10000) is at consolidation.py#L201; L152 is an unrelated decay parameter assignment Corrected anchor to #L201 and repinned to the audited SHA automem@8ff266e:consolidation.py#L201

Also repinned the two automem/runtime_wiring.py permalinks from ebcf5f16 to 8ff266e (that file changed since the old pin). The other seven source-note links point at files that are byte-identical between ebcf5f16 and 8ff266e, so they were left alone.

Verified correct and untouched: the eight-step init_* startup sequence (runtime_wiring.py#L84-L92), every default in the Enrichment / Embedding / Consolidation configuration tables (automem/config.py, app.py#L217-L218), the enrichment retry model (flat ENRICHMENT_FAILURE_BACKOFF_SECONDS sleep, give up after ENRICHMENT_MAX_ATTEMPTS), and the /admin/reembed reference.

Verified against: automem@8ff266e62e65cb2e81719a765b05f64a2361a127

Questions

  • The Threading Model table lists three worker threads but omits the sync worker, even though the page's own "Four Independent Systems" table counts it as one of the four. Intentional simplification, or should the sync worker get a row?
  • "Relationship Count Caching (80% consolidation speedup)" and "Embedding Batching (40-50% cost reduction)" are unsourced performance numbers. Not code-verifiable — left as-is.

Unverified

  • None. Every code-anchored claim on this page resolved to current source.

Follow-ups

  • The Worker Coordination diagram labels the sync worker SyncWorker<br/>sync/runtime_bindings.py. The scheduling closure does live there, but the actual polling loop is automem/sync/runtime_worker.py. Cosmetic, not fixed.
  • /health publishes enrichment.queue_depth from enrichment_queue.qsize(), which excludes inflight jobs — the same "queue depth" wording appears on /docs/operations/health with an explicitly wrong definition ("pending + inflight"). Being handled on that page.

Generated by Claude Code

- /health exposes only enrichment.* metrics; drop fabricated embedding.*
  and consolidation.* rows and point consolidation timings at
  GET /consolidate/status
- only /enrichment/reprocess requires the admin token; /enrichment/status,
  /consolidate and /consolidate/status are covered by the API token guard
- /enrichment/status returns no pending-ID samples
- relationship-count LRU cache is consolidation.py L201, not L152
- repin automem/runtime_wiring.py permalinks to 8ff266e

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EXhZREMSHTSqk3bhiNaX4k
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying automem-website with  Cloudflare Pages  Cloudflare Pages

Latest commit: 8e1319a
Status: ✅  Deploy successful!
Preview URL: https://152bd7f1.automem-website.pages.dev
Branch Preview URL: https://docs-audit-architecture-back.automem-website.pages.dev

View logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants