fix: give memory embedding its own queue so it stops starving behind distillation

[Barsoomx]

What

embed_memory_projection_work_v1 moves from QUEUE_BATCH to QUEUE_REALTIME. One routing
line, two tests.

Why

Embedding gates the entire curation pipeline. corpus_fully_embedded
(curation_shortlist.py:240) is false if any current retrieval document has a NULL
embedding_pgvector, which sets comparison_complete=false project-wide and makes the
judge's publish_new rung ineligible.

It shared QUEUE_BATCH with session distillation and candidate decisions, served by a single
worker at concurrency 3, while worker-realtime sat idle:

worker-realtime    -Q engram-realtime,engram-near-realtime,engram-domain-events -c 3 --prefetch-multiplier=8
worker-highmemory  -Q engram-highmemory,engram-batch                            -c 3 --prefetch-multiplier=1

Measured on the stand tonight, right after deploying #304: six memory_embedding works sat
in ready with queued runs for over twenty minutes, behind a distillation session with 289
outstanding stages. Run by hand they each completed in 0.286–0.598s. They were never
broken — they were never given a worker slot.

The effect is a priority inversion with a feedback loop: every write outcome creates a new
memory version, therefore a new unembedded document, therefore a new embedding work — so
curation continuously generates the work it is blocked on, and that work queues behind the
longest job in the system.

Why QUEUE_REALTIME and not QUEUE_NEAR_REALTIME

Near-realtime is one FIFO queue with observation ingestion
(process_observation_recorded, process_observation_work_v1), and
ReembedMissingEmbeddings bulk-enqueues up to 200 works per beat tick (every 15 minutes).
Prefetch does not reorder anything — an observation behind N embeddings waits for all N. At
0.3s each that is a ~20s ingestion delay; at the configured 30s embedding ceiling it is ~33
minutes. That is the same head-of-line bug being fixed here, with the roles swapped.

QUEUE_REALTIME carried no tasks before this (it appears in celeryconfig.py only at its
definition and in the task_queues declaration, never in task_routes) and already has a
consumer in both worker layouts. A brand-new dedicated queue was rejected deliberately: the
deployed topology lives in a gitignored compose.server.yml that no test pins, so a queue
with no subscriber would silently stall embeddings entirely with no failure signal. Tracked
as B-010.

The name is a stretch for a background projection step. The queue is being used for its
isolation, not its latency class, and the routing table says so.

Not changed

process_candidate_decision_work_v1 stays on batch. It chains a judge chat call plus an
embedding (WORK_TIMEOUT_SPECS[CANDIDATE_DECISION]: chat_calls=1, embedding_calls=1,
soft 240s), so it is long-running by design; moving it would recreate the same inversion and
block the very embeddings this change frees. The asymmetry is the point — embedding is
chat_calls=0.

Tests

• test_embedding_projection_worker_is_registered_on_realtime_queue — the pre-existing route
  test, renamed, all four original assertions kept with only the queue constant flipped.
• test_embedding_projection_does_not_queue_behind_session_distillation
• test_embedding_projection_does_not_share_a_queue_with_observation_ingestion

The last two pin the property rather than the constant, so a future edit that folds these
back together fails even if the queue names change.

Verified: pytest -q engram/memory engram/core → 2098 passed; ruff check and
ruff format --check clean. Route tests were run red before green.

Limits

This fixes scheduling, not generation. Curation still mints one NULL-embedding document per
write outcome, so comparison_complete still oscillates between a decision and its embedding
— one unembedded document can still veto every decision in the project. That coupling is
B-009 and needs a design change (a grace window, or per-shortlist rather than per-corpus
completeness), not a routing change.

[Barsoomx]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[codecov-commenter]

Bundle Report

Bundle size has no change ✅

Affected Assets, Files, and Routes:

view changes for bundle: engram-frontend-client-array-push

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
server/middleware-*.js	2.72kB	3.21kB	548.89% ⚠️
server/middleware-*.js	-2.72kB	495 bytes	-84.59%
static/hKEzOlI8-*.js (New)	1.51kB	1.51kB	100.0% 🚀
static/hKEzOlI8-*.js (New)	77 bytes	77 bytes	100.0% 🚀
static/hra7jyggQ1jlRwRrEr8yv/_buildManifest.js (Deleted)	-1.51kB	0 bytes	-100.0% 🗑️
static/hra7jyggQ1jlRwRrEr8yv/_ssgManifest.js (Deleted)	-77 bytes	0 bytes	-100.0% 🗑️

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.09%. Comparing base (e6b104f) to head (e44adc0).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #305   +/-   ##
=======================================
  Coverage   91.09%   91.09%           
=======================================
  Files         223      223           
  Lines       23344    23344           
=======================================
  Hits        21266    21266           
  Misses       2078     2078           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.