[AEP][high] AEP-20260924-BE001: restore corpus vectors after a failed upsert

[mattmre]

Description

A failed Qdrant chunk during sedimentation stops later chunks and upserts the pre-cycle vectors for ids already written. The collision log is cleared only when every chunk succeeds. Offline distillation and a cycle that started with no adapter file reload the pre-cycle adapter after a failed corpus commit.

Follow-up a9d1225 is the commit this score covers. cf8aab1 did not.

Type of Change

• Bug fix

Testing

Spark Python unittest test_sedimentation_corpus_rollback.py test_sedimentation_trainer.py: 29 passed on a9d1225. Ruff passed on the seven touched files. Review grok-tierb-335b-20260924 re-ran that suite. No live Qdrant server and no browser.

Brutal Honesty

• Compensation is an upsert of the pre-cycle vectors, not a follow-up retrieve. The log says those vectors were written back only when that upsert returns.
• The first review of cf8aab1 scored 44 because the offline snapshot was taken after the trained save. This score is for a9d1225.
• Local commit 176e1f4 was not pushed.

"We don't steal — we give credit to the projects and people who came
before us. We entertain ideas, we research, we try to see if something
from another angle or study can open insights into our own work, and
eventually we will publish something novel that brings a productive return
for humanity."

EVIDENCE: 29 sedimentation tests passed on a9d1225; review grok-tierb-335b-20260924
SMOKE: targeted unittest only; bash scripts/smoke.sh was not run on this branch
BHS_SELF_DRAFT: 100
BHS_SELF_DRAFT_AGENT: grok-aep-fix-20260924
BHS_TIER_B: 100
BHS_TIER_B_AGENT: grok-tierb-335b-20260924
BHS_TIER_B_SEVERITY: none
BHS_OFFICIAL: 100
CARRY_FORWARD: none
DEFERRED_SCOPE: none
LOOP_ITERATIONS: 2
OPERATOR_OVERRIDE: none

[mattmre]

Review of cf8aab1 (no prior threads). The helper does stop later chunks and, when the compensating upsert returns, writes the pre-cycle vector back. These acceptance criteria are still open. Do not open a second PR. Do not replace this branch with local commit 176e1f4.

1. run_offline_distillation saves the trained adapter at antigravity_engine.py:2307 and :2377, then reads prior_adapter at :2388-2390. The failure path writes those same bytes back. Snapshot the pre-cycle file before that save. A failed corpus commit must reload that snapshot. AC3.

2. If no adapter file exists when SafeTrainingContext is entered (antigravity_engine.py:1908, sedimentation.py:132), create_checkpoint copies nothing. The cycle then saves trained weights, the context does not remove them, and adapter.load reloads the trained file. Save the in-memory adapter before entering the context when the file is absent, so the checkpoint has the pre-cycle weights. AC2.

3. Callers always log that pre-cycle vectors were written back (antigravity_engine.py:2085-2088, sedimentation.py:209-212) even when _restore_written returns false (sedimentation_trainer.py:174). Say that only when the compensating upsert returned and at least one id was written. Keep the corpus_not_restored log.

4. Offline passes payload_map=None. Compensation retrieves outside the try (sedimentation_trainer.py:107-113). A retrieve error skips both corpus_not_restored and the adapter rewind. Catch that retrieve and still reload the pre-cycle adapter.

5. test_sedimentation_corpus_rollback.py only calls the helper. Add tests that fail if offline, or a sedimentation/hierarchical cycle with no pre-cycle file, leaves trained weights, and if a failed compensating upsert still logs written back. test_sedimentation_trainer.py is red on this SHA: test_partial_failure now logs corpus_not_restored because originals were omitted (keep that log; update the count), and test_value_error_handling looks for Invalid vector data in a message that now says Update batch 0 failed.

Tier B on this SHA: 44, severity critical. This commit does not merge.

Parent reading note: docs/aep-remediation/20260924-merge/01-corpus-restore-reading.md. Full probe writeup: the review that produced this comment.

[mattmre]

Fix pushed as a9d1225.

Offline distillation snapshots adapter bytes before either trained save and writes those bytes back with os.replace when an upsert fails. Sedimentation and hierarchical sedimentation save a missing adapter file before SafeTrainingContext. sync_vectors_to_qdrant returns compensated, and the failure log says the pre-cycle vectors were written back only when that flag is true. A compensation retrieve that raises is logged as Corpus was not restored. and does not skip the adapter reload.

Parent re-ran test_sedimentation_corpus_rollback.py and test_sedimentation_trainer.py with Spark Python: 29 passed. Ruff passed. Not merged. A fresh Tier B review of a9d1225 is still required. cf8aab1 stays at official 44.