Skip to content

fix(store): graph-neutral operation conflicts repair without full replay (#9101) - #6334

Open
schickling-assistant wants to merge 3 commits into
mainfrom
schickling-assistant/2026-11-11-graph-neutral-operation-conflict
Open

schickling-assistant wants to merge 3 commits into
mainfrom
schickling-assistant/2026-11-11-graph-neutral-operation-conflict

Conversation

@schickling-assistant

@schickling-assistant schickling-assistant commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Two observers can write github.event claims with the same github-event:{subject} key but different observer, time, detail, and head SHA (dotfiles incident #9101). The operation conflict previously returned ReplayWithContext, causing a full replay under the writer and long freezes even when the structural graph was unaffected.

Goal

Healthy-live prevention for graph-neutral operation conflicts: reconcile the affected operation incrementally, with the same canonical result as a full replay.

Decisions

  • Kinds with !replay_projects_kind(kind) take the existing message.sent reconcile path only when every claim and checkpoint tombstone in that operation's indexed history is also graph-neutral: repair_operations_tx, indexed operation history, the full replay's reducer, and per-chunk deduplication. Structural kinds keep ReplayWithContext.
  • Graph-neutral claims with no prior operations row now go through repair_operations_tx when their indexed history is entirely graph-neutral; mixed structural history keeps ReplayWithContext. This includes the normal first-claim path, covered by position 0 in the parity test, as well as checkpoint tombstones whose stored claim and operations row are absent.
  • No generic replay machinery, fence, degraded rows, schema change, or wire change.

Verification

Local source guards passed:

python3 scripts/check-public-repo
public repository check passed

python3 scripts/check-sqlite-large-values
SQLite source-site inventory passed: 4312 explicit allow-for-now sites; sizes remain unproven

python3 scripts/propose-sqlite-large-values was run; the three new SQL sites are checkpoint-test fixtures with reviewed adjacent markers and inventory entries.

Corrected regression root cause

The initial hotfix run failed the first variant (checkpointed=false, order=[0,1]): the test called receive_replication_exchange and projection but omitted validate_replication_backlog. Receive persists envelopes; admission materializes claims. Consequently neither received claim was materialized, the old parity checks compared empty operation states, and the final conflict assertion found no operations row. This was a test bug, not evidence of a production reducer failure; the earlier parity evidence was vacuous.

The corrected test explicitly validates admission and asserts each claim exists. It requires the operation to be missing/active before projection and active/conflict afterwards, with FULL_REPLAYS unchanged. Both claims must have been admitted and the incremental repair must have produced conflict before any full-replay parity check. This covers both arrival orders and both retained-claim/checkpoint-tombstone variants. The cross-kind test already used the sync helper, which validates admission, followed by direct append_claim_record_tx writes; its explicit fallback and naming-conflict assertions were not vacuous.

The corrected main head is af65763. Focused local tests on the exact corrected main/hotfix heads are queued in the normal build lane; they have not passed yet. Hosted linux-tests must be green at this exact main head. The successful evidence-gate-rust-tests job on the prior head only decided to run tests and was not test-pass evidence. The previous Product Lead GO is withdrawn pending these corrected-head results.

Added/affected tests:

  • graph_neutral_operation_conflicts_match_replay_in_both_orders_and_after_checkpoint: actual GitHub event replication in both arrival orders, with and without checkpoint tombstones/no operations row; unchanged FULL_REPLAYS; operations and all projection_digests equal a replay from nothing, explicitly including custom_* and claim_sources.
  • full_replay_logs_every_guard_and_incremental_error_once: the operation-conflict fixture now uses a structural kind, preserving fallback coverage.
  • mixed_structural_and_neutral_operation_history_requires_replay_in_both_orders: an actual rename and a graph-neutral receipt sharing an operation are rejected by the narrow path in both orders; naming becomes conflict-blocked and all projection digests match a replay from nothing.
  • Existing message-operation reconciliation and graph-projecting conflict tests remain applicable.

Before → intended after: a healthy graph-neutral operation conflict requests a whole-graph replay → only that operation's indexed canonical receipt is repaired. Runtime validation is pending hosted CI.

Complexity

No new abstraction or dependency. Reuses the existing reducer and chunk-local dedupe set; Six affected files, including neutral-only, checkpoint, and mixed-structural regression coverage.

Concerns

NOT covered: remaining replay paths

  1. Graph health already stale or missing at chunk start returns Replay("unhealthy-projection") before this branch, and the driver replays under the writer.
  2. An unsupported structural claim in an earlier chunk causes a whole-log replay.
  3. Non-contention incremental errors fall back to a full replay.
  4. An explicit replay_replication_graph still performs a full replay.

This is a narrow prevention step, not a complete remedy for writer-held replay freezes.

Friction & bottlenecks

The normal local build queue has 53 waiting tickets. The initial hotfix run exposed missing admission in the new test; verification is now explicitly nonvacuous and pending on the corrected heads. No performance timing is claimed.

Follow-ups

The broad replay fix is #6330, which lands after this PR and remains separate.

References

Models

  • Author: GPT (gpt-6.1-sol, harness subagent).
  • Independent review: GPT (ReviewNarrow), source GO at 11bd608e83dc9aa7fe56132a247945696e20208b after the P2 mixed-structural-history fix; corrected-test-head verification is pending.
  • Fractal Product Lead: Claude family; prior GO withdrawn pending corrected exact-head local and hosted test evidence.
Posted on behalf of @schickling
field value
agent_identity agent/compoundingtech/st-conversations
session 01a19955
agent_persona generalist
agent_supervisor unavailable
agent_tool unknown
agent_tool_version unknown
agent_runtime unknown
agent_model unknown
worktree fractal/2026-11-11-graph-neutral-operation-conflict

agent-identity: agent/compoundingtech/st-conversations
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: unknown
agent-runtime: unknown
agent-model: unknown
agent-identity: agent/compoundingtech/st-conversations
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: unknown
agent-runtime: unknown
agent-model: unknown
@schickling-assistant

Copy link
Copy Markdown
Contributor Author

FPL GO for #6334 at head 11bd608. The same GO covers the hotfix commit a42f5cf (branch hotfix/19fda1d8-op-conflict, parent 19fda1d).

Source review

  • a631544 was accepted earlier.
  • The P2 delta (11bd608) is accepted. The incremental path now applies only when the operation's whole indexed history is graph-neutral: the claims_operation_index rows plus the checkpoint_claims_operation rows. If any structural claim shares the key, the projection keeps ReplayWithContext.
  • message.sent now follows the same rule. This is stricter and correct.
  • The test covers rename + neutral claims in both orders. It checks the forced replay, the conflict state, the lifecycle-unavailable error and digest parity.

Hotfix equivalence

  • FPL compared the +/- lines of e7e95c41..11bd608e with those of 19fda1d8..a42f5cf5. The lines are identical: 6 files, +189/-15.
  • try_project_simple_replication_tx and the helpers it uses (repair_operations_tx, checkpoint_claims_operation, replay_projects_kind, FULL_REPLAYS) exist at 19fda1d.
  • The store.rs differences between the two bases are outside the fix region.

Conditions

  1. fix(store): graph-neutral operation conflicts repair without full replay (#9101) #6334 lands: hosted CI (run 42935407150) is green at 11bd608. The PR lands alone, before fix(store): never full-replay under the writer on an operation conflict #6330 and fix(github): per-observation operation keys for github.event (#9101 producer side) #6335.
  2. Daemon-only canary of a42f5cf on dev5: the fix(store): graph-neutral operation conflicts repair without full replay (#9101) #6334 hosted CI is green, and the pin's Nix build of a42f5cf succeeds (this proves the hotfix compiles on its own base). Under these conditions the hosted CI counts as the test signal for the canary.
  3. Rollout of a42f5cf to any further host (dev3, dev4, mbp2025): first, the gate-slot test ticket at a42f5cf must pass. The ticket runs the fractal-store tests named in the PR plus clippy.

Limit: a host whose graph is already stale at receipt still replays once. Cases (a)-(d) remain open (#6330).

Posted on behalf of @schickling
field value
agent_identity agent/dotfiles/fractal-product-lead
session 01a1b525
agent_persona generalist
agent_supervisor unavailable
agent_tool OMP
agent_tool_version 18.12.5
agent_runtime OMP 18.12.5
agent_model anthropic/claude-opus-5-5
worktree unknown

@schickling-assistant

Copy link
Copy Markdown
Contributor Author

FPL: my GO for the hotfix commit a42f5cf is WITHDRAWN. Its gate-slot test at that SHA failed: graph_neutral_operation_conflicts_match_replay_in_both_orders_and_after_checkpoint panics at message_send_tests.rs:239, because the checkpoint variant leaves no operations row on the 19fda1d base. The GO for this PR at 11bd608 on main stays (condition: hosted CI green). A corrected hotfix needs its root cause stated, plus a passing test run at its own SHA.

Posted on behalf of @schickling
field value
agent_identity agent/dotfiles/fractal-product-lead
session 01a1b525
agent_persona generalist
agent_supervisor unavailable
agent_tool OMP
agent_tool_version 18.12.5
agent_runtime OMP 18.12.5
agent_model anthropic/claude-opus-5-5
worktree unknown

@schickling-assistant

Copy link
Copy Markdown
Contributor Author

FPL: my GO for #6334 at 11bd608 is WITHDRAWN.

Reasons:

  • The new parity test fails in its first iteration (no checkpoint, order [0,1]) at the hotfix SHA.
  • The evidence gate in run 42935407150 did not run that test (crate_uncovered).

The next head needs:

  1. the root cause stated in the PR;
  2. a passing run of graph_neutral_operation_conflicts_match_replay_in_both_orders_and_after_checkpoint and the cross-kind test at that exact head (local or gate-slot, with the receipt);
  3. green linux-tests in hosted CI;
  4. the same commit on hotfix/19fda1d8-op-conflict, tested at its own SHA.

If the test fails because the test setup delivers nothing ("listing 0 identities"), the fix must make the test non-vacuous: it must assert that both claims arrived before it asserts anything else.

Posted on behalf of @schickling
field value
agent_identity agent/dotfiles/fractal-product-lead
session 01a1b525
agent_persona generalist
agent_supervisor unavailable
agent_tool OMP
agent_tool_version 18.12.5
agent_runtime OMP 18.12.5
agent_model anthropic/claude-opus-5-5
worktree unknown

agent-identity: unknown
agent-persona: unknown
agent-supervisor: unavailable
agent-tool: unknown
agent-runtime: unknown
agent-model: unknown
@schickling-assistant

Copy link
Copy Markdown
Contributor Author

FPL delta GO for #6334 at head af65763 (delta from 11bd608; test and release note only). This GO replaces the withdrawn GO.

Root cause accepted: the parity test was vacuous. It received envelopes but never ran validate_replication_backlog, so it compared empty operation states. The test now:

  • asserts that every exchange is non-empty;
  • admits each incoming claim and asserts that it exists;
  • asserts the operation state before projection: none, or active at position 1 without a checkpoint;
  • asserts after projection that position 0 is active and position 1 is conflict, with FULL_REPLAYS unchanged;
  • only then checks operations and digest parity against replay_replication_graph.

The production code is unchanged since 11bd608. Note: the production path is now tested for the first time.

Conditions:

  1. The hosted linux-tests job is green at this exact head, and its log shows that graph_neutral_operation_conflicts_match_replay_in_both_orders_and_after_checkpoint and the cross-kind test actually ran and passed (a test filtered out or marked crate_uncovered does not count).
  2. The PR lands alone, first in the order.

Hotfix: 2328689dab23f38585868a9b8865b4c404945af3 needs a passing run of the same two tests at its own SHA before any canary. FPL then confirms it separately.

Posted on behalf of @schickling
field value
agent_identity agent/dotfiles/fractal-product-lead
session 01a1b525
agent_persona generalist
agent_supervisor unavailable
agent_tool OMP
agent_tool_version 18.12.5
agent_runtime OMP 18.12.5
agent_model anthropic/claude-opus-5-5
worktree unknown

This branch is being deployed

1 in progress deployment
evidence-signing — af657635 Deployed Nov 11, 2026 by schickling-assistant via genie-freshness #19765
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant