Skip to content

engine_architecture.md

Latest commit

 

History

History
427 lines (335 loc) · 19.7 KB

engine_architecture.md

File metadata and controls

427 lines (335 loc) · 19.7 KB
Raw

Engine architecture

This document codifies the invariants and contributor-facing conventions of the Rust simulation kernel under engine_rs/. It exists so the load-bearing architectural decisions live somewhere greppable, not just in tests and conversation history.

The code enforces most of these rules — CI lockdowns, policy checks, divergence tests — but the doc teaches the right way before CI yells.

Audience. Contributors adding a new pass, a new contract, a new derived-state consumer, or any code that mutates the Simulation IR.

1. The seven invariants

The engine's correctness rests on seven layered invariants. Each has a canonical type and a test that proves it.

1.1 Contracts constrain support before proposals

Sampling is constrain-before-propose, not propose-and-retry. Every contract-aware sampler narrows its distribution's support through a filter pass before drawing a candidate. A draw that returns a value is always admissible by construction.

A permissive mode failure surfaces as the pass's explicit sentinel (e.g. b'N' for ambiguous-base sampling, no-op skip for indel position sampling). A strict mode failure surfaces as PassError::ConstraintSampling.

1.2 Trace = choices/proposals

The trace records what was proposed or sampled at each addressed decision point. It is the persisted artifact that makes runs reproducible across rebuilds.

Two traces are value-equal iff every recorded (address, ChoiceValue) pair matches in order.

1.3 Replay = validated proposal consumption

Trace-injected replay (Option B) consumes the recorded values through a TraceCursor at each sampling site. The cursor's strict-positional contract means the cursor must be drained when the plan completes — trailing records are a structured PassError::Replay.

Critical rule — every replayed value must pass the same admissibility chain a fresh draw would: refdata lookup, distribution-support membership, contract acceptance, feasibility acceptance. The trace supplies proposals; the engine decides whether they apply. See SampleAllelePass::validate_replay_candidate for the canonical example.

strict=True on replay does not re-raise permissive sentinels. Sentinel values (indel site = -1 NoOp, NP length 0, NP base N, trim 0) are themselves valid trace records — they represent "the pass committed nothing" rather than "the pass committed something the contract rejects." Replay consumes them verbatim and they reproduce the original outcome. A user who wants the strict-fresh-sampling behaviour for a recorded seed should call simulator.run(seed=<trace.seed>, strict=True), not replay_from_trace_file(tf, strict=True). See docs/productive_failure_mode_audit.md §5 and §6.2 for the failure matrix and the test pin.

1.4 Simulation events = consequences

SimulationEvent is the typed runtime stream describing what actually happened to the IR. Variants today:

Variant Carried payload
BasePushed handle, base, segment, germline_pos, flags
BaseChanged handle, old_base, new_base, segment, germline_pos
IndelInserted / IndelDeleted at, base/removed_base, segment, flags
AssignmentChanged segment, old, new
TrimChanged segment, end, old, new
RegionAdded / RegionReplaced region or (old, new)
MutationCountChanged old, new, delta
ReverseComplementFlagRecorded applied
BaseDeleted at, removed_base (reserved)

Events are emitted by SimulationBuilder at every mutation site and fanned out to attached SimulationEventSink implementors. The compiled executor captures the per-pass event stream into EventRecord.simulation_events.

1.5 Compile effects = scheduling facts

PassCompileEffect is the static, declarative category of state change a pass produces. It's read at compile time by the schedule analyzer for ordering and dependency reasoning. It is never consulted at runtime by derived-state refresh.

A transitional pub type PassEffect = PassCompileEffect alias keeps the old spelling compiling; new code should use the canonical name. The static-vs-runtime split is asymmetric on purpose: declarations are intent; events are what happened.

1.6 Live-call refresh follows events

The post-pass V/D/J live-call refresh (LiveCallRefreshHook) reads the pass's simulation_events stream — not the compile_effects declarations — and runs the steps produced by LiveCallRefreshPlan::from_events. A pass that declares an effect but emits no event triggers no refresh; a pass that emits an event without declaring the matching effect still triggers refresh.

Two divergence tests pin this: refresh_follows_events_not_effects_effect_without_event_skips_refresh and ..._event_without_effect_triggers_refresh.

1.7 Built-in mutating passes route through SimulationBuilder

The only sanctioned mutation path for production pass code is the event-emitting SimulationBuilder API:

Pass intent Builder method
Assign allele builder.assign_allele(segment, instance)
Update trim builder.update_trim(segment, end, value)
Add region builder.add_region(region)
Replace region builder.replace_region(replacement)
Bump mutation count builder.set_mutation_count(new_count)
Record rev-comp flag builder.record_reverse_complement_flag(applied)
Substitute / insert / delete a base route through MutationTransaction

Calling Simulation::with_* directly from production pass code bypasses event emission, so the LiveCallRefreshHook sees nothing and the live-call sidecar goes stale.

Two CI lockdowns make this unrepresentable:

2. How to add a new pass

A new pass that mutates biology must touch six surfaces. The template below is the canonical order; each step has an existing analog you can copy from.

Practical companion. docs/adding_a_pass.md is the copy-pasteable version of this section: minimal pass template, three required test patterns with helpers from passes::test_support, and a per-mechanism crib sheet of which existing pass to copy from. Read it when you're about to write code; come back here for the why.

2.1 Declare typed choice address patterns

If the pass samples anything, every sampling site needs a ChoiceAddress variant in engine_rs/src/address.rs.

fn declared_choice_patterns(&self) -> Vec<ChoiceAddressPattern> {
    vec![ChoiceAddressPattern::MyNewPassCount, ...]
}

The address vocabulary is schema-versioned via ADDRESS_SCHEMA_VERSION (see address.rs); adding a variant is additive. Pin the spelling via the frozen_address_spellings_for_choice_address_schema_v1 test.

2.2 Declare compile effects — for the scheduler only

fn effects(&self) -> Vec<PassCompileEffect> {
    vec![PassCompileEffect::EditBases]   // or AssembleSegment(seg), etc.
}

This is static. The scheduler reads it to order this pass correctly relative to its dependencies. It is not read at runtime for derived-state refresh.

2.3 Emit events through SimulationBuilder

Mutate via the builder, not Simulation::with_*. For substitutions and indels, route through MutationTransaction. For non-pool consequences (assignment, trim, region, mutation count, rev-comp), call the typed builder methods listed in §1.7.

If you build your own SimulationBuilder instance (e.g. to isolate a region-add step) and the caller passed PassContext.event_log_sink, forward the captured events:

let mut builder = SimulationBuilder::from_simulation(sim);
if ctx.event_log_sink.is_some() {
    builder.attach_event_log_observer();
}
builder.add_region(region);
if let Some(sink) = ctx.event_log_sink.as_deref_mut() {
    sink.extend(builder.seal_event_log_observer());
}
let sim = builder.seal();

2.4 Validate replayed values

When the pass is in replay mode (ctx.replay_cursor.is_some()), the cursor supplies the value. Run the same admissibility chain a fresh draw would before applying. The canonical template is SampleAllelePass::validate_replay_candidate:

  1. Look up resource (refdata, etc.).
  2. Check the value lies in the natural distribution's support.
  3. Run contracts.admits_typed(...).
  4. Run feasibility.admits(...).

Mismatches surface as the structured PassError variant that best names what failed (MissingAllele, InvalidDistributionOutput { reason: ... }, ContractViolation).

2.5 Add contract support if the pass needs to be filter-aware

If the pass should respect the productive bundle (or any future contract bundle), wire it into the sample_filtered_result pattern its category uses. Mutation/PCR/quality passes go through MutationTransaction::substitute_position_constrained; indel passes go through MutationTransaction::insert_base / delete_base_admitting.

2.6 Add coverage

Three layers of test coverage:

3. Anti-patterns

The patterns below are caught by tests today. The doc lists them so a new contributor sees the rule before tripping the CI.

3.1 Direct sim.with_* in production passes

❌ sim.with_allele_assigned(...) / sim.with_trim(...) / sim.with_region_added(...) / sim.with_mutation_count(...) / sim.with_base_changed(...) / sim.with_indel_inserted(...)

The runtime LiveCallRefreshHook follows events. A direct with_* call leaves the event stream silent and the V/D/J live-call sidecar goes stale.

Caught by: no_pass_calls_persistent_with_star_mutators_in_production_code plus the existing low-level lockdown.

Fix: route through SimulationBuilder or MutationTransaction.

3.2 Replay force-apply

❌ A replay path that applies the recorded value without running the admissibility chain.

Replay does not mean "trust the trace." A recorded value can have been invalidated by a refdata swap, an address-schema revision, or a contract change since the trace was written. Force-applying silently corrupts the IR; validation surfaces a structured PassError::Replay or PassError::MissingAllele that the caller can diagnose.

Caught by: replay_rejects_id_missing_from_refdata and friends.

Fix: mirror SampleAllelePass::validate_replay_candidate.

3.3 Contract-violating permissive fallback

❌ Permissive mode falling back to an unconstrained draw.

When the contract narrows the support to empty, the canonical permissive fallback is the pass's explicit sentinel — b'N' for ambiguous-base, no-op skip for trim, etc. Falling back to the unconstrained natural distribution defeats the purpose of declaring the contract.

The one documented exception is SampleAllelePass::sample_allele (an empty allele pool can't sentinel-skip without panicking the next assembly pass). That exception is spelled out in the method-level doc and tested via the productive-only NP1 frame filter.

3.4 Using PassCompileEffect for runtime refresh

❌ A new derived-state consumer that reads effects instead of events.

PassCompileEffect is the intent a pass declared at compile time. SimulationEvent is the consequence stream the pass actually emitted. A consumer reading effects sees declarations the pass may not have honored; a consumer reading events sees the truth.

Caught by: refresh_follows_events_not_effects_effect_without_event_skips_refresh plus the policy-conformance test.

Fix: consume events: &[SimulationEvent] in your hook's apply. Use a LiveCallRefreshPlan-style event-to-step translator if the consumer logic is non-trivial.

3.5 Declaring an effect without emitting matching events

❌ fn effects(&self) -> vec![PassCompileEffect::EditBases] plus mutation via a path that doesn't fire BaseChanged.

The runtime refresh trusts events. Declaring an effect without emitting the matching event variant silently skips refresh — the same anti-pattern as direct sim.with_*, just phrased differently.

Caught by: builtin_passes_emit_events_consistent_with_declared_compile_effects and the per-pass policy unit tests in event::tests.

Fix: either route through the event-emitting builder, or drop the effect declaration if the pass legitimately doesn't produce that consequence.

4. Where the architecture lives

Quick navigation for new contributors:

Concept Canonical home
Contract trait + set engine_rs/src/contract/
Choice address vocabulary engine_rs/src/address.rs
Persisted trace engine_rs/src/trace.rs, trace_file.rs
Replay cursor engine_rs/src/replay.rs
Persistent IR engine_rs/src/ir/
Event-emitting builder engine_rs/src/ir/builder.rs
Simulation event channel engine_rs/src/ir/sim_event.rs
Compile effects + Pass trait engine_rs/src/pass/
Mutation transaction engine_rs/src/passes/mutation_transaction/
Live-call refresh + plan engine_rs/src/live_call/refresh_hook.rs, refresh_plan.rs
Event record / outcome ledger engine_rs/src/event.rs
Compiled execution loop engine_rs/src/compiled/execute.rs
Production passes engine_rs/src/passes/
AIRR record projection engine_rs/src/airr_record/
Productive-contract stress matrix tests/test_productive_stress_matrix.py

5. Glossary

  • Pass. A scheduled unit of simulation work. Implements Pass. Declares requirements, effects, and choice-address patterns; runs inside the compiled executor's transactional loop.
  • Plan. Ordered list of passes via PassPlan or the higher-level Experiment DSL on the Python side.
  • Contract. A typed admissibility predicate over a candidate choice. Constraint-aware samplers narrow their distribution's support via contracts before drawing.
  • Effect. Renamed PassCompileEffect. Static declaration of category of state change. Not a runtime fact.
  • Event. SimulationEvent. Typed runtime consequence emitted by the builder at each mutation site.
  • Outcome. Outcome. The per-run result: revisions, trace, pass names, and the per-pass EventRecord ledger.
  • Live call. The V/D/J allele-call sidecar maintained on each Simulation, refreshed by the LiveCallRefreshHook after every pass that emits relevant events.
  • AIRR record. The flat record dict produced at the Python boundary by outcome_to_airr_record — the user-visible projection of an Outcome.