Skip to content

P0 follow-up: retire unresolved MMS control sessions without blocking reconnect - #615

Draft
masarray wants to merge 20 commits into
fix/p0-command-ied-singleflight-timeout-quarantine-20261017from
fix/p0-command-retired-session-drain-20261017
Draft

masarray wants to merge 20 commits into
fix/p0-command-ied-singleflight-timeout-quarantine-20261017from
fix/p0-command-retired-session-drain-20261017

Conversation

@masarray

@masarray masarray commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

P0 Command continuation (stacked on #614)

Addresses a verified consumer lifecycle hazard: StopDeviceAsync could await NativeIec61850Client.DisposeAsync indefinitely while an unresolved native control held the MMS I/O gate. The process-wide deviceId|controlReference map also allowed a late release from an old association to interfere with a new connection.

Changes

  • Introduce association-scoped, atomic Iec61850ControlSessionLease with sticky quarantine, retirement, and an actual-native-release drain task.
  • Replace process-wide target ownership with association-scoped single-flight admission for control AND ctlModel Write. An old session can no longer release new-session command authority.
  • Retire before awaiting teardown; for unresolved native control, immediately cancel old monitoring, allow UI Stop/reconnect, and defer old client disposal until native control and monitor tasks actually settle.
  • Reject late process/status publication from a retired runtime session and suppress late Command UI results using a per-session GUID. Existing ledger remains the only process-value authority.
  • Advisory Command UI checks actual runtime quarantine as well as view-model busy state.
  • Expand regression tests for native drain ordering, sticky quarantine, atomic retirement, and stale result rejection; update existing ctlModel source-contract test.

Important limitations / safety

  • A native operation that ignores cancellation can still leave the detached old MMS socket allocated indefinitely. We do not force-release the native operation or promise physical wire cancellation. The actual IED state remains UNKNOWN until verified.
  • This is consumer lifecycle recovery, not proof of the CSWI6 root cause or evidence of successful physical control. Requires bench tests, wire capture and engine investigation as applicable.
  • Do not automatically retry Operate on live equipment.

Validation

Pending fresh GitHub CI on this PR and physical acceptance. Preserve ARIEC pin 036e2c00d83a0815b6160147b14ea05975057965 and ArdIrec pin 7dbc7149db668126a493ce338264363a6ec5ec00. No merge to main.

Additional fail-closed evidence and engine follow-up

  • A completed native Task with CompletionState=WireStateUnknown (or explicit timeout state) also quarantines the association before release; no subsequent Operate is admitted.
  • Command Dock distinguishes Command outcome UNKNOWN from Command not sent for both native-unknown and outer-deadline timeout paths.
  • An association-retirement token cancels queued Command/ctlModel operations even when retirement races with the creation of the native Task. Cancellation callbacks never run under the admission lock.
  • The UI-facing facade forwards the new association-fence APIs to the existing inner runtime.
  • Engine investigation tracked separately in ARIEC61850 #172. A potentially unbounded BestEffortCancelAsync(CancellationToken.None) was identified by source audit; it is not yet proven to be the field incident's root cause. No engine repin.

Command snapshot / diagnostic hardening

  • Every native control gets a detached, per-attempt SignalDefinition snapshot; a late ARIEC task cannot write its mutable control fields into the WPF-owned model after old-session retirement.
  • Completed native Cancelled and AssociationLost control outcomes are conservatively quarantined alongside WireStateUnknown and TimedOut.
  • Structured per-attempt diagnostics now correlate attemptId + associationId, admission UTC, native task creation (NOT evidence of an actual wire send), timeout, terminal result and late-native observation.
  • Regression assertions updated from removed global deviceId target map to atomic per-association flight lease; detached command signal mutation test added.

No physical wire-send or invoke ID is claimed without ARIEC evidence.

Verified CI evidence (HEAD b22f0f8e92e76f7d602b7955321c165cf626499e)

  • SCL Interoperability R7 Build: SUCCESS, including Release build, 2211/2211 ARSAS xUnit, 1127/1127 pinned ARIEC xUnit.
  • Remaining workflows should be evaluated at this exact HEAD before changing draft status or considering merge.
  • No hardware/IEDScout physical acceptance yet; server-vs-engine root cause remains unproven.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant