Skip to content

P0: prevent consecutive BCUGE command stalls with single-flight and uncertain-timeout quarantine - #614

Draft
masarray wants to merge 20 commits into
feat/m217h-dataset-verified-mx-initial-seed-20261016from
fix/p0-command-ied-singleflight-timeout-quarantine-20261017
Draft

masarray wants to merge 20 commits into
feat/m217h-dataset-verified-mx-initial-seed-20261016from
fix/p0-command-ied-singleflight-timeout-quarantine-20261017

Conversation

@masarray

@masarray masarray commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

P0 — BCUGE consecutive IEC 61850 control stall / association-wide recovery

Source: operator's 2026-10-16 18:49 diagnostic from ARSAS 29eb0d2d9ce38286feb433f5aa9411d5263d91f6 / pinned engine 036e2c00d83a0815b6160147b14ea05975057965.

Reproduced factual log sequence

  • 18:48:09 CSWI7.Pos Open: native MMS Operate response and status feedback confirmed in 39.129 ms (one ordered wire step, ctlNum=1).
  • 18:48:18 CSWI6.Pos Open: dispatch and Control execution requested logged, no terminal result by diagnostic 18:49:16.
  • 18:48:55–18:49:01 CSWI5/4/3/2.Pos also accepted by distinct per-object UI claims; no result for any.
  • 18:49:16 Copy Diagnostics succeeded; full WPF UI deadlock is not proven. The shared monitor loop waits while session.ControlCommandActive > 0; native client serializes controls behind one gate while per-object UI permitted queue buildup.
  • The diagnostic does not prove whether CSWI6 wire Operate was sent or accepted. Its outcome is UNKNOWN until transport evidence, not safe for blind retry.
  • This report predates M2.17-G/H and is not attributable to their code changes.

Consumer fixes

  1. Make an association-wide atomic single-flight gate, shared by Operate and ctlModel configuration, while retaining existing per-target lock and canonical engine service. Reject commands on other CSWIs immediately before any second MMS control operation; no queue growth.
  2. Add a 45-second outer deadline across descriptor, native Operate, and process feedback; use the existing cancellation token. If timeout/cancel occurs and native task is still running, keep physical ownership until that original task actually settles. WaitAsync timeout is NOT equated to native completion; observe and log late task errors/results without publishing process status.
  3. Quarantine that session after uncertain timeout. A late completion releases its active-flight gate but does not clear quarantine. Only a new full association/session can admit another command. This is fail-closed; it cannot guarantee wire cancellation or prove device state.
  4. Prevent UI confirmation/quick-action staging while device flight is active/quarantined. Surface explicit UNKNOWN wire outcome, instruct operator to confirm actual IED status and reconnect before retrying.
  5. Reject overlapping read-only control inspection while control flight is active.
  6. Five regression tests exercise cross-object exclusion, concurrent admission, sticky quarantine, and source contract verifying timeout/late-flight and UI handling.

Limits / acceptance

This fixes the demonstrated consumer queue amplification and missing overall timeout, not the ultimate native timeout root cause. A native operation ignoring cancellation can still retain the old association; do not force-release it and accidentally send a second command on the same socket. Follow-up may require ARIEC engine diagnostics of MMS receive pump / cancelled native control cleanup. No engine repin, MMS Write semantic change, or new value cache.
Physical acceptance: first successful command, subsequent different-CSWI command, fake nonresponding MMS endpoint, UI responsiveness, actual control wire outcomes and reconnect semantics. Keep draft and unmerged until CI and bench evidence.

Continuation: P0 native lifetime / reconnect isolation (2026-10-17)

Additional commits continue directly on this existing draft PR (no engine repin, no main merge).

  • Per-association target claims: new ControlSessionIdentity prevents an old native completion from clearing a different session's target ownership.
  • Bounded disconnect: old client disposal uses the established cleanup budget rather than an unbounded native await.
  • Uncertain ctlModel CF Write: shares the 45 s native wait/quarantine boundary and late result observer with Operate. A canceled native Task is not proof that a write was NotSent.
  • Monitor fail-closed: quarantined association stops MMS acquisition explicitly. Monitor start is denied until recovery. RCB teardown does not enqueue behind an active/uncertain command.
  • Session retirement: cancels the per-session linked command token before teardown, fences old result publication, and suppresses late WPF SignalDefinition control metadata writes (Operate/ctlModel/inspection).
  • Deterministic regressions: timeout without native completion, canceled outstanding native write, already-canceled native task, normal completion and source-bound integration guards.

Verification limits

  • Windows build, SCL interoperability and regression CI must be checked on the latest head, not inferred from prior green commits.
  • Root cause of the 2026-10-16 CSWI6 missing terminal result remains UNKNOWN. Pinned ARIEC code has potentially unbounded Iec61850ControlObjectSession.DisposeAsync gate wait and BestEffortCancelAsync(CancellationToken.None) cleanup; these are static review findings, not proof that either caused the incident.
  • Physical bench evidence still required: request-sent/invokeID/response correlation; first/second CSWI controls; server silent response; reconnect while old task unresolved. Never blind-retry uncertain Operate against live equipment.
  • Keep PR draft / unmerged until current-HEAD CI and bench acceptance.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant