Skip to content

P0 investigation: consecutive BCUGE Operate stalls and unbounded cancellation cleanup #172

Open

Description

@masarray

Symptom / scope

ARSAS diagnostic on 2026-10-16 (consumer commit 29eb0d2d9ce38286feb433f5aa9411d5263d91f6, pinned engine 036e2c00d83a0815b6160147b14ea05975057965):

  • 18:48:09 CSWI7.Pos Open succeeded, MMS response and feedback confirmed (~39 ms).
  • 18:48:18 CSWI6.Pos Open dispatched into consumer native path but no terminal result observed by 18:49:16.
  • Later CSWIs queued on same association; consumer #614/#615 address queue amplification, UNKNOWN outcomes and detach/recovery.
  • No wire capture proves whether CSWI6 Operate was sent or received. Do not ascribe fault to simulator or ARIEC without evidence.

Engine code paths that require targeted inspection

  • src/AR.Iec61850/Control/Iec61850ControlObjectSession.cs: OperateCoreAsync awaits BestEffortCancelAsync(CancellationToken.None) in expiry, mismatch, and cancellation paths.
  • BestEffortCancelAsync can await a live MMS Cancel write without caller cancellation or a proven completion deadline. If simulator/transport never answers and its operation ignores cancellation, cleanup can retain the engine control lock indefinitely.
  • src/AR.Iec61850/Mms/MmsClientSession.cs: SendConfirmedPresentationPayloadAsync registers invoke ID and waits on the receive pump or synchronous receive path. Needs per-operation evidence (invoke ID, send timestamp, response receipt, routing, cancellation and native exit).

These are source-level risk findings, not a proven explanation of the CSWI6 incident.

Required evidence / acceptance

  1. Capture first successful and second stalled CSWI request/response with exact invoke IDs and control model, using IEDScout Server Simulator.
  2. Reproduce delayed reply, absent reply, cancellation before send, cancellation after send, late response, and association teardown with deterministic fake transport.
  3. Prove cleanup honors a bounded native cancellation policy without releasing control/socket ownership while the old MMS operation is still unresolved; do not force-open a second overlapping Operate.
  4. Compare simulator with authorized lab IED under safe operating procedure. No uncontrolled breaker Operate or automatic blind retry.
  5. Run engine tests and consumer integration against an explicit new engine commit; preserve the currently pinned SHA until separately approved.

Related consumer: ARSAS #614, ARSAS #615.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions