Morning-planner log never reaches S3 on the runs that matter — EXIT trap can't fire on SSM's SIGKILL

[cipher813]

Problem

The morning-planner SSM command uploads /var/log/executor.log to S3 via a shell trap:

trap 'aws s3 cp /var/log/executor.log "s3://alpha-engine-research/_ssm_logs/executor/$(date -u +%Y-%m-%d)/$(hostname)-$(date -u +%H%M%SZ).log" --only-show-errors || true' EXIT
python executor/main.py 2>&1 | tee -a /var/log/executor.log

A shell EXIT trap does not run when the process is SIGKILLed, which is exactly what SSM does at executionTimeout (600 s for RunMorningPlanner). So the runs that hang — the ones whose logs are most needed — are precisely the ones that never upload.

Verified live during the 2026-07-27 incident:

$ aws s3 ls s3://alpha-engine-research/_ssm_logs/executor/
... PRE 2026-07-23/  PRE 2026-07-24/          # no 2026-07-27/ at all
$ aws s3 ls s3://alpha-engine-research/_ssm_logs/executor/2026-07-24/
2026-07-24 08:16:51  ip-...-151650Z.log       # the 15:16 rerun (succeeded)
2026-07-24 08:37:00  ip-...-153659Z.log       # the 15:36 rerun (succeeded)

On 7/24 only the two successful reruns uploaded; the 12:25 run that hung and killed the pipeline did not. On 7/27 nothing uploaded at all. Diagnosing both required SSM-exec'ing onto the trading box (i-018eb3307a21329bf) and reading /var/log/executor.log by hand — the box is SSM-only, so this is a multi-step interactive detour on the critical path of a live trading outage.

This is a pure observability gap, not a functional one — but it directly lengthened time-to-diagnosis on two lost trading days (see crucible-executor-PR441).

Why the obvious fix is not enough

Adding trap ... TERM INT does not fix it: SSM's executionTimeout kill is SIGKILL, which is uncatchable by design. The upload must not depend on the doomed process running anything at exit.

Suggested approaches (pick at implementation time)

1. Periodic flush — a background while sleep 60; do aws s3 cp ...; done loop alongside the planner, so a partial log exists in S3 at all times regardless of how the process dies. Simplest; costs one PUT/min for the run's duration.
2. CloudWatch Logs on the SSM command — set CloudWatchOutputConfig.CloudWatchOutputEnabled=true on RunMorningPlanner (currently false, CloudWatchLogGroupName=""). The agent streams stdout/stderr as it is produced, so nothing is lost on SIGKILL, and it removes the bespoke S3 path entirely. Note: SSM truncates the StandardOutputContent field at 24 000 characters, but the CloudWatch stream itself is not truncated.
3. Self-imposed inner timeout — run the planner under a timeout shorter than executionTimeout so the shell regains control and the trap fires normally. Weakest of the three: still loses the log if the box or agent dies.

Option 2 is likely the SOTA answer — it deletes the custom log-shipping path rather than hardening it, and puts planner output on the same surface as everything else. Whichever is chosen, apply the same treatment to every SSM step that ships logs this way (RunMorningPlanner, the EOD reconcile step, the data-spot steps).

Anchors

• SF: ne-preopen-trading-pipeline, state RunMorningPlanner (definition in nousergon-data/infrastructure/step_function_daily.json)
• The trap is in that state's Parameters.commands array
• S3 prefix: s3://alpha-engine-research/_ssm_logs/executor/{date}/
• Poll budget that surfaces the timeout: WaitForMorningPlanner, max_attempts: 30, ~15 s each

Closes when

• A morning-planner run that is SIGKILLed at executionTimeout leaves its output retrievable without SSM-exec'ing onto the box.
• Verifiable by: force a planner run to exceed executionTimeout (e.g. a temporary sleep), confirm the output is readable afterwards from S3 or CloudWatch Logs.

Re-exam: 2026-08-10