The AWS Batch executor is growing a coordinator mode (fulcrumgenomics/snakemake-executor-plugin-aws-batch — "coordinator mode" PR): snakemake itself runs as a Batch job and the local terminal disconnects. That moves .snakemake/ and the snakesee logger plugin's .snakesee_events.jsonl into the coordinator container, so snakesee's passive local monitoring sees nothing.
Sketch:
snakemake-logger-plugin-snakesee: optional sink that ships the event stream to a reachable location — the natural choice is the workflow's S3 prefix (the executor already requires S3 storage), e.g. append/rotate JSONL objects under <prefix>/.snakesee/. Needs batching to avoid per-event PUTs.- snakesee: an S3-aware
EventReader (poll the object(s), track offset/etag) selected when the workflow dir argument is an s3:// URL.
- Auth/perms: reuse the ambient AWS credentials; document
s3:GetObject/ListBucket needs.
This is the missing piece for monitoring fire-and-forget remote workflows end-to-end, and extends the remote-job-state contract work (the snakesee_remote events would travel the same stream).
The AWS Batch executor is growing a coordinator mode (fulcrumgenomics/snakemake-executor-plugin-aws-batch — "coordinator mode" PR): snakemake itself runs as a Batch job and the local terminal disconnects. That moves
.snakemake/and the snakesee logger plugin's.snakesee_events.jsonlinto the coordinator container, so snakesee's passive local monitoring sees nothing.Sketch:
snakemake-logger-plugin-snakesee: optional sink that ships the event stream to a reachable location — the natural choice is the workflow's S3 prefix (the executor already requires S3 storage), e.g. append/rotate JSONL objects under<prefix>/.snakesee/. Needs batching to avoid per-event PUTs.EventReader(poll the object(s), track offset/etag) selected when the workflow dir argument is ans3://URL.s3:GetObject/ListBucketneeds.This is the missing piece for monitoring fire-and-forget remote workflows end-to-end, and extends the remote-job-state contract work (the
snakesee_remoteevents would travel the same stream).