Ensure Atelet API is idempotent

[juli4n]

Atelet's Run/Checkpoint/Restore methods need to be idempotent to make sure we can recover from workflows that fail half-way through. I believe this is not true today.

This task is to go over atelet (and ateom) and ensure operations can be retried.

[mjlshen]

I'm seeing that idempotency for Run, Checkpoint, and Restore is not guaranteed, especially if a gRPC Unavailable (atelet --> ateom connection interrupted, e.g. an atelet restart) or DeadlineExceeded (10min deadline) is returned in the middle of these operations. In these cases, the caller has no information about whether the initial operation never started, partially ran, or completed without a response. I have a small repro for Checkpoint below:

Checkpoint-specific reproduction

1. Setup locally with demo counter

KIND_CLUSTER_NAME=issue-372 hack/create-kind-cluster.sh
kubectl config use-context kind-issue-372

hack/install-ate-kind.sh --deploy-ate-system
hack/install-ate-kind.sh --deploy-demo-counter

Relevant output:

/dev/kvm not available: micro-VM support disabled (gVisor still works).
Creating cluster "issue-372" ...
Set kubectl context to "kind-issue-372"
daemon set "atelet" successfully rolled out
deployment "counter" successfully rolled out
actortemplate.ate.dev/counter condition met

2. Create and resume a counter actor

kubectl ate create atespace demo

kubectl ate create actor repro-main \
  -a demo \
  --template ate-demo-counter/counter

kubectl ate resume actor repro-main -a demo
kubectl ate get actor repro-main -a demo

Observed:

ATESPACE   NAME         TEMPLATE                   STATUS           ATEOM POD                                   ATEOM IP      VERSION
demo       repro-main   ate-demo-counter/counter   STATUS_RUNNING   ate-demo-counter/counter-7c877c5cfd-qt9ml   10.244.0.30   3

Save the worker location before suspending:

kubectl ate get actor repro-main -a demo -o json \
  > /private/tmp/repro-main-running.json

export WORKER_NS="$(
  jq -r '.actors[0].ateomPodNamespace' \
    /private/tmp/repro-main-running.json
)"

export WORKER_POD="$(
  jq -r '.actors[0].ateomPodName' \
    /private/tmp/repro-main-running.json
)"

echo "$WORKER_NS/$WORKER_POD"

Observed:

ate-demo-counter/counter-7c877c5cfd-qt9ml

3. Complete the first Checkpoint

kubectl ate suspend actor repro-main -a demo

Observed:

ATESPACE   NAME         TEMPLATE                   STATUS
demo       repro-main   ate-demo-counter/counter   STATUS_SUSPENDED

Ateom logged a successful destructive checkpoint:

Actor checkpointing
About to run runsc fscheckpoint
Actor checkpointed

Its RPC result contained the snapshot:

/ateom.Ateom/CheckpointWorkload
  resp.snapshot_files=[
    fscheckpoint.json
    multitar.img
    pages.img
    pages_meta.img
  ]
  err=null
Atelet also returned success:
/atelet.AteomHerder/Checkpoint
  resp={}
  err=null

4. Reconstruct the same Checkpoint request
   Get the suspended actor and template:

kubectl ate get actor repro-main -a demo -o json \
  > /private/tmp/repro-main-suspended.json

kubectl get actortemplate counter \
  -n ate-demo-counter \
  -o json \
  > /private/tmp/counter-template.json

Generate the atelet protobuf JSON request:

jq -n \
  --slurpfile actor /private/tmp/repro-main-suspended.json \
  --slurpfile template /private/tmp/counter-template.json '
    ($actor[0].actors[0]) as $a |
    ($template[0]) as $t |
    {
      targetAteomUid: $a.ateomPodUid,
      atespace: $a.metadata.atespace,
      actorName: $a.metadata.name,
      actorUid: $a.metadata.uid,
      actorTemplateNamespace: $a.actorTemplateNamespace,
      actorTemplateName: $a.actorTemplateName,
      spec: {
        pauseImage: $t.spec.pauseImage,
        containers: [
          $t.spec.containers[] |
          {
            name: .name,
            image: .image,
            command: (.command // []),
            args: (.args // []),
            readyz: (.readyz // null),
            volumeMounts: (.volumeMounts // [])
          }
        ],
        volumes: [
          ($t.spec.volumes // [])[] |
          if has("durableDir") then
            {
              name: .name,
              type: "VOLUME_TYPE_DURABLE_DIR",
              durableDir: {}
            }
          else
            empty
          end
        ]
      },
      type: "CHECKPOINT_TYPE_EXTERNAL",
      externalConfig: {
        snapshotUriPrefix:
          $a.latestSnapshotInfo.external.snapshotUriPrefix
      },
      scope:
        if $t.spec.snapshotsConfig.onCommit == "Data" then
          "SNAPSHOT_SCOPE_DATA"
        else
          "SNAPSHOT_SCOPE_FULL"
        end
    }
  ' > /private/tmp/checkpoint-main.json

Check its identity fields:

jq '{
  targetAteomUid,
  actorUid,
  externalConfig,
  scope
}' /private/tmp/checkpoint-main.json

Observed during the experiment:

{
  "targetAteomUid": "40b649d9-72e2-4e92-9e36-db4d3a4ab95c",
  "actorUid": "9a2fa872-a205-4faa-b719-f9d5c13f4205",
  "externalConfig": {
    "snapshotUriPrefix": "gs://ate-snapshots/ate-demo-counter/repro-main/2026-07-24T16:51:32Z-4GE3TODKVSWMRHQG7EGMMKOLH4"
  },
  "scope": "SNAPSHOT_SCOPE_DATA"
}

5. Port-forward atelet
   In a second terminal:

export ATELET_POD="$(
  kubectl get pods \
    -n ate-system \
    -l app=atelet \
    -o jsonpath='{.items[0].metadata.name}'
)"

kubectl port-forward \
  -n ate-system \
  "pod/$ATELET_POD" \
  18085:8085

6. Replay Checkpoint
   In the first terminal:

grpcurl \
  -plaintext \
  -d @ \
  127.0.0.1:18085 \
  atelet.AteomHerder/Checkpoint \
  < /private/tmp/checkpoint-main.json

echo "exit status: $?"

Observed:

ERROR:
  Code: Internal
  Message: while fscheckpointing durable-dir "/home/counter": while running `runsc fscheckpoint`: exit status 128

exit status: 77

Ateom confirms that the replay attempted the destructive runtime operation again:

Actor checkpointing
About to run runsc fscheckpoint
FetchSpec failed: loading container: file does not exist

The final ateom RPC log was:

/ateom.Ateom/CheckpointWorkload
  resp=null
  err="while fscheckpointing durable-dir \"/home/counter\": while running `runsc fscheckpoint`: exit status 128"
Atelet propagated that failure:
/atelet.AteomHerder/Checkpoint
  resp=null
  err="while calling ateom.CheckpointWorkload: rpc error: code = Internal desc = while fscheckpointing durable-dir \"/home/counter\": while running `runsc fscheckpoint`: exit status 128"

7. Cleanup

KIND_CLUSTER_NAME=issue-372 \
  hack/delete-kind-cluster.sh

Summary

The first Checkpoint succeeded and tore down the runtime and replaying the same semantic request caused ateom to execute runsc fscheckpoint again against the already-removed container state. A real lost response produces the same server-side situation: ateom completed, but the caller cannot know that and retries. The retry fails because neither atelet nor ateom records a durable result that can be replayed.

[troychiu]

I can help with this Zoe Zhao (@zoez7) . Thank you