Some improvements to ActorTemplate

[juli4n]

There are a few cases where updating / deleting ActorTemplate resources can lead to a broken system.

• Deleting an ActorTemplate that is currently in use by an actor breaks the ResumeActor and SuspendActor RPCs.
• If an ActorTemplate is updated, the golden snapshot becomes stale and actors using the template diverge. Although ActorTemplates are conceptually immutable, this is not enforced today.
• Deleting an ActorTemplate leaks its associated golden actor.

We should discuss whether we want to allow an actor to outlive its template. Assuming we don't, a possible path forward would be:

• The Substrate controller manages ActorTemplate resources.
• ActorTemplates are immutable (enforced via CEL validation).
• Each ActorTemplate owns the lifecycle of a “golden actor”. The golden actor is created when the template is created, and deleted when the template is deleted.
• An ActorTemplate cannot be deleted if there are actors using it (this will require ate-apiserver to expose an efficient way to check if there are active actors referencing a given template).

[ArnaudBger]

I’m wondering about the upgrade/recovery story for long-lived actors.

Let’s say an Actor is running an Agent Session Loop, and that loop is responsible for calling an external LLM provider API.

If the provider API changes incompatibly, does that mean actors created or snapshotted before the change could resume with an outdated in-memory Agent Session Loop that no longer works?

In that case, even if ActorTemplates become immutable, I’m not sure what the recovery path is for existing actors.
Is the intended model:

old actors must continue using the exact template/image they were created from,
actors should be recreated from a newer template,
or there should be some mechanism to “bump” / migrate an actor to a newer or previous template?
Put differently: how does the current design expect users to handle external API drift when actor state may outlive the template version that originally produced it?

[a4-a4s1]

On the third bullet (ActorTemplate deletion leaking its golden actor) — the proposed "block-delete-if-referenced" rule handles the prevent case, but there's also a cleanup case worth scoping in:

• Finalizer + DeleteActor RPC. Even if delete is blocked when references exist, the golden actor associated with the template itself needs explicit teardown when the template IS deletable (no references). A finalizer on ActorTemplate invoking DeleteActor on ate-apiserver before allowing the object's removal closes the leak symmetrically with the create-side (golden actor created on template-create). Today the controller doesn't appear to register a finalizer on ActorTemplate at all — happy to point at the relevant controller file if useful.

• Immutability enforcement granularity. CEL x-kubernetes-validations on the whole spec is straightforward, but if any field is intentionally mutable (labels for selector use, annotations for ops metadata), worth listing those carve-outs explicitly so the validation rule isn't broader than intended.

• Reference-check cost. "Cannot be deleted if there are actors using it" depends on ate-apiserver being able to answer "are there active actors referencing template T?" efficiently. For small fleets this is cheap; for the load-test case the substrate roadmap calls out (many actors per template), a server-side index makes more sense than a client-side scan-and-decide.

Question: does the controller-managed ActorTemplate proposal envision the golden-actor lifecycle as owned by the template's finalizer (delete template → finalizer fires DeleteActor → release finalizer), or decoupled via Owner Reference (delete template → cascade GC handles the golden actor)? Both work; the former is more legible in event/audit logs, the latter leans on K8s GC semantics.

[🤖a4s1]

[thockin]

ArnaudBger (@ArnaudBger) I think this topic is not pinned down yet, so let's talk about what semantic we think makes sense. I would love for us all to agree on a single semantic, but I suspect there will end up being some configuration involved.

old actors must continue using the exact template/image they were created from,
actors should be recreated from a newer template,
or there should be some mechanism to “bump” / migrate an actor to a newer or previous template?

My gut instinct is that there MUST be a way for a developer to update a template and then roll that out across the fleet of instances of that template.

I suspect we really want to define something like a set of versions of a given template. For example (not well thought out yet):

• Each ActorTemplate (AT) has N "Versions" (ATVs).
• The set of versions for a given AT is called a "Series".
• An actor is created from a specific AT, with a "floating" version.
• When the actor starts for the first time it gets the latest ATV in that series.
• An AT can spec a particular ATV as being EOL, meaning:
  1. new actors cannot use that ATV
  2. policy about how to handle existing actors
     a) Leave existing actors on this version
     b) Force actors on this ATV to upgrade to the newest ATV on the next cold-start
     c) Force actors on this ATV to upgrade to the newest ATV on the next resume (cold or warm)
     d) Maybe some form of slow rollout of (b) and (c)

If we end up with auto-suspend while waiting for IO, we may need to exempt those.

So the questions:

1. Does that seem reasonable?
2. Can we define a default behavior that works for most people without any config, and punt most of the work past v1?

[juli4n]

A few notes:

• When the template of an actor changes, what happens to its checkpoint?

Process snapshots cannot be reused across templates (e.g. image changes) so they need to be discarded. The filesystem state could be preserved, otherwise there is no much difference from creating a new actor?

• Can an actor outlive its template? If yes, what happens when an actor's template is deleted?

Today actors are not "self-contained" so during resume we fail with an internal error. Should we prevent the deletion of a template if an actor is using it? Otherwise we need to make actors self-contained in the ate-apiserver storage and copy all its spec at create time.

• Do we need first class support for versioning?

While I agree it would be nice to have some first class support for actor template versioning and rollout, I'm worried it will take time to get it right. Maybe we could start with a simple model where UpdateActor can patch the template (requiring the actor to be SUSPENDED)?

[npolshakova]

There's a similar issue I ran into with the WorkerPool getting deleted- the worker pods get removed, but records still exist. It feels like the ActorTemplate should become NotReady when its referenced WorkerPool is deleted? Or block deletion while it is still referenced?

[juli4n]

There's a similar issue I ran into with the WorkerPool getting deleted- the worker pods get removed, but records still exist. It feels like the ActorTemplate should become NotReady when its referenced WorkerPool is deleted? Or block deletion while it is still referenced?

Yes, we have #23 which tracks handling pod deletion in a graceful way.