feat: Add structured Kubernetes Events for sandbox lifecycle transitions

[Maximus-08]

What would you like to be added:
Emit Kubernetes Events via client-go/tools/record.EventRecorder at key sandbox lifecycle
points in the Workload Manager. Currently, sandbox creation, readiness, rollback, and garbage
collection are tracked only through internal Go channels and klog statements, no Events are
emitted on the parent AgentRuntime or CodeInterpreter resources.

Why is this needed:

Emitting Events is standard Kubernetes controller practice, every core controller does it.
Without Events, operators using kubectl describe agentruntime/<name> see no history of what
happened to their sandboxes. Debugging a slow or failed sandbox startup today requires tailing
workload-manager logs directly rather than using kubectl get events.

This also becomes critical for the upcoming multi-AgentCube orchestration feature, where
a single task spans multiple coordinated sandboxes. Operators and platform teams will need a
reliable, per-resource audit trail to monitor lifecycle transitions at scale.

[Maximus-08]

/assign

[Maximus-08]

Hey @hzxuzhonghu Does this look good to implement, I would like your feedback on this before I proceed with implementation.

[hzxuzhonghu]

Great proposal. A few thoughts:

On EventRecorder usage: Tie the recorder to the specific resource object (e.g. AgentRuntime or CodeInterpreter) as the "regarding" object — this makes events show up in kubectl describe on those resources, which is exactly what operators/users expect.

On event placement:

• Emit Normal events for: sandbox requested, sandbox ready, sandbox GC'd
• Emit Warning events for: sandbox creation failed, rollback triggered, idle-timeout expired

Key lifecycle points to cover (per the proposal):

1. SandboxCreating — when the warm-pool claim or fresh sandbox request is issued
2. SandboxReady — when the sandbox transitions to Running
3. SandboxRollback — on warm-pool rollback
4. SandboxGCed — when GC removes a session

Caution: avoid logging secrets/session tokens in event messages — event objects are readable by anyone with get events RBAC.

Happy to help implement once the event taxonomy is agreed on.

[Maximus-08]

Thanks for the detailed feedback @hzxuzhonghu !

Confirmed on the regarding object - I'll tie the recorder to the AgentRuntime (or CodeInterpreter) instance so events surface in kubectl describe on those resources directly.

Proposed final taxonomy based on your review:

Event Reason	Type	Trigger
SandboxCreating	Normal	warm-pool claim or fresh sandbox request issued
SandboxReady	Normal	sandbox transitions to Running
SandboxCreationFailed	Warning	sandbox creation errors out
SandboxRollback	Warning	warm-pool rollback triggered
SandboxIdleTimeout	Warning	idle-timeout expiry (per your note)
SandboxGCed	Normal	GC removes a session

Will also make sure no session tokens or secret values appear in any message strings.

Happy to open PR1 (wiring EventRecorder + Normal events) once you confirm the above.

[hzxuzhonghu]

One thing i want to remind is: agentruntime and codeinterpreter can map to multiple sandboxes. This model need to be carefully processed.

[Maximus-08]

Hey @hzxuzhonghu yes after a further look at this,
Since AgentRuntime/CodeInterpreter can back many concurrent sandboxes, emitting events on the parent for every instance would be a firehose - etcd write amplification and kubectl describe becoming unreadable.

The fix I'm thinking: scope events to the individual child Sandbox object with the parent name in the message, e.g. Sandbox default/python-sandbox-a1b2c3d4 (AgentRuntime: my-agent, session: abc) transitioned to Running.

Two things to confirm before I finalize:

1. For SandboxGCed - GC deletes the Sandbox before it can emit on it. Should this go on the parent (the only object that survives), or is GC-level observability out of scope here?
2. No EventRecorder pattern exists in the repo today. Can I establish the convention using mgr.GetEventRecorderFor() in reconcilers and a raw EventBroadcaster from the clientset for the GC goroutine?

Will update the issue once I hear back.

[hzxuzhonghu]

The fix I'm thinking: scope events to the individual child Sandbox object with the parent name in the message, e.g. Sandbox default/python-sandbox-a1b2c3d4 (AgentRuntime: my-agent, session: abc) transitioned to Running.

Operating a object event should be the operator of it. So i think the event should be emitted by agent-sandbox controller, not by agetcube

1. For SandboxGCed - GC deletes the Sandbox before it can emit on it. Should this go on the parent (the only object that survives), or is GC-level observability out of scope here?

I think a log is ok

[Maximus-08]

Thanks for the guidance! I believe then I have to look at the upstream kubernetes-sigs/agent-sandbox (controllers/sandbox_controller.go).

If that's correct, I'll open the PRs there:

• PR1: Wire EventRecorder into SandboxReconciler, emit SandboxCreating and SandboxReady on the child Sandbox object with AgentRuntime + session context in the message
• PR2: Emit SandboxCreationFailed and SandboxRolledBack similarly; replace SandboxGCed with a structured log

Could you confirm, and let me know if there's anything specific to align with the kubernetes-sigs/agent-sandbox maintainers before I send a PR?

[hzxuzhonghu]

No further suggestion now, i donot have time to read that code now.