From 3e4460ffe04f65b4f3357880facb8cc49bef342d Mon Sep 17 00:00:00 2001 From: likun Date: Thu, 3 Sep 2026 18:43:30 +0800 Subject: [PATCH 1/2] docs(blog): add Beyond Function Calling Generated-by: Codex --- docs/blogs/beyond-function-calling.md | 518 ++++++++++++++++++++ docs/blogs/beyond-function-calling.zh-CN.md | 518 ++++++++++++++++++++ 2 files changed, 1036 insertions(+) create mode 100644 docs/blogs/beyond-function-calling.md create mode 100644 docs/blogs/beyond-function-calling.zh-CN.md diff --git a/docs/blogs/beyond-function-calling.md b/docs/blogs/beyond-function-calling.md new file mode 100644 index 0000000000..7dd5f0bacb --- /dev/null +++ b/docs/blogs/beyond-function-calling.md @@ -0,0 +1,518 @@ + + +[简体中文](./beyond-function-calling.zh-CN.md) + +# Beyond Function Calling: How Agents Reach the Real World + +## Deferred Tools: Even an Unused Tool Has a Cost + +In an ordinary program, a function that is never called has almost no runtime cost. It can sit in a codebase or a dynamic library without consuming CPU or occupying the call stack. + +Tools in an agent do not work that way. + +Before a model can call a tool, it must know the tool's name, purpose, and argument format. The runtime therefore sends tool definitions to the model together with the system prompt and conversation history. Even when a tool is never invoked, its description and JSON Schema have already participated in every inference. + +A tool starts costing tokens before it starts executing. Its schema occupies context, influences the model's next-action decision, and changes the request prefix available for provider-side caching. More tools give the model a larger action space, but leave less room for the task itself and introduce more competing choices. + +This is barely noticeable when an agent has only a few tools such as `Read`, `Write`, and `Bash`. It becomes a scaling problem once browsers, computer use, subagents, external services, and MCP connectors join the tool registry. Keeping every schema resident in every model request is not a sustainable architecture. + +Maka's deferred tools begin with this observation. "Deferred" does not mean delayed execution or background execution. It means delaying the moment when the complete tool schema becomes visible to the model. + +The runtime still holds every tool binding available to the current run. On the first model request, however, the model sees only a small set of frequently used tools and a lightweight `tool_search`. Other tools appear in a compact search inventory by group and name, without their full descriptions or argument schemas. + +```text +Bound Tool Registry + │ + ├── Direct Tools ───────────────→ Full schemas in this request + │ + └── Deferred Tools + │ + └── Lightweight Search Inventory + │ + tool_search + │ + Bounded matches + │ + ▼ + Next provider step + injects matched schemas +``` + +`tool_search` does not search files, web pages, or application data. It searches capabilities already owned by the runtime. Maka performs the lookup locally against tool names, descriptions, and capability groups, then selects a bounded set of matches with bounded schema size. The result contains only the activated tool names. Full schemas are not duplicated inside the tool result; they appear through the normal tool projection in the next provider request. + +This separates several concepts that are easy to conflate: + +- **Bound:** the runtime owns an executable tool binding. This defines the capability ceiling of the run. +- **Discoverable:** the tool appears in the lightweight inventory, so the model knows that the capability exists. +- **Visible:** the complete schema is present in the current provider request, so the model can construct a valid call. + +Search does not bind a new tool and cannot exceed the run's binding ceiling. It changes only the tool projection visible to the next model call. + +"Next" is an important boundary. Once a provider step begins, its tool schemas are fixed. If the model emits both of these calls in one response: + +```text +tool_search("browser click") +browser_click(...) +``` + +Maka still rejects the second call. A search result can affect a later request, but it cannot rewrite the schema set of a request already sent to the provider. Only in the next step does the full `browser_click` definition enter context, allowing the model to generate arguments for an interface it has actually seen. + +Deferred activation is scoped to the current turn. Tools discovered during a turn accumulate monotonically, and provider retries inherit that working set. When the turn ends, the activation set is released. The next user turn starts again from the stable base set instead of permanently paying for every capability used in the past. + +Visibility is also not authorization. A visible tool still passes through permission checks, argument validation, and runtime execution boundaries when called. `tool_search` manages the model's cognitive action space, not the user's permission space. + +Deferred tools therefore do not answer "how should a tool execute?" They answer "which tools deserve to enter the model's next thought?" The runtime retains the complete capability space while the model sees only the working set relevant to the current task. + +## Tool Calls: Giving the LLM Hands and Feet + +Once a tool schema enters context, the model merely knows which actions are available. Until it emits a tool call, everything remains tokens. + +An LLM cannot read a file, start a process, or click a screen. It consumes input and predicts output. Even if it says, "I have modified the file," that sentence changes nothing on disk. Language describes the world; by itself, it does not alter the world. + +A tool call creates a channel between the two. Instead of producing only natural language, the model emits a structured action request: a tool name, arguments, and an ID that associates the eventual result with the call. The runtime receives the request, executes the corresponding operation in a real environment, and returns the observation to the model. + +```text +LLM + │ + │ function_call(name, arguments, call_id) + ▼ +Runtime + │ + ├── Resolve the tool binding + ├── Validate arguments and execution boundaries + ├── Request permission when necessary + ├── Invoke the real implementation + ▼ +Filesystem / Process / Browser / Network / Human + │ + │ function_response(call_id, result) + ▼ +The LLM's next inference +``` + +This closed loop is where a model becomes an agent. File reads give it observations of a codebase. Commands expose compiler and test feedback. File edits let it change the workspace. Browser and network tools connect it to systems outside the local process. Questions let it pause for new facts when information is missing. + +If tools are the agent's hands and feet, tool results are its senses. Without feedback, the model cannot tell whether an action succeeded or whether reality matches its prediction. A complete agent step is therefore not simply "the model thought once." It combines intention, execution, and observation: + +```text +Reason → Act → Observe → Reason +``` + +This resembles a function call, but differs in a fundamental way. When a program calls an internal function, caller and callee usually share one deterministic execution environment. A model issuing a tool call is proposing an action from a probability distribution. Its arguments may be incomplete, its target may have changed, and its understanding of the environment may be wrong. + +It is more accurate to say that the LLM does not grow its own hands and feet. The runtime lends it a controlled set. + +In Maka, a model-generated call cannot bypass the runtime and reach the outside world directly. The runtime verifies that the binding exists, that it is visible in the current step, and that the arguments conform to its schema. The call must also pass concurrency limits, permission policies, and execution boundaries before the implementation can run. + +This boundary separates model intent from system authority. A model may request an action, but emitting a syntactically valid call does not create a capability or grant permission. The schema teaches the model how to express the request, the binding determines whether the runtime possesses the capability, and permission determines whether this particular invocation may proceed. + +After execution, the runtime converts the outcome into a provider-independent tool result and pairs it with the original call ID. In Maka's `RuntimeEvent Log`, the two sides become `function_call` and `function_response`. What the model requested and what the runtime actually returned both become replayable, auditable facts. + +The call ID is more than a message-format field. A turn may launch several calls at once, and completion order may differ from call order. Stable identities allow the runtime to route every result to the correct call and reconstruct the same causal relationships during recovery. + +Tool calling completes a crucial transition: model output is no longer only language for a human reader. It can become a request to inspect private data, consume resources, start processes, or mutate state. Deferred tools decide which capabilities enter the model's field of thought. A tool call lets one selected capability cross the language boundary and attempt to change reality. + +At that moment, the systems problem changes. A failed generation produces disappointing text. A failed tool call may occur after the real-world effect happened but before its result returned. Once the model has hands and feet, the runtime must become responsible for the consequences. + +## Reliable Tool Calls: Resume Replays History, Not Actions + +Tool calling connects the model to the real world, and imports the real world's uncertainty into the agent runtime. + +Suppose the model calls `Edit` to change a configuration port from `3000` to `4000`. The file write finishes, and the Maka process crashes immediately afterward. After restart, the runtime can see that the call has no result, but that does not prove the file was never modified. + +A missing result can represent several realities: dispatch never began; the tool is still running; the side effect completed but its result was never persisted; or the external state changed again after execution. If resume simply executes the call again, it can duplicate writes, messages, object creation, or even payments. + +This is the most important difference between a tool call and text generation. Missing text can be regenerated. An action that already crossed a process boundary cannot be assumed absent merely because the runtime did not receive its result. + +Maka places two durable boundaries around real tool execution: + +```text +Model emits function_call + │ + ▼ +Arguments, availability, permission, and boundary checks + │ + ▼ +T1: Commit Tool Dispatch + │ + ▼ +Execute the real-world operation + │ + ▼ +T2: Commit function_response + │ + ▼ +Expose Tool Result to the model +``` + +T1 means that the runtime has completed every pre-execution check and has formally crossed the dispatch boundary. From this point onward, the system can no longer safely claim that the tool did not run. T1 must commit before the implementation begins; if the commit fails, the side effect is not allowed to start. + +T2 means that the outcome has become a durable `function_response`. Only after T2 commits may the result enter the next model inference. Even if a tool returns successfully, the runtime cannot show the model a result that it would be unable to reconstruct after restart. + +Maka does not try to wrap the entire tool call in a database transaction. Filesystem operations, shell commands, browser actions, and network requests can take seconds or hours, and SQLite cannot participate in a true distributed transaction with all of those systems. Maka instead uses two short transactions to make the side-effect window explicit: + +```text +Committed T1 → External Side Effect → Committed T2 +``` + +Wherever the process crashes, the committed append-only prefix gives the restarted runtime a precise classification: + +| Durable facts | Runtime conclusion | +|---|---| +| T1 was never crossed | The tool was definitely not dispatched | +| Both T1 and T2 exist | The tool completed; reuse the existing result and never execute it again | +| T1 exists but T2 is missing | The side-effect state is unknown; reconcile or park | +| Call, dispatch, or response identities conflict | The ledger is corrupt; fail closed | + +The interval between T1 and T2 is the dangerous case. The system knows that execution was authorized, but not whether the external effect finished. Maka does not let the model guess, and does not reinterpret "no result" as "not executed." Tool bindings can declare recovery semantics, such as natural idempotency, support for observing an existing outcome, or a prohibition on automatic retry. Without enough evidence, the runtime parks the operation for stronger observation or human intervention. + +Recovery remains append-only. The runtime does not edit the old `function_call` or fabricate a past that never happened. Dispatch, outcome, reconciliation, and recovery decisions are appended as new facts. Old facts remain unchanged; later facts explain how the operation eventually converged. + +Resume becomes safe only after every tool call has been classified as completed or definitely not dispatched. + +"Replay" is easy to misunderstand here. Maka does not execute historical tools again, nor does it resurrect the old process's promises, JavaScript stack, sockets, or child processes. It replays the valid history that the model had already observed: user messages, model output, paired `function_call` and `function_response` events, and other facts admissible to provider context. + +```text +Immutable RuntimeEvent Prefix + │ + ├── Resolve tool operations + ├── Discard streaming partials + ├── Preserve paired calls and responses + ├── Trim an interrupted, non-replayable suffix + └── Validate high-water and digest + │ + ▼ + Verified Provider Replay + │ + ▼ + New Run / Invocation / Turn +``` + +The append-only structure makes this natural. Resume does not infer progress from objects left in old process memory or reconstruct execution from UI state. It reads the immutable event prefix through a recorded high-water mark, verifies its digest, and projects the provider history required for the next inference. + +The continuation receives new run, invocation, and turn identities, and records the source run and event high-water from which it continues. It does not duplicate the original user message, and completed tools do not execute again. A continuation inherits verified causal history, not a list of commands waiting to be rerun. + +Before invoking the model, Maka also rechecks the external conditions on which that history depends: whether the workspace is still the same workspace, whether required tool bindings still exist, whether background processes and child tasks have converged, and whether another continuation already claimed the same recovery boundary. If any condition cannot be proven, resume parks instead of carrying old conclusions into a changed world. + +Maka's Resume is therefore not "continue executing code from the crash instruction pointer." It first gives every real-world action a trustworthy conclusion in the log, then creates a new execution from an immutable and verified history. Tool-call recovery answers whether an action happened. The append-only log answers which facts the model may continue from. + +Once real-world actions reliably settle into log facts, resume stops being an attempt to rescue an old process. It becomes the problem of constructing a new runtime from history. + +## Code Mode: When a Tool Call Becomes a Program + +So far, every tool call in this discussion has happened one at a time. + +The model chooses a next action, the runtime executes it, and the result returns to context. The model reads the observation, reasons again, and decides whether to call another tool. When every step requires semantic judgment, this is exactly how an agent should work. + +But not every step deserves another model invocation. + +Imagine an agent that must read twenty files, identify those containing a dependency, inspect each configuration, and report only projects with inconsistent versions. With ordinary tool calling, the model may request one read, inspect the result, request the next, and repeat. Every intermediate result enters context, while loops, filtering, and aggregation advance through repeated inference. + +```text +Reason → Call → Observe → Reason → Call → Observe → ... +``` + +The task may require model judgment only when forming the initial plan and interpreting the final anomalies. Most of the middle is deterministic control flow. Asking an LLM to impersonate a `for` loop is slow, and burdens future context with every raw result. + +Code Mode changes this layer. + +Rather than emitting a separate top-level call for every action, the model writes a small program that invokes multiple tools. Loops, parallelism, branches, field extraction, and aggregation run inside a constrained code environment. The model sees only what the program elects to return. + +```text + ┌─ Tool A ─┐ +Reason → Program ─┼─ Tool B ─┼→ Filter / Join / Reduce → Observe → Reason + └─ Tool C ─┘ +``` + +OpenAI Codex calls this execution shape Code Mode. The public Responses API describes the same class of capability as Programmatic Tool Calling: the model writes JavaScript that orchestrates available tools through `tools.*` in an isolated V8 runtime. Claude also provides Programmatic Tool Calling, using Python in a Code Execution Container and `allowed_callers` to specify which tools code may invoke. + +The protocols differ, but express the same judgment: LLMs are good at forming plans and resolving semantic uncertainty; programs are better at executing control flow that has already become explicit. + +This does not give the model an unbounded machine. A Code Mode program can reach only the capabilities exposed by the runtime. Writing network code does not create network access, and writing filesystem code does not bypass filesystem permissions. The program is an orchestration layer over tools, not a new source of authority. + +Nor does it replace tool calling. Programmatic Tool Calling turns a linear sequence into a call tree: a model-generated program sits at the root, and the tools invoked by that program become its children. Every leaf still requires runtime validation, authorization, and execution. + +```text +Program / exec +├── Tool Call 1 +├── Tool Call 2 +│ └── Tool Result 2 +└── Tool Call 3 + └── Tool Result 3 + │ + ▼ + Program Result +``` + +The most visible gain is fewer model round trips. A loop or batch query that once required repeated sampling can run inside one program. Equally important, programmatic execution reduces context pollution. Code can process dozens of raw results and return only the few lines that matter. Tool results have not vanished; the portions that require no model understanding simply never enter the model's state space. + +Code Mode and deferred tools therefore address two different kinds of tool-context pressure. Deferred tools reduce tool definitions loaded before inference. Code Mode reduces tool-result accumulation and model round trips during execution. The first controls the working set of capability descriptions; the second controls the working set of observations. + +Not every sequence belongs inside a program. A write may need human approval. A search result may change the direction of an investigation. An unexpected UI message may require fresh semantic interpretation. Irreversible effects are also often easier for humans to understand and control as explicit top-level calls. Code Mode should move deterministic work downward, not hide every agent decision inside code. + +Maka's Code Mode preserves that boundary. The model submits a JavaScript cell through an `exec` tool. The cell can invoke only currently active tools that explicitly allow nesting. The execution environment has no ambient process, filesystem, or network capability, and it enforces limits on time, memory, source size, result size, call count, and concurrency. + +More importantly, every nested invocation returns to the same `ToolRuntime`. Argument validation, permissions, execution boundaries, and the T1/T2 durability semantics from the previous section do not disappear merely because code issued the call. Maka assigns each nested invocation its own identity and records its parent relationship to the outer `exec`. + +Those internal calls are durable, but they do not reenter model history as a long sequence of calls and results. Runtime events mark them as originating from Code Mode and hidden from provider replay. The model sees the outer `exec` and its final result. Again, Maka follows the same architecture: the log preserves complete facts, while provider context is a projection of those facts. + +Code Mode also sharpens the recovery problem. A program may finish three tools and crash while awaiting the fourth. Rerunning the entire program after restart would repeat real actions that already completed. Maka therefore never automatically retries an interrupted `exec`. Existing nested outcomes remain in the log, while the outer cell receives an explicit interrupted result. A new model inference then decides how to continue. + +The program is not a shortcut around reliability. It compresses reasoning round trips between model and runtime, but cannot compress facts that already happened in the world. The program and its call stack may be ephemeral. Every tool call that crosses a real execution boundary must still leave an auditable, recoverable record. + +Tool calling moves the model from language into action. Code Mode takes another step: the model produces not just an action, but the structure among actions. + +## Parallel Tool Calls: Async I/O for Agent Runtimes + +Code Mode can call several tools concurrently from a program. Even without Code Mode, modern models can emit multiple tool calls in one assistant step. + +This is commonly called Parallel Tool Calling, but "parallel" needs a precise meaning. The model does not observe the first result while deciding the second call. It commits the entire batch in one generation, so calls in that batch cannot have data dependencies based on tool results. + +If the second action must consume the first result, it belongs in the next model step rather than the same batch. + +```text +One Assistant Step + + ┌── Tool Call A ──→ Result A ──┐ +Model ──┼── Tool Call B ──→ Result B ──┼──→ Next Model Step + └── Tool Call C ──→ Result C ──┘ + + Fan-out / Fan-in +``` + +From the runtime's perspective, this resembles classic asynchronous I/O. Each tool call becomes an independently awaitable task. Once a task starts, the runtime does not need to hold a synchronous call stack for it. It can start other ready work, then wake the corresponding continuation when the filesystem, process, network, or remote service produces a result. Only after every task reaches a terminal state does the runtime hand the batch of results to the next model step. + +The benefit is not merely that execution is "faster." Waiting overlaps. While one web search is waiting on the network, another search, file read, or child agent need not wait alongside it. End-to-end latency moves from the sum of independent I/O delays toward the longest delay on the critical path. + +But an absence of result dependencies does not imply an absence of resource conflicts. + +A model can emit `Read(a)` and `Edit(a)` together. It can ask two tools to replace the same session state. Neither call consumes the other's result, but both contend for one real resource. If the runtime simply hands the batch to `Promise.allSettled()`, observation order, write order, and overwrite behavior depend on unpredictable execution timing. + +Maka [PR #4542](https://github.com/apache/maka/pull/4542) discusses this exact problem: how to preserve concurrency among independent I/O while giving conflicting operations a deterministic order. + +It is tempting to place all responsibility in a central tool scheduler. Such a scheduler can predict which resources each call reads or writes, start non-conflicting work immediately, and queue conflicts in model-generated order. This provides a clear batch orchestration policy, but should not become the only source of resource correctness. + +Classic async I/O offers a useful separation of concerns: executors schedule tasks; resource authorities manage resources. + +A Tokio executor does not inspect futures to discover whether they touch the same Redis key or file. It runs futures that are ready. Mutual exclusion, reader/writer fairness, capacity, and wakeups live closer to the resource in an async mutex, an RwLock, a semaphore, or an actor that exclusively owns the state. + +The same boundary applies to an agent runtime: + +```text +Tool Batch + │ Create tasks, retain result slots, propagate cancellation + ▼ +Resource Authority + │ Resolve identity, queue, exclude, check versions, wake waiters + ▼ +Filesystem / Terminal / Browser / Session / Remote Service +``` + +Why must the authority resolve resource identity? Because the real resource is often not the string in a tool argument. `link/a` and `real/a` may refer to the same file through a symbolic link. Different UI tools may target the same browser tab. Different MCP tools may share one remote session. Only the layer that owns or executes against the resource can know whether two names identify the same thing and where the operation actually linearizes. + +A lock that exists only inside the current tool-batch scheduler cannot protect against another turn, another agent, another process, or another code path reaching the same resource. Correctness must still hold at the point closest to the side effect. A batch scheduler remains valuable for reducing contention and creating deterministic orchestration, but it should not be the only lock. + +Different resources need not pretend to share one conflict model. Files fit canonical-path, writer-fair read/write leases. Terminals and browsers resemble actors with exclusive state ownership. Concurrency limits for remote providers, MCP servers, and child agents are capacity concerns and fit semaphores. Revisioned session state may use compare-and-swap. These systems share an asynchronous lifecycle, not one universal lock. + +This is also why resource conflict and capacity must remain separate: + +- Resource conflict asks whether two actions can happen concurrently without violating correctness. +- Capacity asks how much work the system is willing to run concurrently. + +Representing an API rate limit as a global resource conflict can reduce concurrency, but introduces unrelated head-of-line blocking: a slow request stalls a file read that shares no resource with it. Async I/O instead blocks only work that is genuinely not ready and lets independent work proceed. + +For actual conflicts, provider array order can serve as a stable tie-breaker. It must not be misread as a data dependency. The model did not see any intermediate result while generating the batch. Order can say who acquires a contended resource first; it cannot mean that a later call consumed an earlier result. + +Parallel tool calling therefore contains at least four distinct orders: + +```text +Model generation order + ≠ Task start order + ≠ Task completion order + ≠ Runtime event arrival order +``` + +An independent later task may start or finish first. Live events should enter the log in the order facts actually occur, carrying tool call IDs for causal association. Results sent to the provider can still be reassembled in original call order. Factual order and provider-protocol order are different projections of the same execution. + +Cancellation and failure must also obey the async lifecycle. A queued task that is cancelled must never start later. A task that already crossed T1 cannot be treated as nonexistent; the runtime must let it settle and record its outcome. An ordinary tool failure can return as one result alongside its siblings. A T1 or T2 persistence failure, however, should prevent queued work from acquiring dispatch permission. Active work must wind down safely while not-yet-started work freezes. + +This has the flavor of structured concurrency. A parent batch does not launch a collection of promises and walk away. It owns their lifetimes. Before the next model inference begins, every child task must have completed, been cancelled, or reached an explicit recoverable state. + +Parallel Tool Calling is therefore not fully described by saying "tools run at the same time." The hard part is drawing boundaries among three goals: overlap independent I/O, preserve correctness for shared resources, and give the batch a coherent lifecycle under cancellation, failure, and recovery. + +The model expresses concurrent intent. The batch runtime joins it structurally. Resource authorities decide which concurrency reality permits. + +## Sandboxes and Serverless: Giving an Agent a Disposable Computer + +A tool call ultimately has to run somewhere. + +The model can emit an invocation and write an orchestration program, but it cannot conjure CPU, memory, filesystems, or network connections. JavaScript execution, Python processes, dependency installation, test runs, and browser automation all consume real computing resources. + +The lightest environment may be a JavaScript V8 isolate. It starts quickly and provides a narrow boundary suitable for short Code Mode control flow. Data analysis and large library ecosystems may call for a Python runtime. Tools that need a complete filesystem, system commands, compilers, and background processes naturally lead to containers or even microVMs. + +```text +LLM emits intent + │ + ▼ +Agent Runtime + │ Select environment and capabilities + ▼ +┌──────────┬──────────────┬─────────────┐ +│ V8 │ Python │ MicroVM │ +│ Orchestr.│ Data/scripts │ Full OS tools│ +└──────────┴──────────────┴─────────────┘ + │ + ▼ +Filesystem / Process / Network / Browser +``` + +Heavier is not always better. Starting a VM for every small tool call is wasteful; running untrusted system commands inside the runtime process is unsafe. The runtime should select an execution substrate that is light enough for the task and strong enough for the isolation it requires. + +A sandbox is therefore more than a wall around dangerous model-generated code. It is the resource boundary, fault boundary, and lifecycle boundary of one agent execution. + +The runtime can limit CPU, memory, disk, concurrency, and elapsed time. It can decide whether the sandbox has network access, which paths it can see, and which external services it can invoke. If code loops forever, exhausts memory, or crashes a process, the environment can be terminated without spreading the failure across the agent system. + +More importantly, the sandbox separates an agent from the machine currently running it. + +Traditional desktop software often assumes that a process and its local state persist. Agent execution environments should be assumed to disappear at any moment. A V8 cell ends when its code finishes. An idle container can be reclaimed. A microVM can vanish because of timeout, migration, preemption, or host failure. Recovery becomes nearly impossible if the agent's authoritative state lives inside those temporary environments. + +This is where the append-only log returns. + +Conversations, tool calls, results, permission decisions, and recovery facts live in a durable log. Files, media, and oversized results live in external artifact storage. Workspaces can be reconstructed from persistent volumes, snapshots, or objects. The sandbox carries only the computation currently in progress. It can be destroyed and recreated on another machine. + +```text +Durable State Ephemeral Compute + +RuntimeEvent Log ─┐ ┌─ V8 Isolate +Artifact Storage ─┼─→ Rehydrate ────┼─ Container +Workspace Snapshot┘ └─ MicroVM + + Preserves what happened Executes what happens next +``` + +Serverless and agents fit naturally because agent workloads are bursty. While the model reasons, the sandbox may have nothing to do. When a call arrives, it may suddenly need computation. Some tasks last milliseconds; others compile a large project or wait on long-running I/O. An ideal compute layer appears on demand, scales to zero while idle, and assigns different resource shapes according to tool requirements. + +Agent serverless cannot, however, be a simple copy of traditional Function as a Service. A conventional function receives input, computes, and returns. An agent also maintains a workspace, starts background processes, waits for approvals, calls external tools, and resumes hours later. It does not need an immortal process. It needs a protocol that reconnects durable state to ephemeral compute. + +When a sandbox disappears, the runtime should not try to restore its heap, promises, or stack frames. It should use the log to determine which tools ran and which outcomes committed, mount the necessary workspace and artifacts into a fresh environment, and start the next execution from a trustworthy historical prefix. + +Serverless does not mean the agent has no state. It means the state belongs to no individual computer. + +This architecture also changes permissions. A sandbox need not hold permanent credentials for every cloud service or receive ambient network access. It gets only the capabilities needed by the current task. Secrets, approvals, and resource authorities remain outside. Code may request an action, but the external runtime still decides whether that action may cross the boundary. + +Once execution is cheap enough, agents can scale in a new way. A short orchestration rents V8, data processing rents a Python container, and a complete software build rents a microVM. Child agents can run concurrently in isolated workspaces, then release every resource when they finish. + +Take this one step further: put every session in cheap S3-compatible object storage, and make the compute layer entirely out of inexpensive, short-lived, replaceable execution resources. + +This is complete disaggregation of storage and compute. + +A session no longer corresponds to an object in one process or a directory on one machine. It becomes a set of durable objects: append-only event segments, artifacts, workspace snapshots, compaction projections, and a manifest pointing to the current trustworthy prefix. After a conversation turn, no runtime needs to remain resident in memory. The session can rest in object storage while consuming almost no compute. + +```text + Cheap Durable Storage + +Session A ── Events / Artifacts / Workspace Snapshots ─┐ +Session B ── Events / Artifacts / Workspace Snapshots ─┼── S3 +Session C ── Events / Artifacts / Workspace Snapshots ─┘ + │ + Event / User / Schedule │ + │ │ + ▼ │ + Rehydrate a Session ◀─────────┘ + │ + ┌───────────┼───────────┐ + ▼ ▼ ▼ + V8 Python MicroVM + │ │ │ + └───────────┴───────────┘ + │ + Append Facts + │ + └──────────────→ S3 +``` + +A "long-running agent" no longer requires a long-running machine. + +It can remain dormant most of the time. When a user message, timer, webhook, or background completion arrives, the scheduler reads the session manifest, loads the required log prefix and workspace snapshot, and assigns a new sandbox. When the task finishes, new facts and artifacts return to object storage and the compute environment is released. + +The agent is not continuously alive. It is continuously awakenable. + +S3 is no longer merely backup media in this design. It can hold the factual state of the agent. Memory, SQLite, local SSD, vector indexes, and provider context on hot machines become caches or projections. They can accelerate reads, but they should not determine whether the session still exists. Lose the machine and rebuild the cache. Preserve the trustworthy history in object storage and the agent survives. + +Putting a session in S3 does not mean repeatedly appending in place to one giant object. A natural design writes immutable event segments and artifacts, then advances a small manifest or head pointer to the latest committed prefix. Leases, compare-and-swap, idempotency keys, and in-flight operation state still require a strongly consistent control plane. The large bodies of history, tool output, filesystem snapshots, and media can live in cheap object storage. + +The system separates naturally into two layers: + +- The data plane stores immutable, voluminous, rarely modified session state. +- The control plane stores small, strongly consistent heads, leases, admissions, and operation state. + +This resembles storage-compute disaggregation in modern databases. Object storage provides vast, inexpensive, durable capacity. Compute nodes appear only when a query or mutation needs them. Here the object being queried and continued is not a table. It is an agent's history. + +From this perspective, model context is itself a query. The runtime reads durable session state from S3 and applies compaction, tool-result pruning, visibility, and provider-compatibility projections to construct what the model should see now. The model's next output does not rewrite the past; it appends new facts. + +```text +Session on S3 + │ + ├── Projection ──→ Model Context ──→ LLM + │ │ + ├── Rehydrate ───→ Sandbox ───────→ Tool Call + │ │ + └──────────────── Append New Facts ◀───┘ +``` + +Both the LLM and the sandbox now become compute resources. + +Model capacity can be rented according to task difficulty: an inexpensive model for routine work, a stronger model for difficult judgment. Execution capacity can likewise match capabilities: an isolate for orchestration, a container for scripts, a microVM for operating-system tools. A session belongs to no particular model and no particular sandbox. + +This creates a different agent economy. Cost no longer depends primarily on how many sessions exist, but on how many are thinking and acting now. Ten million dormant sessions can be ten million object prefixes. Only the small active fraction consumes model tokens, CPU, and memory. + +The cheapest agent is not an agent running on a smaller server. It is an agent with no server at all while asleep. + +Disaggregation also makes preemptible compute practical. Workers can come from inexpensive instances, shared pools, or capacity that may disappear at any time. Previously, killing a machine running an agent meant losing the conversation. Once state is externalized, losing a worker means losing only a temporary execution. The runtime classifies real-world effects through T1 and T2, then continues the session on another compute resource. + +Branching and forking also become cheap. Append-only history and copy-on-write workspace snapshots let several agents share one historical prefix and grow independent suffixes. Spawning a child agent need not copy the entire session. It records the prefix and snapshot from which it starts. Unchanged artifacts remain shared; only new facts consume new storage. + +Even model upgrades need not migrate sessions. History retains provider-neutral runtime facts, and a new model receives a projection suited to its protocol. One durable session can run on one model today and resume on another months later. The identity of an agent comes from the history it has lived through, not from the model weights currently loading it. + +The security boundary becomes cleaner as well. S3 holds encrypted, auditable long-term state. A sandbox receives minimal capabilities only for its short lifetime. Secrets need not enter workspace snapshots, and permanent cloud credentials need not enter microVMs. When code needs an external resource, it asks an outside authority for one constrained operation. If the compute environment is compromised, its authority expires with the environment. + +Cheap compute does not automatically create correctness. An inexpensive microVM cannot make a duplicate payment safe. A restartable container cannot determine whether an email was sent before a crash. The more disposable workers become, the more the system depends on reliable tool calls, idempotent operations, resource authorities, and append-only logs to prove what happened in the real world. + +This is more than "run agents on serverless." We are assembling a new kind of computer for agents: + +```text +S3 is its inexpensive durable disk +Append-Only Log is its recoverable state +LLM is its rented reasoning unit +Sandbox / MicroVM is its rented body +Agent Runtime is the operating system connecting them +``` + +Discussion of agents today often centers on models. But models produce judgment and intent. Applying that intent to reality safely, reliably, and economically requires vast amounts of on-demand execution, together with session state that outlives every execution environment. + +The core infrastructure of future agents will include extremely cheap storage and extremely cheap compute. Storage lets hundreds of millions of sessions persist. Compute lets any one of them wake immediately when needed. The bridge is not the memory of one machine, but a history that can be replayed, verified, and extended. + +Look back across the tool stack. Deferred tools decide which capabilities deserve the model's attention. Tool calls turn language into action. Reliable execution makes action a trustworthy fact. Code Mode expresses structure among actions. The async runtime overlaps waiting. Sandboxes and serverless provide the CPU, memory, and isolation that all of those layers consume. + +Ultimately, an agent is not a long-lived process that happens to save some state. + +**An agent is durable state that temporarily rents a model and a computer whenever it needs to think and act.** + +It sleeps in cheap S3. When an event arrives, the log tells it who it used to be, the sandbox defines what it may do now, and inexpensive compute lets it move forward. diff --git a/docs/blogs/beyond-function-calling.zh-CN.md b/docs/blogs/beyond-function-calling.zh-CN.md new file mode 100644 index 0000000000..ab5bda0a0a --- /dev/null +++ b/docs/blogs/beyond-function-calling.zh-CN.md @@ -0,0 +1,518 @@ + + +[ENGLISH](./beyond-function-calling.md) + +# Tool Call 不只是 Function Calling：Agent 如何真正触碰现实世界 + +## Deferred Tool：没有被调用的 Tool 也有成本 + +在普通程序里，一个从未被调用的函数几乎不会产生运行时成本。它可以存在于代码库或动态链接库中，只要执行路径没有经过它，就不会消耗 CPU，也不会占用调用栈。 + +Agent 里的 Tool 不是这样。 + +模型想要调用一个 Tool，首先必须知道这个 Tool 的名称、用途以及参数格式。因此，Runtime 会把 Tool Definition 连同 System Prompt 和对话历史一起发送给模型。一个 Tool 即使从未被调用，它的 Description 和 JSON Schema 也已经进入了每一次推理。 + +这意味着 Tool 在执行之前就开始产生成本。Schema 会占用上下文窗口，会参与模型对下一步动作的判断，也会改变可供 Provider 缓存的请求前缀。Tool 越多，模型能够采取的动作越多，但留给任务本身的上下文越少，选择动作时需要面对的干扰也越大。 + +当 Agent 只有 `Read`、`Write`、`Bash` 等少数工具时，这个问题并不明显。但随着 Browser、Computer Use、子 Agent、外部服务和 MCP Connector 不断加入，把所有 Tool Schema 常驻在每一次模型请求里，就不再是一种可以持续扩展的方式。 + +Maka 的 Deferred Tool 从这里出发。这里的 Deferred 不是延迟执行，也不是让 Tool 在后台异步完成，而是延迟向模型暴露完整的 Tool Schema。 + +Runtime 仍然持有当前 Run 可以使用的全部 Tool Binding，但模型在第一次推理时只看到一组高频基础工具，以及一个轻量的 `tool_search`。其余 Tool 只以分组和名称出现在 Search Inventory 中，不携带完整的 Description 和参数 Schema。 + +```text +Bound Tool Registry + │ + ├── Direct Tools ───────────────→ 当前请求中的完整 Schema + │ + └── Deferred Tools + │ + └── 轻量 Search Inventory + │ + tool_search + │ + 有界的匹配结果 + │ + ▼ + 下一次 Provider Step + 注入匹配 Tool 的 Schema +``` + +`tool_search` 搜索的不是文件、网页或业务数据，而是 Runtime 已经拥有的能力。Maka 在本地根据 Tool 的名称、Description 和所属能力分组完成匹配，再选择数量与 Schema 体积都受限制的一组结果。返回给模型的只是被激活的 Tool 名称，完整 Schema 不会重复塞进 Tool Result，而是在下一次 Provider Request 中通过正常的 Tool Projection 出现。 + +这套机制把过去容易混在一起的几个概念拆开了： + +- **Bound**：Runtime 持有可执行的 Tool Binding，它定义了本次运行的能力上限。 +- **Discoverable**：Tool 出现在轻量 Inventory 中，模型知道某类能力存在。 +- **Visible**：完整 Tool Schema 已经进入当前 Provider Request，模型可以据此生成调用。 + +搜索不会绑定新的 Tool，也不能突破当前 Run 已有的 Binding Ceiling。它只是改变下一次模型调用看到的 Tool Projection。 + +“下一次”是这里很重要的边界。Provider Step 开始后，这次请求包含哪些 Tool Schema 就已经确定。假如模型在同一个响应里同时生成下面两个调用： + +```text +tool_search("browser click") +browser_click(...) +``` + +第二个调用仍然会被 Maka 拒绝。`tool_search` 的结果只能影响后续请求，不能反过来改写一份已经发送给 Provider 的 Schema 集合。直到下一个 Step，`browser_click` 的完整定义才会进入模型上下文，模型也才能基于自己真正见过的接口生成参数。 + +Deferred Tool 的激活状态只保留在当前 Turn 中。同一个 Turn 内，搜索得到的工具会单调累积，Provider 重试也会继承这份工作集；Turn 结束后，激活集合随之释放。下一轮对话重新从稳定的基础工具集开始，不会因为此前偶然使用过某项能力，就永久背负它的 Schema 成本。 + +Tool 的可见性也不等于执行授权。已经进入模型上下文的 Tool，真正调用时仍然需要经过权限判断、参数校验和 Runtime 的执行边界。`tool_search` 管理的是模型的认知范围，不是用户授予的权限范围。 + +因此，Deferred Tool 解决的并不是“工具怎样执行”，而是“哪些工具值得进入模型的下一次思考”。Runtime 保存完整的能力空间，模型看到的则是当前任务真正需要的工作集。 + +## Tool Call：让 LLM 长出手脚 + +Tool Schema 进入上下文之后，模型只是知道自己有哪些动作可以选择。直到它生成一个 Tool Call，一切仍然只是 Token。 + +LLM 本身不会读取文件，不会启动进程，也不会点击屏幕。它接收一段输入，再预测一段输出。即使模型回答“我已经修改了文件”，这句话本身也不会在磁盘上产生任何变化。语言描述的是世界，不能直接改变世界。 + +Tool Call 在两者之间建立了一条通道。模型不再只生成自然语言，而是按照 Tool Schema 输出一份结构化的动作意图，其中包含要调用的 Tool、传入的参数，以及用于关联结果的 Call ID。Runtime 接住这份意图，在真实环境中执行对应操作，再把执行结果送回模型。 + +```text +LLM + │ + │ function_call(name, arguments, call_id) + ▼ +Runtime + │ + ├── 查找 Tool Binding + ├── 校验参数与执行边界 + ├── 请求必要的权限 + ├── 调用真实实现 + ▼ +Filesystem / Process / Browser / Network / Human + │ + │ function_response(call_id, result) + ▼ +LLM 的下一次推理 +``` + +从这个闭环开始，模型才真正成为 Agent。读取文件让它获得对代码库的观察，执行命令让它得到编译器和测试系统的反馈，修改文件让它能够改变工作区，浏览器和网络工具把它连接到本地进程之外的环境，向用户提问则让它能够在信息不足时暂停并等待新的事实。 + +如果把 Tool 看作 Agent 的手脚，那么 Tool Result 就是感觉反馈。只有动作没有反馈，模型无法判断调用是否成功，也无法知道现实世界是否与自己的预测一致。一个完整的 Agent Step 因此不是“模型想了一次”，而是由意图、执行和观察共同组成： + +```text +Reason → Act → Observe → Reason +``` + +这个循环看起来像普通的函数调用，但它们之间存在一个根本区别。程序调用内部函数时，调用者和被调用者通常共享同一个确定性的执行环境；模型生成 Tool Call 时，它只是在根据概率分布提出下一步动作。参数可能不完整，目标可能已经变化，对环境的理解也可能是错的。 + +因此，更准确地说，并不是 LLM 自己长出了手脚，而是 Runtime 把一组受控的手脚借给了它。 + +在 Maka 中，模型生成的调用不能直接越过 Runtime 接触外部世界。Runtime 会先确认 Tool 确实存在于当前 Binding 中，并检查它是否已经对当前 Step 可见；参数必须符合 Tool Schema，调用还要经过并发限制、权限策略和执行边界。只有这些条件都成立，Tool 的真实实现才会运行。 + +这条边界区分了模型的意图与系统的授权。模型可以请求执行某个动作，但不能仅凭生成了一个合法 Tool Call，就为自己创造能力或取得权限。Tool Schema 告诉模型怎样表达请求，Tool Binding 决定 Runtime 是否拥有这种能力，Permission 则决定这一次具体请求能否执行。 + +执行结束后，Runtime 会把结果转换成与 Provider 无关的 Tool Result，再通过 Call ID 与原始调用配对。在 Maka 的 `RuntimeEvent Log` 中，这两端分别成为 `function_call` 和 `function_response`。这样，模型提出过什么动作、Runtime 实际返回了什么结果，都会成为可以重放和审计的运行事实。 + +Call ID 在这里不只是消息格式中的一个字段。一个 Turn 可能同时发起多个 Tool Call，执行完成顺序也未必与发起顺序一致。Runtime 必须依靠稳定的身份关联，才能把每份 Result 送回正确的 Call，并在恢复历史时重新构造同一组因果关系。 + +Tool Call 由此完成了一次关键转换：模型输出的不再只是供人阅读的语言，而是可能读取隐私、消耗资源、启动进程或者修改数据的操作请求。Deferred Tool 决定哪些能力进入模型的思考范围，Tool Call 则让其中一个选择越过语言边界，成为对现实世界的一次尝试。 + +从这一刻开始，Agent 系统面对的问题也发生了变化。一次生成失败，最多得到一段不理想的文本；一次 Tool Call 失败，却可能发生在现实效果已经产生、结果尚未返回的时候。模型有了手脚之后，Runtime 就必须开始对这些动作的后果负责。 + +## Reliable Tool Call：Resume 重放历史，而不是重做动作 + +Tool Call 把模型连接到现实世界，也把现实世界的不确定性带进了 Agent Runtime。 + +假设模型调用 `Edit`，要求把配置文件中的端口从 `3000` 改成 `4000`。文件刚刚写完，Maka 的进程恰好崩溃。重启之后，Runtime 只能看到这次 Tool Call 没有返回结果，但这并不能说明文件没有被修改。 + +缺少 Tool Result 可能对应完全不同的现实：调用尚未开始，工具正在执行，副作用已经完成但结果没有落盘，或者外部状态在执行后又被其他进程改变。如果 Resume 简单地把这次调用重新执行一遍，就可能制造重复写入、重复发送、重复创建甚至重复付款。 + +这也是 Tool Call 和普通文本生成之间最重要的差异。文本没有返回，可以重新生成；一个已经越过进程边界的现实动作，却不能因为 Runtime 没看到结果就假定它没有发生。 + +Maka 用两个持久化边界夹住 Tool 的真实执行： + +```text +Model 生成 function_call + │ + ▼ +参数、可用性、权限与执行边界检查 + │ + ▼ +T1：提交 Tool Dispatch + │ + ▼ +执行现实世界中的操作 + │ + ▼ +T2：提交 function_response + │ + ▼ +把 Tool Result 交给模型 +``` + +T1 表示 Runtime 已经完成所有执行前检查，并正式跨过了派发边界。从这一刻开始，系统不能再安全地声称 Tool 一定没有运行。T1 必须先提交，Tool 的真实实现才会被调用；如果 T1 提交失败，副作用就不允许开始。 + +T2 表示 Tool 的结果已经成为持久化的 `function_response`。只有 T2 提交成功，这份结果才可以进入下一次模型推理。即使 Tool 已经返回成功，如果 T2 没有落盘，Runtime 也不能把一个无法在重启后重建的结果临时交给模型。 + +Maka 没有尝试用一个数据库事务包住整个 Tool Call。文件操作、Shell 命令、浏览器动作和网络请求可能持续几秒甚至几小时，SQLite 不可能与这些外部系统共同完成一个真正的分布式事务。Maka 能做的是用两个很短的事务明确副作用窗口： + +```text +Committed T1 → External Side Effect → Committed T2 +``` + +这样一来，进程无论在哪里崩溃，重启后的 Runtime 都可以根据 Append-Only Log 中已经提交的前缀做出确定判断： + +| 日志事实 | Runtime 能够得出的结论 | +|---|---| +| 没有跨过 T1 | Tool 确定没有被派发 | +| T1 和 T2 都存在 | Tool 已经完成，直接使用既有 Result，不能重复执行 | +| T1 存在但 T2 缺失 | 副作用状态未知，需要 Reconcile 或 Park | +| Call、Dispatch、Response 的身份或顺序冲突 | Ledger 损坏，Fail Closed | + +其中最危险的是 T1 与 T2 之间。系统只知道 Tool 已经获得执行资格，却不知道现实效果是否完成。Maka 不会让模型根据上下文猜测，也不会把“没有 Result”自动解释成“没有执行”。Tool Binding 可以声明自己的恢复语义，例如操作是否天然幂等、能否重新观察结果，或者永远不能自动重试；缺少足够证据时，Runtime 会把这次操作 Park，等待更可靠的观察或人工处理。 + +这种恢复同样遵循 Append-Only。Runtime 不会回头修改原来的 `function_call` 或假装补上一段过去没有发生的历史。正常的 Dispatch、Outcome，以及后续可能产生的 Reconcile 和 Recovery Decision，都会作为新的事实继续追加到 Log 尾部。旧事实保持不变，新的事实负责解释旧操作最终收敛到了什么状态。 + +当所有 Tool Call 都已经被判定为 Completed 或 Definitely Not Dispatched，Resume 才具备安全重放的基础。 + +这里的“重放”很容易被误解。Maka 不会重新执行历史中的 Tool，也不会复活崩溃前的 Promise、JavaScript 调用栈、网络连接或宿主进程。它重放的是模型当时已经看到的合法历史：User Message、模型输出、成对的 `function_call` 与 `function_response`，以及其他可以进入 Provider Context 的确定事实。 + +```text +Immutable RuntimeEvent Prefix + │ + ├── 解析并收敛 Tool Operation + ├── 丢弃流式 Partial + ├── 保留成对的 Call / Response + ├── 裁掉无法构成合法历史的中断尾部 + └── 校验 High-Water 与 Digest + │ + ▼ + Verified Provider Replay + │ + ▼ + New Run / Invocation / Turn +``` + +Append-Only 结构让这件事变得自然。Resume 不需要猜测旧进程内存中曾经有哪些对象，也不需要从 UI 状态反推出执行进度。它只读取截至某个 High-Water 的不可变事件前缀，验证这段前缀的 Digest，再从中投影出下一次 Provider 调用需要看到的历史。 + +新的执行会获得全新的 Run、Invocation 和 Turn 身份，并记录自己从哪个 Source Run、哪个 Event High-Water 继续。原始 User Message 不会再复制一遍，已经完成的 Tool Call 也不会再次执行。Continuation 继承的是一段经过验证的因果历史，而不是一份准备重新运行的命令列表。 + +在真正调用模型之前，Maka 还会重新检查这段历史赖以成立的外部条件：Workspace 是否仍是同一个 Workspace，历史中使用过的 Tool 是否仍然存在，后台进程和子任务是否已经收敛，以及是否已经有另一个 Continuation 占用了同一恢复边界。任何一个条件无法证明，Resume 都会停在 Park，而不是带着旧结论进入一个已经变化的现实世界。 + +因此，Maka 的 Resume 并不是“从崩溃的位置继续执行代码”，而是先让每一次现实动作在日志中获得可信的结论，再从一段不可变、可验证的历史创建新的执行。Tool Call Recovery 解决了动作是否已经发生的问题，Append-Only Log 解决了模型应该从哪些事实继续的问题。 + +一旦现实动作能够稳定地沉淀为 Log 中的事实，Resume 就不再是对旧进程的抢救，而变成了一个从历史构造新 Runtime 的 Replay 问题。 + +## Code Mode：当 Tool Call 变成一段程序 + +到这里为止，我们讨论的 Tool Call 都是一次一个的。 + +模型先判断下一步要调用什么，Runtime 执行 Tool，再把 Result 放回上下文。模型读到结果之后，重新推理，决定是否调用下一个 Tool。对于每一步都需要语义判断的任务，这正是 Agent 应有的工作方式。 + +但并不是每一步都值得重新调用一次模型。 + +假设 Agent 需要读取二十个文件，找出包含某个依赖的文件，再分别读取它们的配置，最后只把版本不一致的项目列出来。如果沿用普通 Tool Call，整个过程会变成：模型发起一次读取，看到结果，再发起下一次读取；所有中间结果都进入上下文，循环、筛选和聚合也都靠一次又一次推理来推进。 + +```text +Reason → Call → Observe → Reason → Call → Observe → ... +``` + +这里真正需要模型判断的，也许只有任务开始时的执行计划，以及最后如何解释异常。中间大量工作只是确定性的控制流。让 LLM 逐步扮演 `for` 循环，不仅慢，也会让每一份原始 Tool Result 都成为后续上下文的负担。 + +Code Mode 改变的就是这一层。 + +模型不再为每个动作分别生成一次顶层 Tool Call，而是先生成一小段程序，由这段程序调用多个 Tool。循环、并发、条件分支、字段提取和结果聚合在受限的代码执行环境中完成，模型只需要看到程序最终选择输出的内容。 + +```text + ┌─ Tool A ─┐ +Reason → Program ─┼─ Tool B ─┼→ Filter / Join / Reduce → Observe → Reason + └─ Tool C ─┘ +``` + +OpenAI 的 Codex 把这种执行形态称作 Code Mode。在公开的 Responses API 中，同一类能力被称为 Programmatic Tool Calling：模型生成 JavaScript，在隔离的 V8 Runtime 中通过 `tools.*` 编排可用工具。Claude 也提供 Programmatic Tool Calling，只是让 Claude 在 Code Execution Container 中生成 Python，并通过 `allowed_callers` 指定哪些 Tool 可以从程序内部调用。 + +两种协议的实现细节不同，但表达的是同一个判断：LLM 擅长提出计划和处理语义不确定性，程序更适合执行已经明确的控制流。 + +这不是给模型一台没有边界的机器。Code Mode 中的程序能够触达什么，仍然由 Runtime 提供的 Tool 集合决定。它不能仅凭写下一段网络请求代码就获得网络，也不能因为生成了文件操作代码就绕过文件系统权限。程序只是 Tool 的编排层，不是新的权限来源。 + +它也没有取代 Tool Call。恰恰相反，Programmatic Tool Calling 把一个线性的 Tool Call 序列变成了一棵调用树：最外层是模型生成的 Program，下面是程序实际发起的 Tool Call。每个叶子节点最终仍要由 Runtime 校验、授权和执行。 + +```text +Program / exec +├── Tool Call 1 +├── Tool Call 2 +│ └── Tool Result 2 +└── Tool Call 3 + └── Tool Result 3 + │ + ▼ + Program Result +``` + +这种结构最直接的收益是减少模型往返。原本需要多次采样才能完成的循环或批量查询，可以在一个 Program 中执行。另一个同样重要的收益是减少上下文污染：程序可以先处理几十份原始结果，只把筛选后的几行结论交回模型。Tool Result 没有消失，只是其中不需要模型理解的部分没有进入它的状态空间。 + +因此，Code Mode 与 Deferred Tool 正好解决 Tool Context 的两个不同问题。Deferred Tool 减少的是推理开始前加载的 Tool Definition；Code Mode 减少的是执行过程中积累的 Tool Result 和模型往返。前者控制能力说明的工作集，后者控制执行结果的工作集。 + +当然，并不是 Tool Call 越多，越应该塞进一段程序。一次写入是否需要用户批准，搜索结果是否改变了下一步调查方向，页面上一个异常提示究竟意味着什么，这些都需要模型在观察之后重新判断。涉及不可逆副作用时，让动作保持为清晰、独立的顶层 Tool Call，往往也更容易被人理解和控制。Code Mode 适合下沉确定性的部分，不适合把所有 Agent 决策藏进代码。 + +Maka 的 Code Mode 延续了这个边界。模型通过一个 `exec` Tool 提交 JavaScript Cell，Cell 只能调用当前已经激活、并且允许嵌套的 Tool。执行环境本身没有进程、文件系统或网络能力，并受到运行时间、内存、源码体积、结果体积、调用次数和并发数的限制。 + +更关键的是，Cell 内部的调用仍然回到同一个 `ToolRuntime`。参数校验、权限判断、执行边界以及上一节讨论的 T1/T2 持久化语义都不会因为调用来自代码而消失。Maka 会为这些嵌套调用分配独立身份，并记录它们与外层 `exec` 的父子关系。 + +这些内部调用是 Durable 的，但不会作为一长串 Call / Result 再次塞给模型。它们在 Runtime Event Log 中标记为来自 Code Mode，对模型历史则是 Hidden；模型看到的是外层 `exec` 及其最终结果。这里又出现了 Maka 一贯的结构：Log 保存完整事实，Provider Context 只是对事实的一种 Projection。 + +Code Mode 也让恢复问题变得更尖锐。一段程序可能已经成功执行了前三个 Tool，却在第四个 Tool 等待结果时崩溃。如果重启后把整段程序重新运行一次，就会把已经完成的现实动作也重新做一遍。因此，Maka 不会自动重试一个中断的 `exec`。嵌套 Tool 的既有结果保留在日志中，外层 Cell 则获得一个明确的 Interrupted Result，之后由新的模型推理决定如何继续。 + +这说明 Program 并没有成为绕过可靠性的捷径。它压缩了模型与 Runtime 之间的推理回合，却不能压缩现实世界已经发生过的事实。程序可以是临时的，调用栈可以随着 Cell 一起消失；但每一次真正越过边界的 Tool Call，仍然必须留下可审计、可恢复的 Log。 + +Tool Call 让模型从语言走向行动。Code Mode 又向前走了一步：模型开始生成的不只是一个动作，而是动作之间的结构。 + +## Parallel Tool Call：Agent Runtime 里的 Async I/O + +Code Mode 可以在程序中并发调用多个 Tool。即使没有 Code Mode，今天的模型也可以在一个 Assistant Step 中一次生成多个 Tool Call。 + +这通常被叫作 Parallel Tool Call，但这里的“并行”需要先说清楚。模型并不是一边观察第一个调用的结果，一边决定第二个调用。它在同一次生成中已经把整组调用全部交给了 Runtime，因此这些调用之间不可能存在基于 Tool Result 的数据依赖。 + +如果第二个动作必须读取第一个动作的结果，它就不属于这一批，而应该出现在下一次模型推理中。 + +```text +同一个 Assistant Step + + ┌── Tool Call A ──→ Result A ──┐ +Model ──┼── Tool Call B ──→ Result B ──┼──→ 下一次 Model Step + └── Tool Call C ──→ Result C ──┘ + + Fan-out / Fan-in +``` + +从 Runtime 的角度看，这与经典 Async I/O 非常接近。每个 Tool Call 被转换成一个可以独立等待的 Task。Task 开始之后，Runtime 不需要为它占住一个同步调用栈，可以继续启动其他已经 Ready 的 Task；等到底层文件系统、进程、网络或远端服务返回结果，再唤醒对应的 Continuation。整批 Task 全部进入终态后，Runtime 才把 Tool Results 交给模型，开始下一轮推理。 + +这种结构的价值并不只是“更快”。更准确地说，它让等待可以重叠。一个 Web Search 正在等待网络时，另一个 Search、文件读取或子 Agent 不必陪它一起空等。Agent 的执行时间从多个 I/O 延迟之和，逐渐接近关键路径上的最长延迟。 + +但没有数据依赖，不等于没有资源冲突。 + +模型可以同时生成 `Read(a)` 和 `Edit(a)`，也可以同时要求两个 Tool 改写同一份 Session State。两个调用都不依赖对方的返回值，却可能争用同一个现实资源。如果 Runtime 只是把这一批调用全部交给 `Promise.allSettled()`，那么谁先观察、谁先写入、后写是否覆盖前写，就会取决于不可预测的执行时序。 + +Maka 在 [PR #4542](https://github.com/apache/maka/pull/4542) 中讨论的正是这个问题：一批 Tool Call 应该如何在保留独立 I/O 并发的同时，让访问同一资源的操作获得确定顺序。 + +这里很容易把所有责任都放进一个中央 Tool Scheduler。Scheduler 预先计算每个调用会读取或写入哪些资源，不冲突的立即执行，冲突的按照模型生成顺序排队。这种做法能够提供清晰的 Batch 编排，却不应该成为资源正确性的唯一来源。 + +经典 Async I/O 对这件事有一个很有用的职责划分：Executor 调度 Task，Resource Authority 管理资源。 + +一个 Tokio Executor 不会分析 Future 是否访问了同一个 Redis Key，也不会猜测两段异步代码最终是否写入同一个文件。它负责运行已经 Ready 的 Future。互斥、读写公平性、容量和唤醒通常由更靠近资源的一层负责，例如 Async Mutex、RwLock、Semaphore，或者独占状态的 Actor。 + +同样的边界也适用于 Agent Runtime： + +```text +Tool Batch + │ 创建 Task、保留结果槽位、传播取消 + ▼ +Resource Authority + │ 确认资源身份、排队、互斥、版本检查、唤醒 + ▼ +Filesystem / Terminal / Browser / Session / Remote Service +``` + +为什么资源身份必须由 Authority 确认？因为真正的资源往往不是 Tool 参数中的那段字符串。`link/a` 和 `real/a` 可能通过符号链接指向同一个文件；两个不同的 UI Tool 可能操作同一个 Browser Tab；两个 MCP Tool 也可能共享同一个远端 Session。只有实际拥有或执行这个资源的一层，才能知道它们是否是同一个东西，以及操作在哪一个瞬间真正生效。 + +如果互斥只存在于当前 Tool Batch 的 Scheduler 中，它也无法约束另一个 Turn、另一个 Agent、另一个进程，或者任何绕过该 Scheduler 到达同一资源的执行路径。正确性必须在最靠近副作用的位置依然成立。Batch Scheduler 可以减少无谓竞争并提供确定性，但它更适合成为编排层，而不是唯一的锁。 + +不同资源也不必被塞进同一种冲突模型。文件适合按 Canonical Path 建立带写者公平性的读写 Lease；Terminal 和 Browser 更像拥有单一状态的 Actor；远端 Provider、MCP Server 和子 Agent 的并发上限是 Capacity 问题，更适合用 Semaphore 表达；带 Revision 的 Session State 则可以使用 CAS 检查。它们共享的是异步生命周期，不是同一种锁。 + +这也解释了为什么“资源冲突”和“容量限制”必须分开： + +- 资源冲突回答两个动作能否正确地同时发生。 +- 容量限制回答系统愿意同时承担多少个动作。 + +把 API QPS 限制伪装成一个与所有资源都冲突的全局锁，虽然能降低并发，却会制造不必要的 Head-of-Line Blocking。一个慢请求会挡住与它完全无关的文件读取。相反，Async I/O 追求的是只阻塞真正尚未 Ready 的 Task，让独立工作继续前进。 + +对于确实冲突的调用，Provider 返回的数组顺序可以作为一个稳定的 Tie-breaker，但不能被解释为数据依赖。模型在生成这一批调用时没有看见任何中间结果，这个顺序只能表示“发生冲突时谁先获得资源”，不能表示后一个调用消费了前一个调用的结果。 + +因此，Parallel Tool Call 中至少存在四种不同的顺序： + +```text +模型生成顺序 + ≠ Task 启动顺序 + ≠ Task 完成顺序 + ≠ Runtime Event 到达顺序 +``` + +不冲突的后续 Task 可以先启动，也可以先完成。实时事件应该按照实际发生的时序进入 Log，并通过 Tool Call ID 保持因果关联；而发送给 Provider 的 Tool Result，则可以按照原始调用顺序重新组装。事实顺序与模型协议顺序不必相同，它们是同一次执行的不同 Projection。 + +取消和失败同样要遵守 Async I/O 的生命周期。还在队列中的 Task 被取消后不能偷偷开始；已经跨过 T1 的 Task 则不能假装不存在，Runtime 必须等待它收敛并记录结果。普通的 Tool 业务失败可以作为一个 Result 与同批其他任务一起返回，但如果 T1/T2 持久化失败，新的排队任务就不应继续获得派发资格。已经 Active 的工作需要安全结束，尚未开始的工作应该被冻结。 + +这正是经典 Async Runtime 中 Structured Concurrency 的味道：父级 Batch 不只是启动一堆 Promise 然后离开，它拥有这些 Task 的生命周期。下一次模型推理开始之前，每个子 Task 都必须已经完成、被取消，或者进入一个明确可恢复的状态。 + +Parallel Tool Call 因而不是一句“工具可以同时调用”就结束了。真正困难的是在三个目标之间建立边界：让独立 I/O 充分重叠，让共享资源保持正确，让整个 Batch 在取消、失败和恢复时仍然拥有清楚的生命周期。 + +模型提供并发的意图，Batch Runtime 负责结构化地汇合，Resource Authority 决定现实世界允许怎样的并发。 + +## Sandbox 与 Serverless：给 Agent 一台随时可以丢掉的计算机 + +Tool Call 最终必须在某个地方运行。 + +模型可以生成调用意图，可以写出一段编排程序，却不能凭空产生 CPU、内存、文件系统和网络连接。真正执行 JavaScript、启动 Python、安装依赖、运行测试或操作浏览器的，始终是一块现实中的计算资源。 + +最轻量的环境可以是一个 JavaScript V8 Isolate。它启动快、边界清楚，适合运行 Code Mode 中短小的控制流。需要数据分析和丰富 Library 时，可以给 Agent 一个 Python Runtime。再往下，当 Tool 需要完整文件系统、系统命令、编译器和后台进程时，自然会走向 Container，甚至 MicroVM。 + +```text +LLM 生成意图 + │ + ▼ +Agent Runtime + │ 选择执行环境与 Capability + ▼ +┌──────────┬──────────────┬─────────────┐ +│ V8 │ Python │ MicroVM │ +│ 编排调用 │ 数据与脚本 │ 完整 OS Tool│ +└──────────┴──────────────┴─────────────┘ + │ + ▼ +Filesystem / Process / Network / Browser +``` + +这些环境不是越重越好。让每个简单 Tool Call 都启动一台 VM 很浪费，让不受信任的系统命令与 Runtime 运行在同一个进程里又过于危险。Agent Runtime 需要根据任务真正需要的能力，选择足够轻、同时又足够隔离的执行载体。 + +Sandbox 因此不只是防止模型运行危险代码的围墙。它还是一次 Agent Execution 的资源边界、故障边界和生命周期边界。 + +Runtime 可以限制一个 Sandbox 能使用多少 CPU、内存、磁盘、并发和运行时间；可以决定它是否拥有网络、可以看到哪些目录、能够调用哪些外部服务；也可以在代码死循环、内存耗尽或进程崩溃时，直接终止这个环境，而不让故障扩散到整个 Agent 系统。 + +更重要的是，Sandbox 把“Agent”与“运行 Agent 的那台机器”分开了。 + +传统桌面程序往往默认进程和本地状态长期存在。Agent 的执行环境则应该被假设为随时可能消失：V8 Cell 运行结束就销毁，Container 空闲后可以回收，MicroVM 可以因为超时、迁移或宿主故障而终止。只要系统把 Agent 的真实状态寄托在这些临时环境里，恢复就会变得异常困难。 + +这也是 Append-Only Log 再次出现的地方。 + +对话、Tool Call、Tool Result、权限决定和恢复结论保存在 Durable Log 中；文件、图片和大型结果进入外部 Artifact Storage；Workspace 可以通过持久卷、快照或对象存储恢复。Sandbox 只承载当前正在运行的计算。它可以被销毁，也可以在另一台机器上重新创建。 + +```text +Durable State Ephemeral Compute + +RuntimeEvent Log ─┐ ┌─ V8 Isolate +Artifact Storage ─┼─→ Rehydrate ────┼─ Container +Workspace Snapshot┘ └─ MicroVM + + 保存“发生过什么” 执行“下一步做什么” +``` + +Serverless 与 Agent 天然契合的原因也在这里。Agent 工作负载通常是突发的：模型思考时 Sandbox 可能无事可做，Tool Call 到来时又需要迅速获得计算；有些任务只运行几十毫秒，有些任务要编译大型项目或等待长时间 I/O。理想的计算层应该能够按需创建、闲时归零，并根据 Tool 的资源声明分配不同规格。 + +但 Agent Serverless 不能只是传统 Function as a Service 的简单翻版。普通函数通常接收输入、计算并返回结果；Agent 还会保有 Workspace，启动后台进程，等待用户批准，调用外部 Tool，并在数小时后 Resume。它需要的不是一段永不消失的进程，而是一套能够把 Durable State 与 Ephemeral Compute 重新接合起来的协议。 + +一个 Sandbox 消失以后，Runtime 不应该尝试恢复它原来的内存、Promise 和调用栈。它应该先根据 Log 判断哪些 Tool 已经发生、哪些结果已经提交，再把 Workspace 和必要 Artifact 装载到新的环境，从可信的历史前缀开始下一段执行。 + +换句话说，Serverless 的重点不是 Agent 没有状态，而是它的状态不属于任何一台计算机。 + +这种架构也会改变权限的实现方式。Sandbox 不需要持有所有云服务的永久凭证，也不应该天然拥有完整网络。它只得到本次任务需要的 Capability；真正的 Secret、审批和 Resource Authority 留在 Sandbox 外部。代码可以请求一个动作，但外部 Runtime 仍然决定这个动作能否越过边界。 + +当这样的计算层足够便宜之后，Agent 才能真正扩大规模。一个 Agent 可以为一次短暂编排申请 V8，为一次数据处理申请 Python Container，为一次完整软件构建申请 MicroVM；也可以同时创建多个隔离环境，让子 Agent 在不同 Workspace 中并行工作，结束后立即释放资源。 + +再沿着这个方向往前走一步，会得到一个更有意思的形态：所有 Session 都沉到廉价的 S3-Compatible Object Storage 中，计算层则完全由廉价、短暂、可以被替换的执行资源组成。 + +这是一种彻底的存算分离。 + +Session 不再对应某个进程中的对象，也不对应某台机器上的目录。它是一组持久对象：Append-Only Event Segments、Artifact、Workspace Snapshot、Compaction Projection，以及指向当前可信前缀的 Manifest。一次对话结束之后，不需要有任何 Runtime 继续驻留在内存里。Session 可以安静地躺在对象存储中，除了存储本身几乎不消耗计算资源。 + +```text + Cheap Durable Storage + +Session A ── Events / Artifacts / Workspace Snapshots ─┐ +Session B ── Events / Artifacts / Workspace Snapshots ─┼── S3 +Session C ── Events / Artifacts / Workspace Snapshots ─┘ + │ + Event / User / Schedule │ + │ │ + ▼ │ + Rehydrate a Session ◀─────────┘ + │ + ┌───────────┼───────────┐ + ▼ ▼ ▼ + V8 Python MicroVM + │ │ │ + └───────────┴───────────┘ + │ + Append Facts + │ + └──────────────→ S3 +``` + +所谓“长期运行的 Agent”，也就不再要求一台机器长期运行。 + +它可以绝大多数时间都处于休眠状态。用户发来消息、定时器到期、Webhook 抵达或者后台任务完成时，调度层读取 Session Manifest，加载必要的 Log Prefix 与 Workspace Snapshot，为它分配一个新的 Sandbox。任务完成后，新事实和 Artifact 回写对象存储，计算环境随即释放。 + +Agent 不是一直活着，只是随时可以被重新唤醒。 + +这里的 S3 也不再只是备份介质，而可以成为 Agent State 的事实存储。热机器上的内存、SQLite、Local SSD、向量索引和 Provider Context 都只是缓存或 Projection。它们可以提高读取速度，却不应该决定 Session 是否仍然存在。机器丢了，缓存可以重建；只要对象存储里的可信历史仍在，Agent 就仍在。 + +当然，把 Session 放进 S3，不意味着对同一个大对象不断执行原地 Append。更自然的实现是写入不可变的 Event Segment 和 Artifact，再用很小的 Manifest 或 Head Pointer 指向最新的已提交前缀。Lease、CAS、幂等键和正在执行的 Operation 仍然需要一个强一致的控制面，但庞大的历史正文、Tool Result、文件快照和媒体内容都可以进入廉价对象存储。 + +于是整个系统会自然分成两层： + +- Data Plane 保存不可变、体积巨大、很少修改的 Session State。 +- Control Plane 保存体积很小、需要强一致的 Head、Lease、Admission 和 Operation 状态。 + +这与现代数据库的存算分离很像。对象存储提供近乎无限、廉价而持久的容量，计算节点只在查询或写入发生时出现。只不过这里被查询和继续执行的，不是一张表，而是一个 Agent 的历史。 + +从这个角度看，Model Context 本身也是一次 Query。Runtime 从 S3 中读取 Session 的 Durable State，应用 Compaction、Tool Result Prune、Visibility 和 Provider Compatibility 等 Projection，构造出这一轮模型真正需要看到的上下文。模型完成推理后，新的输出不去修改过去，而是继续追加新的事实。 + +```text +Session on S3 + │ + ├── Projection ──→ Model Context ──→ LLM + │ │ + ├── Rehydrate ───→ Sandbox ───────→ Tool Call + │ │ + └──────────────── Append New Facts ◀───┘ +``` + +这样一来，LLM 和 Sandbox 都只是计算资源。 + +模型可以根据任务难度临时选择，轻任务使用便宜模型，复杂决策使用更强模型；执行环境也可以根据 Capability 临时选择，简单编排进入 Isolate，普通脚本进入 Container，完整系统操作进入 MicroVM。同一个 Session 不属于任何一个模型，也不属于任何一种 Sandbox。 + +这会带来一种新的 Agent Economics。系统成本不再主要取决于保存了多少 Session，而取决于此刻有多少 Session 正在思考和行动。一千万个休眠 Session 可以只是对象存储中的一千万组前缀；只有被事件唤醒的那一小部分，才占用模型 Token、CPU 和内存。 + +最便宜的 Agent，不是运行在一台更小的服务器上，而是睡着时根本没有服务器。 + +存算分离也让抢占式计算真正可用。计算节点可以来自低价实例、共享 Worker Pool，甚至随时可能消失的 Capacity。过去，杀死一台正在运行 Agent 的机器意味着丢失整个会话；当状态已经外置，失去一个 Worker 只是失去一份临时执行。Runtime 根据 T1/T2 判断现实动作的状态，再把 Session 放到另一块计算资源上继续。 + +Branch 和 Fork 也会变得非常便宜。Append-Only History 与 Copy-on-Write Workspace Snapshot 天然允许多个 Agent 共享同一段历史前缀，再从不同位置长出各自的后缀。创建一个子 Agent 不必复制整个 Session，只需要记录它从哪个 Prefix 和 Snapshot 出发。没有修改的 Artifact 继续共享，只有新的事实产生新的存储。 + +甚至模型升级也不必迁移 Session。历史保留的是 Provider-Neutral 的 Runtime Fact，新模型只需要获得适合自己的 Context Projection。同一份 Durable Session 可以在今天由一个模型执行，几个月后由另一个模型 Resume。Agent 的身份来自它经历过的历史，而不是当前装载它的模型权重。 + +安全边界也因此变得更干净。S3 保存的是加密且可审计的长期状态，Sandbox 只在短暂生命周期内获得最小 Capability。Secret 不必写进 Workspace Snapshot，云账户的永久凭证也不必进入 MicroVM；需要访问外部资源时，Sandbox 通过外部 Authority 请求一次受约束的操作。计算环境被攻破之后，其权限会随着环境销毁而失效。 + +当然，廉价计算不会自动带来正确性。一个 MicroVM 再便宜，也不能让重复付款变得安全；一个 Container 再容易重启，也不能回答崩溃前的邮件是否已经发送。越是把 Worker 视为可以随时抛弃，越需要 Reliable Tool Call、幂等 Operation、Resource Authority 和 Append-Only Log 来证明现实世界中发生过什么。 + +所以这并不是一句简单的“把 Agent 跑在 Serverless 上”。更准确的说法是，我们正在为 Agent 构造一种新的计算机： + +```text +S3 是它廉价而持久的磁盘 +Append-Only Log 是它可恢复的状态 +LLM 是它按需租用的推理单元 +Sandbox / MicroVM 是它按需租用的身体 +Agent Runtime 是连接这一切的操作系统 +``` + +今天谈 Agent，注意力往往集中在模型上。但模型只负责产生判断和意图。让这些意图安全、可靠、低成本地作用于现实世界，需要大量随取随用的执行环境，以及比这些环境活得更久的 Session State。 + +未来 Agent 的核心基础设施，一定包含极其廉价的存储和极其廉价的计算。存储让数以亿计的 Session 可以长期存在，计算让其中任何一个 Session 都能在需要时迅速醒来。两者之间依靠的不是某台机器的内存，而是一条可以重放、验证和继续追加的历史。 + +回头看整条 Tool 链路，Deferred Tool 决定模型此刻需要知道哪些能力，Tool Call 把语言转换成行动，Reliable Execution 让行动成为可信事实，Code Mode 组织动作之间的结构，Async Runtime 让等待彼此重叠，而 Sandbox 与 Serverless 则为这一切提供真正可以消耗的 CPU、内存和隔离边界。 + +最终，Agent 不是一个恰好会保存状态的长驻进程。 + +**Agent 是一份持久状态，在需要思考和行动时，暂时租用一个模型和一台计算机。** + +它沉睡在廉价的 S3 中。事件到来时，Log 告诉它曾经是谁，Sandbox 决定它现在能够做什么，廉价计算让它继续向前。 From 092b0c2c55fcfafcab26df670ab542464dec8782 Mon Sep 17 00:00:00 2001 From: likun Date: Thu, 3 Sep 2026 18:56:24 +0800 Subject: [PATCH 2/2] docs(blog): adopt reviewer-polished drafts Generated-by: Gemini Flash --- docs/blogs/beyond-function-calling.md | 385 +++++++++----------- docs/blogs/beyond-function-calling.zh-CN.md | 363 ++++++++---------- 2 files changed, 323 insertions(+), 425 deletions(-) diff --git a/docs/blogs/beyond-function-calling.md b/docs/blogs/beyond-function-calling.md index 7dd5f0bacb..a59e09b148 100644 --- a/docs/blogs/beyond-function-calling.md +++ b/docs/blogs/beyond-function-calling.md @@ -23,28 +23,28 @@ ## Deferred Tools: Even an Unused Tool Has a Cost -In an ordinary program, a function that is never called has almost no runtime cost. It can sit in a codebase or a dynamic library without consuming CPU or occupying the call stack. +In standard programs, an uncalled function incurs negligible runtime overhead. It can sit in a source repository or a dynamic library without consuming CPU cycles or occupying stack frames. -Tools in an agent do not work that way. +Tools in an agent architecture behave fundamentally differently. -Before a model can call a tool, it must know the tool's name, purpose, and argument format. The runtime therefore sends tool definitions to the model together with the system prompt and conversation history. Even when a tool is never invoked, its description and JSON Schema have already participated in every inference. +Before a large language model can invoke a tool, it requires explicit awareness of the tool name, behavioral description, and structured parameter schema. Consequently, the runtime must transmit these definitions alongside the system prompt and conversation history on every request. Even if a tool is never triggered during an entire session, its schema continually consumes input tokens across every inference step. -A tool starts costing tokens before it starts executing. Its schema occupies context, influences the model's next-action decision, and changes the request prefix available for provider-side caching. More tools give the model a larger action space, but leave less room for the task itself and introduce more competing choices. +Tool overhead begins long before execution starts. Schemas occupy scarce context windows, dilute model attention during planning, and degrade prefix cache reuse across provider endpoints. As registries expand, the broader action space comes at the expense of task-specific context and introduces higher decision variance. -This is barely noticeable when an agent has only a few tools such as `Read`, `Write`, and `Bash`. It becomes a scaling problem once browsers, computer use, subagents, external services, and MCP connectors join the tool registry. Keeping every schema resident in every model request is not a sustainable architecture. +When an agent exposes only elementary tools like `Read`, `Write`, and `Bash`, the overhead remains manageable. Once the registry includes browser drivers, OS automation routines, subagent delegations, enterprise services, and dozens of Model Context Protocol (MCP) connectors, keeping all schemas resident across all requests breaks scalability. -Maka's deferred tools begin with this observation. "Deferred" does not mean delayed execution or background execution. It means delaying the moment when the complete tool schema becomes visible to the model. +Maka addresses this limitation through Deferred Tools. The mechanism does not alter execution timing; its purpose is to control when complete schemas become visible to the model. -The runtime still holds every tool binding available to the current run. On the first model request, however, the model sees only a small set of frequently used tools and a lightweight `tool_search`. Other tools appear in a compact search inventory by group and name, without their full descriptions or argument schemas. +The runtime continuously retains all registered tool bindings for the active run. However, the initial request exposes only high-frequency primitives alongside a compact `tool_search` utility. Extended tools register solely by name and category in a lightweight inventory, omitting full descriptions and parameter schemas. ```text Bound Tool Registry │ - ├── Direct Tools ───────────────→ Full schemas in this request + ├──── Direct Tools ───────────────→ Full schemas in this request │ - └── Deferred Tools + └──── Deferred Tools │ - └── Lightweight Search Inventory + └──── Lightweight Search Inventory │ tool_search │ @@ -55,38 +55,38 @@ Bound Tool Registry injects matched schemas ``` -`tool_search` does not search files, web pages, or application data. It searches capabilities already owned by the runtime. Maka performs the lookup locally against tool names, descriptions, and capability groups, then selects a bounded set of matches with bounded schema size. The result contains only the activated tool names. Full schemas are not duplicated inside the tool result; they appear through the normal tool projection in the next provider request. +The `tool_search` utility performs local lookups across capabilities already registered with the runtime. Maka matches queries against tool names, descriptions, and functional categories, returning a size-bounded candidate set. The payload returned to the model contains only the activated tool identifiers. Full schemas are never dumped directly into the tool result payload; they are injected into the subsequent model turn through standard tool projection. -This separates several concepts that are easy to conflate: +In Maka, tool state is organized into three distinct tiers: -- **Bound:** the runtime owns an executable tool binding. This defines the capability ceiling of the run. -- **Discoverable:** the tool appears in the lightweight inventory, so the model knows that the capability exists. -- **Visible:** the complete schema is present in the current provider request, so the model can construct a valid call. +- **Bound:** The runtime possesses an executable implementation, defining the absolute capability ceiling of the run. +- **Discoverable:** The tool is cataloged in the lightweight inventory, making the model aware of its availability. +- **Visible:** The complete schema is injected into the active provider request, enabling the model to construct valid calls. -Search does not bind a new tool and cannot exceed the run's binding ceiling. It changes only the tool projection visible to the next model call. +Capability discovery does not introduce unregistered implementations or exceed the binding ceiling. It serves exclusively to reshape the tool projection presented to subsequent inference steps. -"Next" is an important boundary. Once a provider step begins, its tool schemas are fixed. If the model emits both of these calls in one response: +Step boundaries enforce strict temporal separation. Once a provider step is dispatched, its schema set is immutable. If a model generates the following sequence within a single completion: ```text tool_search("browser click") browser_click(...) ``` -Maka still rejects the second call. A search result can affect a later request, but it cannot rewrite the schema set of a request already sent to the provider. Only in the next step does the full `browser_click` definition enter context, allowing the model to generate arguments for an interface it has actually seen. +Maka rejects the second call. Results from `tool_search` apply only to subsequent provider interactions; they cannot retroactively amend schemas already committed to the provider. The complete definition of `browser_click` enters context in the next step, allowing the model to construct arguments against a validated interface. -Deferred activation is scoped to the current turn. Tools discovered during a turn accumulate monotonically, and provider retries inherit that working set. When the turn ends, the activation set is released. The next user turn starts again from the stable base set instead of permanently paying for every capability used in the past. +Deferred activation is strictly scoped to the active turn. Discovered tools accumulate monotonically across retries within the turn, and release upon completion. Subsequent user turns reset to the baseline tool set, preventing intermittent tool usage from permanently burdening long-term inference context. -Visibility is also not authorization. A visible tool still passes through permission checks, argument validation, and runtime execution boundaries when called. `tool_search` manages the model's cognitive action space, not the user's permission space. +Visibility does not equate to authorization. A visible schema still requires parameter validation, concurrency checks, and permission gates upon invocation. The `tool_search` mechanism regulates cognitive surface area; system safety remains the sole responsibility of runtime enforcement. -Deferred tools therefore do not answer "how should a tool execute?" They answer "which tools deserve to enter the model's next thought?" The runtime retains the complete capability space while the model sees only the working set relevant to the current task. +Deferred Tools constrain the action space presented to the model. The runtime preserves comprehensive capabilities while exposing only task-relevant subsets per step. -## Tool Calls: Giving the LLM Hands and Feet +## The Action Boundary: Bridging Probability and System Side Effects -Once a tool schema enters context, the model merely knows which actions are available. Until it emits a tool call, everything remains tokens. +Injecting a tool schema into context only informs the model of available actions. Until the model produces a tool call, the interaction remains strictly within the domain of text tokens. -An LLM cannot read a file, start a process, or click a screen. It consumes input and predicts output. Even if it says, "I have modified the file," that sentence changes nothing on disk. Language describes the world; by itself, it does not alter the world. +Language models cannot directly manipulate host systems. They consume input sequences and predict subsequent tokens. Emitting the statement "I have updated the configuration" does not alter any byte on disk. A physical boundary separates descriptive language from concrete system state. -A tool call creates a channel between the two. Instead of producing only natural language, the model emits a structured action request: a tool name, arguments, and an ID that associates the eventual result with the call. The runtime receives the request, executes the corresponding operation in a real environment, and returns the observation to the model. +Tool calls bridge this boundary. The model ceases freeform generation and emits a structured action intent specifying the target tool, call arguments, and a correlation identifier (Call ID). The runtime intercepts this intent, executes the real operation within a sandboxed environment, and returns observed outcomes to the model. ```text LLM @@ -95,145 +95,139 @@ LLM ▼ Runtime │ - ├── Resolve the tool binding - ├── Validate arguments and execution boundaries - ├── Request permission when necessary - ├── Invoke the real implementation + ├── Resolve tool binding + ├── Validate arguments and execution bounds + ├── Request required permissions + ├── Invoke concrete system operation ▼ Filesystem / Process / Browser / Network / Human │ │ function_response(call_id, result) ▼ -The LLM's next inference +Next LLM inference step ``` -This closed loop is where a model becomes an agent. File reads give it observations of a codebase. Commands expose compiler and test feedback. File edits let it change the workspace. Browser and network tools connect it to systems outside the local process. Questions let it pause for new facts when information is missing. +This feedback loop allows the model to interact with external environments. File reads inspect workspace state, command executions capture compiler and test diagnostics, file modifications update working trees, network calls interface with external services, and user interaction tools pause for clarification. -If tools are the agent's hands and feet, tool results are its senses. Without feedback, the model cannot tell whether an action succeeded or whether reality matches its prediction. A complete agent step is therefore not simply "the model thought once." It combines intention, execution, and observation: +Tool results provide the empirical ground truth for subsequent decisions. Without observational feedback, models cannot verify whether operations succeeded or correct invalid assumptions. An end-to-end agent step consists of a closed loop across reasoning, dispatch, and observation: ```text Reason → Act → Observe → Reason ``` -This resembles a function call, but differs in a fundamental way. When a program calls an internal function, caller and callee usually share one deterministic execution environment. A model issuing a tool call is proposing an action from a probability distribution. Its arguments may be incomplete, its target may have changed, and its understanding of the environment may be wrong. +This interaction differs fundamentally from regular software invocation. Conventional programs link callers and callees inside deterministic execution environments. Tool calls generated by language models represent probabilistic action proposals. Arguments may be malformed, environmental preconditions may be stale, and assumptions regarding system state may be incorrect. -It is more accurate to say that the LLM does not grow its own hands and feet. The runtime lends it a controlled set. +Language models do not possess ambient execution authority. External side effects occur strictly through runtime arbitration, validation, and policy checks. -In Maka, a model-generated call cannot bypass the runtime and reach the outside world directly. The runtime verifies that the binding exists, that it is visible in the current step, and that the arguments conform to its schema. The call must also pass concurrency limits, permission policies, and execution boundaries before the implementation can run. +In Maka, invocations face rigorous validation before dispatch: bindings are checked, turn visibility is verified, parameter schemas are enforced, and concurrency policies are applied. Underlying system implementations execute only after all checks pass. -This boundary separates model intent from system authority. A model may request an action, but emitting a syntactically valid call does not create a capability or grant permission. The schema teaches the model how to express the request, the binding determines whether the runtime possesses the capability, and permission determines whether this particular invocation may proceed. +This boundary decouples model intent from system authorization. The model possesses proposal authority; producing syntactically valid JSON cannot grant environmental privileges. Tool schemas define the wire format for proposals, bindings register available capabilities, and runtime permissions arbitrate individual calls. -After execution, the runtime converts the outcome into a provider-independent tool result and pairs it with the original call ID. In Maka's `RuntimeEvent Log`, the two sides become `function_call` and `function_response`. What the model requested and what the runtime actually returned both become replayable, auditable facts. +Upon completion, the runtime normalizes external payloads into provider-neutral tool results, paired via stable Call IDs. Within Maka's `RuntimeEvent Log`, these events are committed as immutable `function_call` and `function_response` entries, establishing an auditable factual foundation for replay and crash recovery. -The call ID is more than a message-format field. A turn may launch several calls at once, and completion order may differ from call order. Stable identities allow the runtime to route every result to the correct call and reconstruct the same causal relationships during recovery. +Call IDs serve as architectural anchors. When an agent dispatches multiple concurrent calls, disparate I/O latencies shuffle completion order. The runtime relies on deterministic identifiers to route results back to their respective causal chains and preserve structural topology during replay. -Tool calling completes a crucial transition: model output is no longer only language for a human reader. It can become a request to inspect private data, consume resources, start processes, or mutate state. Deferred tools decide which capabilities enter the model's field of thought. A tool call lets one selected capability cross the language boundary and attempt to change reality. +Tool calls transform model output from human-directed prose into system invocations with irreversible side effects. The runtime must therefore implement rigorous engineering boundaries to manage external consequences safely. -At that moment, the systems problem changes. A failed generation produces disappointing text. A failed tool call may occur after the real-world effect happened but before its result returned. Once the model has hands and feet, the runtime must become responsible for the consequences. +## Reliable Execution: Crash Recovery Over Committed History -## Reliable Tool Calls: Resume Replays History, Not Actions +Introducing external side effects exposes the agent runtime to real-world infrastructure failures. -Tool calling connects the model to the real world, and imports the real world's uncertainty into the agent runtime. +Consider a scenario where a model calls `Edit` to update a port from `3000` to `4000`. The disk write completes, but the host process loses power immediately afterward. Upon restart, the runtime observes a dangling call without an associated result. This absence does not mean the filesystem remained untouched. -Suppose the model calls `Edit` to change a configuration port from `3000` to `4000`. The file write finishes, and the Maka process crashes immediately afterward. After restart, the runtime can see that the call has no result, but that does not prove the file was never modified. +A missing tool result can signify several conflicting states: the call was never dispatched, execution is still in progress, disk writes succeeded while metadata commits failed, or external processes modified state post-write. Blindly re-executing such calls risks duplicate writes, redundant financial transactions, or persistent data corruption. -A missing result can represent several realities: dispatch never began; the tool is still running; the side effect completed but its result was never persisted; or the external state changed again after execution. If resume simply executes the call again, it can duplicate writes, messages, object creation, or even payments. +Unlike text generation, external system actions cannot be assumed nonexistent simply because the runtime missed the return signal. -This is the most important difference between a tool call and text generation. Missing text can be regenerated. An action that already crossed a process boundary cannot be assumed absent merely because the runtime did not receive its result. - -Maka places two durable boundaries around real tool execution: +Maka encloses every external tool invocation within a lightweight two-phase persistence boundary: ```text -Model emits function_call +Model generates function_call │ ▼ -Arguments, availability, permission, and boundary checks +Validate parameters, visibility, permissions, and bounds │ ▼ T1: Commit Tool Dispatch │ ▼ -Execute the real-world operation +Execute real-world operation │ ▼ T2: Commit function_response │ ▼ -Expose Tool Result to the model +Deliver Tool Result to model ``` -T1 means that the runtime has completed every pre-execution check and has formally crossed the dispatch boundary. From this point onward, the system can no longer safely claim that the tool did not run. T1 must commit before the implementation begins; if the commit fails, the side effect is not allowed to start. +T1 signifies that all pre-flight validations passed and execution crossed the dispatch threshold. From this point forward, the runtime cannot safely assume the operation never occurred. T1 must commit before concrete implementations are invoked; if T1 persistence fails, external actions remain blocked. -T2 means that the outcome has become a durable `function_response`. Only after T2 commits may the result enter the next model inference. Even if a tool returns successfully, the runtime cannot show the model a result that it would be unable to reconstruct after restart. +T2 certifies that execution results have committed as an immutable `function_response` event. Only after T2 commits may the outcome enter subsequent model inference steps. Even if an external operation succeeds, missing T2 persistence prohibits feeding unverified state into the active reasoning loop. -Maka does not try to wrap the entire tool call in a database transaction. Filesystem operations, shell commands, browser actions, and network requests can take seconds or hours, and SQLite cannot participate in a true distributed transaction with all of those systems. Maka instead uses two short transactions to make the side-effect window explicit: +Maka avoids distributed database transactions across external systems. File I/O, shell tasks, browser drivers, and network requests exhibit wide variance in latency, making global ACID transactions impractical. Maka uses two localized storage transactions to bound the external side-effect window: ```text Committed T1 → External Side Effect → Committed T2 ``` -Wherever the process crashes, the committed append-only prefix gives the restarted runtime a precise classification: +When unexpected crashes occur, recovery logic derives exact status from the append-only event prefix: -| Durable facts | Runtime conclusion | +| Log State | Recovery Disposition | |---|---| -| T1 was never crossed | The tool was definitely not dispatched | -| Both T1 and T2 exist | The tool completed; reuse the existing result and never execute it again | -| T1 exists but T2 is missing | The side-effect state is unknown; reconcile or park | -| Call, dispatch, or response identities conflict | The ledger is corrupt; fail closed | +| No T1 committed | Operation never dispatched; safe to discard or re-evaluate | +| Both T1 and T2 present | Operation completed; reuse committed result without re-execution | +| T1 present, T2 missing | State indeterminate; force reconcile or park | +| Broken ID causality or ordering conflicts | Ledger corrupted; fail closed | -The interval between T1 and T2 is the dangerous case. The system knows that execution was authorized, but not whether the external effect finished. Maka does not let the model guess, and does not reinterpret "no result" as "not executed." Tool bindings can declare recovery semantics, such as natural idempotency, support for observing an existing outcome, or a prohibition on automatic retry. Without enough evidence, the runtime parks the operation for stronger observation or human intervention. +The interval between T1 and T2 represents the critical failure window. The system knows dispatch was authorized, but cannot confirm external completion. Maka prohibits speculative guessing and never defaults missing outcomes to failure. Tool bindings declare specific recovery policies: natural idempotency, queryable status checks, or strict prohibition of automatic retries. When definitive evidence is lacking, the runtime parks the operation, awaiting automated probes or operator intervention. -Recovery remains append-only. The runtime does not edit the old `function_call` or fabricate a past that never happened. Dispatch, outcome, reconciliation, and recovery decisions are appended as new facts. Old facts remain unchanged; later facts explain how the operation eventually converged. +Recovery operations remain append-only. The runtime never edits prior `function_call` events or fabricates missing history. Dispatches, outcomes, reconciliations, and operator decisions append to the log tail as new facts. Historical facts remain immutable; subsequent events record how dangling operations converged. -Resume becomes safe only after every tool call has been classified as completed or definitely not dispatched. +Resume routines initiate fresh execution cycles only after all pending operations resolve to Completed or Definitely Not Dispatched. -"Replay" is easy to misunderstand here. Maka does not execute historical tools again, nor does it resurrect the old process's promises, JavaScript stack, sockets, or child processes. It replays the valid history that the model had already observed: user messages, model output, paired `function_call` and `function_response` events, and other facts admissible to provider context. +Replay follows strict architectural boundaries. Maka never re-runs historical tool implementations, nor does it resurrect transient in-memory objects, unresolved promises, or dropped sockets. Replay reconstructs verified causal history: user inputs, reasoning traces, and paired `function_call` and `function_response` events. ```text Immutable RuntimeEvent Prefix │ - ├── Resolve tool operations - ├── Discard streaming partials - ├── Preserve paired calls and responses - ├── Trim an interrupted, non-replayable suffix - └── Validate high-water and digest + ├── Resolve and converge tool states + ├── Strip transient streaming chunks + ├── Retain paired Call / Response events + ├── Prune uncommitted dangling suffixes + └── Verify High-Water mark and Digest │ ▼ - Verified Provider Replay + Verified Provider Replay Plan │ ▼ - New Run / Invocation / Turn + Fresh Run / Invocation Instance ``` -The append-only structure makes this natural. Resume does not infer progress from objects left in old process memory or reconstruct execution from UI state. It reads the immutable event prefix through a recorded high-water mark, verifies its digest, and projects the provider history required for the next inference. - -The continuation receives new run, invocation, and turn identities, and records the source run and event high-water from which it continues. It does not duplicate the original user message, and completed tools do not execute again. A continuation inherits verified causal history, not a list of commands waiting to be rerun. +Leveraging append-only logs, recovery operates independently of volatile memory dumps. The runtime reads the immutable event slice up to the recorded high-water mark, validates its cryptographic digest, and projects canonical context for the subsequent step. -Before invoking the model, Maka also rechecks the external conditions on which that history depends: whether the workspace is still the same workspace, whether required tool bindings still exist, whether background processes and child tasks have converged, and whether another continuation already claimed the same recovery boundary. If any condition cannot be proven, resume parks instead of carrying old conclusions into a changed world. +The resumed instance receives distinct Run and Invocation identifiers, noting its parent run and high-water anchor. Original user prompts are not duplicated, and finished operations do not re-run. The continuation inherits verified historical facts rather than an imperative re-execution script. -Maka's Resume is therefore not "continue executing code from the crash instruction pointer." It first gives every real-world action a trustworthy conclusion in the log, then creates a new execution from an immutable and verified history. Tool-call recovery answers whether an action happened. The append-only log answers which facts the model may continue from. +Before dispatching model requests, Maka verifies fundamental environmental invariants: matching workspace paths, active tool bindings, converged background tasks, and absence of conflicting recoveries. If any condition cannot be confirmed, resume aborts to a parked state, preventing execution within compromised environments. -Once real-world actions reliably settle into log facts, resume stops being an attempt to rescue an old process. It becomes the problem of constructing a new runtime from history. +Maka crash recovery reconstructs execution from verified append-only history, rather than attempting to resurrect volatile process state. -## Code Mode: When a Tool Call Becomes a Program +## Code Mode: Programmatic Orchestration and Folded Call Trees -So far, every tool call in this discussion has happened one at a time. +Standard tool calling adheres to a sequential turn pattern: the model proposes an action, the runtime executes it, and the model re-evaluates the prompt. For workflows requiring continuous semantic reasoning at every step, this pattern provides necessary control. -The model chooses a next action, the runtime executes it, and the result returns to context. The model reads the observation, reasons again, and decides whether to call another tool. When every step requires semantic judgment, this is exactly how an agent should work. +However, for deterministic data transformations, this round-trip structure creates severe latency and token overhead. -But not every step deserves another model invocation. - -Imagine an agent that must read twenty files, identify those containing a dependency, inspect each configuration, and report only projects with inconsistent versions. With ordinary tool calling, the model may request one read, inspect the result, request the next, and repeat. Every intermediate result enters context, while loops, filtering, and aggregation advance through repeated inference. +Consider multi-package dependency audits: an agent must traverse dozens of directories, inspect `package.json` files, extract version constraints, and report discrepancies. Under sequential tool calling, the agent repeats dozens of inference cycles: generating read requests, waiting for file contents, parsing results, and emitting subsequent calls. Raw file contents flood the context window, and model round trips compound latency. ```text Reason → Call → Observe → Reason → Call → Observe → ... ``` -The task may require model judgment only when forming the initial plan and interpreting the final anomalies. Most of the middle is deterministic control flow. Asking an LLM to impersonate a `for` loop is slow, and burdens future context with every raw result. +In these workflows, reasoning is essential for initial planning and final error analysis, while intermediate steps involve deterministic control flow. Forcing models to emulate loops and string parsers incurs unnecessary inference cost and pollutes context with intermediate noise. -Code Mode changes this layer. +Code Mode replaces discrete invocations with programmatic orchestration. -Rather than emitting a separate top-level call for every action, the model writes a small program that invokes multiple tools. Loops, parallelism, branches, field extraction, and aggregation run inside a constrained code environment. The model sees only what the program elects to return. +Instead of emitting fragmented tool calls, the model produces an executable program. Iteration, concurrency, branching, parsing, and aggregation execute inside a sandboxed interpreter. The model receives only the final structured output. ```text ┌─ Tool A ─┐ @@ -241,13 +235,13 @@ Reason → Program ─┼─ Tool B ─┼→ Filter / Join / Reduce → Observe └─ Tool C ─┘ ``` -OpenAI Codex calls this execution shape Code Mode. The public Responses API describes the same class of capability as Programmatic Tool Calling: the model writes JavaScript that orchestrates available tools through `tools.*` in an isolated V8 runtime. Claude also provides Programmatic Tool Calling, using Python in a Code Execution Container and `allowed_callers` to specify which tools code may invoke. +Implementations vary across ecosystem providers: OpenAI exposes Programmatic Tool Calling within the Responses API, executing model-generated JavaScript in a secure V8 environment with access to `tools.*`; Anthropic allows Claude to execute Python scripts within a containerized environment, calling whitelisted tools programmatically. -The protocols differ, but express the same judgment: LLMs are good at forming plans and resolving semantic uncertainty; programs are better at executing control flow that has already become explicit. +Both approaches share common architectural principles: delegating non-deterministic planning to the model while offloading deterministic control flow to an execution engine. -This does not give the model an unbounded machine. A Code Mode program can reach only the capabilities exposed by the runtime. Writing network code does not create network access, and writing filesystem code does not bypass filesystem permissions. The program is an orchestration layer over tools, not a new source of authority. +Sandboxes operate under strict isolation. Code executed within the container accesses only tools explicitly surfaced by the runtime. Writing custom network or filesystem logic cannot bypass runtime permissions. The script acts as an orchestration layer, not an escalation of privilege. -Nor does it replace tool calling. Programmatic Tool Calling turns a linear sequence into a call tree: a model-generated program sits at the root, and the tools invoked by that program become its children. Every leaf still requires runtime validation, authorization, and execution. +Code Mode does not displace standard tool mechanisms. Instead, it reorganizes linear call sequences into a hierarchical call tree: the root node contains the program payload, while branch nodes represent concrete tool calls issued by the script. Each leaf operation must still pass through runtime validation, permission gates, and transaction boundaries. ```text Program / exec @@ -261,34 +255,30 @@ Program / exec Program Result ``` -The most visible gain is fewer model round trips. A loop or batch query that once required repeated sampling can run inside one program. Equally important, programmatic execution reduces context pollution. Code can process dozens of raw results and return only the few lines that matter. Tool results have not vanished; the portions that require no model understanding simply never enter the model's state space. - -Code Mode and deferred tools therefore address two different kinds of tool-context pressure. Deferred tools reduce tool definitions loaded before inference. Code Mode reduces tool-result accumulation and model round trips during execution. The first controls the working set of capability descriptions; the second controls the working set of observations. +Folding invocations into trees yields two primary benefits: it minimizes round-trip inference steps, and it shields the context window from intermediate telemetry. The sandbox absorbs raw operational payloads, returning only consolidated summaries to the outer context. Full operational details are preserved in audit logs without consuming inference memory. -Not every sequence belongs inside a program. A write may need human approval. A search result may change the direction of an investigation. An unexpected UI message may require fresh semantic interpretation. Irreversible effects are also often easier for humans to understand and control as explicit top-level calls. Code Mode should move deterministic work downward, not hide every agent decision inside code. +Programmatic orchestration should not be applied universally. Irreversible side effects, actions requiring human authorization, or workflows where subsequent steps depend on unstructured semantic observations benefit from explicit top-level tool calls. Code Mode is designed for deterministic data pipelines, not for concealing agent decisions. -Maka's Code Mode preserves that boundary. The model submits a JavaScript cell through an `exec` tool. The cell can invoke only currently active tools that explicitly allow nesting. The execution environment has no ambient process, filesystem, or network capability, and it enforces limits on time, memory, source size, result size, call count, and concurrency. +Maka enforces clear operational bounds within Code Mode. The model submits JavaScript cells via an `exec` primitive, restricted to registered tools marked for nested invocation. The execution environment lacks ambient OS capabilities, constrained by quotas on execution time, memory usage, script size, response size, and concurrency. -More importantly, every nested invocation returns to the same `ToolRuntime`. Argument validation, permissions, execution boundaries, and the T1/T2 durability semantics from the previous section do not disappear merely because code issued the call. Maka assigns each nested invocation its own identity and records its parent relationship to the outer `exec`. +Crucially, nested invocations within a cell route through the central `ToolRuntime`. Validation, permission evaluation, and T1/T2 transactions apply uniformly. Maka assigns discrete identifiers to nested calls, maintaining parent-child links with the host `exec` event. -Those internal calls are durable, but they do not reenter model history as a long sequence of calls and results. Runtime events mark them as originating from Code Mode and hidden from provider replay. The model sees the outer `exec` and its final result. Again, Maka follows the same architecture: the log preserves complete facts, while provider context is a projection of those facts. +Nested calls retain durable persistence semantics without inflating the model prompt. They commit to the `RuntimeEvent Log` with `modelVisibility: hidden`, while the model sees only the outer `exec` boundary and its aggregated result. Maka preserves complete factual history in storage while projecting a clean abstraction for inference. -Code Mode also sharpens the recovery problem. A program may finish three tools and crash while awaiting the fourth. Rerunning the entire program after restart would repeat real actions that already completed. Maka therefore never automatically retries an interrupted `exec`. Existing nested outcomes remain in the log, while the outer cell receives an explicit interrupted result. A new model inference then decides how to continue. +Crash recovery also accounts for programmatic execution. A script may execute three nested operations before crashing on the fourth. Re-running the entire script upon recovery would duplicate completed side effects. Maka prohibits automatic retries of unfinalized `exec` cells. Settled nested operations remain in the log, while the outer cell marks an interrupted state, leaving resumption strategy to subsequent model evaluation. -The program is not a shortcut around reliability. It compresses reasoning round trips between model and runtime, but cannot compress facts that already happened in the world. The program and its call stack may be ephemeral. Every tool call that crosses a real execution boundary must still leave an auditable, recoverable record. +Script environments are ephemeral, yet every nested tool invocation crossing the system boundary remains durably logged and auditable. -Tool calling moves the model from language into action. Code Mode takes another step: the model produces not just an action, but the structure among actions. +## Parallel Tool Execution: Decoupling Task Concurrency from Resource Authority -## Parallel Tool Calls: Async I/O for Agent Runtimes +Agents can emit concurrent tool calls within Code Mode programs or output parallel invocations in a single standard completion step. -Code Mode can call several tools concurrently from a program. Even without Code Mode, modern models can emit multiple tool calls in one assistant step. +Parallel tool calling requires clear architectural definition. When a model outputs a batch of calls in one step, it does so without observing any interim results. Therefore, invocations within that batch cannot possess causal data dependencies on each other. -This is commonly called Parallel Tool Calling, but "parallel" needs a precise meaning. The model does not observe the first result while deciding the second call. It commits the entire batch in one generation, so calls in that batch cannot have data dependencies based on tool results. - -If the second action must consume the first result, it belongs in the next model step rather than the same batch. +If an operation depends on data produced by another call, it must be scheduled in a subsequent reasoning step. ```text -One Assistant Step +Single Assistant Step ┌── Tool Call A ──→ Result A ──┐ Model ──┼── Tool Call B ──→ Result B ──┼──→ Next Model Step @@ -297,105 +287,104 @@ Model ──┼── Tool Call B ──→ Result B ──┼──→ Next Mod Fan-out / Fan-in ``` -From the runtime's perspective, this resembles classic asynchronous I/O. Each tool call becomes an independently awaitable task. Once a task starts, the runtime does not need to hold a synchronous call stack for it. It can start other ready work, then wake the corresponding continuation when the filesystem, process, network, or remote service produces a result. Only after every task reaches a terminal state does the runtime hand the batch of results to the next model step. +From an architectural standpoint, batch calls mirror asynchronous I/O primitives. Each tool call is handled as an independently awaitable task. The runtime avoids thread blocking, advancing concurrent tasks until external filesystems, processes, or networks respond. Once the entire batch settles, the runtime aggregates results for the next inference step. -The benefit is not merely that execution is "faster." Waiting overlaps. While one web search is waiting on the network, another search, file read, or child agent need not wait alongside it. End-to-end latency moves from the sum of independent I/O delays toward the longest delay on the critical path. +This structure allows independent wait states to overlap. A slow web query does not delay local file inspection or subagent execution. Overall latency converges toward the critical path rather than the sum of independent operations. -But an absence of result dependencies does not imply an absence of resource conflicts. +However, an absence of data dependencies does not guarantee an absence of resource conflicts. -A model can emit `Read(a)` and `Edit(a)` together. It can ask two tools to replace the same session state. Neither call consumes the other's result, but both contend for one real resource. If the runtime simply hands the batch to `Promise.allSettled()`, observation order, write order, and overwrite behavior depend on unpredictable execution timing. +A model may emit `Read(a)` and `Edit(a)` within the same batch, or instruct multiple tools to update shared session state simultaneously. While neither call consumes the other's return value, both contend for identical physical resources. Handing such batches directly to uncoordinated primitives like `Promise.allSettled()` introduces race conditions governed by nondeterministic execution timing. -Maka [PR #4542](https://github.com/apache/maka/pull/4542) discusses this exact problem: how to preserve concurrency among independent I/O while giving conflicting operations a deterministic order. +Maka addresses this challenge in [PR #4542](https://github.com/apache/maka/pull/4542): the runtime must maximize independent I/O parallelism while guaranteeing deterministic ordering for conflicting operations. -It is tempting to place all responsibility in a central tool scheduler. Such a scheduler can predict which resources each call reads or writes, start non-conflicting work immediately, and queue conflicts in model-generated order. This provides a clear batch orchestration policy, but should not become the only source of resource correctness. +Centralizing all concurrency constraints inside a single Tool Scheduler introduces architectural bottlenecks. Expecting a scheduler to statically deduce read/write sets from tool arguments creates brittle abstractions that fail across dynamic host environments. -Classic async I/O offers a useful separation of concerns: executors schedule tasks; resource authorities manage resources. +Asynchronous system design provides a clear separation of concerns: executors schedule task lifecycles, while resource authorities govern access constraints. -A Tokio executor does not inspect futures to discover whether they touch the same Redis key or file. It runs futures that are ready. Mutual exclusion, reader/writer fairness, capacity, and wakeups live closer to the resource in an async mutex, an RwLock, a semaphore, or an actor that exclusively owns the state. +An executor drives ready tasks forward. Mutual exclusion, reader-writer fairness, capacity limits, and wakeup signals belong to authorities positioned beside the underlying resources: asynchronous mutexes, reader-writer locks, semaphores, or state-owning actors. -The same boundary applies to an agent runtime: +Agent runtimes follow this division: ```text Tool Batch - │ Create tasks, retain result slots, propagate cancellation + │ Create tasks, allocate result slots, broadcast cancellation ▼ Resource Authority - │ Resolve identity, queue, exclude, check versions, wake waiters + │ Resolve identity, order, enforce exclusivity, check versions, wake ▼ Filesystem / Terminal / Browser / Session / Remote Service ``` -Why must the authority resolve resource identity? Because the real resource is often not the string in a tool argument. `link/a` and `real/a` may refer to the same file through a symbolic link. Different UI tools may target the same browser tab. Different MCP tools may share one remote session. Only the layer that owns or executes against the resource can know whether two names identify the same thing and where the operation actually linearizes. - -A lock that exists only inside the current tool-batch scheduler cannot protect against another turn, another agent, another process, or another code path reaching the same resource. Correctness must still hold at the point closest to the side effect. A batch scheduler remains valuable for reducing contention and creating deterministic orchestration, but it should not be the only lock. +Resource authorities must resolve true resource identity. Raw path arguments cannot reveal physical aliases: distinct paths may point to the same file via symlinks, multiple tools may manipulate the same browser tab, and separate MCP calls may target the same remote session. Only the authority directly managing the resource can arbitrate true contention and commit order. -Different resources need not pretend to share one conflict model. Files fit canonical-path, writer-fair read/write leases. Terminals and browsers resemble actors with exclusive state ownership. Concurrency limits for remote providers, MCP servers, and child agents are capacity concerns and fit semaphores. Revisioned session state may use compare-and-swap. These systems share an asynchronous lifecycle, not one universal lock. +Batch schedulers reduce unnecessary contention, but cannot serve as the sole source of safety. Schedulers cannot enforce exclusivity across independent turns, concurrent subagents, or external system processes. Safety must close at the resource authority layer. -This is also why resource conflict and capacity must remain separate: +Different resource types require tailored synchronization models: -- Resource conflict asks whether two actions can happen concurrently without violating correctness. -- Capacity asks how much work the system is willing to run concurrently. +- **Filesystems:** Canonical path leases with writer-priority or read-write fairness. +- **Terminals and Browsers:** Single-state actors enforcing strict sequential operations. +- **External APIs and MCP Servers:** Counting semaphores regulating concurrency and request quotas. +- **Versioned Session State:** Optimistic concurrency control via Compare-And-Swap (CAS) on revisions. -Representing an API rate limit as a global resource conflict can reduce concurrency, but introduces unrelated head-of-line blocking: a slow request stalls a file read that shares no resource with it. Async I/O instead blocks only work that is genuinely not ready and lets independent work proceed. +These patterns share an asynchronous lifecycle without forcing heterogeneous resources into a single locking model. -For actual conflicts, provider array order can serve as a stable tie-breaker. It must not be misread as a data dependency. The model did not see any intermediate result while generating the batch. Order can say who acquires a contended resource first; it cannot mean that a later call consumed an earlier result. +This separation clarifies the distinction between resource contention and capacity limits: -Parallel tool calling therefore contains at least four distinct orders: +- Resource contention determines whether operations can safely execute concurrently without corrupting state. +- Capacity limits determine how many concurrent operations the infrastructure can support. -```text -Model generation order - ≠ Task start order - ≠ Task completion order - ≠ Runtime event arrival order -``` +Treating upstream API rate limits as a global mutex introduces head-of-line blocking, allowing slow network calls to stall unrelated disk reads. Asynchronous runtimes should restrict blocking strictly to genuine physical conflicts, keeping independent work unhindered. -An independent later task may start or finish first. Live events should enter the log in the order facts actually occur, carrying tool call IDs for causal association. Results sent to the provider can still be reassembled in original call order. Factual order and provider-protocol order are different projections of the same execution. +When batch invocations contend for identical resources, the model's generated array order acts as a deterministic tie-breaker. This sequence establishes prioritization during contention, but does not represent causal data flow. -Cancellation and failure must also obey the async lifecycle. A queued task that is cancelled must never start later. A task that already crossed T1 cannot be treated as nonexistent; the runtime must let it settle and record its outcome. An ordinary tool failure can return as one result alongside its siblings. A T1 or T2 persistence failure, however, should prevent queued work from acquiring dispatch permission. Active work must wind down safely while not-yet-started work freezes. +Parallel execution involves four distinct temporal sequences: -This has the flavor of structured concurrency. A parent batch does not launch a collection of promises and walk away. It owns their lifetimes. Before the next model inference begins, every child task must have completed, been cancelled, or reached an explicit recoverable state. - -Parallel Tool Calling is therefore not fully described by saying "tools run at the same time." The hard part is drawing boundaries among three goals: overlap independent I/O, preserve correctness for shared resources, and give the batch a coherent lifecycle under cancellation, failure, and recovery. +```text +Model Generation Order + ≠ Task Start Order + ≠ Task Completion Order + ≠ Runtime Event Arrival Order +``` -The model expresses concurrent intent. The batch runtime joins it structurally. Resource authorities decide which concurrency reality permits. +Independent tasks start and complete out of order. Raw execution events commit to the log as they occur, linked through Tool Call IDs, while payloads returned to the provider reassemble to match original prompt ordering. Historical logs preserve physical facts, while context projection satisfies model protocol requirements. -## Sandboxes and Serverless: Giving an Agent a Disposable Computer +Aborts and timeouts adhere strictly to structured concurrency rules. Queued tasks that are canceled must not begin execution; tasks that have crossed T1 dispatch cannot simply be abandoned. The runtime must await their convergence and record final dispositions. The batch manager maintains ownership across child tasks, ensuring every operation completes, aborts, or reaches a verifiable state before the next inference step begins. -A tool call ultimately has to run somewhere. +## Sandboxes, Serverless, and Disaggregated State -The model can emit an invocation and write an orchestration program, but it cannot conjure CPU, memory, filesystems, or network connections. JavaScript execution, Python processes, dependency installation, test runs, and browser automation all consume real computing resources. +Tool invocations must ultimately execute on concrete computing infrastructure. -The lightest environment may be a JavaScript V8 isolate. It starts quickly and provides a narrow boundary suitable for short Code Mode control flow. Data analysis and large library ecosystems may call for a Python runtime. Tools that need a complete filesystem, system commands, compilers, and background processes naturally lead to containers or even microVMs. +Models generate action plans and programs coordinate control flow, but operating system processes, memory spaces, and network interfaces require physical or virtual resources. Execution targets span a broad spectrum: lightweight JavaScript V8 isolates, Python container environments with data science toolchains, and full MicroVMs with dedicated kernels and hardware virtualization. ```text -LLM emits intent +LLM Generates Intent │ ▼ Agent Runtime - │ Select environment and capabilities + │ Select execution environment and capabilities ▼ ┌──────────┬──────────────┬─────────────┐ │ V8 │ Python │ MicroVM │ -│ Orchestr.│ Data/scripts │ Full OS tools│ +│ Program │ Data/Scripts │ Full OS Tool│ └──────────┴──────────────┴─────────────┘ │ ▼ Filesystem / Process / Network / Browser ``` -Heavier is not always better. Starting a VM for every small tool call is wasteful; running untrusted system commands inside the runtime process is unsafe. The runtime should select an execution substrate that is light enough for the task and strong enough for the isolation it requires. +Heavier execution environments carry distinct trade-offs. Booting a full virtual machine for basic string manipulation introduces unnecessary latency, while running untrusted shell scripts directly within the host process creates severe security risks. Runtimes must dynamically match tool requirements against lightweight, securely isolated substrates. -A sandbox is therefore more than a wall around dangerous model-generated code. It is the resource boundary, fault boundary, and lifecycle boundary of one agent execution. +Sandboxes define more than security perimeters; they establish resource, failure, and lifecycle boundaries for agent execution. -The runtime can limit CPU, memory, disk, concurrency, and elapsed time. It can decide whether the sandbox has network access, which paths it can see, and which external services it can invoke. If code loops forever, exhausts memory, or crashes a process, the environment can be terminated without spreading the failure across the agent system. +Runtimes enforce strict quotas at the sandbox layer: limiting CPU, memory, storage, concurrency, and execution time; restricting network domains; and terminating environments upon memory exhaustion or process failures to prevent systemic instability. -More importantly, the sandbox separates an agent from the machine currently running it. +Sandboxing also enables decoupling agent state from host infrastructure. -Traditional desktop software often assumes that a process and its local state persist. Agent execution environments should be assumed to disappear at any moment. A V8 cell ends when its code finishes. An idle container can be reclaimed. A microVM can vanish because of timeout, migration, preemption, or host failure. Recovery becomes nearly impossible if the agent's authoritative state lives inside those temporary environments. +Conventional applications assume long-running local processes. In contrast, modern agent environments treat compute substrates as disposable: V8 cells terminate upon completion, containers recycle after idle timeouts, and MicroVMs drain during host migrations. Binding persistent agent state to ephemeral compute nodes undermines system reliability. -This is where the append-only log returns. +Append-only logging provides the foundation for this separation. -Conversations, tool calls, results, permission decisions, and recovery facts live in a durable log. Files, media, and oversized results live in external artifact storage. Workspaces can be reconstructed from persistent volumes, snapshots, or objects. The sandbox carries only the computation currently in progress. It can be destroyed and recreated on another machine. +Conversation traces, tool calls, results, permission records, and recovery events reside in durable logs. Large artifacts and binary outputs persist in object storage, while workspaces mount via copy-on-write snapshots or persistent volumes. Sandboxes act as stateless execution engines, disposable and recreatable across nodes. ```text Durable State Ephemeral Compute @@ -404,115 +393,75 @@ RuntimeEvent Log ─┐ ┌─ V8 Isolate Artifact Storage ─┼─→ Rehydrate ────┼─ Container Workspace Snapshot┘ └─ MicroVM - Preserves what happened Executes what happens next + Preserves "What happened" Executes "Next action" ``` -Serverless and agents fit naturally because agent workloads are bursty. While the model reasons, the sandbox may have nothing to do. When a call arrives, it may suddenly need computation. Some tasks last milliseconds; others compile a large project or wait on long-running I/O. An ideal compute layer appears on demand, scales to zero while idle, and assigns different resource shapes according to tool requirements. - -Agent serverless cannot, however, be a simple copy of traditional Function as a Service. A conventional function receives input, computes, and returns. An agent also maintains a workspace, starts background processes, waits for approvals, calls external tools, and resumes hours later. It does not need an immortal process. It needs a protocol that reconnects durable state to ephemeral compute. +Agent workloads are bursty: sandboxes sit idle during model reasoning, followed by intense spikes during compilation or batch processing. Certain tools run in milliseconds, while others block for hours on external feedback. Modern infrastructure must support rapid scaling to zero during idle periods, provisioning specialized capacity only when invoked. -When a sandbox disappears, the runtime should not try to restore its heap, promises, or stack frames. It should use the log to determine which tools ran and which outcomes committed, mount the necessary workspace and artifacts into a fresh environment, and start the next execution from a trustworthy historical prefix. +This differs from traditional Function-as-a-Service (FaaS) abstractions. Standard serverless functions assume brief, stateless execution; agents maintain stateful workspaces, spawn long-running background tasks, pause for human review, and resume hours later. -Serverless does not mean the agent has no state. It means the state belongs to no individual computer. +Agent Serverless decouples session state entirely from compute lifecycle. -This architecture also changes permissions. A sandbox need not hold permanent credentials for every cloud service or receive ambient network access. It gets only the capabilities needed by the current task. Secrets, approvals, and resource authorities remain outside. Code may request an action, but the external runtime still decides whether that action may cross the boundary. +When a sandbox terminates, the runtime does not attempt to reconstruct volatile process heaps or unresolved sockets. Instead, it inspects durable logs, rehydrates workspace snapshots into a newly provisioned sandbox, and resumes execution from a verified factual history. -Once execution is cheap enough, agents can scale in a new way. A short orchestration rents V8, data processing rents a Python container, and a complete software build rents a microVM. Child agents can run concurrently in isolated workspaces, then release every resource when they finish. +Security architectures also benefit from this model. Disposable sandboxes do not hold static administrative credentials or broad network access. They receive short-lived, minimum-privilege capabilities per task. Credential storage and policy authorization remain within the trusted runtime outside the container. If a sandbox environment is compromised, its authorization scope expires immediately upon termination. -Take this one step further: put every session in cheap S3-compatible object storage, and make the compute layer entirely out of inexpensive, short-lived, replaceable execution resources. +Affordable compute substrates enable fleet-scale agent deployments. A single session can provision lightweight V8 isolates for script orchestration, Python containers for data analysis, and MicroVMs for software compilation, terminating resources as each task completes. -This is complete disaggregation of storage and compute. +The natural extension of this architecture is complete state disaggregation: persisting session state in cost-effective object storage (such as S3-compatible systems) while compute executes across on-demand, stateless workers. -A session no longer corresponds to an object in one process or a directory on one machine. It becomes a set of durable objects: append-only event segments, artifacts, workspace snapshots, compaction projections, and a manifest pointing to the current trustworthy prefix. After a conversation turn, no runtime needs to remain resident in memory. The session can rest in object storage while consuming almost no compute. +Sessions cease to correlate with static processes or host directories. They exist as collections of durable objects: append-only event segments, binary artifacts, workspace snapshots, compaction projections, and manifest metadata pointing to current commit boundaries. Between interactions, sessions persist passively in object storage at minimal cost. ```text - Cheap Durable Storage + Cost-Effective Object Storage (S3) Session A ── Events / Artifacts / Workspace Snapshots ─┐ Session B ── Events / Artifacts / Workspace Snapshots ─┼── S3 Session C ── Events / Artifacts / Workspace Snapshots ─┘ │ - Event / User / Schedule │ + External Event / User / Schedule │ │ ▼ │ - Rehydrate a Session ◀─────────┘ + Rehydrate Session Context ◀───┘ │ ┌───────────┼───────────┐ ▼ ▼ ▼ V8 Python MicroVM │ │ │ - └───────────┴───────────┘ + └───────────┼───────────┘ │ Append Facts │ └──────────────→ S3 ``` -A "long-running agent" no longer requires a long-running machine. - -It can remain dormant most of the time. When a user message, timer, webhook, or background completion arrives, the scheduler reads the session manifest, loads the required log prefix and workspace snapshot, and assigns a new sandbox. When the task finishes, new facts and artifacts return to object storage and the compute environment is released. - -The agent is not continuously alive. It is continuously awakenable. - -S3 is no longer merely backup media in this design. It can hold the factual state of the agent. Memory, SQLite, local SSD, vector indexes, and provider context on hot machines become caches or projections. They can accelerate reads, but they should not determine whether the session still exists. Lose the machine and rebuild the cache. Preserve the trustworthy history in object storage and the agent survives. +Long-running agents no longer require persistent, dedicated servers. -Putting a session in S3 does not mean repeatedly appending in place to one giant object. A natural design writes immutable event segments and artifacts, then advances a small manifest or head pointer to the latest committed prefix. Leases, compare-and-swap, idempotency keys, and in-flight operation state still require a strongly consistent control plane. The large bodies of history, tool output, filesystem snapshots, and media can live in cheap object storage. +They remain dormant most of the time. When messages arrive, webhooks trigger, or schedules elapse, the control plane reads the session manifest, mounts the required log prefix and workspace snapshot, and provisions an appropriate sandbox. Once execution settles, new facts sync back to storage, and compute resources release immediately. -The system separates naturally into two layers: +The architecture organizes into two coordinated tiers: -- The data plane stores immutable, voluminous, rarely modified session state. -- The control plane stores small, strongly consistent heads, leases, admissions, and operation state. +- **Data Plane:** Object storage managing immutable, high-volume event logs and filesystem snapshots. +- **Control Plane:** Low-latency storage tracking authoritative head pointers, resource leases, quotas, and pending operations. -This resembles storage-compute disaggregation in modern databases. Object storage provides vast, inexpensive, durable capacity. Compute nodes appear only when a query or mutation needs them. Here the object being queried and continued is not a table. It is an agent's history. - -From this perspective, model context is itself a query. The runtime reads durable session state from S3 and applies compaction, tool-result pruning, visibility, and provider-compatibility projections to construct what the model should see now. The model's next output does not rewrite the past; it appends new facts. +This mirrors disaggregated database architectures. Object storage provides durable, cost-effective persistence, while compute resources provision strictly on demand. Context assembly operates like a materialized view query: the runtime reads durable state, applies compaction and result pruning, and projects bounded context for model inference. ```text Session on S3 │ ├── Projection ──→ Model Context ──→ LLM │ │ - ├── Rehydrate ───→ Sandbox ───────→ Tool Call + ├── Rehydrate ───→ Sandbox ──────────→ Tool Call │ │ └──────────────── Append New Facts ◀───┘ ``` -Both the LLM and the sandbox now become compute resources. - -Model capacity can be rented according to task difficulty: an inexpensive model for routine work, a stronger model for difficult judgment. Execution capacity can likewise match capabilities: an isolate for orchestration, a container for scripts, a microVM for operating-system tools. A session belongs to no particular model and no particular sandbox. - -This creates a different agent economy. Cost no longer depends primarily on how many sessions exist, but on how many are thinking and acting now. Ten million dormant sessions can be ten million object prefixes. Only the small active fraction consumes model tokens, CPU, and memory. - -The cheapest agent is not an agent running on a smaller server. It is an agent with no server at all while asleep. - -Disaggregation also makes preemptible compute practical. Workers can come from inexpensive instances, shared pools, or capacity that may disappear at any time. Previously, killing a machine running an agent meant losing the conversation. Once state is externalized, losing a worker means losing only a temporary execution. The runtime classifies real-world effects through T1 and T2, then continues the session on another compute resource. - -Branching and forking also become cheap. Append-only history and copy-on-write workspace snapshots let several agents share one historical prefix and grow independent suffixes. Spawning a child agent need not copy the entire session. It records the prefix and snapshot from which it starts. Unchanged artifacts remain shared; only new facts consume new storage. - -Even model upgrades need not migrate sessions. History retains provider-neutral runtime facts, and a new model receives a projection suited to its protocol. One durable session can run on one model today and resume on another months later. The identity of an agent comes from the history it has lived through, not from the model weights currently loading it. - -The security boundary becomes cleaner as well. S3 holds encrypted, auditable long-term state. A sandbox receives minimal capabilities only for its short lifetime. Secrets need not enter workspace snapshots, and permanent cloud credentials need not enter microVMs. When code needs an external resource, it asks an outside authority for one constrained operation. If the compute environment is compromised, its authority expires with the environment. - -Cheap compute does not automatically create correctness. An inexpensive microVM cannot make a duplicate payment safe. A restartable container cannot determine whether an email was sent before a crash. The more disposable workers become, the more the system depends on reliable tool calls, idempotent operations, resource authorities, and append-only logs to prove what happened in the real world. - -This is more than "run agents on serverless." We are assembling a new kind of computer for agents: - -```text -S3 is its inexpensive durable disk -Append-Only Log is its recoverable state -LLM is its rented reasoning unit -Sandbox / MicroVM is its rented body -Agent Runtime is the operating system connecting them -``` - -Discussion of agents today often centers on models. But models produce judgment and intent. Applying that intent to reality safely, reliably, and economically requires vast amounts of on-demand execution, together with session state that outlives every execution environment. - -The core infrastructure of future agents will include extremely cheap storage and extremely cheap compute. Storage lets hundreds of millions of sessions persist. Compute lets any one of them wake immediately when needed. The bridge is not the memory of one machine, but a history that can be replayed, verified, and extended. +Both models and sandboxes function as interchangeable compute utilities. -Look back across the tool stack. Deferred tools decide which capabilities deserve the model's attention. Tool calls turn language into action. Reliable execution makes action a trustworthy fact. Code Mode expresses structure among actions. The async runtime overlaps waiting. Sandboxes and serverless provide the CPU, memory, and isolation that all of those layers consume. +Model selection scales with reasoning complexity, and sandbox sizing matches workload requirements. A session is never coupled to a single model provider or execution substrate. -Ultimately, an agent is not a long-lived process that happens to save some state. +Cost efficiency stems from ensuring dormant sessions consume zero active compute. -**An agent is durable state that temporarily rents a model and a computer whenever it needs to think and act.** +Disaggregated storage also facilitates spot-instance execution and instantaneous branching. Using append-only logs and copy-on-write snapshots, subagents fork from parent histories without copying storage, writing only delta records going forward. -It sleeps in cheap S3. When an event arrives, the log tells it who it used to be, the sandbox defines what it may do now, and inexpensive compute lets it move forward. +The future of agent runtime engineering is clear: establishing immutable logs as the authoritative source of truth, relying on disposable sandboxes for safe execution, decoupling resource governance from task scheduling, and dynamically managing schema projections to preserve model focus. Extending language models into external systems requires robust runtime engineering to ensure safety and reliability. diff --git a/docs/blogs/beyond-function-calling.zh-CN.md b/docs/blogs/beyond-function-calling.zh-CN.md index ab5bda0a0a..3f82d07e0e 100644 --- a/docs/blogs/beyond-function-calling.zh-CN.md +++ b/docs/blogs/beyond-function-calling.zh-CN.md @@ -23,28 +23,28 @@ ## Deferred Tool：没有被调用的 Tool 也有成本 -在普通程序里，一个从未被调用的函数几乎不会产生运行时成本。它可以存在于代码库或动态链接库中，只要执行路径没有经过它，就不会消耗 CPU，也不会占用调用栈。 +在普通程序中，未被调用的函数几乎不产生运行时开销。它可以静态存放于代码库或动态链接库中，只要执行路径不经过，就不会消耗 CPU，也不会占用调用栈。 -Agent 里的 Tool 不是这样。 +Agent 中的 Tool 则完全不同。 -模型想要调用一个 Tool，首先必须知道这个 Tool 的名称、用途以及参数格式。因此，Runtime 会把 Tool Definition 连同 System Prompt 和对话历史一起发送给模型。一个 Tool 即使从未被调用，它的 Description 和 JSON Schema 也已经进入了每一次推理。 +大语言模型若要调用某个工具，必须预先获知该工具的名称、用途描述以及结构化的参数定义。因此，Runtime 需要将 Tool Definition 连同 System Prompt 与会话历史一同打包发送给模型。一个工具即使在整个会话中从未被触发，其描述文本和 JSON Schema 也会持续消耗每一次推理的输入 Token。 -这意味着 Tool 在执行之前就开始产生成本。Schema 会占用上下文窗口，会参与模型对下一步动作的判断，也会改变可供 Provider 缓存的请求前缀。Tool 越多，模型能够采取的动作越多，但留给任务本身的上下文越少，选择动作时需要面对的干扰也越大。 +工具的开销从执行前就已经产生。Schema 挤占宝贵的上下文窗口，分散模型对核心任务的注意力，并可能改变 Provider 侧 KV Cache 的命中前缀。随着可用工具数量的膨胀，模型的行动空间虽然得以拓展，但留给任务本身的有效上下文被不断压缩，动作决策面临的噪声干扰也显著增加。 -当 Agent 只有 `Read`、`Write`、`Bash` 等少数工具时，这个问题并不明显。但随着 Browser、Computer Use、子 Agent、外部服务和 MCP Connector 不断加入，把所有 Tool Schema 常驻在每一次模型请求里，就不再是一种可以持续扩展的方式。 +当 Agent 仅配置 `Read`、`Write`、`Bash` 等基础工具时，这种开销尚可接受。然而，一旦接入浏览器自动化、系统级 GUI 操作、子 Agent 委派、外部企业服务以及大量 MCP Connector，在每一次请求中常驻全量 Tool Schema 将迅速突破架构的可扩展性上限。 -Maka 的 Deferred Tool 从这里出发。这里的 Deferred 不是延迟执行，也不是让 Tool 在后台异步完成，而是延迟向模型暴露完整的 Tool Schema。 +Maka 引入 Deferred Tool 机制来应对这一瓶颈。Deferred Tool 并不改变工具的执行时机，核心在于按需控制 Tool Schema 暴露给模型的时间点。 -Runtime 仍然持有当前 Run 可以使用的全部 Tool Binding，但模型在第一次推理时只看到一组高频基础工具，以及一个轻量的 `tool_search`。其余 Tool 只以分组和名称出现在 Search Inventory 中，不携带完整的 Description 和参数 Schema。 +Runtime 始终完整持有当前 Run 所注册的全部 Tool Binding，但在初次调用模型时，仅向其暴露高频基础工具集以及一个专用的轻量级 `tool_search` 工具。其余扩展工具仅以能力分组和名称的形式登记在检索清单（Search Inventory）中，不携带详细的描述文本与参数 Schema。 ```text Bound Tool Registry │ - ├── Direct Tools ───────────────→ 当前请求中的完整 Schema + ├──── Direct Tools ───────────────→ 当前请求中的完整 Schema │ - └── Deferred Tools + └──── Deferred Tools │ - └── 轻量 Search Inventory + └──── 轻量 Search Inventory │ tool_search │ @@ -55,38 +55,38 @@ Bound Tool Registry 注入匹配 Tool 的 Schema ``` -`tool_search` 搜索的不是文件、网页或业务数据，而是 Runtime 已经拥有的能力。Maka 在本地根据 Tool 的名称、Description 和所属能力分组完成匹配，再选择数量与 Schema 体积都受限制的一组结果。返回给模型的只是被激活的 Tool 名称，完整 Schema 不会重复塞进 Tool Result，而是在下一次 Provider Request 中通过正常的 Tool Projection 出现。 +`tool_search` 面向 Runtime 已注册的内部能力执行本地检索。Maka 依据工具名称、描述语义及所属分类完成本地过滤，提取体积受限且条目数量有界的一组候选结果。返回给模型的内容仅包含被激活工具的名称标识，完整的 Schema 规范不会直接倾倒进 Tool Result，而是在下一次向 Provider 发起请求时，以标准的 Tool Projection 格式透明注入。 -这套机制把过去容易混在一起的几个概念拆开了： +在 Maka 中，工具状态严格划分为三个层级： -- **Bound**：Runtime 持有可执行的 Tool Binding，它定义了本次运行的能力上限。 -- **Discoverable**：Tool 出现在轻量 Inventory 中，模型知道某类能力存在。 -- **Visible**：完整 Tool Schema 已经进入当前 Provider Request，模型可以据此生成调用。 +- **Bound**：Runtime 已完成工具绑定的可执行实现，确立当前 Run 的能力上限。 +- **Discoverable**：工具登记于轻量级 Inventory 中，模型感知到该能力的存在。 +- **Visible**：完整 Tool Schema 已注入当前 Provider 请求上下文，模型具备构造合法调用的必要信息。 -搜索不会绑定新的 Tool，也不能突破当前 Run 已有的 Binding Ceiling。它只是改变下一次模型调用看到的 Tool Projection。 +能力检索不会动态引入未经注册的实现，无法突破当前 Run 固有的 Binding 上限，它仅用于动态调整下一次模型交互所见的 Tool Projection。 -“下一次”是这里很重要的边界。Provider Step 开始后，这次请求包含哪些 Tool Schema 就已经确定。假如模型在同一个响应里同时生成下面两个调用： +这里的“下一次”构成了严格的时序边界。Provider Step 一旦建立，本次请求携带的 Tool Schema 集合即行固化。若模型在单次回复中同时生成如下调用序列： ```text tool_search("browser click") browser_click(...) ``` -第二个调用仍然会被 Maka 拒绝。`tool_search` 的结果只能影响后续请求，不能反过来改写一份已经发送给 Provider 的 Schema 集合。直到下一个 Step，`browser_click` 的完整定义才会进入模型上下文，模型也才能基于自己真正见过的接口生成参数。 +Maka 会明确拒绝执行第二个调用。`tool_search` 的生效结果只能投影至后续请求，绝不能反向修改已经交由 Provider 解析的 Schema 集合。只有推进至下一个推理步骤，`browser_click` 的完整规范才会进入上下文，模型方可基于实际观测到的接口定义生成准确参数。 -Deferred Tool 的激活状态只保留在当前 Turn 中。同一个 Turn 内，搜索得到的工具会单调累积，Provider 重试也会继承这份工作集；Turn 结束后，激活集合随之释放。下一轮对话重新从稳定的基础工具集开始，不会因为此前偶然使用过某项能力，就永久背负它的 Schema 成本。 +Deferred Tool 的激活范围被严格限定在当前 Turn 之内。在同一次用户交互轮次中，搜索激活的工具单调累积，Provider 侧的重试请求亦完整继承该工作集；一旦 Turn 执行结束，激活集合即刻释放。下一轮交互重新从基线工具集启动，避免偶发性调用的 Schema 成本永久滞留在后续的推理历史中。 -Tool 的可见性也不等于执行授权。已经进入模型上下文的 Tool，真正调用时仍然需要经过权限判断、参数校验和 Runtime 的执行边界。`tool_search` 管理的是模型的认知范围，不是用户授予的权限范围。 +工具的可见性亦不等同于执行授权。即使 Schema 已完全可见，真实的调用请求仍须通过参数类型校验、并发配额审计以及环境权限判定。`tool_search` 仅管理模型的认知视野，系统安全边界始终由 Runtime 统一把关。 -因此，Deferred Tool 解决的并不是“工具怎样执行”，而是“哪些工具值得进入模型的下一次思考”。Runtime 保存完整的能力空间，模型看到的则是当前任务真正需要的工作集。 +Deferred Tool 专注于约束进入模型注意力窗口的能力集合。Runtime 维护全局能力空间，模型则按需获取当前任务所需的紧凑子集。 -## Tool Call：让 LLM 长出手脚 +## 意图与边界：从概率生成到系统副作用 -Tool Schema 进入上下文之后，模型只是知道自己有哪些动作可以选择。直到它生成一个 Tool Call，一切仍然只是 Token。 +Tool Schema 进入上下文后，模型仅获知了可选操作的规范定义。在真正发出 Tool Call 之前，所有交互仍处于纯文本 Token 的范畴。 -LLM 本身不会读取文件，不会启动进程，也不会点击屏幕。它接收一段输入，再预测一段输出。即使模型回答“我已经修改了文件”，这句话本身也不会在磁盘上产生任何变化。语言描述的是世界，不能直接改变世界。 +大语言模型本身不具备直接操作宿主环境的能力。它接受上下文输入，预测下一个 Token；即便输出“已成功修改文件”，磁盘上的实际数据也不会发生任何改变。文本表述与客观系统状态之间存在物理隔离。 -Tool Call 在两者之间建立了一条通道。模型不再只生成自然语言，而是按照 Tool Schema 输出一份结构化的动作意图，其中包含要调用的 Tool、传入的参数，以及用于关联结果的 Call ID。Runtime 接住这份意图，在真实环境中执行对应操作，再把执行结果送回模型。 +Tool Call 建立了跨越这一隔离的控制通道。模型停止输出自然语言，转而依据 Schema 生成结构化的动作意图：包含目标工具名称、调用入参以及用于因果追踪的 Call ID。Runtime 拦截该结构化意图，在受控环境中代为触发实际系统的操作，并将执行得到的客观观测回传给模型。 ```text LLM @@ -97,8 +97,8 @@ Runtime │ ├── 查找 Tool Binding ├── 校验参数与执行边界 - ├── 请求必要的权限 - ├── 调用真实实现 + ├── 请求必要的执行权限 + ├── 触发真实环境操作 ▼ Filesystem / Process / Browser / Network / Human │ @@ -107,133 +107,127 @@ Filesystem / Process / Browser / Network / Human LLM 的下一次推理 ``` -从这个闭环开始，模型才真正成为 Agent。读取文件让它获得对代码库的观察，执行命令让它得到编译器和测试系统的反馈，修改文件让它能够改变工作区，浏览器和网络工具把它连接到本地进程之外的环境，向用户提问则让它能够在信息不足时暂停并等待新的事实。 +通过这一反馈闭环，模型得以介入外部系统。读取文件获取代码仓的客观状态，执行命令获取编译器与测试套件的即时反馈，修改文件改变工作区结构，网络调用接入外部服务，人机交互工具则在关键决策点获取外部确认。 -如果把 Tool 看作 Agent 的手脚，那么 Tool Result 就是感觉反馈。只有动作没有反馈，模型无法判断调用是否成功，也无法知道现实世界是否与自己的预测一致。一个完整的 Agent Step 因此不是“模型想了一次”，而是由意图、执行和观察共同组成： +Tool Result 构成模型感知外部环境的观测凭据。缺失观测反馈，模型无法评估操作的实际效果，也无法修正认知偏差。一个完整的 Agent 执行步由意图生成、系统执行与结果观测三部分闭环构成： ```text Reason → Act → Observe → Reason ``` -这个循环看起来像普通的函数调用，但它们之间存在一个根本区别。程序调用内部函数时，调用者和被调用者通常共享同一个确定性的执行环境；模型生成 Tool Call 时，它只是在根据概率分布提出下一步动作。参数可能不完整，目标可能已经变化，对环境的理解也可能是错的。 +这一循环与普通函数调用存在本质区别。常规程序中的调用方与被调方通常运行在确定性受控的运行时中；而 LLM 生成的 Tool Call 本质上是基于概率分布给出的动作建议。参数可能残缺，环境前提可能已经失效，模型对当前系统状态的假设亦可能存在偏差。 -因此，更准确地说，并不是 LLM 自己长出了手脚，而是 Runtime 把一组受控的手脚借给了它。 +LLM 本身不具备环境执行能力；所有的外部交互都由 Runtime 经过校验和鉴权后代为发起。 -在 Maka 中，模型生成的调用不能直接越过 Runtime 接触外部世界。Runtime 会先确认 Tool 确实存在于当前 Binding 中，并检查它是否已经对当前 Step 可见；参数必须符合 Tool Schema，调用还要经过并发限制、权限策略和执行边界。只有这些条件都成立，Tool 的真实实现才会运行。 +在 Maka 中，模型产出的调用请求必须经由 Runtime 屏障的严格审查：核实工具绑定有效性、确认当前 Step 可见性、比对参数 Schema、执行并发与权限策略。唯有全部条件满足，真实的底层操作才被允许触发。 -这条边界区分了模型的意图与系统的授权。模型可以请求执行某个动作，但不能仅凭生成了一个合法 Tool Call，就为自己创造能力或取得权限。Tool Schema 告诉模型怎样表达请求，Tool Binding 决定 Runtime 是否拥有这种能力，Permission 则决定这一次具体请求能否执行。 +这条边界清晰分离了模型意图与系统授权。模型拥有提议权，但生成合法的调用报文并不能凭空赋予自身系统权限。Tool Schema 规范了意图的表达格式，Tool Binding 定义了 Runtime 的实现能力，Permission 机制则裁定特定调用的合法性。 -执行结束后，Runtime 会把结果转换成与 Provider 无关的 Tool Result，再通过 Call ID 与原始调用配对。在 Maka 的 `RuntimeEvent Log` 中，这两端分别成为 `function_call` 和 `function_response`。这样，模型提出过什么动作、Runtime 实际返回了什么结果，都会成为可以重放和审计的运行事实。 +执行完成后，Runtime 将环境返回的原始负载规范化为中立的 Tool Result，借助稳定的 Call ID 与原调用精确配对。在 Maka 的 `RuntimeEvent Log` 中，两者分别沉淀为不可变的 `function_call` 与 `function_response` 事件，作为审计追踪与崩溃重放的法定事实。 -Call ID 在这里不只是消息格式中的一个字段。一个 Turn 可能同时发起多个 Tool Call，执行完成顺序也未必与发起顺序一致。Runtime 必须依靠稳定的身份关联，才能把每份 Result 送回正确的 Call，并在恢复历史时重新构造同一组因果关系。 +Call ID 不仅是消息载荷中的关联字段。在单轮多工具并发调用的场景下，各个调用的实际完成顺序受 I/O 延迟影响可能完全打乱。Runtime 必须依托确定性的标识映射，将各路结果分发至正确的因果链路，并在后续重放时复原精确的逻辑拓扑。 -Tool Call 由此完成了一次关键转换：模型输出的不再只是供人阅读的语言，而是可能读取隐私、消耗资源、启动进程或者修改数据的操作请求。Deferred Tool 决定哪些能力进入模型的思考范围，Tool Call 则让其中一个选择越过语言边界，成为对现实世界的一次尝试。 +由此，Tool Call 实现了质的跨越：模型输出从面向人类消费的语言描述，转变为可能产生不可逆外部副作用的系统调用。这要求 Runtime 必须建立严谨的工程机制，对每一次真实动作的执行后果负责。 -从这一刻开始，Agent 系统面对的问题也发生了变化。一次生成失败，最多得到一段不理想的文本；一次 Tool Call 失败，却可能发生在现实效果已经产生、结果尚未返回的时候。模型有了手脚之后，Runtime 就必须开始对这些动作的后果负责。 +## 可靠执行：基于已提交历史的崩溃恢复 -## Reliable Tool Call：Resume 重放历史，而不是重做动作 +系统副作用的引入，不可避免地将真实世界的不确定性带入了 Agent 运行时。 -Tool Call 把模型连接到现实世界，也把现实世界的不确定性带进了 Agent Runtime。 +设想模型调用 `Edit` 工具将配置端口从 `3000` 修改为 `4000`。底层磁盘写入刚刚完成，宿主机器遭遇突发断电。进程重启后，Runtime 观测到的仅是一条缺失对应结果的悬空调用，但这绝不意味着文件未曾被篡改。 -假设模型调用 `Edit`，要求把配置文件中的端口从 `3000` 改成 `4000`。文件刚刚写完，Maka 的进程恰好崩溃。重启之后，Runtime 只能看到这次 Tool Call 没有返回结果，但这并不能说明文件没有被修改。 +缺少 Tool Result 对应着多种互斥的现实状态：操作尚未派发、操作仍在执行、磁盘已变更但元数据落盘失败、或者状态在变更后已被第三方进程二次覆写。若在恢复时盲目重跑该调用，极易引发重复写盘、重复扣款或数据污染等灾难性后果。 -缺少 Tool Result 可能对应完全不同的现实：调用尚未开始，工具正在执行，副作用已经完成但结果没有落盘，或者外部状态在执行后又被其他进程改变。如果 Resume 简单地把这次调用重新执行一遍，就可能制造重复写入、重复发送、重复创建甚至重复付款。 +与纯文本生成不同，未决的外部系统动作绝不能因为未收到确认信号就被假设为“从未发生”。 -这也是 Tool Call 和普通文本生成之间最重要的差异。文本没有返回，可以重新生成；一个已经越过进程边界的现实动作，却不能因为 Runtime 没看到结果就假定它没有发生。 - -Maka 用两个持久化边界夹住 Tool 的真实执行： +Maka 采用轻量级两阶段持久化边界保护所有的外部工具操作： ```text Model 生成 function_call │ ▼ -参数、可用性、权限与执行边界检查 +参数、可见性、权限与执行边界检查 │ ▼ T1：提交 Tool Dispatch │ ▼ -执行现实世界中的操作 +执行真实系统操作 │ ▼ T2：提交 function_response │ ▼ -把 Tool Result 交给模型 +将 Tool Result 移交模型 ``` -T1 表示 Runtime 已经完成所有执行前检查，并正式跨过了派发边界。从这一刻开始，系统不能再安全地声称 Tool 一定没有运行。T1 必须先提交，Tool 的真实实现才会被调用；如果 T1 提交失败，副作用就不允许开始。 +T1 标志着 Runtime 已通过全部前置审查，正式越过调用派发点。自此，系统不再允许做出“该操作绝对未执行”的乐观假设。T1 必须在真实逻辑触发前完成持久化；若 T1 提交失败，外部副作用严禁启动。 -T2 表示 Tool 的结果已经成为持久化的 `function_response`。只有 T2 提交成功，这份结果才可以进入下一次模型推理。即使 Tool 已经返回成功，如果 T2 没有落盘，Runtime 也不能把一个无法在重启后重建的结果临时交给模型。 +T2 标志着工具执行结果已转化为持久化的 `function_response` 事件。唯有 T2 落盘确认，该结果才获准进入下一次模型推理。即便外部操作已成功返回，若 T2 写入失败，Runtime 亦不得将无法持久复现的状态交给模型使用。 -Maka 没有尝试用一个数据库事务包住整个 Tool Call。文件操作、Shell 命令、浏览器动作和网络请求可能持续几秒甚至几小时，SQLite 不可能与这些外部系统共同完成一个真正的分布式事务。Maka 能做的是用两个很短的事务明确副作用窗口： +Maka 并不试图将异构的外部操作纳入分布式数据库事务。文件 I/O、Shell 执行、浏览器渲染与远程网络调用的耗时跨度极大，强行追求全局 ACID 既不切实际亦不可行。Maka 选择以两次极短的局部数据库事务，清晰标定外部副作用的发生区间： ```text Committed T1 → External Side Effect → Committed T2 ``` -这样一来，进程无论在哪里崩溃，重启后的 Runtime 都可以根据 Append-Only Log 中已经提交的前缀做出确定判断： +当进程发生异常崩溃，恢复逻辑依据 Append-Only Log 中已固化的事件前缀做出精确的状态判定： -| 日志事实 | Runtime 能够得出的结论 | +| 日志状态 | 恢复判决 | |---|---| -| 没有跨过 T1 | Tool 确定没有被派发 | -| T1 和 T2 都存在 | Tool 已经完成，直接使用既有 Result，不能重复执行 | -| T1 存在但 T2 缺失 | 副作用状态未知，需要 Reconcile 或 Park | -| Call、Dispatch、Response 的身份或顺序冲突 | Ledger 损坏，Fail Closed | +| 未记录 T1 | 工具确定未派发，可安全忽略或重新评估 | +| T1 与 T2 均完整存在 | 工具已完成闭环，直接复用既有结果，严禁重复执行 | +| 仅存在 T1，缺失 T2 | 副作用状态处于未决区间，强制进入 Reconcile 或 Park | +| 标识乱序或因果链断裂 | 账本完整性受损，立即 Fail Closed | -其中最危险的是 T1 与 T2 之间。系统只知道 Tool 已经获得执行资格，却不知道现实效果是否完成。Maka 不会让模型根据上下文猜测，也不会把“没有 Result”自动解释成“没有执行”。Tool Binding 可以声明自己的恢复语义，例如操作是否天然幂等、能否重新观察结果，或者永远不能自动重试；缺少足够证据时，Runtime 会把这次操作 Park，等待更可靠的观察或人工处理。 +处于 T1 与 T2 之间的悬空操作具有最高风险。此时系统确知操作已获得派发许可，但无法证实外部效果是否生效。Maka 严禁模型依靠幻觉猜测执行状态，亦不将缺失结果默认视作失败。Tool Binding 支持声明专属的恢复策略（例如：操作具备幂等性、支持外部状态探查、或是严格禁止自动重试）。若无法取得明确证据，Runtime 将该操作置于挂起状态（Park），交由外部探针或人工接入处理。 -这种恢复同样遵循 Append-Only。Runtime 不会回头修改原来的 `function_call` 或假装补上一段过去没有发生的历史。正常的 Dispatch、Outcome，以及后续可能产生的 Reconcile 和 Recovery Decision，都会作为新的事实继续追加到 Log 尾部。旧事实保持不变，新的事实负责解释旧操作最终收敛到了什么状态。 +恢复机制严格遵循追加写原则。Runtime 不会就地篡改早先的 `function_call`，亦不凭空伪造执行记录。正常的 Dispatch、Outcome，以及后续的 Reconcile 和人工判决，均作为全新的增量事实追加至日志尾端。既有事实保持不可变，新增事实明确旧操作的最终收敛结局。 -当所有 Tool Call 都已经被判定为 Completed 或 Definitely Not Dispatched，Resume 才具备安全重放的基础。 +唯有当全部并发调用均被确定性收敛为已完成（Completed）或确定未派发（Definitely Not Dispatched）时，Resume 流程才获准基于已验证的历史构建新轮次。 -这里的“重放”很容易被误解。Maka 不会重新执行历史中的 Tool，也不会复活崩溃前的 Promise、JavaScript 调用栈、网络连接或宿主进程。它重放的是模型当时已经看到的合法历史：User Message、模型输出、成对的 `function_call` 与 `function_response`，以及其他可以进入 Provider Context 的确定事实。 +这里的“重放（Replay）”具有严格的系统边界。Maka 从不重新触发历史工具的执行代码，亦不试图恢复崩溃前进程的瞬时内存指针、Promise 状态机或未结 Socket。它所重放的，纯粹是已经过校验的因果事实序列：用户输入、思考过程、配对完整的 `function_call` 与 `function_response`。 ```text -Immutable RuntimeEvent Prefix +不可变 RuntimeEvent 日志前缀 │ - ├── 解析并收敛 Tool Operation - ├── 丢弃流式 Partial + ├── 解析并收敛 Tool 状态 + ├── 剥离流式临时切片 ├── 保留成对的 Call / Response - ├── 裁掉无法构成合法历史的中断尾部 + ├── 截断未闭合的悬空后缀 └── 校验 High-Water 与 Digest │ ▼ - Verified Provider Replay + 已验证的 Provider Replay Plan │ ▼ - New Run / Invocation / Turn + 全新 Run / Invocation 实例 ``` -Append-Only 结构让这件事变得自然。Resume 不需要猜测旧进程内存中曾经有哪些对象，也不需要从 UI 状态反推出执行进度。它只读取截至某个 High-Water 的不可变事件前缀，验证这段前缀的 Digest，再从中投影出下一次 Provider 调用需要看到的历史。 - -新的执行会获得全新的 Run、Invocation 和 Turn 身份，并记录自己从哪个 Source Run、哪个 Event High-Water 继续。原始 User Message 不会再复制一遍，已经完成的 Tool Call 也不会再次执行。Continuation 继承的是一段经过验证的因果历史，而不是一份准备重新运行的命令列表。 +基于 Append-Only 的事件模型，恢复流程无须依赖易失的内存快照。系统仅读取截止至特定水位线（High-Water Mark）的不可变事件切片，校验其摘要哈希，随即向新实例提供规范的上下文投影。 -在真正调用模型之前，Maka 还会重新检查这段历史赖以成立的外部条件：Workspace 是否仍是同一个 Workspace，历史中使用过的 Tool 是否仍然存在，后台进程和子任务是否已经收敛，以及是否已经有另一个 Continuation 占用了同一恢复边界。任何一个条件无法证明，Resume 都会停在 Park，而不是带着旧结论进入一个已经变化的现实世界。 +恢复实例获得全新的 Run 与 Invocation 标识，明确记录其承接的源 Run 标识与水位线坐标。原有的用户指令不会发生重复拷贝，已结案的工具调用亦不会二次触发。执行链条继承的是经受验证的客观事实，而非一份重新执行的指令清单。 -因此，Maka 的 Resume 并不是“从崩溃的位置继续执行代码”，而是先让每一次现实动作在日志中获得可信的结论，再从一段不可变、可验证的历史创建新的执行。Tool Call Recovery 解决了动作是否已经发生的问题，Append-Only Log 解决了模型应该从哪些事实继续的问题。 +在拉起模型调用前，Maka 会重新验证底层环境的基线约束：确认 Workspace 物理路径未变、所需 Tool Binding 依然注册就绪、关联的后台进程处于收敛状态、且不存在并发的竞争性恢复实例。任何一项前提无法自证，恢复流程立即转入安全挂起（Park），杜绝在不一致的外部环境中继续执行。 -一旦现实动作能够稳定地沉淀为 Log 中的事实，Resume 就不再是对旧进程的抢救，而变成了一个从历史构造新 Runtime 的 Replay 问题。 +Maka 的 Resume 逻辑不试图恢复崩溃进程的内存堆栈，而是先核实未决动作在日志中的最终收敛状态，再基于已验证的不可变历史前缀启动全新的执行轮次。 -## Code Mode：当 Tool Call 变成一段程序 +## Code Mode：程序化编排与调用树折叠 -到这里为止，我们讨论的 Tool Call 都是一次一个的。 +常规的 Tool Call 遵循单步交互模型：模型判定下一步动作，Runtime 触发工具并返回结果，模型基于最新上下文重新推演后续行动。在每一步骤均高度依赖动态语义决策的场景中，这种模式提供了精细的控制粒度。 -模型先判断下一步要调用什么，Runtime 执行 Tool，再把 Result 放回上下文。模型读到结果之后，重新推理，决定是否调用下一个 Tool。对于每一步都需要语义判断的任务，这正是 Agent 应有的工作方式。 +然而，在处理确定性较强的组合逻辑时，这一模式的效率瓶颈十分显著。 -但并不是每一步都值得重新调用一次模型。 - -假设 Agent 需要读取二十个文件，找出包含某个依赖的文件，再分别读取它们的配置，最后只把版本不一致的项目列出来。如果沿用普通 Tool Call，整个过程会变成：模型发起一次读取，看到结果，再发起下一次读取；所有中间结果都进入上下文，循环、筛选和聚合也都靠一次又一次推理来推进。 +以排查依赖冲突为例：Agent 需遍历数十个项目目录，读取 `package.json`，提取特定依赖项的版本号，最终输出存在版本分歧的项目列表。若采用逐步 Tool Call 模式，模型需要连续经历数十轮循环：生成单次读取指令、等待结果灌入上下文、重新解析后再生成下一次读取。全部中间内容不仅大幅消耗 Token 配额，亦导致多轮往返的严重网络延迟。 ```text Reason → Call → Observe → Reason → Call → Observe → ... ``` -这里真正需要模型判断的，也许只有任务开始时的执行计划，以及最后如何解释异常。中间大量工作只是确定性的控制流。让 LLM 逐步扮演 `for` 循环，不仅慢，也会让每一份原始 Tool Result 都成为后续上下文的负担。 +在此类场景中，除初始任务拆解与最终异常评估外，中间环节本质上属于确定性的控制流操作。由大模型反复充当代码解释器去模拟 `for` 循环与字符串过滤，不仅执行迟缓，还会将海量低价值的原始数据永久滞留在推理历史中。 -Code Mode 改变的就是这一层。 +Code Mode 改变了工具的组织形态。 -模型不再为每个动作分别生成一次顶层 Tool Call，而是先生成一小段程序，由这段程序调用多个 Tool。循环、并发、条件分支、字段提取和结果聚合在受限的代码执行环境中完成，模型只需要看到程序最终选择输出的内容。 +模型不再针对每个细粒度动作生成离散的顶层 Tool Call，而是编写一段结构化的程序脚本。循环、并发抓取、条件分支、数据提取与结果聚合，均在受约束的代码沙箱内自主运行。模型最终仅需消费脚本计算后显式输出的高价值结论。 ```text ┌─ Tool A ─┐ @@ -241,13 +235,13 @@ Reason → Program ─┼─ Tool B ─┼→ Filter / Join / Reduce → Observe └─ Tool C ─┘ ``` -OpenAI 的 Codex 把这种执行形态称作 Code Mode。在公开的 Responses API 中，同一类能力被称为 Programmatic Tool Calling：模型生成 JavaScript，在隔离的 V8 Runtime 中通过 `tools.*` 编排可用工具。Claude 也提供 Programmatic Tool Calling，只是让 Claude 在 Code Execution Container 中生成 Python，并通过 `allowed_callers` 指定哪些 Tool 可以从程序内部调用。 +该机制在业内存在不同实现路径：OpenAI 在 Responses API 中通过 Programmatic Tool Calling 提供该能力，模型产出 JavaScript 代码并在隔离的 V8 运行环境中通过 `tools.*` 操纵工具；Anthropic 亦在 Claude 的代码执行容器中引入了类似的机制，由模型生成 Python 脚本并通过权限白名单调度受限工具。 -两种协议的实现细节不同，但表达的是同一个判断：LLM 擅长提出计划和处理语义不确定性，程序更适合执行已经明确的控制流。 +两者的工程细节虽有差异，但遵循着一致的设计逻辑：让概率模型专注于目标规划与语义解析，由底层程序环境承载确定性的控制流编排。 -这不是给模型一台没有边界的机器。Code Mode 中的程序能够触达什么，仍然由 Runtime 提供的 Tool 集合决定。它不能仅凭写下一段网络请求代码就获得网络，也不能因为生成了文件操作代码就绕过文件系统权限。程序只是 Tool 的编排层，不是新的权限来源。 +代码沙箱绝非无边界的特权环境。脚本所能触达的全部能力，依然严格局限于 Runtime 所授予的工具子集。脚本无法因编写了原生系统调用就突破沙箱隔离，亦不能直接绕过既有的权限管控。程序仅扮演工具的编排逻辑层，不构成独立的安全越权来源。 -它也没有取代 Tool Call。恰恰相反，Programmatic Tool Calling 把一个线性的 Tool Call 序列变成了一棵调用树：最外层是模型生成的 Program，下面是程序实际发起的 Tool Call。每个叶子节点最终仍要由 Runtime 校验、授权和执行。 +Code Mode 并没有取代底层的 Tool Call 体系。相反，它将扁平的调用序列重构为层次化的调用树：根节点为模型提交的程序载荷，子节点为脚本运行时实际派发的具体工具调用。每一个叶子节点的操作，仍须完整穿透 Runtime 的校验、鉴权与事务边界。 ```text Program / exec @@ -261,31 +255,27 @@ Program / exec Program Result ``` -这种结构最直接的收益是减少模型往返。原本需要多次采样才能完成的循环或批量查询，可以在一个 Program 中执行。另一个同样重要的收益是减少上下文污染：程序可以先处理几十份原始结果，只把筛选后的几行结论交回模型。Tool Result 没有消失，只是其中不需要模型理解的部分没有进入它的状态空间。 - -因此，Code Mode 与 Deferred Tool 正好解决 Tool Context 的两个不同问题。Deferred Tool 减少的是推理开始前加载的 Tool Definition；Code Mode 减少的是执行过程中积累的 Tool Result 和模型往返。前者控制能力说明的工作集，后者控制执行结果的工作集。 +调用树折叠带来了显著的工程效益。首先，它大幅削减了模型交互的往返频次；原本需耗费多次采样的批量检索，在单次程序执行内即可完成。其次，它有效阻断了上下文污染：沙箱可就地消化海量原始数据，仅向外层模型上下文抛出结构精炼的最终结论。底层工具调用产生的明细未被丢失，只是其中不具推理价值的噪音被精准过滤在模型工作内存之外。 -当然，并不是 Tool Call 越多，越应该塞进一段程序。一次写入是否需要用户批准，搜索结果是否改变了下一步调查方向，页面上一个异常提示究竟意味着什么，这些都需要模型在观察之后重新判断。涉及不可逆副作用时，让动作保持为清晰、独立的顶层 Tool Call，往往也更容易被人理解和控制。Code Mode 适合下沉确定性的部分，不适合把所有 Agent 决策藏进代码。 +然而，程序化编排并非适用于所有场景。涉及不可逆外部副作用、需要显式人工审批、或是后续操作方向高度取决于非结构化观测结果的环节，保持显式、单步的顶层 Tool Call 能够提供更为清晰的审查线索与干预切入点。Code Mode 专为确定性计算的下沉而设计，不宜用于隐藏关键的 Agent 决策分支。 -Maka 的 Code Mode 延续了这个边界。模型通过一个 `exec` Tool 提交 JavaScript Cell，Cell 只能调用当前已经激活、并且允许嵌套的 Tool。执行环境本身没有进程、文件系统或网络能力，并受到运行时间、内存、源码体积、结果体积、调用次数和并发数的限制。 +Maka 的 Code Mode 深度整合了上述边界。模型借助 `exec` 工具向系统提交包含 JavaScript 代码的单元（Cell），该单元仅允许调度当前已激活且声明支持嵌套调用的工具接口。执行容器自身剥离了直接的操作系统原生访问能力，并在执行超时、内存配额、源码长度、输出体积及并发深度等多个维度受到硬性配额约束。 -更关键的是，Cell 内部的调用仍然回到同一个 `ToolRuntime`。参数校验、权限判断、执行边界以及上一节讨论的 T1/T2 持久化语义都不会因为调用来自代码而消失。Maka 会为这些嵌套调用分配独立身份，并记录它们与外层 `exec` 的父子关系。 +在事务保证上，Cell 内部发起的每一处工具调用，均受控回调至核心 `ToolRuntime`。参数校验、权限审批以及前文阐述的 T1/T2 事务保障全量生效。Maka 会为每个嵌套动作生成独立的调用凭证，并精确记录其与宿主 `exec` 之间的父子关联。 -这些内部调用是 Durable 的，但不会作为一长串 Call / Result 再次塞给模型。它们在 Runtime Event Log 中标记为来自 Code Mode，对模型历史则是 Hidden；模型看到的是外层 `exec` 及其最终结果。这里又出现了 Maka 一贯的结构：Log 保存完整事实，Provider Context 只是对事实的一种 Projection。 +此类嵌套操作被赋予完整的持久化语义，但不会以冗长的调用链形式直接灌入后续模型上下文。它们在 `RuntimeEvent Log` 中均标记为源自 Code Mode，并被设为 `modelVisibility: hidden`；模型仅在上下文中看到顶层的 `exec` 及其聚合产物。这体现了 Maka 一贯的架构哲学：底层日志负责记录全量系统事实，模型上下文仅作为面向推理场景的特定投影。 -Code Mode 也让恢复问题变得更尖锐。一段程序可能已经成功执行了前三个 Tool，却在第四个 Tool 等待结果时崩溃。如果重启后把整段程序重新运行一次，就会把已经完成的现实动作也重新做一遍。因此，Maka 不会自动重试一个中断的 `exec`。嵌套 Tool 的既有结果保留在日志中，外层 Cell 则获得一个明确的 Interrupted Result，之后由新的模型推理决定如何继续。 +Code Mode 同样对崩溃恢复提出了严格要求。一段脚本可能在连续完成三项操作后，在第四项操作等待响应时遭遇崩溃。若在恢复时全量重新执行该脚本，先前已完成的系统副作用势必遭受重复触发。因此，Maka 严禁自动重跑未完结的 `exec` 任务。沙箱内已沉淀的工具结果完整封存于日志中，外层 Cell 记录明确的中断事实，交由后续推理轮次评估下一步策略。 -这说明 Program 并没有成为绕过可靠性的捷径。它压缩了模型与 Runtime 之间的推理回合，却不能压缩现实世界已经发生过的事实。程序可以是临时的，调用栈可以随着 Cell 一起消失；但每一次真正越过边界的 Tool Call，仍然必须留下可审计、可恢复的 Log。 +脚本环境本质上是瞬时的，其内存栈随着沙箱销毁而湮灭；但其中每一次越过系统边界的真实工具操作，都必须作为不可变记录永久留存于底层审计账本中。 -Tool Call 让模型从语言走向行动。Code Mode 又向前走了一步：模型开始生成的不只是一个动作，而是动作之间的结构。 +## 并发调度与资源权威分离 -## Parallel Tool Call：Agent Runtime 里的 Async I/O +模型不仅可以在 Code Mode 内部并发拉起多个工具，亦能在单次常规推理输出中并列生成多个 Tool Call。 -Code Mode 可以在程序中并发调用多个 Tool。即使没有 Code Mode，今天的模型也可以在一个 Assistant Step 中一次生成多个 Tool Call。 +这类通常被称为并行工具调用（Parallel Tool Call）的机制需要厘清其本质特征：模型在生成这一批调用指令时，尚未接收到其中任何一项操作的实际执行反馈。因此，同一批次内的调用之间不存在基于返回结果的数据依赖。 -这通常被叫作 Parallel Tool Call，但这里的“并行”需要先说清楚。模型并不是一边观察第一个调用的结果，一边决定第二个调用。它在同一次生成中已经把整组调用全部交给了 Runtime，因此这些调用之间不可能存在基于 Tool Result 的数据依赖。 - -如果第二个动作必须读取第一个动作的结果，它就不属于这一批，而应该出现在下一次模型推理中。 +若某个操作的前提条件依赖于另一操作的输出载荷，则该操作必须等待下一次推理步骤，而不应归入当前并发批次。 ```text 同一个 Assistant Step @@ -297,50 +287,57 @@ Model ──┼── Tool Call B ──→ Result B ──┼──→ 下一 Fan-out / Fan-in ``` -从 Runtime 的角度看，这与经典 Async I/O 非常接近。每个 Tool Call 被转换成一个可以独立等待的 Task。Task 开始之后，Runtime 不需要为它占住一个同步调用栈，可以继续启动其他已经 Ready 的 Task；等到底层文件系统、进程、网络或远端服务返回结果，再唤醒对应的 Continuation。整批 Task 全部进入终态后，Runtime 才把 Tool Results 交给模型，开始下一轮推理。 +在 Runtime 的系统视角下，这与经典的异步 I/O 模型高度契合。每一个 Tool Call 被抽象为可独立等待的任务单元。任务派发后，Runtime 无须同步阻塞宿主线程，可继续推进其他就绪任务；底层文件系统、子进程、网络套件或外部微服务产生响应后，依序唤醒对应的调度点。待整批任务全部收敛至终止状态，Runtime 再将聚合后的结果集提交给下一轮模型推理。 -这种结构的价值并不只是“更快”。更准确地说，它让等待可以重叠。一个 Web Search 正在等待网络时，另一个 Search、文件读取或子 Agent 不必陪它一起空等。Agent 的执行时间从多个 I/O 延迟之和，逐渐接近关键路径上的最长延迟。 +该机制的核心收益在于让等待时延充分重叠。网络检索发起后，无须阻碍本地文件的读取或子 Agent 的运算；端到端的系统耗时得以从各个 I/O 延迟的代数累加，收敛至关键路径上的最大时延。 -但没有数据依赖，不等于没有资源冲突。 +数据依赖的缺位，并不意味着物理资源冲突的消除。 -模型可以同时生成 `Read(a)` 和 `Edit(a)`，也可以同时要求两个 Tool 改写同一份 Session State。两个调用都不依赖对方的返回值，却可能争用同一个现实资源。如果 Runtime 只是把这一批调用全部交给 `Promise.allSettled()`，那么谁先观察、谁先写入、后写是否覆盖前写，就会取决于不可预测的执行时序。 +模型可能在同一批次中同时发起 `Read(a)` 与 `Edit(a)`，亦可能驱动两个工具并发写入同一份会话上下文。此类操作虽无返回值的先后依赖，却直接竞争同一系统资源。若 Runtime 仅仅将批次整体交付 `Promise.allSettled()` 盲目并发，底层的读写次序与覆写结局将完全沦为由调度抖动决定的不可控竞态。 -Maka 在 [PR #4542](https://github.com/apache/maka/pull/4542) 中讨论的正是这个问题：一批 Tool Call 应该如何在保留独立 I/O 并发的同时，让访问同一资源的操作获得确定顺序。 +Maka 在 [PR #4542](https://github.com/apache/maka/pull/4542) 中深入探讨并明确了这一设计：必须在保障独立 I/O 充分并发的前提下，为共享同一底层资源的互斥操作确立确定性的时序控制。 -这里很容易把所有责任都放进一个中央 Tool Scheduler。Scheduler 预先计算每个调用会读取或写入哪些资源，不冲突的立即执行，冲突的按照模型生成顺序排队。这种做法能够提供清晰的 Batch 编排，却不应该成为资源正确性的唯一来源。 +将全部排他逻辑集中寄托于单一的调度器（Tool Scheduler）并非最佳实践。由中心调度器通过解析调用参数静态预判资源读写集合，虽然能够完成批次内的初步排队，但无法覆盖复杂的动态系统环境。 -经典 Async I/O 对这件事有一个很有用的职责划分：Executor 调度 Task，Resource Authority 管理资源。 +经典异步系统的实践提供了清晰的职责边界划分：由执行器（Executor）编排任务流，由资源权威（Resource Authority）掌管具体资源的互斥约束。 -一个 Tokio Executor 不会分析 Future 是否访问了同一个 Redis Key，也不会猜测两段异步代码最终是否写入同一个文件。它负责运行已经 Ready 的 Future。互斥、读写公平性、容量和唤醒通常由更靠近资源的一层负责，例如 Async Mutex、RwLock、Semaphore，或者独占状态的 Actor。 +底层执行器专注于调度已就绪的任务单元；互斥仲裁、读写公平性调度、容量水位管理与唤醒逻辑，则下沉由直接管理具体资源的权威层（如 Async Mutex、RwLock、Semaphore 或独占状态的 Actor）负责裁决。 -同样的边界也适用于 Agent Runtime： +Agent Runtime 适用同样的架构分工： ```text Tool Batch - │ 创建 Task、保留结果槽位、传播取消 + │ 创建 Task、预留结果槽位、广播取消信号 ▼ Resource Authority - │ 确认资源身份、排队、互斥、版本检查、唤醒 + │ 确认资源身份、排队、互斥排他、版本核对、唤醒 ▼ Filesystem / Terminal / Browser / Session / Remote Service ``` -为什么资源身份必须由 Authority 确认？因为真正的资源往往不是 Tool 参数中的那段字符串。`link/a` 和 `real/a` 可能通过符号链接指向同一个文件；两个不同的 UI Tool 可能操作同一个 Browser Tab；两个 MCP Tool 也可能共享同一个远端 Session。只有实际拥有或执行这个资源的一层，才能知道它们是否是同一个东西，以及操作在哪一个瞬间真正生效。 +资源身份的最终确认必须由权威层执行。入参中的路径字符串并不能代表真实的底层资源对象：不同路径可能通过符号链接汇聚至同一物理文件；不同的操作可能作用于同一个浏览器标签页；多个工具调用可能共享同一远程工作会话。唯有直接操纵物理资源的权威组件，方能确立操作间的真实冲突关系与生效时序。 + +若互斥保障仅维系在单次批处理的调度器内部，该机制将无法约束其他交互轮次、并发 Agent 实例或底层系统内部发起的竞争访问。安全性必须在最贴近系统副作用的层级闭环。Batch Scheduler 负责消除不必要的批内冲突并优化吞吐，但具体的互斥锁机制应下沉至资源实体。 -如果互斥只存在于当前 Tool Batch 的 Scheduler 中，它也无法约束另一个 Turn、另一个 Agent、另一个进程，或者任何绕过该 Scheduler 到达同一资源的执行路径。正确性必须在最靠近副作用的位置依然成立。Batch Scheduler 可以减少无谓竞争并提供确定性，但它更适合成为编排层，而不是唯一的锁。 +针对不同类型的系统资源，应当采用差异化的仲裁策略： -不同资源也不必被塞进同一种冲突模型。文件适合按 Canonical Path 建立带写者公平性的读写 Lease；Terminal 和 Browser 更像拥有单一状态的 Actor；远端 Provider、MCP Server 和子 Agent 的并发上限是 Capacity 问题，更适合用 Semaphore 表达；带 Revision 的 Session State 则可以使用 CAS 检查。它们共享的是异步生命周期，不是同一种锁。 +- **文件系统**：依据 Canonical Path 构建具备写优先或读写公平的租约（Lease）控制。 +- **交互终端与浏览器**：抽象为具备单一有序状态的串行 Actor。 +- **外部接口与 MCP 服务**：针对 QPS 与并发上限配置容量型信号量（Semaphore）。 +- **版本化会话状态**：采用基于 Revision 的 CAS（Compare-And-Swap）机制实施乐观校验。 -这也解释了为什么“资源冲突”和“容量限制”必须分开： +各类资源共享统一的异步生命周期模型，但其互斥语义保持独立适配。 -- 资源冲突回答两个动作能否正确地同时发生。 -- 容量限制回答系统愿意同时承担多少个动作。 +这亦要求在架构上明确切分“资源互斥”与“容量限额”： -把 API QPS 限制伪装成一个与所有资源都冲突的全局锁，虽然能降低并发，却会制造不必要的 Head-of-Line Blocking。一个慢请求会挡住与它完全无关的文件读取。相反，Async I/O 追求的是只阻塞真正尚未 Ready 的 Task，让独立工作继续前进。 +- 资源互斥仲裁多项并发操作是否会破坏状态的一致性。 +- 容量限额裁决系统当前所能承受的最大并发负荷。 -对于确实冲突的调用，Provider 返回的数组顺序可以作为一个稳定的 Tie-breaker，但不能被解释为数据依赖。模型在生成这一批调用时没有看见任何中间结果，这个顺序只能表示“发生冲突时谁先获得资源”，不能表示后一个调用消费了前一个调用的结果。 +将服务端的 QPS 限流粗暴包装为全局互斥锁，势必引发不必要的线头阻塞（Head-of-Line Blocking），导致长耗时的网络请求无端拖死完全无关的本地文件读取。异步运行时应致力于仅阻塞存在真实物理冲突的任务，确保无关联操作顺畅推进。 -因此，Parallel Tool Call 中至少存在四种不同的顺序： +对于批次内确实存在物理冲突的调用，模型输出的原始数组序可作为确定性的仲裁决胜依据（Tie-breaker）。但须注意，该顺序仅表示在遭遇资源争用时仲裁所有权的先后，并不代表调用之间存在任何因果数据传递。 + +在并行工具调用中，系统在不同层面面临四种各异的时序状态： ```text 模型生成顺序 @@ -349,23 +346,15 @@ Filesystem / Terminal / Browser / Session / Remote Service ≠ Runtime Event 到达顺序 ``` -不冲突的后续 Task 可以先启动，也可以先完成。实时事件应该按照实际发生的时序进入 Log，并通过 Tool Call ID 保持因果关联；而发送给 Provider 的 Tool Result，则可以按照原始调用顺序重新组装。事实顺序与模型协议顺序不必相同，它们是同一次执行的不同 Projection。 - -取消和失败同样要遵守 Async I/O 的生命周期。还在队列中的 Task 被取消后不能偷偷开始；已经跨过 T1 的 Task 则不能假装不存在，Runtime 必须等待它收敛并记录结果。普通的 Tool 业务失败可以作为一个 Result 与同批其他任务一起返回，但如果 T1/T2 持久化失败，新的排队任务就不应继续获得派发资格。已经 Active 的工作需要安全结束，尚未开始的工作应该被冻结。 +互不干扰的任务可以乱序拉起，亦能乱序终结。客观发生的操作事件依实际触发时序落入底层日志，并借助 Tool Call ID 保持逻辑因果；向模型组装回传报文时，则按模型原始请求顺序对号入座。底层日志记录物理事实，上下文装配满足模型协议，两者互为同一执行过程的独立投影。 -这正是经典 Async Runtime 中 Structured Concurrency 的味道：父级 Batch 不只是启动一堆 Promise 然后离开，它拥有这些 Task 的生命周期。下一次模型推理开始之前，每个子 Task 都必须已经完成、被取消，或者进入一个明确可恢复的状态。 +异常中断与超时取消亦须严格遵守结构化并发（Structured Concurrency）的约束规范。处于等待队列中的任务被取消后严禁悄然启动；一旦越过 T1 派发边界的任务则严禁直接丢弃，Runtime 必须挂起并静待其状态明确收敛。顶层批处理机制对其派生的全部子任务生命周期负有终极管理责任；在下一轮模型推理拉起前，每一项并发动作必须已确切终结或转入可审计的受控状态。 -Parallel Tool Call 因而不是一句“工具可以同时调用”就结束了。真正困难的是在三个目标之间建立边界：让独立 I/O 充分重叠，让共享资源保持正确，让整个 Batch 在取消、失败和恢复时仍然拥有清楚的生命周期。 +## 沙箱、Serverless 与存算分离 -模型提供并发的意图，Batch Runtime 负责结构化地汇合，Resource Authority 决定现实世界允许怎样的并发。 +所有工具调用终归需要在特定的物理或虚拟计算载体上运行。 -## Sandbox 与 Serverless：给 Agent 一台随时可以丢掉的计算机 - -Tool Call 最终必须在某个地方运行。 - -模型可以生成调用意图，可以写出一段编排程序，却不能凭空产生 CPU、内存、文件系统和网络连接。真正执行 JavaScript、启动 Python、安装依赖、运行测试或操作浏览器的，始终是一块现实中的计算资源。 - -最轻量的环境可以是一个 JavaScript V8 Isolate。它启动快、边界清楚，适合运行 Code Mode 中短小的控制流。需要数据分析和丰富 Library 时，可以给 Agent 一个 Python Runtime。再往下，当 Tool 需要完整文件系统、系统命令、编译器和后台进程时，自然会走向 Container，甚至 MicroVM。 +模型产出结构化动作，脚本编排业务控制流，但底层 CPU、内存堆栈、文件系统与网络协议栈必须依托真实的物理环境。从快速启动的 JavaScript V8 Isolate，到承载数据处理与脚本生态的 Python 容器，再到提供完整操作系统能力、独立网络栈与编译环境的 MicroVM，不同工具所需的环境规格存在显著阶梯。 ```text LLM 生成意图 @@ -383,136 +372,96 @@ Agent Runtime Filesystem / Process / Network / Browser ``` -这些环境不是越重越好。让每个简单 Tool Call 都启动一台 VM 很浪费，让不受信任的系统命令与 Runtime 运行在同一个进程里又过于危险。Agent Runtime 需要根据任务真正需要的能力，选择足够轻、同时又足够隔离的执行载体。 +执行环境并非越重越好。为轻量级的文本解析动用完整虚拟机将带来不合理的开销，而将不受信任的复杂命令置于宿主进程同构运行则构成严重的安全隐患。Runtime 需依据操作所需的最小特权与资源形态，动态适配隔离强度与消耗成本均衡的执行载体。 -Sandbox 因此不只是防止模型运行危险代码的围墙。它还是一次 Agent Execution 的资源边界、故障边界和生命周期边界。 +沙箱绝非仅是防范恶意代码的隔离护栏，它同时界定了单次 Agent 执行的资源边界、故障域以及生命周期轮廓。 -Runtime 可以限制一个 Sandbox 能使用多少 CPU、内存、磁盘、并发和运行时间；可以决定它是否拥有网络、可以看到哪些目录、能够调用哪些外部服务；也可以在代码死循环、内存耗尽或进程崩溃时，直接终止这个环境，而不让故障扩散到整个 Agent 系统。 +Runtime 可以在沙箱层面施加精细配额：限制 CPU、内存、存储用量、并发线程与最大执行窗口；限制网络访问域与外部服务白名单；并在出现内存耗尽、死循环或进程异常时迅速熔断隔离环境，阻断单点故障向整个 Agent 核心运行时的蔓延。 -更重要的是，Sandbox 把“Agent”与“运行 Agent 的那台机器”分开了。 +更核心的架构演进，在于沙箱实现了 Agent 运行时与特定计算节点的彻底解耦。 -传统桌面程序往往默认进程和本地状态长期存在。Agent 的执行环境则应该被假设为随时可能消失：V8 Cell 运行结束就销毁，Container 空闲后可以回收，MicroVM 可以因为超时、迁移或宿主故障而终止。只要系统把 Agent 的真实状态寄托在这些临时环境里，恢复就会变得异常困难。 +传统桌面软件通常假定宿主进程与本地环境长期稳定存续。而现代 Agent 的执行载体应当被视为随时可被回收的易失资源：V8 Cell 运行完毕即行销毁，容器在空闲超时后自动回收，MicroVM 亦可因节点迁移、负载均衡或硬件异常被随时下线。若将 Agent 的真实状态绑死在单台计算节点的易失内存中，系统的可靠性与扩展性将无从谈起。 -这也是 Append-Only Log 再次出现的地方。 +不可变追加写日志在此构成了解耦的核心支柱。 -对话、Tool Call、Tool Result、权限决定和恢复结论保存在 Durable Log 中；文件、图片和大型结果进入外部 Artifact Storage；Workspace 可以通过持久卷、快照或对象存储恢复。Sandbox 只承载当前正在运行的计算。它可以被销毁，也可以在另一台机器上重新创建。 +会话流程、工具派发与响应、授权记录及恢复决议全量固化于持久化日志中；产生的文件实体、媒体流及大体积结果存入对象存储；工作区结构则依托写时复制（Copy-on-Write）快照或持久卷管理。计算沙箱退化为单纯消费状态并执行任务的无状态处理器，可被随时销毁并在任意其他节点按需重构。 ```text -Durable State Ephemeral Compute +持久化状态 (Durable State) 易失计算 (Ephemeral Compute) RuntimeEvent Log ─┐ ┌─ V8 Isolate Artifact Storage ─┼─→ Rehydrate ────┼─ Container Workspace Snapshot┘ └─ MicroVM - 保存“发生过什么” 执行“下一步做什么” + 保存“发生过什么” 执行“下一步做什么” ``` -Serverless 与 Agent 天然契合的原因也在这里。Agent 工作负载通常是突发的：模型思考时 Sandbox 可能无事可做，Tool Call 到来时又需要迅速获得计算；有些任务只运行几十毫秒，有些任务要编译大型项目或等待长时间 I/O。理想的计算层应该能够按需创建、闲时归零，并根据 Tool 的资源声明分配不同规格。 - -但 Agent Serverless 不能只是传统 Function as a Service 的简单翻版。普通函数通常接收输入、计算并返回结果；Agent 还会保有 Workspace，启动后台进程，等待用户批准，调用外部 Tool，并在数小时后 Resume。它需要的不是一段永不消失的进程，而是一套能够把 Durable State 与 Ephemeral Compute 重新接合起来的协议。 +Agent 的负载特征天生具备极高的突发性（Burstiness）：在模型深度推理阶段，计算沙箱处于完全空闲；而一旦批处理或编译任务触发，计算需求瞬时拉升；部分操作耗时数毫秒，部分任务则需持续挂起数小时等待网络回调或人工确认。理想的计算架构应当支持按需秒级拉起、闲时归零（Scale to Zero），并精准依据工具的资源画像调配不同梯度的计算单元。 -一个 Sandbox 消失以后，Runtime 不应该尝试恢复它原来的内存、Promise 和调用栈。它应该先根据 Log 判断哪些 Tool 已经发生、哪些结果已经提交，再把 Workspace 和必要 Artifact 装载到新的环境，从可信的历史前缀开始下一段执行。 +但这并不等同于将传统 FaaS（Function as a Service）做简单套用。无状态函数假定单次调用即完成计算；而 Agent 执行具有长周期的上下文依赖，需要保留持久工作区、支持后台挂起长任务、接纳异步人工审批，并在数小时后依然能够无缝续接。 -换句话说，Serverless 的重点不是 Agent 没有状态，而是它的状态不属于任何一台计算机。 +Agent Serverless 的核心在于状态的彻底解耦：Session 状态脱离任何特定计算节点的生命周期。 -这种架构也会改变权限的实现方式。Sandbox 不需要持有所有云服务的永久凭证，也不应该天然拥有完整网络。它只得到本次任务需要的 Capability；真正的 Secret、审批和 Resource Authority 留在 Sandbox 外部。代码可以请求一个动作，但外部 Runtime 仍然决定这个动作能否越过边界。 +当某个执行沙箱因故障或超时回收后，Runtime 绝不试图从内存垃圾中拼凑还原原有的指针与未决调用栈。它依托持久化日志核验先前操作的最终收敛状态，将工作区快照与必要产物水合（Rehydrate）至全新分配的沙箱环境中，基于经过严密验证的历史前缀启动后续执行。 -当这样的计算层足够便宜之后，Agent 才能真正扩大规模。一个 Agent 可以为一次短暂编排申请 V8，为一次数据处理申请 Python Container，为一次完整软件构建申请 MicroVM；也可以同时创建多个隔离环境，让子 Agent 在不同 Workspace 中并行工作，结束后立即释放资源。 +权限管理架构亦由此迎来重构。易失沙箱不持有云服务特权凭证，亦不默认具备泛化公网权限。沙箱内部仅获得当前任务所需的临时最小能力上下文；真正的密钥管理、审计仲裁与核心资源权限始终收拢于沙箱外部的受信 Runtime 中。沙箱内代码提出操作请求，外部权威执行安全鉴权与副作用分发。即使沙箱环境遭受入侵，其权限边界亦随沙箱的销毁而即时作废。 -再沿着这个方向往前走一步，会得到一个更有意思的形态：所有 Session 都沉到廉价的 S3-Compatible Object Storage 中，计算层则完全由廉价、短暂、可以被替换的执行资源组成。 +当执行载体的边际成本大幅下降，Agent 体系的规模化应用才具备落地可行性：单一 Agent 会话可针对不同任务阶梯式申请 V8、Python 容器乃至微型虚拟机；亦可并发实例化多个彼此隔离的工作沙箱，驱动多子 Agent 协同作业并在收工时立即释放硬件配额。 -这是一种彻底的存算分离。 +这一架构的极致演进，正是存算分离的全面深化：将会话的全量状态沉淀于廉价可靠的对象存储（如 S3-Compatible Storage）中，计算层则完全由轻量、异构、即用即弃的执行节点按需供给。 -Session 不再对应某个进程中的对象，也不对应某台机器上的目录。它是一组持久对象：Append-Only Event Segments、Artifact、Workspace Snapshot、Compaction Projection，以及指向当前可信前缀的 Manifest。一次对话结束之后，不需要有任何 Runtime 继续驻留在内存里。Session 可以安静地躺在对象存储中，除了存储本身几乎不消耗计算资源。 +会话不再绑定于特定机器的目录或常驻进程。它体现为对象存储中一组不可变文件的集合：追加写的事件切片、外部产物实体、工作区文件快照、压缩投影快照以及指向当前合法提交前缀的元数据清单（Manifest）。交互间歇期，无须任何守护进程驻留内存；会话静止存放于存储池中，仅产生极低的静态数据存储成本。 ```text - Cheap Durable Storage + 廉价持久化存储 (Object Storage) Session A ── Events / Artifacts / Workspace Snapshots ─┐ Session B ── Events / Artifacts / Workspace Snapshots ─┼── S3 Session C ── Events / Artifacts / Workspace Snapshots ─┘ │ - Event / User / Schedule │ + 外部事件 / 用户交互 / 定时触发 │ │ │ ▼ │ - Rehydrate a Session ◀─────────┘ + Rehydrate 会话上下文 ◀─────────┘ │ ┌───────────┼───────────┐ ▼ ▼ ▼ V8 Python MicroVM │ │ │ - └───────────┴───────────┘ + └───────────┼───────────┘ │ - Append Facts + 追加增量事实 │ └──────────────→ S3 ``` -所谓“长期运行的 Agent”，也就不再要求一台机器长期运行。 - -它可以绝大多数时间都处于休眠状态。用户发来消息、定时器到期、Webhook 抵达或者后台任务完成时，调度层读取 Session Manifest，加载必要的 Log Prefix 与 Workspace Snapshot，为它分配一个新的 Sandbox。任务完成后，新事实和 Artifact 回写对象存储，计算环境随即释放。 - -Agent 不是一直活着，只是随时可以被重新唤醒。 - -这里的 S3 也不再只是备份介质，而可以成为 Agent State 的事实存储。热机器上的内存、SQLite、Local SSD、向量索引和 Provider Context 都只是缓存或 Projection。它们可以提高读取速度，却不应该决定 Session 是否仍然存在。机器丢了，缓存可以重建；只要对象存储里的可信历史仍在，Agent 就仍在。 +长效存续的 Agent 系统，不再依赖长期不关机的物理服务器。 -当然，把 Session 放进 S3，不意味着对同一个大对象不断执行原地 Append。更自然的实现是写入不可变的 Event Segment 和 Artifact，再用很小的 Manifest 或 Head Pointer 指向最新的已提交前缀。Lease、CAS、幂等键和正在执行的 Operation 仍然需要一个强一致的控制面，但庞大的历史正文、Tool Result、文件快照和媒体内容都可以进入廉价对象存储。 +它可以在绝大部分时间内处于完全休眠。用户发送指令、定时周期唤醒、Webhook 抵达或异步监控任务就绪时，调度控制面迅速检索 Session Manifest，挂载所需的日志切片与工作区快照，动态唤起适配规格的沙箱实例。任务执行收敛后，最新增量事实与工作区差分同步写回对象存储，计算资源随即完全归还。 -于是整个系统会自然分成两层： +系统由两层核心架构协同运作： -- Data Plane 保存不可变、体积巨大、很少修改的 Session State。 -- Control Plane 保存体积很小、需要强一致的 Head、Lease、Admission 和 Operation 状态。 +- **数据面（Data Plane）**：由对象存储托管体积庞大、不可变、极低频修改的历史数据与快照实体。 +- **控制面（Control Plane）**：依托轻量级存储维护强一致的 Head 指针、资源租约、全局限额及未决操作状态。 -这与现代数据库的存算分离很像。对象存储提供近乎无限、廉价而持久的容量，计算节点只在查询或写入发生时出现。只不过这里被查询和继续执行的，不是一张表，而是一个 Agent 的历史。 - -从这个角度看，Model Context 本身也是一次 Query。Runtime 从 S3 中读取 Session 的 Durable State，应用 Compaction、Tool Result Prune、Visibility 和 Provider Compatibility 等 Projection，构造出这一轮模型真正需要看到的上下文。模型完成推理后，新的输出不去修改过去，而是继续追加新的事实。 +这与现代分布式云原生数据库的解耦逻辑如出一辙。对象存储提供近乎无限的耐用性与低成本空间，计算算力仅在执行读写请求时按需供给。在此架构下，模型上下文的装配本质上等价于一次物化视图查询：Runtime 自对象存储提取 Session 状态，执行 Compaction 压缩与 Tool Result 裁剪投影，交付模型展开推理；模型输出的新事实再次以追加写形式沉淀回存储层。 ```text Session on S3 │ ├── Projection ──→ Model Context ──→ LLM │ │ - ├── Rehydrate ───→ Sandbox ───────→ Tool Call + ├── Rehydrate ───→ Sandbox ──────────→ Tool Call │ │ └──────────────── Append New Facts ◀───┘ ``` -这样一来，LLM 和 Sandbox 都只是计算资源。 - -模型可以根据任务难度临时选择，轻任务使用便宜模型，复杂决策使用更强模型；执行环境也可以根据 Capability 临时选择，简单编排进入 Isolate，普通脚本进入 Container，完整系统操作进入 MicroVM。同一个 Session 不属于任何一个模型，也不属于任何一种 Sandbox。 - -这会带来一种新的 Agent Economics。系统成本不再主要取决于保存了多少 Session，而取决于此刻有多少 Session 正在思考和行动。一千万个休眠 Session 可以只是对象存储中的一千万组前缀；只有被事件唤醒的那一小部分，才占用模型 Token、CPU 和内存。 - -最便宜的 Agent，不是运行在一台更小的服务器上，而是睡着时根本没有服务器。 - -存算分离也让抢占式计算真正可用。计算节点可以来自低价实例、共享 Worker Pool，甚至随时可能消失的 Capacity。过去，杀死一台正在运行 Agent 的机器意味着丢失整个会话；当状态已经外置，失去一个 Worker 只是失去一份临时执行。Runtime 根据 T1/T2 判断现实动作的状态，再把 Session 放到另一块计算资源上继续。 - -Branch 和 Fork 也会变得非常便宜。Append-Only History 与 Copy-on-Write Workspace Snapshot 天然允许多个 Agent 共享同一段历史前缀，再从不同位置长出各自的后缀。创建一个子 Agent 不必复制整个 Session，只需要记录它从哪个 Prefix 和 Snapshot 出发。没有修改的 Artifact 继续共享，只有新的事实产生新的存储。 - -甚至模型升级也不必迁移 Session。历史保留的是 Provider-Neutral 的 Runtime Fact，新模型只需要获得适合自己的 Context Projection。同一份 Durable Session 可以在今天由一个模型执行，几个月后由另一个模型 Resume。Agent 的身份来自它经历过的历史，而不是当前装载它的模型权重。 - -安全边界也因此变得更干净。S3 保存的是加密且可审计的长期状态，Sandbox 只在短暂生命周期内获得最小 Capability。Secret 不必写进 Workspace Snapshot，云账户的永久凭证也不必进入 MicroVM；需要访问外部资源时，Sandbox 通过外部 Authority 请求一次受约束的操作。计算环境被攻破之后，其权限会随着环境销毁而失效。 - -当然，廉价计算不会自动带来正确性。一个 MicroVM 再便宜，也不能让重复付款变得安全；一个 Container 再容易重启，也不能回答崩溃前的邮件是否已经发送。越是把 Worker 视为可以随时抛弃，越需要 Reliable Tool Call、幂等 Operation、Resource Authority 和 Append-Only Log 来证明现实世界中发生过什么。 - -所以这并不是一句简单的“把 Agent 跑在 Serverless 上”。更准确的说法是，我们正在为 Agent 构造一种新的计算机： - -```text -S3 是它廉价而持久的磁盘 -Append-Only Log 是它可恢复的状态 -LLM 是它按需租用的推理单元 -Sandbox / MicroVM 是它按需租用的身体 -Agent Runtime 是连接这一切的操作系统 -``` - -今天谈 Agent，注意力往往集中在模型上。但模型只负责产生判断和意图。让这些意图安全、可靠、低成本地作用于现实世界，需要大量随取随用的执行环境，以及比这些环境活得更久的 Session State。 - -未来 Agent 的核心基础设施，一定包含极其廉价的存储和极其廉价的计算。存储让数以亿计的 Session 可以长期存在，计算让其中任何一个 Session 都能在需要时迅速醒来。两者之间依靠的不是某台机器的内存，而是一条可以重放、验证和继续追加的历史。 +在此架构中，LLM 与 Sandbox 均回归为纯粹的按需计算资源。 -回头看整条 Tool 链路，Deferred Tool 决定模型此刻需要知道哪些能力，Tool Call 把语言转换成行动，Reliable Execution 让行动成为可信事实，Code Mode 组织动作之间的结构，Async Runtime 让等待彼此重叠，而 Sandbox 与 Serverless 则为这一切提供真正可以消耗的 CPU、内存和隔离边界。 +模型选型可依推理复杂度动态调配，简单逻辑调用轻量模型，复杂决策转交旗舰模型；计算载体则依能力需求即时匹配，轻量编排交付 Isolate，工程编译挂载 MicroVM。同一个 Session 不与特定模型或特定计算硬件存在排他绑定。 -最终，Agent 不是一个恰好会保存状态的长驻进程。 +降低 Agent 运行成本的关键，在于会话处于休眠状态时不持有任何常驻计算资源。 -**Agent 是一份持久状态，在需要思考和行动时，暂时租用一个模型和一台计算机。** +存算分离架构亦为分布式抢占式调度（Spot Instances）与灵活分支（Fork / Branch）扫清了障碍。借助不可变日志与写时复制快照，派生子 Agent 仅需引用既有历史前缀与快照标识，即可零开销衍生出独立的并行分支，仅增量数据占用独立存储空间。 -它沉睡在廉价的 S3 中。事件到来时，Log 告诉它曾经是谁，Sandbox 决定它现在能够做什么，廉价计算让它继续向前。 +面向未来，Agent 运行时的演进方向正逐步明晰：以不可变的真实事实日志作为状态权威，以多级轻量的弹性沙箱作为执行载体，以解耦的资源权限保障并发安全，以按需加载的 Schema 投影消除模型注意力干扰。从概率语言模型向客观系统的每一次延伸，都由严谨的运行时边界构筑起安全与可靠的工程基石。