Add supervisor pattern: /add-supervisor skill (custom LangGraph + Supervisor API) - #33
Add supervisor pattern: /add-supervisor skill (custom LangGraph + Supervisor API)#33alexbaur wants to merge 3 commits into
Conversation
Introduces a supervisor agent as a post-scaffold pattern (like eval gates,
governance, monitoring) applied once a project has >1 agent. A supervisor
routes user queries across the project's agents and other managed sub-agents.
The core contribution is a Selection Matrix that picks the best-fit supervisor
pattern from three options based on the user's needs, architecture, and the
complexity required — because the three are NOT interchangeable; they differ in
who owns the routing loop and what artifact lands in the bundle:
custom Hand-written LangGraph supervisor. GA, fully DAB-declarable
(an agent App under src/agents/), gated by the same CI eval
loop as any agent. The default.
supervisor_api Databricks-managed loop (Responses API) wrapped in a
declarable App. Beta (AI Gateway + UC OTel-traces preview).
agent_bricks_mas Agent Bricks Supervisor tile. NOT a DAB resource — scaffolds
a bundle-declared bootstrap job that provisions it imperatively
and a consumed-endpoint reference. UI GA / SDK Beta.
Integration is deliberately lightweight and maps to what already exists:
- Models the supervisor as "just another agent" (custom/api) so CI's
detect_patterns -> eval_gate picks it up with zero workflow changes.
- Reuses the add_agent.py wiring approach; renders graph/tools/deps variants.
- Records the choice in .agentops-stacks/manifest.yml under `supervisor:` — the
same contract CI/tooling already read.
- No new bundle-init input (choice made post-scaffold when it's meaningful).
- Honors the standing "declarative over notebook-created; document why" rule:
the MAS tile can't be a DAB resource, so it's a documented bootstrap job.
Files:
- plugin/skills/add-supervisor/SKILL.md — skill + Selection Matrix + security posture
- plugin/commands/add-supervisor.md — /add-supervisor command
- plugin/skills/agentops-stacks/scripts/add_supervisor.py — engine (3 patterns)
- .../scripts/supervisor_templates/ — graph (custom, api), tools, MAS job+notebook
- template/.../docs/supervisor-patterns.md.tmpl — pattern deep-dive in rendered project
- template/.../.agentops-stacks/manifest.yml.tmpl — supervisor contract placeholder
- workflows/single-account-single-agent.json — machine-readable decision matrix
- plugin/skills/agentops-lifecycle/SKILL.md — Step 3.5 routing note
- plugin/README.md, install_skills.sh — register the new skill/command
- tests/test_add_supervisor.py — 13 unit tests (no CLI needed), all passing
Co-authored-by: Isaac
Review pass found two defects that would break the generated supervisor App: 1. supervisor_api graph: LangChain message .type "ai" was passed to the Responses API as role "ai" (invalid — expects "assistant"). Added an explicit _ROLE_MAP so human/ai/system/tool map to valid roles. 2. agent.py inheritance: the script copied the source agent's agent.py, which — when the base agent had Lakebase memory — imports get_async_checkpointer from graph. The supervisor graph.py doesn't define it, so the App crashed on import. Now overwrite agent.py with a dedicated stateless supervisor handler (agent_supervisor.py.tmpl). Adds tests covering the agent.py overwrite, including the Lakebase-base case. Co-authored-by: Isaac
Agent Bricks Supervisor (MAS) is being deprecated and will not be available going forward, so the /add-supervisor feature is reduced to the two patterns that are first-class DAB citizens and remain supported: custom Hand-written LangGraph supervisor (GA, default). supervisor_api Databricks-managed loop wrapped in a declarable App (Beta). Both scaffold the supervisor as an agent App, so it deploys and is eval-gated like any other agent — unchanged from before. Removed: - agent_bricks_mas from VALID_TYPES, the scaffold_mas_bootstrap() path, the manifest endpoint branch, and the MAS next-steps in add_supervisor.py - supervisor_templates/mas_bootstrap_notebook.py.tmpl and mas_bootstrap_job.yml.tmpl - the D3a (no-code MAS) gate and the agent_bricks_mas pattern entry from the workflow decision matrix (D3 is now a single supervisor_api gate) - MAS rows/prose from SKILL.md, docs/supervisor-patterns.md, the lifecycle Step 3.5 table, the manifest template comment, the command file, README, and the installer help text - the two MAS-specific unit tests Verified: 12 unit tests pass; workflow JSON valid; end-to-end scaffold of both patterns renders databricks.yml that parses (yaml.safe_load) and graph/agent.py that compile, with no leftover placeholders; an invalid --type agent_bricks_mas is now rejected by argparse. No residual MAS references remain in feature files. Co-authored-by: Isaac
sdonohoo-db
left a comment
sdonohoo-db
left a comment
There was a problem hiding this comment.
Reviewed and tested this end-to-end against the dev workspace — scaffolded a project, added a second agent, ran /add-supervisor --type custom, and deployed to dev, and sent requests.
Requesting changes: two independent blockers make every scaffolded supervisor dead on arrival, and neither is caught by bundle validate/deploy — both pass clean.
-
Custom supervisor crashes on import (graph_custom.py.tmpl:89). create_supervisor(..., state_schema=MessagesState) fails under the resolved deps (langgraph 1.2.10): ValueError: Missing required key(s) {'remaining_steps'} in state_schema. This is the GA default path. Fix: drop state_schema or use AgentState (both verified).
-
Default model endpoint is deprecated and already 400s (graph_custom.py.tmpl:29, graph_supervisor_api.py.tmpl:30, plus agentops-lifecycle/SKILL.md:234). databricks-claude-sonnet-4 returns BAD_REQUEST: This endpoint ... is deprecated on every call today (fully unavailable in October). It reports READY, so only a live query catches it. This was raised in the prior review and is still unfixed. Note the same dead default also lives in the base agent template outside this PR's diff (graph.py.tmpl:20, app.yaml.tmpl:9, .env.example.tmpl:14) — the app.yaml one bakes it into the deployed App's env — so it's a repo-wide fix worth doing in the same pass. Replace with databricks-claude-sonnet-4-5, databricks-claude-sonnet-5, or databricks-claude-opus-4-8 (all verified working in dev).
Also needs addressing before merge:
- supervisor_api uses a DatabricksOpenAI client that likely doesn't exist in databricks_openai (graph_supervisor_api.py.tmpl:63) — verify against the package; if wrong, that whole path ImportErrors at startup.
- Empty --routes scaffolds an invalid create_supervisor([]) (add_supervisor.py:254) — require ≥1 route.
- Naive content.replace(source, name) corrupts identifiers containing the source name as a substring (add_supervisor.py:123) — shared with add_agent.py.
- _add_supervisor_deps silently drops the dependency if the langgraph pin isn't in the exact expected format (add_supervisor.py:163).
What works well: the DAB wiring, manifest recording, experiment/app resource injection, and bundle validate/deploy mechanics are solid — all three Apps deployed cleanly. The 12 unit tests pass. The issues are runtime/startup problems that unit tests and deploy-time validation structurally can't catch — they need a live inference call against the running app.
| model=supervisor_llm, | ||
| prompt=SUPERVISOR_PROMPT, | ||
| output_mode="full_history", | ||
| state_schema=MessagesState, |
There was a problem hiding this comment.
Blocking — the generated custom supervisor crashes on import.
With the pinned deps (langgraph-supervisor>=0.0.5 resolves to 0.0.31, pulling langgraph 1.2.10), create_supervisor(..., state_schema=MessagesState) fails at module import:
ValueError: Missing required key(s) {'remaining_steps'} in state_schema
MessagesState only carries messages; langgraph 1.x's create_react_agent (called inside create_supervisor) requires remaining_steps. Since graph_builder = build_graph() runs at import (line 92), import graph in agent.py fails and the App never starts. This is the GA/default path, so every custom supervisor is dead on arrival.
Verified fix (both work): drop state_schema=MessagesState entirely (use the default), or import and pass AgentState:
from langgraph.prebuilt.chat_agent_executor import AgentState
...
state_schema=AgentState,
Tested end-to-end against langgraph 1.2.10 in the dev workspace.
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
| LLM_ENDPOINT = os.environ.get("LLM_ENDPOINT", "databricks-claude-sonnet-4") |
There was a problem hiding this comment.
Blocking — deprecated model endpoint, already non-functional.
databricks-claude-sonnet-4 is deprecated and returns a 400 on every inference call today (goes fully unavailable in October). It reports state.ready: READY, so bundle validate/deploy pass clean — only a live query surfaces it:
BAD_REQUEST: This endpoint databricks-claude-sonnet-4 is deprecated.
Suggest a current endpoint — databricks-claude-sonnet-4-5, databricks-claude-sonnet-5, or databricks-claude-opus-4-8 (all verified working in dev).
|
|
||
| from langgraph.graph import MessagesState | ||
| from langgraph.prebuilt import create_react_agent | ||
| from langgraph_supervisor import create_supervisor |
There was a problem hiding this comment.
The >=0.0.5 floor resolves to 0.0.31 against langgraph 1.2.10, which is the source of the state_schema crash above and emits a deprecation warning (create_react_agent moving to langchain.agents in V2). Once the state_schema fix is in, pin a known-good langgraph + langgraph-supervisor pair and re-run an import test so the resolved versions are the ones you validated against.
|
|
||
| def _call_supervisor(messages: list[dict]) -> str: | ||
| """Invoke the managed Supervisor API and return the final text answer.""" | ||
| from databricks_openai import DatabricksOpenAI |
There was a problem hiding this comment.
Likely blocking for the supervisor_api path — verify this client exists.
databricks_openai exposes tool helpers (VectorSearchRetrieverTool, UCFunctionToolkit), not a DatabricksOpenAI client class. The documented OpenAI-compatible client is:
from databricks.sdk import WorkspaceClient
client = WorkspaceClient().serving_endpoints.get_open_ai_client()
As written, from databricks_openai import DatabricksOpenAI (and DatabricksOpenAI(use_ai_gateway=True) on line 65) will raise ImportError at startup. I didn't deploy-test this path (custom is the default), so please confirm against the installed package — if it's wrong, every supervisor_api scaffold fails on import.
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
| SUPERVISOR_MODEL = os.environ.get("SUPERVISOR_MODEL", "databricks-claude-sonnet-4") |
There was a problem hiding this comment.
Same deprecated-endpoint issue as the custom template — databricks-claude-sonnet-4 400s today.
| content = filepath.read_text() | ||
| except UnicodeDecodeError: | ||
| continue | ||
| updated = content.replace(source, name) |
There was a problem hiding this comment.
Substring replace corrupts identifiers that merely contain the source name. E.g. source rag → supervisor router turns storage into stoROUTERe, integration into inteROUTERtion, across every copied file. It didn't trigger in my rag→router test only because those files happened not to contain such words — it's latent. (Inherited from add_agent.py's copy logic, so worth fixing in both.) Recommend a word-boundary/token-aware rename rather than a raw str.replace.
| help="Supervisor agent name (lowercase, underscores, min 3 chars)") | ||
| parser.add_argument("--type", required=True, choices=sorted(VALID_TYPES), | ||
| help="Supervisor pattern (see the Selection Matrix in SKILL.md)") | ||
| parser.add_argument("--routes", default="", |
There was a problem hiding this comment.
Empty routes produces ROUTES = [] → create_supervisor([]) at import, which also crashes (same remaining_steps error surfaces first, but create_supervisor with no members is invalid regardless). Recommend requiring ≥1 route: fail fast in main() if routes is empty, rather than scaffolding a graph that can't start.
| if marker in content: | ||
| return | ||
| # Insert right after the langgraph pin, which every agent has. | ||
| content = re.sub( |
There was a problem hiding this comment.
The dependency insert only fires if pyproject.toml contains "langgraph>=..." in that exact shape. A different pin (langgraph==1.1.0, langgraph~=1.1) makes re.sub a silent no-op — langgraph-supervisor/databricks-openai never get added, and the failure only shows up later as ModuleNotFoundError at uv sync. At minimum, detect the miss and print a warning (or error) instead of silently continuing.
2 participants
What
Introduces a supervisor agent to AgentOps Stacks as a post-scaffold pattern (applied once a project has >1 agent, alongside eval gates / governance / monitoring). A supervisor routes user queries across the project's agents and other managed sub-agents.
The core contribution is a Selection Matrix that picks the best-fit supervisor pattern based on the user's needs, architecture, and required complexity. This PR ships two patterns — both first-class DAB citizens scaffolded as agent Apps:
src/agents/src/agents/Decision matrix (first gate that fires wins)
databricks.yml) → customWhy this fits AgentOps Stacks
detect_patterns → eval_gatepicks it up with zero workflow changes.add_agent.pywiring approach; renders graph/agent/tools/deps variants per pattern..agentops-stacks/manifest.ymlundersupervisor:— the same contract CI/tooling already read.bundle initinput — the choice is made post-scaffold, when it's meaningful.Files
plugin/skills/add-supervisor/SKILL.md— skill + Selection Matrix + per-pattern security postureplugin/commands/add-supervisor.md—/add-supervisorcommandplugin/skills/agentops-stacks/scripts/add_supervisor.py— engine (2 patterns).../scripts/supervisor_templates/— graph (custom, api), agent handler, toolstemplate/.../docs/supervisor-patterns.md.tmpl— pattern deep-dive in the rendered projecttemplate/.../.agentops-stacks/manifest.yml.tmpl— supervisor contract placeholderworkflows/single-account-single-agent.json— machine-readable decision matrix inrouting_notesplugin/skills/agentops-lifecycle/SKILL.md— Step 3.5 routing noteplugin/README.md,install_skills.sh— register the new skill/commandtests/test_add_supervisor.py— 12 unit tests (no CLI needed), all passingTesting
add_supervisor.pypass — build a fake project tree, assert both patterns wiredatabricks.yml/ manifest / layout correctly; no Databricks CLI needed.databricks.ymlparses (yaml.safe_load), renderedgraph.py/agent.pycompile, no leftover placeholders; an invalid--type agent_bricks_masis now rejected by argparse.test_create_project.pytests require Databricks CLI v1.1.0 (unrelated to this change; unaffected).This pull request and its description were written by Isaac.