[samschifman]

BPMN automation templates: extraction, review, synthesis and publication (interim)

Offline suites, UI builds, chart lints and shell tests are green. The compose proof, the Java decision-service tests and a full cluster deploy and run at this branch's images have been completed; results and the honest gaps are in Verification at the end.

What this adds

Carewright gains a second computable output from guidelines: automation templates, BPMN process definitions expressed as a JSON Process IR, compiled deterministically to Kogito-valid BPMN and validated through a static ladder that ends at the Kogito engine.

• cpg-ingester identifies care processes in a CPG, has the LLM express each as Process IR, compiles and validates it, lets the informaticist review it in a new Automations tab (diagram, findings, parameter defaults, escalation acknowledgement), and publishes accepted templates alongside DMN and recommendations.
• acp-writer registers templates in its decision engine, selects templates for the activities of a generated care plan, has the LLM bind their parameters from the plan and DMN audit trail, validates every instance with the same ladder plus a grounding validator that proves each element came from a template, shows the result read-only inside the activity in the review UI and on the persisted plan, and on approval writes it to the FHIR bundle as a DocumentReference plus Task and publishes it to the automation service.
• automation is a new receive, validate and store service. It does not execute processes. Execution remains a later phase.
• shared carries the contract: Process IR, expression grammar, compiler, validation ladder, capability catalog, AutomationTemplate, and a shared BPMN viewer for both UIs.

This is the Phase 1–3 milestone of the design plus the fixes from the first end-to-end demo. The design is captured point-in-time in dev_docs/design/automation-design.md; the user-facing overview is docs/automation.md. Read those two in that order.

What runs where

Environment	Behaviour
Compose, default profile	Monolith path. Templates register on the ingester's artifact notification. Approval publishes to the automation service with a shared dev token.
Compose, orchestrated profile	SonataFlow dev-mode plus split pods. Same behaviour; the fhir-server service now carries the automation URL and token.
Cluster	Inert by design. Extraction, template registration (via deploy/load-published-artifacts.sh) and read-only review all run. The automation service is not deployed; approvals record publication state unconfigured (or not-applicable when no automation is enabled) and the CarePlan is written exactly as before. No retry threads are started. The LLM gateway can be the in-cluster MaaS or the external EnMaaS gateway; both are configured through cluster.env and the sandbox egress policies allow whichever host is configured.

Behaviour changes to be aware of

• Every CPG upload now runs BPMN extraction when the item identifier finds process items. There is no opt-out in this PR; extraction adds model calls and time per upload.
• AUTOMATION_SERVICE_URL no longer defaults to a localhost address. Unset means publication is skipped with state unconfigured; a plan with no automations records not-applicable. Both states are terminal.
• acp-writer moved sentence-transformers to an optional local-embeddings extra. Local setups using that provider need pip install -e ".[local-embeddings]".
• shared gains one runtime dependency, regex, for timeout-bounded constraint matching.
• The artifact loader treats a rejected template as a warning and continues; only guideline, recommendation and DMN failures abort it. The decision-service image must be rebuilt at the release tag because registration calls its new /jit/bpmn/validate route.
• The retry-publication route returns a status view only; it used to return the whole job including the bundle.
• Deploy fixes found while running this on the cluster: the platform MinIO manifest pins a public image (the quay.io/minio/minio image was withdrawn) and uses a Recreate strategy so a rollout can reuse its single volume; the OpenShell LLM policies allow the configured gateway host on 443; an empty MAAS_ROUTE_SEGMENT no longer produces a trailing slash in the LLM base URL; the MCP proxy image pins mcp<2 (2.x removed the module it imports; images built before the pin crash at start); async pod callbacks to the SonataFlow engine retry three times instead of once; the BFF detects the care-plan gate from the workflow's own marker instead of inferring it from data presence, so it no longer offers Approve during FHIR review rounds.

New configuration

Variable	Where	Default
AUTOMATION_SERVICE_URL	acp-writer fhir-server / monolith	unset (publication skipped)
AUTOMATION_SERVICE_TOKEN	acp-writer and automation service	compose dev value only; never a cluster credential
BPMN_PREFLIGHT_URL	cpg-ingester	unset (optional Kogito preflight)
ACP_JOB_STORE_DIR	acp-writer	data/automation-publications when no PHI store
AUTOMATION_STORE_DIR	automation service	data/automations when no PHI store
ACP_AUTOMATION_DEFAULT_MONITORING_PERIOD	acp-writer	P90D

Security

A review of this branch (working/security/security-review-2026-10-12.md, not in the repo) listed findings introduced by the feature. Status in this PR:

• Fixed with tests: cross-bucket read through both BFF artifact proxies (H1); LLM-controlled process and template ids reaching paths and object keys (M2); unbounded regex in constraints (M3); unbounded expression parsing (M4); super-linear fuzzy grounding (M5).
• Partially mitigated: the automation service's write routes now require a shared token with constant-time comparison, and its host port is no longer published by default (C2). List and get routes stay unauthenticated on the compose network; the service is documented as trusted-local only until Phase 5 adds proper service identity.
• Accepted for this PR, named here on purpose: PHI reaches MLflow spans from the job store, publisher and ladder (C1). The pre-existing autolog spans already carry bundles, so this is treated as one general problem and will be fixed separately, not piecemeal here. Service-to-service auth on approve, reject and template registration (H2) stays with the deferred backlog. Minor items (M7, M9, M12, M13, L1–L5, L8–L10) are recorded with reasons; L5 (non-finite numeric literals) is a small parser follow-up.

Known gaps and follow-up PRs, in order

1. Phase 4: apply_edits at the review gate, the parameter form, incomplete-instance handling, composition and pruning, analyst visibility of automations.
2. Phase 5: automation service chart, NetworkPolicy and OpenShell policy on the cluster; ordering guards and set semantics; service identity beyond a shared secret.
3. Repair-loop observability on the orchestrated profile: persist each creator/reviewer attempt to the artifact store (today a temp dir is deleted per step), start MLflow with --allowed-hosts so pod traces are no longer rejected, and include prior attempts' findings in the repair feedback. Needed before any further prompt tuning.
4. Terminology lookup retry for alternate lab phrasings; age derivation in the binder; registry rehydration on decision-engine restart (today: rerun the loader).
5. Phase 6 benchmarks and prompt baselines.
6. Phase 7 remaining documentation.
7. The deferred security backlog, starting with PHI in traces.
8. Sync the Carewright operator's embedded SonataFlow workflows with the component workflows changed here (six lines; the operator was merged from main after this work).
9. Cluster hygiene found during verification: make the Medplum loader idempotent against an existing database (today a Helm hook failure blocks the mock-EHR install on redeploy); fix cpg-ingester/deploy/verify.sh's published-BPMN check, which matches the wrong path prefix and reports a false negative; document in deploy/README.md that setup-secrets.sh writes LLM_API_KEY while the in-cluster MaaS ExternalModel needs a Secret keyed api-key, and that an OpenShell sandbox is restarted by re-running the component deploy, never by deleting its pod (the pod returns without its supervised process); have the deploy scripts wait on their background sandbox-create sessions before exiting.

Verification

Green on the merged tree: shared, cpg-ingester, acp-writer and automation suites; all three UI typechecks, tests and builds; five Helm chart lints; loader and slice-access shell tests; golden regeneration check; mvn test in acp-writer/decision-service (11 tests).

Compose, orchestrated profile, all pods rebuilt: one guideline through both gates; one care plan generated, approved and written to Medplum; the attached automation was incomplete (a required interval left unbound by the guideline), so the approval recorded not-applicable with no publication attempt and no retry thread.

Cluster, sschifma-cpg-acp, images at c98ee9e, EnMaaS as the LLM gateway: deploy-all.sh and the component verifiers pass; all nine OpenShell sandboxes answer their routed health check; one guideline ingested through both gates and delivered; the loader registered the DMN models and the accepted template (one template was rejected by the engine's static ladder, which the loader now reports as a warning and continues past); one care plan generated with the template attached read-only, approved, active in Medplum, publication not-applicable, no retry thread; a second plan set to entered-in-error through the FHIR status endpoint; a fresh decision-engine sandbox starts with an empty registry and the loader re-registers the template.

What the live runs did not exercise, stated plainly:

• The unconfigured publication state and the published token path. Neither compose nor cluster run produced an enabled, fully bound automation from the demo guidelines (that needs the Phase 4 parameter form). Both states are covered by the acp-writer unit tests, and published by the in-process slice test against the real automation service app.
• A reject decision at the review gate. The workflow's gate accepts approve and request_changes only; rejection of a persisted plan goes through the FHIR status endpoint, which writes the withdrawal tombstone. The description above reflects that.

Suggested merge: squash.

🤖 Generated with Claude Code and Codex

[jaideepr97]

thanks sam, approving with a couple minor comments

[jaideepr97]

what's the reasoning behind this change specific to automations?

[samschifman]

Not automation-specific as such: this branch made the cpg-ingester UI consume the shared UI package (shared/ui, which now carries the BPMN viewer both UIs use). Its Containerfile copies shared/ui and so must build from the repository root, the same way the acp-writer UI entry already does. The matrix entry still pointed at cpg-ingester/ui, and the first publish run on this branch failed for that image until it was fixed. The cluster BuildConfig in cpg-ingester/deploy/setup-images.sh got the same root-context change.

[jaideepr97]

is there a specific reason for restricting mcp to the 1.x?

[samschifman]

Yes: the unpinned install started pulling mcp 2.x, which renamed FastMCP to MCPServer and removed the mcp.server.fastmcp module that mcp_proxy.py imports, so the freshly built MCP image crash-looped on the cluster at startup. The pin keeps the proxy working as written; nothing on the branch touched that code. Migrating mcp_proxy.py to the 2.x API is a small follow-up, tracked in the PR’s follow-up list. Worth knowing for the operator too: any image of this proxy built after the 2.x release has the same problem without the pin.