chores(cti)!: remove dead infrastructure capabilities and add a CALDERA fact bridge - #28
Merged
Conversation
…oint conf/default.yml shipped a MITRE gateway hostname as the default api_base for both the llm and cti profiles. Both now ship empty and name MCP_LLM_API_BASE through a new api_base_env key, mirroring the existing api_key_env pattern, so no deployment's endpoint is baked into the repo. An unresolved api_base is now refused rather than passed through. dspy_lm_kwargs_from_settings drops a falsy api_base entirely, which leaves LiteLLM routing openai/* models to its built-in https://api.openai.com/v1, so a deployment that missed the env var would have quietly sent CTI prompts to a provider it never chose. resolve_llm_config now raises, matching the api_key check beside it and the identical check get_llm_provenance already performs on the cti path. Whitespace is stripped before normalization because normalize_openai_api_base treats a blank but truthy string as a real URL and would turn it into the relative path "/v1". Also removes the hostname from two tests and one code comment. BREAKING CHANGE: deployments must set MCP_LLM_API_BASE in plugins/mcp/.env or pin llm.api_base in conf/local.yml. Runs raise ValueError otherwise.
llm_defaults() was welded to the `llm` block while get_llm_provenance() took a profile argument, so the two paths each grew their own copy of the api_key_env and api_base_env lookups. LLMClient carried a third, with a comment noting it existed only because that path bypassed config.py. resolve_env_indirection() in llm_client.py is now the single implementation. It reads the *_env indirection for every field named in ENV_INDIRECT_FIELDS, consumes those keys so they never reach the settings dict handed to DSPy, strips values before normalization, and coerces provider with `or` rather than setdefault so an explicit yaml null no longer survives into the callers that branch on it. config.py gains profile_defaults(profile) and keeps llm_defaults() as a thin alias, so the cti profile resolves through exactly the same code as the llm profile. The api_key fallback in _openai_compatible_generate is removed: llm_cfg now arrives resolved, so the workaround it documented no longer applies.
The guard sat inside `if provider == 'openai_compatible'`, so a request body naming any other provider skipped it while model stayed openai/gpt-oss-120b. dspy_lm_kwargs_from_settings then dropped the empty api_base and LiteLLM routed the call, and the deployment's key, to its built-in https://api.openai.com/v1. provider reaches resolve_llm_config straight from the /execute request body with no whitelist, and conf/default.yml locks only model and api_base, so a lm_config of {"provider": "ollama"} was enough to bypass the check. Only the normalization is provider-specific; the requirement is not, and the ollama path needs a base to post to just as much. Normalizes provider before branching as well, so an explicit yaml null no longer reads as "not openai_compatible".
The override filter tested `value not in ("", None)` with no strip, so a
field containing only spaces counted as a real override. An api_base of
" " is truthy, normalize_openai_api_base turns it into the relative
path "/v1", and that satisfies the api_base guard, so the run failed
deep inside the HTTP client instead of at the resolver with an
actionable message. A pasted URL with a trailing space had the same
shape and became "https://host/v1 /v1".
Both modelSelector.vue and the chat sidebar bind these inputs with a
bare v-model, so nothing trims them client side either. The yaml and env
tiers were already stripped in resolve_env_indirection; this makes the
stated invariant true on all three.
All four dspy.LM() constructions in the plugin take their kwargs from
dspy_lm_kwargs_from_settings, which makes it the only place that can
guarantee the fallback never happens:
dspy_env.py:132 plan_execute.py:60
workflows/author.py:248 utilities/llm_client.py:172
resolve_llm_config guards the /execute path, but plan_execute and author
fall back to llm_defaults() when no lm_obj is passed, and llm_defaults()
returns an empty api_base by design. That was safe while yaml always
carried a base and is not any more. plan_execute then forwards the empty
DSPY_API_BASE to every spawned MCP subprocess, which repeats it.
Dropping the falsy api_base was what let LiteLLM apply its own default,
so this checks before assembling the kwargs rather than after. The
`and settings.get("api_base")` on custom_llm_provider goes away with it,
since a base is now guaranteed to be present.
llm_configured came from the api_key alone, which was a complete precondition until api_base moved out of the shipped yaml. Pull this branch onto a deployment whose .env carries only MCP_LLM_API_KEY and the splash page shows a green "configured" tag, omits the empty base rather than showing it blank, suppresses the warning banner, and then every run fails in the resolver. The suppressed banner also claimed that setting MCP_LLM_API_KEY was sufficient, which is no longer true. The page now reports the specific env vars that failed to resolve, so the diagnostic names the thing the operator has to fix instead of a fixed string. This is the one page whose stated job is telling the operator whether the plugin is wired up.
Nothing in the suite imported app/config.py, so both guards could be deleted with a byte-identical test result. These 26 cases pin the three merge tiers, the shared env indirection, and the two refusals. Mutation checked rather than assumed. Re-nesting the api_base guard under `if provider == "openai_compatible"` fails the three test_api_base_required_for_every_provider cases; disabling the dspy_env guard fails all four TestDspyLmKwargs refusal cases; restoring either makes them pass again. _load_defaults is stubbed via a fixture so the tests read neither conf/default.yml nor the developer's .env.
The env var is required now but appeared only in .env.example. Install step 5 still said "configure model credentials through the UI or environment/local config" without naming either variable, so a new operator had no way to reach a working install from the README alone. The local.yml advice was worse than absent. load_config returns local.yml INSTEAD of default.yml with no merge, so an operator following the old "pin llm.api_base in conf/local.yml" wrote a two-line file, lost api_key_env with it, and was then told to set MCP_LLM_API_KEY, which they had already set correctly. Both the yaml comment and the README now say to copy the whole file and spell out that local.yml replaces rather than merges. The README sample also nested the llm block under a top-level `mcp:` key that nothing reads, and omitted the *_env keys entirely. Corrected to the shape config.py actually loads.
Several comments added on this branch restated what the code already says or carried three lines of history where one would do. Trimmed to the reason a reader cannot infer from the line below it. No behaviour change; tests unchanged and still passing.
Consolidating both fields under one precedence inverted api_key. On main a named api_key_env always won; the shared resolver let a yaml literal shadow it, so a key written to conf/local.yml silently pinned itself and rotating MCP_LLM_API_KEY did nothing. Secrets now resolve env-first, endpoints stay yaml-first. Verified against main's semantics across all four cases. Also covers the yaml-side strip, which no test reached.
The Save button posts api_key with every save and set_config wrote the payload verbatim, leaving the key in plaintext on disk beside tracked config. Secrets are stripped from every section on write, so a save also clears a key an earlier build left behind. Reload now goes through reload_config: load_config is lru_cached, so the old call read back the pre-write contents.
globalConfig and every saved endpoint profile were serialized to localStorage verbatim, so the key sat in plaintext readable by anything on the origin. Stripping at the storage boundary covers both, and load purges a key an earlier build already wrote. The Save payload no longer sends api_key either, since set_config refuses to persist it.
get_llm_provenance was the last resolver still gating its api_base raise on openai_compatible, contradicting the invariant the other two now hold and test for. author.run() prechecked only api_key, so a partial lm_obj failed at dspy.LM() after the AsyncExitStack had already spawned every MCP subprocess. Both credentials are now checked together, before that. Adds provenance guard tests and the first coverage of the readiness payload, including that neither leaks the key.
It still described two credentials and claimed the UI never writes to disk. Now names MCP_LLM_API_BASE, records that set_config strips secrets from conf/local.yml, and states the per-field precedence.
requirements.txt asked for an unpinned mcp, so a fresh install resolves to 2.x, which dropped mcp.server.fastmcp in favour of MCPServer. Both mcp_server.py and app/mcp_server.py still import FastMCP, so every stdio subprocess died at import and the client reported only "Connection closed" from the closed pipe. 1.29.1 is the newest release that still ships fastmcp. Pinning restores the servers without a port; migrating to the 2.x server API is a separate piece of work.
author.py and plan_execute.py called set_tracking_uri, set_experiment and dspy.autolog at module scope. set_experiment is a network round trip, so importing either module blocked whenever the tracking server was down, which put both workflows out of reach of a test, a linter, or anything running before MLflow comes up. llm_client.init_mlflow already documents the rule these two broke. Both now initialise lazily on the first run() call, which makes the author credential precheck testable for the first time.
globalConfig.apiBase is restored from localStorage, and applyServerDefaults only fills it when blank, so a stored value survives forever and ships as a UI override on every /execute, where it outranks the env-resolved base. A browser that cached the old vendor endpoint kept reaching it after the URL was removed from the repo, and editing .env did nothing for that user. The stored blob now carries a schema version, and loading a pre-v2 one drops the ambient apiBase so the server default applies again. Named endpoint profiles keep theirs: those are explicit user artifacts.
The first pass walked only the top level of each section. set_config
accepts arbitrary top-level keys, and a caller posting the documented
{"config": {...}} envelope produces {"config": {"cti": {api_key: ...}}},
which the one-level scrub walked straight past. Confirmed against a real
local.yml that still held a key after a save.
Now recursive through dicts and lists.
The range plugin does not ship with this open-source build, so the three helpers behind GET /plugin/mcp/features all resolved against a service that is never registered and an import that cannot succeed. Removes _range_supported_providers, _range_images_for_provider and _range_feature_catalog, along with _count_catalog_items, which had no other caller. The features() response loses its top-level "range" key. BREAKING CHANGE: GET /plugin/mcp/features no longer returns a "range" key. No in-repo consumer reads it.
…ng stages
cti_deploy_spec.py existed only to build a POST body for
/plugin/range/onprem/manage/deploy, and read its image catalogs and
Ansible playbooks straight out of plugins/range, which is absent here.
The ae-e2e workflow loses deploy_spec, deploy, meta and sandcat. Those
four stages hardcoded plugins/range paths and a microvm runtime base, so
none of them could run. STAGES is trimmed to match the surviving methods,
since run() dispatches through getattr(self, f"_stage_{name}") and a
stale entry would raise AttributeError at request time.
preflight no longer requires range_svc. The topology stage keeps
build_range_topology but passes an empty image catalog.
BREAKING CHANGE: the ae-e2e workflow no longer provisions infrastructure.
It runs preflight, cti, topology, agents, adversary, operation,
detections and report against an already-running Caldera.
…ne server Five of the thirteen tools existed only to drive plugins/range over HTTP: range_capabilities, wait_for_range, import_ansible_feature, deploy_range and apply_features. Every one targeted a /plugin/range/* endpoint that does not exist in this build. Removing them orphans _range_get_json and _range_post_json, so those go too. The image catalog helpers stay for now because build_topology and fuse still thread a catalog into build_range_topology. The server now exposes seven tools: ingest_cti, build_topology, fuse, refine_topology, run_operation, validate_detections and wait_for_agents. BREAKING CHANGE: the cti_pipeline MCP server no longer exposes range provisioning tools.
image_candidates was the only substrate-derived field in the topology SDO, and it resolved against plugins/range/conf/onprem_images.yml. With no range plugin the catalog never loaded, so the field was already always empty and stage 4 logged "image catalogs absent" on every run. build_range_topology and derive_hosts lose their images_catalog parameter, and _enrich_topology_with_ae_plan and _process_bundle lose theirs. The catalog loaders go with them in both stage 4 and the cti_pipeline MCP server. Everything else in the topology stays: hosts, subnets, roles, platforms, services, software, vulnerabilities, attack surface and identities are all inferred from the STIX bundle and need no deployment substrate.
The planner prompts described a Range execution contract covering providers, hypervisors, image catalogs, Ansible feature playbooks and microvm substrate status. None of that context can ever be populated without the range plugin, so the model was being told to plan against capabilities that do not exist. Rewrites the two agent docs and the operation-context formatter around what this build actually does: ingest CTI, infer topology, build an adversary, run an operation against agents that have checked in. The agent_start and identity_options blocks are kept. They read as range context because of where they sit, but both are generic operator preferences consumed independently of any deployment substrate. Replaces tests/test_range_integration.py with tests/test_discovery_and_tool_merge.py, preserving the generic coverage of discover_mcp_servers and _safe_load_metadata. The new test asserts the registry is exactly caldera_core plus cti_pipeline.
The section drove /plugin/range/onprem/providers, /images, /features and /microvm/substrate-status. With no range plugin every one of those calls failed and the panel rendered "Range unavailable". The identity checkboxes and the Agent selector were nested inside v-if="buildRange" but are not range controls: prompts/plan_execute.py reads identity_options and agent_start on their own. Both are re-parented into the CTI section so they stay reachable, and the CTI section is renamed from "CTI / Range". Drops the range refs, computed properties, loaders and watchers from the script, the range keys from the emitted workflow context, and the now unused .provider-row styles.
Comment-only and documentation change. No executable line moves. Many extraction rules were justified by naming the range plugin as the consumer, for example "the range plugin's LLM deploy step needs the version to pre-stage binaries". The rules themselves are sound and stay as they are, so the comments now describe what the data is for rather than which absent plugin wanted it. The README loses the Range Integration section, the range-aware planning bullet, the range entry in the example plugin list, and the range troubleshooting note.
The SDO describes an inferred victim topology, not a deployable range, so the vocabulary no longer matches what it holds. x-cti-range-topology -> x-cti-topology x-cti-range-host -> x-cti-host build_range_topology -> build_topology host name prefix "range-" -> "host-" The host prefix had three coupled sites with nothing holding them together: derive_hosts builds the name, derive_networks independently rebuilds it to map back to an infrastructure id, and stage 4's _host_key strips it so AE-plan hostnames match. Drift between them does not raise, it silently moves every host into "unanchored-net" with a null anchor_identity, so the topology still looks well formed. Both producers now read HOST_PREFIX. _host_key still strips "range-" as well, so topologies written by an earlier build keep matching. Adds test_host_names_stay_anchored_to_their_infrastructure_sdo, which asserts on the identity anchoring rather than on membership. Verified it fails when the drift is injected and passes when it is not; asserting on membership alone does not catch this. BREAKING CHANGE: consumers matching the x-cti-range-topology or x-cti-range-host type literals, or the "range-" host prefix, must be updated. The measuring-stick fixture is updated in this commit.
main landed the LLM credential consolidation as a squashed PR (#26) while this branch carried the same work unsquashed from its base, so every config conflict was one change in two forms. Main's form supersedes and was taken for app/config.py, conf/default.yml, app/workflows/author.py, the secret-scrubbing helpers in app/mcp_api.py, the local.yml overlay note in README.md, and the three config test files. Range removal is preserved everywhere the two overlapped. Re-applied on top of main's versions: app/config.py tool-surface comment drops range app/workflows/plan_execute.py optional_servers drops range gui/views/mcp.vue pipeline prose drops the deploy spec Also drops a range reference in data/measuring-sticks/README.md that the earlier rename missed. Verified after resolution: no range outside Python's range(), the back-compat prefix strip, and English usage; mcp_server.py still exposes seven tools; cti_deploy_spec.py and test_range_integration.py stay deleted; compile and the E9,F63,F7,F821 gate clean. main's own config tests give identical results before and after the merge.
The topology SDO described machines to stand up. Core CALDERA has nothing that reads it, so it was a deployment blueprint with no consumer. Deletes cti_topology_inference, cti_pipeline_stage4_topology, cti_pipeline_stage3, cti_infra_aggregation, cti_refinement and cti_knowledge_graph, plus the build_topology and refine_topology MCP tools. Stage 3 emitted infrastructure hypotheses that only stage 4 read. The pipeline is now stage 1 to stage 2: raw to IR to STIX. Entity extraction stays, retargeted at CALDERA facts in the next commit. BREAKING CHANGE: the x-cti-topology SDO, outputs_topology/, the cti_pipeline_build_topology and cti_pipeline_refine_topology tools, and the stage3-infra and stage4-topology pipeline steps are all removed.
Extracted hosts, accounts and domains had no consumer once topology went. They now become a CALDERA fact source, so an operation runs on values the report named instead of placeholders. Traits are the ones stockpile abilities reference: remote.host.fqdn, remote.host.name, remote.host.ip, domain.user.name, domain.user.password, host.user.name, target.org.domain. Two filters keep the output usable. Values that are not hostname shaped are dropped, since extractors surface prose like "Internet-facing Exchange server" that would expand into a command and break it. Names resolved by DNS dedupe case-insensitively; usernames and passwords do not. Only a password the report actually stated is emitted, never a redacted one. bundle_to_facts is pure and testable without CALDERA. build_source and the cti_pipeline_build_source tool are the adapters that need it. On the tracked BlackCat bundle this yields 25 facts: 11 hostnames, 7 addresses, 4 domain accounts, 2 local accounts, 1 domain.
The identity checkboxes and the Agent selector were provisioning inputs. Domain, domain users and local users told a range builder what accounts to create; the Agent mode and CTI entry host told it which implant to install and where. With no provisioning step none of it can be acted on. They had one consumer: the prompt formatter stringified them into "Agent start preference: ..." and "Identity/domain options: ...". No tool read them. Keeping them fed the model preferences it could not honour, which invites it to claim work it did not do. Also drops provisioning language from the model-facing prose in the RAG capability description, the shared prompt fragments and the Workflow docstring. The workflow context now carries only CTI and RAG selection, which is what the pipeline is for: better grounding for the agent's reasoning.
Review of the branch turned up four real defects. build_source could never succeed. SourceSchema's pre_load stamps every fact with input_data["id"], so a body without one raised KeyError before validation and returned a 500. The id is now minted client side, and two tests load the body through CALDERA's own schema. remote.host.ip is a live target: stockpile nmaps it and SMB-mounts it. A report names the attacker's C2 and other victims beside the estate, so the tool now previews by default and reports which addresses are publicly routable. Committing takes an explicit flag. start_stage raised a bare ValueError from list.index for an unknown stage, becoming a 500, while only_stage returned a clear message. Both paths now match. Removes provisioning residue the deletions missed: two README bullets promising infrastructure provisioning and agent placement, a stale REST docstring documenting four removed parameters, an orphaned comment block, and the range deploy shape in the measuring stick, which was reachable by a model through cti_pipeline_fuse.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Removes capabilities that could not run. They resolved against a
rangepluginthat is not present, so every one of these paths failed at runtime:
_range_feature_catalogimportedplugins.range.app.onprem_svc, which doesnot resolve
deploy_range,wait_for_range,apply_featuresandimport_ansible_featurePOSTed to/plugin/range/*endpoints that 404RuntimeError: OnPremService not foundimage_candidateswas always empty; stage 4 logged "image catalogs absent"on every run
test_range_integration.pyfailedThe topology layer existed to feed those paths. Nothing in CALDERA core reads a
topology object, so it went with them. CTI ingest now produces context for agent
reasoning rather than a deployment blueprint.
Removed
Build Range section, provisioning workflow stages and prompts
cti_topology_inference,cti_pipeline_stage4_topology,cti_pipeline_stage3,cti_infra_aggregation,cti_refinement,cti_knowledge_graphreads
CTI entry host) and their prompt plumbing. These had one consumer, the prompt
formatter, and no tool acted on them.
Added
app/utilities/cti_caldera_facts.pymaps a STIX bundle onto CALDERA factsusing the traits stockpile abilities actually consume
cti_pipeline_build_sourceMCP toolThe pipeline is now stage 1 to stage 2: raw to IR to STIX. The cti_pipeline MCP
server exposes 6 tools.
build_sourcepreviews by default.remote.host.ipis a live target forstockpile abilities that nmap and SMB-mount it, and a report names the
attacker's C2 beside the victim estate, so the tool reports which addresses are
publicly routable and requires an explicit
commit=true.Type of change
Breaking: the custom topology SDO,
outputs_topology/, thecti_pipeline_build_topologyandcti_pipeline_refine_topologytools, thestage3-infraandstage4-topologysteps, and therangekey onGET /plugin/mcp/featuresare all removed.How Has This Been Tested?
Full suite against main in an identical harness, excluding the four files that
need a live CALDERA on localhost:8888:
Zero regressions. The 4 removed failures are
tests/test_range_integration.py,already failing on main. Remaining failures are the five modules missing from
origin/CTIand are identical on both sides.17 new tests, all passing.
build_sourceverified end to end against a running CALDERA: preview returned25 facts and flagged 2 routable addresses, commit created the source, and
GET /api/v2/sources/<id>confirmed 25 facts with origin IMPORTED. Test sourcedeleted afterwards.
Also verified:
compileallclean,flake8 --select=E9,F63,F7,F821clean, 26 MCPtools register, all 25
hook.pyroutes resolve to live methods, both changedSFCs compile under magma's own compiler.
Not verified: stage 1 IR extraction end to end, which needs an LLM that is not
configured in this environment.
Checklist: