Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified assets/MessageFoundry-Secure-Development-Standards.pdf
Binary file not shown.
Binary file modified assets/docs/MessageFoundry-Architecture.pdf
Binary file not shown.
Binary file modified assets/docs/MessageFoundry-CI-Quality.pdf
Binary file not shown.
Binary file modified assets/docs/MessageFoundry-Configuration.pdf
Binary file not shown.
Binary file modified assets/docs/MessageFoundry-Deployment-Guide.pdf
Binary file not shown.
Binary file modified assets/docs/MessageFoundry-Install-Guide.pdf
Binary file not shown.
Binary file modified assets/docs/MessageFoundry-System-Requirements.pdf
Binary file not shown.
Binary file modified assets/docs/MessageFoundry-Throughput-and-Capacity.pdf
Binary file not shown.
Binary file modified assets/docs/MessageFoundry-User-Guide.pdf
Binary file not shown.
2 changes: 1 addition & 1 deletion assets/docs/_md/Architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ configured per connection, with an off-loopback plaintext listener refused at st
|---|---|
| `messagefoundry.config` | Connector models (`models.py`) + code-first wiring registry/loader (`wiring.py`) + service settings (`settings.py`) |
| `messagefoundry.parsing` | Tolerant peek (python-hl7) + strict validate (hl7apy); parse tree (`tree.py`) and the `Message` transform model (`message.py`); pure non-HL7 codecs — X12 EDI (`x12/`), DICOM headers/SR over pydicom (`dicom/`, ADR 0025), and the base64 binary-carriage codec (`binary.py`, ADR 0028) |
| `messagefoundry.store` | Durable message store / **staged queue** (one `queue` table, `stage` = ingress\|outbound), SQLite WAL; every receipt logged with a disposition that flows with the message. `Store` protocol + `open_store` factory in `base.py`; production SQL Server backend in `sqlserver.py` |
| `messagefoundry.store` | Durable message store / **staged queue** (one `queue` table, `stage` = ingress\|routed\|outbound), SQLite WAL; every receipt logged with a disposition that flows with the message. `Store` protocol + `open_store` factory in `base.py`; production server-DB backends in `postgres.py` and `sqlserver.py` |
| `messagefoundry.transports` | Inbound & outbound connections (MLLP, file, X12 raw-TCP, DICOM C-STORE SCP inbound over pynetdicom — ADR 0025, …), resolved through a registry (`base.py`) — never special-cased in `pipeline/` |
| `messagefoundry.anon` | Deterministic, secret-per-dataset pseudonymization / de-identification (fail-closed; ADR 0030) — exposed to the tee (`anonymize-captures`) and the test harness |
| `messagefoundry.pipeline` | Per-message routing/handling (`RegistryRunner` in `wiring_runner.py`) + per-inbound-connection supervision (`engine.py`); offline `dryrun.py` |
Expand Down
2 changes: 1 addition & 1 deletion assets/docs/_md/CI-Quality.md
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ Ordered by anti-slop leverage, not effort (build placement per §5). **✅ = shi

*All gates are now shipped.* The ruff sweep (#10, \#1047) was run in a quiescent-worktree window (a 100+-file import sort would collide with in-flight parallel sessions) after pruning the stale worktrees to a minimal set. Mutation and diff-coverage were built *blind via CI* — verified by their own gate runs, since this repo's sessions can't stand up a local venv (see \#1040).

**Rollout record (measured 2026-07-13 — how the \#1047 sweep was executed):** `B,C4,SIM,UP,I` = **853 violations** (238 `B008` FastAPI false positives to exclude; 111 `I001` repo-wide import reorder); `C901` = **85 hits**. Safe rollout: (a) exclude framework-idiom rules (`B008` on `api/`); (b) **grandfather** the existing backlog so the *required* gate stays green (per-file-ignores / ratchet — new code only); (c) run the repo-wide import sort as a **dedicated pass when parallel worktrees are quiescent** — a 100+-file sweep conflicts with in-flight sessions; (d) keep `C901` **advisory**. (The built coverage/mutation gates install their tools CI-side via an ephemeral `uv pip install`, so they needed **no** `requirements.lock` change — DEP-1 unaffected.)
**Rollout record (measured 2026-07-13 — how the \#1047 sweep was executed):** `B,C4,SIM,UP,I` = **853 violations** (238 `B008` FastAPI false positives to exclude; 111 `I001` repo-wide import reorder); `C901` = **85 hits**. Safe rollout: (a) exclude framework-idiom rules (`B008` on `api/`); (b) **grandfather** the existing backlog so the *required* gate stays green (per-file-ignores / ratchet — new code only); (c) run the repo-wide import sort as a **dedicated pass when parallel worktrees are quiescent** — a 100+-file sweep conflicts with in-flight sessions; (d) keep `C901` **advisory**. (The built coverage/mutation gates install their tools from a **hash-pinned CI toolchain lock** — `pip install --require-hashes -r ci/locks/ci-quality.lock`, exported from `pyproject.toml`'s PEP 735 `[dependency-groups].ci-quality` — because a *version* pin alone does not satisfy Scorecard's `PinnedDependenciesID`. `pyproject.toml` therefore **did** change, and the lock sits **inside** the DEP-1 export machinery rather than beside it: a hash-pinned toolchain kept outside it rots into a pinned, stale, unpatched one. `requirements.lock` itself is unaffected.)

### A.4 Documented caveat — solo-maintainer review

Expand Down
8 changes: 5 additions & 3 deletions assets/docs/_md/Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -867,7 +867,7 @@ silences an event you didn't name. Matching is pure config (no code/`eval`).

| Key | Type | Default | Notes |
|---|---|---|---|
| `event_type` | str | `any` | match this event: `any`, `connection_stopped`, `queue_buildup`, `storage_threshold`, `cert_expiry`, `connection_error`, `message_stall`, `integrity_drift`, or `gcm_invocations` |
| `event_type` | str | `any` | match this event. The validator accepts `any` plus exactly these **18**, and **rejects anything else at config load**, so a typo is loud rather than a rule that silently never matches: `backup_failed`, `bootstrap_admin_expiring`, `cert_expiry`, `connection_error`, `connection_stopped`, `content_match`, `dr_activated`, `gcm_invocations`, `integrity_drift`, `lane_stuck`, `leadership_acquired`, `message_stall`, `queue_buildup`, `rcsi_off_degraded`, `saturation`, `secret_rotation`, `storage_threshold`, `update_available` |
| `connection` | str (glob) | `*` | glob over the connection name (e.g. `OB_*`, `IB_ACME_*`) |
| `min_depth` | int | _unset_ | `queue_buildup` only — match only when pending depth is at/over this |
| `min_oldest_seconds` | num | _unset_ | `queue_buildup` only — …or the oldest pending message has waited at least this long |
Expand Down Expand Up @@ -970,8 +970,10 @@ the secret-side twin of [`[cert_monitor]`](#cert_monitor). A TLS cert carries it
long-lived secret (the **store data-encryption key** today; connector credentials in a future
`SecretProvider`) has none, so a stale key can sit unrotated with no in-engine signal. The engine
periodically compares each tracked secret's operator-recorded **last-rotated date** against its **max
age** and raises a **`secret_rotation_due`** alert (an [`[alerts]`](#alerts) event — route it with a
`[[alerts.rules]]` rule) when it is overdue or within `warn_days` of due. It reads only the rotation
age** and raises a rotation-due alert (an [`[alerts]`](#alerts) event — route it with a
`[[alerts.rules]]` rule as **`event_type = "secret_rotation"`**, which is the wire name the validator
accepts; the longer `secret_rotation_due` is the internal `AlertSink` method name and is **rejected at
config load** if you write it in a rule) when it is overdue or within `warn_days` of due. It reads only the rotation
**dates** you configure here — **never any secret value** (PHI-free). This is a *reminder*, not
enforcement: it never rotates a key or blocks startup (run `rotate-key` to rotate the store DEK). Under
`[security].enforcement = enforce`, a store DEK past `store_key_max_age_days + enforce_grace_days`
Expand Down
9 changes: 7 additions & 2 deletions assets/docs/_md/Deployment-Guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ authentication on the channel · **Egress gate** = the `[egress]` allow-list tha
| **Raw TCP source** | `[inbound].bind_host` = `127.0.0.1` | **No** — plaintext only | None | — | **Yes** — non-loopback plaintext refused (`check_tcp_tls_exposure`, PR #558); no TLS to enable, so keep loopback / firewall-segment / proxy-terminate |
| **X12 source** (ISA/IEA framed) | `[inbound].bind_host` = `127.0.0.1` | **No** — plaintext only (same socket plumbing as raw TCP) | None | — | **Yes** — non-loopback plaintext refused (`check_tcp_tls_exposure`, PR #558); keep loopback / firewall-segment / proxy-terminate |
| **File source** | local filesystem | n/a (no network) | n/a | — | n/a |
| **Database poll source** | connects to `[store]` DB | **Yes** — inherits the store DB connection TLS (`[store].encrypt` default true) | Store DB auth | `[egress].allowed_db` | n/a (outbound DB connection) |
| **Database poll source** | **binds no socket** — dials an operator-configured DB host: its own `server` (required) and `port` (default `1433`). **Not** the `[store]` database, and it inherits nothing from `[store]` | **Yes, per connection** — its own `encrypt` (default true) + `trust_server_certificate` (default false) on the default `dialect="sqlserver"`. On `dialect="generic"` TLS is the ODBC driver's own keyword and is **not** engine-enforced | its own `auth` / `username` / `password` | `[egress].allowed_db` | n/a (outbound DB connection) |

### Outbound (the engine dials a destination)

Expand Down Expand Up @@ -339,11 +339,16 @@ does not resolve to a listed `host:port` makes the config **fail at load / reloa
|---|---|
| `[egress].allowed_mllp` | MLLP destinations |
| `[egress].allowed_tcp` | raw TCP **and** X12 destinations |
| `[egress].allowed_http` | REST, SOAP, and alert-webhook destinations |
| `[egress].allowed_http` | REST, SOAP, **FHIR**, **DICOMweb (STOW-RS)** destinations, the **SMART token endpoint**, and the read-only `fhir_lookup` |
| `[egress].allowed_db` | DATABASE destination + the DB poll source |
| `[egress].allowed_remote` | RemoteFile SFTP/FTPS/FTP (source + destination) |
| `[egress].allowed_file_dirs` | File destination directories |

**The alert sinks are *not* on this table and are not covered by `[egress]`.** The webhook and SMTP
*alert* sinks carry no PHI bodies and keep their **own** host allow-lists —
`[alerts].webhook_allowed_hosts` and `[alerts].smtp_allowed_hosts`. Populate those separately; an
`[egress].allowed_http` entry does nothing for a webhook alert.

For an off-loopback deployment, populate the lists you use so a transform cannot exfiltrate to an
unapproved address. The **global deny-by-default toggle is built** — `[security].block_unlisted_outbound`
— and on a **PHI instance the serve gate turns it on for you** unless you set it explicitly, so a
Expand Down
7 changes: 4 additions & 3 deletions assets/docs/_md/Install-Guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,10 @@ Three ownership tiers — keeping them separate is the whole design:
- **Python 3.14+** on each engine host (the engine requires 3.14+).
- **git**, plus a **private git host** — GitHub (private repo), GitLab, Azure DevOps, Bitbucket, or a
self-hosted server. Nothing about MessageFoundry requires a public repo; your config repo is yours.
- A source for the engine wheel: **public PyPI** is the target distribution channel; until the project
publishes its first release, the signed **GitHub Release wheel** or an **internal package index**
(Artifactory, Azure Artifacts, a private PyPI) serves the same role.
- A source for the engine wheel: **public PyPI** is the distribution channel and the recommended
install (`pip install "messagefoundry==<version>"`). For estates that cannot install from the
public index, the signed **GitHub Release wheel** or an **internal package index** (Artifactory,
Azure Artifacts, a private PyPI) mirroring it serves the same role.
- Administrator/elevation on the host if you will install the engine as a Windows service (see
[SERVICE.md](SERVICE.md)).

Expand Down
6 changes: 4 additions & 2 deletions assets/docs/_md/System-Requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,9 @@ only the host can. On a host that does not provide it:
operator can satisfy on Windows, so refusing by default would break deployments over something they
cannot change. An estate that has standardized on confidential-computing hosts can make the missing
declaration fatal with `[security].require_memory_encryption_declaration = true` (default `false`).
Loopback and synthetic instances are unaffected and silent. See
Loopback and synthetic instances are **silent — but not exempt**: the warning is keyed on exposure
to keep startup output off unwatched deployments, not because the risk differs, and
`GET /security/posture` states the property for every instance. See
[CONFIGURATION.md](CONFIGURATION.md) `[security]` and
OFF-LOOPBACK-DEPLOYMENT.md.

Expand Down Expand Up @@ -123,7 +125,7 @@ SEV-SNP needs EPYC 7003+ and TDX needs 5th Gen Xeon Scalable+, which is newer th

| Client | Requirement |
|---|---|
| **Web console** (the operator UI) | A modern browser — **nothing to install on the operator's machine.** The engine serves the console same-origin under `/ui` from its own FastAPI app ([ADR 0065](adr/0065-web-ops-dashboard.md)), **on by default** since [ADR 0143](adr/0143-web-console-on-by-default-disableable-with-loopback-secure-context-browser-hardening.md) (`[security].serve_web_console`; set it to `false` for a JSON-API-only deployment). It ships as a separately-versioned wheel, `messagefoundry-webconsole`, mounted in-process. This is the **sole operator console** — the PySide6 desktop console was retired ([ADR 0032](adr/0032-console-desktop-launch.md)). |
| **Web console** (the operator UI) | A modern browser — **nothing to install on the operator's machine.** The engine serves the console same-origin under `/ui` from its own FastAPI app ([ADR 0065](adr/0065-web-ops-dashboard.md)), **on by default for loopback binds** since [ADR 0143](adr/0143-web-console-on-by-default-disableable-with-loopback-secure-context-browser-hardening.md) (`[security].serve_web_console`; set it to `false` for a JSON-API-only deployment). On an **exposed** instance — a non-loopback host, a declared TLS terminator, or a set public address — a default-on console **auto-degrades to JSON-only** with a warning and `/ui` is not served; serving it off-box is opt-in and needs `serve_web_console = true` *plus* TLS *plus* `[security].web_console_public_address`. It ships as a separately-versioned wheel, `messagefoundry-webconsole`, mounted in-process. This is the **sole operator console** — the PySide6 desktop console was retired ([ADR 0032](adr/0032-console-desktop-launch.md)). |
| **VS Code extension** | Visual Studio Code (current stable) — route wizard, validate-on-save, test bench, stage→promote. |
| Test harness — *optional, not needed to run the engine* | The standalone synthetic send/receive/load harness (`python -m harness`) is the **only** PySide6 (Qt) surface left. It ships as its **own distribution**, released in lockstep with the engine and deliberately **not** included in the engine wheel: `pip install messagefoundry-harness` (which pulls `messagefoundry[harness]`, i.e. PySide6). Windows / Linux / macOS, a separate process reaching the engine only over the HTTP API. It is a **testing tool** — an engine host that does not run it needs no Qt and no GUI at all. |

Expand Down
9 changes: 7 additions & 2 deletions assets/docs/_md/Throughput-and-Capacity.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,13 @@ preserve the ordering and at-least-once delivery guarantees the engine makes. Se
not something more hardware relieves.

That has a directly practical consequence: **adding processor cores to the database does not
raise this ceiling.** We measured that rather than assuming it. The levers that would raise it
are fewer database transactions per message and faster transaction-log storage.
raise this ceiling.** We measured that rather than assuming it. Nor do the two changes people
usually reach for next. Cutting the number of database transactions per message was measured
directly: a 28% reduction in committed transactions moved sustained throughput by less than one
percent — inside measurement noise — so transaction reduction is a measured dead end rather than a
lever. Faster transaction-log storage is on the same footing: the store commits far below its
measured ceiling, so the log is not what the pipeline is waiting on. We have no identified
throughput lever, and we would rather say so than name one we have already falsified.

We describe the precise mechanism as well-corroborated rather than proven — it is consistent
with everything we measured, but we have not isolated it to the exclusion of all alternatives,
Expand Down
4 changes: 2 additions & 2 deletions assets/docs/_md/User-Guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -300,7 +300,7 @@ Key points the sample demonstrates:

- **Inbound MLLP takes only a `port`** — passing `host` is a wiring error. The listen interface is the service-level `[inbound].bind_host` (loopback in DEV, a specific NIC in PROD), an operator setting, not authored here.
- **Outbound MLLP needs a `host` and `port`.** Anything that differs by environment (a downstream peer, a credential) uses `env("key")`, resolved per instance from `environments/<env>.toml` (and `MEFOR_VALUE_<KEY>` for secrets) — so one module runs unchanged in every environment. A referenced-but-undefined value fails loud at load, never a silent blank host.
- For a **File** endpoint, use `File(directory="./out/adt")` (in) / `File(directory=..., filename="{MSH-10}.hl7")` (out). For non-HL7 bodies, set the inbound's `content_type` so the body routes as a `RawMessage` instead of being HL7-parsed — the shipped set is `hl7v2` (default), `json`, `xml`, `text`, `x12`, `fhir`, `binary`, and `dicom`. `x12` rides any transport (see [samples/config/IB_PARTNER_X12.py](../samples/config/IB_PARTNER_X12.py)); `fhir` ([ADR 0022](adr/0022-fhir-resource-codec-rest-client.md)) and `dicom` ([ADR 0025](adr/0025-dicom-codec-store-connectors.md)) add their own on-demand codecs, and arbitrary bytes carry NUL-safely over the base64 `binary` path ([ADR 0028](adr/0028-base64-binary-carriage-codec.md)). Two of the newer connectors are direction-specific: `FHIR()` is **outbound-only** and the `DICOM()` C-STORE listener is **inbound-only**.
- For a **File** endpoint, use `File(directory="./out/adt")` (in) / `File(directory=..., filename="{MSH-10}.hl7")` (out). For non-HL7 bodies, set the inbound's `content_type` so the body routes as a `RawMessage` instead of being HL7-parsed — the shipped set is `hl7v2` (default), `json`, `xml`, `text`, `x12`, `fhir`, `binary`, and `dicom`. `x12` rides any transport (see [samples/config/IB_PARTNER_X12.py](../samples/config/IB_PARTNER_X12.py)); `fhir` ([ADR 0022](adr/0022-fhir-resource-codec-rest-client.md)) and `dicom` ([ADR 0025](adr/0025-dicom-codec-store-connectors.md)) add their own on-demand codecs, and arbitrary bytes carry NUL-safely over the base64 `binary` path ([ADR 0028](adr/0028-base64-binary-carriage-codec.md)). `FHIR()` is **outbound-only** (a FHIR REST *server* facade is not shipped). `DICOM()` runs in **both** directions from the one factory — an inbound C-STORE SCP listener and an outbound C-STORE SCU / C-ECHO sender to a downstream PACS (`host` / `called_ae_title` configure the outbound peer).
- An **outbound `FHIR()`/`Rest()` destination to a SMART-secured server** (e.g. Epic, Oracle Health) can be wrapped with `with_smart_backend(...)` for OAuth2 client-credentials + signed-JWT authentication ([ADR 0024](adr/0024-smart-backend-services-token-provider.md)). Import it as `from messagefoundry.transports.smart import with_smart_backend` — it is not re-exported from the top-level package.

The complete per-connector settings (TLS, retry, DoS guards, ACK mode, `simulate`, etc.) are documented in [CONNECTIONS.md](CONNECTIONS.md#settings--whats-supported-today); each factory in [messagefoundry/config/wiring.py](../messagefoundry/config/wiring.py) **is the schema** for its transport.
Expand Down Expand Up @@ -450,7 +450,7 @@ A save is validated against the **same loader the engine uses** (no duplicate ke

Routers and Handlers **must be pure**: message in → message(s) out, no external side effects. At-least-once delivery re-runs a transform after a crash and relies on the re-run producing identical output. Side effects (network, file, DB writes) belong in outbound Connections, not in your functions.

The **one sanctioned exception** is a Handler making a **live, read-only** `db_lookup(connection, statement, params)` for enrichment/gating — its result may differ on a re-run, and that is accepted by design. It is read-only, gated by `[egress].allowed_db`, runs off the event loop, and is **unavailable on a Router or in dry-run** (it raises). See [ADR 0010](adr/0010-handler-callable-db-lookup.md).
There are **two sanctioned exceptions**, both **read-only** and both available only inside a live Handler: a database read, `db_lookup(connection, statement, params)`, and a FHIR read/search, `fhir_lookup(connection, query)` — a read-by-id (`"Patient/123"`) or a search (`"Patient?identifier=MRN|123"`), GET-only. Either may return different data on a re-run, and that is accepted by design. Both are gated fail-closed — `db_lookup` by `[egress].allowed_db`, `fhir_lookup` by `[egress].allowed_http` — run off the event loop, and are **unavailable on a Router or in dry-run** (they raise). See [ADR 0010](adr/0010-handler-callable-db-lookup.md) and [ADR 0043](adr/0043-fhir-read-lookup.md).

### 6. The authoring dev loop

Expand Down
Loading