feat(streaming): controlled-conformance mutation workflow and batched baseline framing by mikemcdougall · Pull Request #3154 · honua-io/honua-server

mikemcdougall · 2026-08-04T19:46:46Z

Pull Request

Issue Link

Completes the issue: #3072 landed REQ-001..REQ-004 and NFR-002; this lands REQ-005, REQ-006, and NFR-001, plus the two wire details that kept the merged SDK harness from executing against the protocol work.

Summary

honua-io/honua-sdk-js#818's scheduled live lane has to prove that a real deployment delivers a baseline followed by a correlated mutation on every transport it advertises. Proving it requires writing to that deployment, which is exactly the thing a scheduled job must never do casually. This PR adds the narrow, bounded, reversible way to do it, and closes two contract gaps found by reading the merged SDK consumer (scripts/realtime-conformance-evidence.mjs, PR #831) against what #3072 actually emits.

Controlled-conformance mutation (REQ-005/REQ-006/NFR-001)

New Features/Streaming/Conformance slice. Lease a run, apply bounded insert / update / touch / delete through the canonical edit pipeline, release in a finally block.

Off by default, fails closed. A deployment that has not deliberately provisioned a dedicated conformance source cannot be driven into a mutation by any caller, however authorized.
Dedicated source. Every write targets the single service/layer named by configuration. There is no request parameter that can redirect a mutation, so no combination of inputs reaches an ordinary demo or user record (NFR-001).
Ownership, not trust. Every controlled record carries a self-describing marker — prefix, run id, absolute deadline — in a configured attribute. Update, touch, delete, and cleanup re-read that marker from the stored row, so two concurrent runs cannot claim or destroy each other's records even holding the same credential. In-memory run bookkeeping can never authorize a write the row itself does not.
Canonical pipeline. FeatureMutationEventService.ResolveOutboxScopeAsync → IFeatureWriter.ApplyEditsAsync → post-commit publish (self-gated on OutboxEnabled), the same sequence every protocol adapter uses. A conformance mutation is observable on the stream for the same reason a real edit is, not because of a test-only side channel.
touch is load-bearing, not a curiosity. It rewrites a record with its current values: state is unchanged but the write path still publishes an event. That is what lets a cross-transport conformance run observe an identical baseline and an identical mutation on transports it can only open sequentially — otherwise transport 2's baseline already contains transport 1's mutation and the SDK's cross-transport-state-divergence check fails every executed transport.
Bounds enforced regardless of the opt-in rate limiter. MaxConcurrentRuns (default 1, so a scheduled run's baseline stays deterministic — snapshot subscriptions cannot carry attribute filters by design, so a second run's records would otherwise appear in the first's baseline), MaxMutationsPerRun, MaxRecordsPerRun, RunTtl/MaxRunTtl, MaxSweepRecords. [RateLimit] additionally meters each route when RateLimiting:Enabled.
TTL sweeper. A PeriodicTimer BackgroundService deletes records whose marker deadline has passed, using only what is stored on the row — the case it exists for is exactly the one where the process that created them is gone.
Tamper-evident baseline. A sha256 digest over every record no run owns, returned at lease and again at cleanup. Equal digests prove the run left the source exactly as it found it. POST /api/v1/admin/streaming/conformance/reset is the operator lever that drops every lease and deletes every controlled record.

Authorization is two gates: a separately-named ConformanceMutate policy (admin baseline today, so it can be tightened to a dedicated grant without touching endpoint code — the TemporalRollbackExecute pattern) plus a per-run bearer token issued once at lease time. A valid conformance credential alone is not enough to act as a particular run.

Fail-closed matrix (REQ-006): disabled deployment → 403; unresolvable source or no immutable deployment revision → 503; mismatched expected revision/source, or exhausted lease/mutation/record budget → 409; unknown run or foreign record → 404, deliberately indistinguishable so the surface cannot be used to confirm that another run's records exist.

Wire alignment with the merged SDK harness

Reading @honua/sdk-js's live lane against what #3072 emits surfaced two mismatches that would have left #818 degraded no matter how good the server semantics were:

Baseline framing. The SDK reduces a single {"type":"snapshot","features":[…],"sequence":n,"replace":true} frame; snapshot-begin/snapshot-feature/snapshot-end appear nowhere in it. mode=snapshot-then-delta (previously a bare alias for snapshot, added one day earlier in feat(streaming): snapshot-then-delta subscriptions, subscription-local sequence, and immutable deployment revision #3072 and consumed by nobody) now selects a batched baseline: one snapshot frame carrying a features array, consuming exactly one subscription-local sequence so the first delta continues at sequence + 1. mode=snapshot keeps the streamed framing for large baselines. Boundary cursor, replacement-snapshot reasons, truncation reporting, and delta resumption are identical between them — only the framing differs, and both run through the same emitter with a framing-aware sequence allocator (a batched baseline must not burn sequences on buffered frames it never writes).
Revision field name. The SDK reads serverRevision / gitRevision / commitSha / imageDigest / revision — never deploymentRevision. The immutable revision is now published as serverRevision alongside deploymentRevision on both /api/v1/streaming/features/capabilities and /api/v1/capabilities/manifest. A manifest that advertises the revision under a name no client reads is, to that client, indistinguishable from a deployment that has none.

Batched baseline entries carry the delta envelope's exact identity (featureId string form as id, serviceId as sourceId) so a client keys the baseline record and its later deltas identically, and geometry/properties are always written even when null — a GeoJSON Feature must carry them, and a consumer cannot otherwise tell absent from withheld.

Changes Made

Added src/Honua.Server/Features/Streaming/Conformance/: options + startup validator, run registry (lease, constant-time token check, budgets, TTL reclamation), ownership marker, workflow service, endpoints, TTL sweeper, source-generated JSON context and logger.
Added POST /api/v1/streaming/conformance/runs, POST /api/v1/streaming/conformance/runs/{runId}/mutations, DELETE /api/v1/streaming/conformance/runs/{runId}, and POST /api/v1/admin/streaming/conformance/reset; registered them in EndpointRegistry.
Added the ConformanceMutate authorization policy and RequireConformanceMutateAuthorization(); registered the helper in EndpointAuthorizationGuardTests' opt-in marker list.
Added FeatureStreamSubscriptionMode.SnapshotThenDelta with batched SSE/WebSocket sinks over the existing snapshot emitter, and made the emitter's sequence allocation framing-aware.
Added serverRevision to the streaming capability response and to the capability manifest's server block; added snapshot-then-delta to advertised modes and a credential-free conformance block to streaming capabilities.
Widened ResolveStreamService / ResolveStreamLayer / StreamLayerDescriptor from private to internal so the conformance workflow resolves its source through the same catalog lookup the subscription path uses instead of a second one that could drift.
Documented the batched framing, the controlled-mutation workflow, and the new troubleshooting rows in docs/guides/edit/react-to-changes.md; documented every config key in .env.example.
Documented the admin reset route in docs/developer/api-specs/admin-api.json; regenerated docs/gis/data/feature-catalog.json and docs/gis/data/capability-matrix.v1.json.

Testing

Unit tests added/updated
Integration tests added/updated
Architecture tests pass
Manual testing performed

Executed locally:

dotnet build Release with TreatWarningsAsErrors=true — src/Honua.Server, tests/dotnet/Honua.Server.Tests, tests/dotnet/Honua.Architecture.Tests: 0 errors, 0 warnings.
dotnet format Honua.sln --include <changed .cs> --verify-no-changes — clean.
dotnet test tests/dotnet/Honua.Architecture.Tests — 195/195 passed, including the API-surface coverage guard, the endpoint-authorization guard, the DI/god-object ceilings, and the feature-catalog drift guard.
dotnet test tests/dotnet/Honua.Server.Tests --filter Category=Unit — 3614 passed. The 48 failures in that run are all DockerUnavailableException from Testcontainers-backed fixtures in unrelated areas (OIDC options, bbox validation); this dev environment has no Docker daemon.
dotnet test --filter FullyQualifiedName~FeatureStreamConformanceRunRegistryTests — 16/16 passed.
python3 scripts/ci/openapi-drift-check.py — PASS, 0 drift. scripts/ci/validate-openapi-contracts.sh — passed.

New integration suite FeatureStreamConformanceEndpointsTests (16 tests) covers: lease binding to the deployment revision and configured source; insert/update/touch/delete; unknown-operation rejection; exhausted mutation budget; two concurrent runs unable to update, delete, or clean up each other's records; missing run token; exhausted lease; revision and source-identity mismatch; unauthenticated refusal; admin reset; the capability and manifest revision projections; cleanup restoring the baseline digest; disabled and revision-less deployments; and observation of a controlled insert on the live SSE stream after a batched baseline, with the run marker on the streamed after-image. FeatureStreamSnapshotEndpointsTests gains batched-framing coverage on SSE and WebSocket plus unknown-mode rejection.

These require Docker (Testcontainers) and this dev environment has no Docker daemon, so the integration tests were not executed locally — they build clean and will run on CI. Flagging that explicitly rather than implying a green local run.

Gate Impact

PR gates (build, test, governance)
Deploy gates (promotion, post-apply validation)

Docs or Contract Impact

OpenAPI spec changed
Documentation updated

admin-api.json gains one additive operation (POST /streaming/conformance/reset) plus its response schemas. No existing operation changed, so no breaking-change acknowledgement is needed.

Release/Deploy Impact

Requires environment variable or secret changes

The controlled-mutation surface stays closed until a demo/staging deployment sets FeatureStreaming__Conformance__Enabled=true and points ServiceId/LayerId at a dedicated conformance source whose schema carries the marker field. That source still has to be provisioned in the target environment — it is deliberately not auto-created, because auto-creating a writable layer in every deployment is the opposite of the guarantee this feature makes. Deployment__ImageDigest (or Deployment__Revision) must also be set, or run leasing fails closed with 503.

Breaking Changes

None for shipped clients. One behavior refinement inside a one-day-old surface: mode=snapshot-then-delta was added in #3072 (merged 2026-08-03) as a bare alias for mode=snapshot and now selects the batched framing instead. mode=snapshot is byte-for-byte unchanged, mode=delta remains the default, and serverRevision / conformance / snapshot-then-delta-in-modes are additive.

Pre-PR Checklist

Ran scripts/ci/pre-pr-check.sh and all checks passed
Commit messages follow conventional format: type: description (#issue)
PR title matches main commit message
Issue number linked above
Tests added for new functionality
If protocol/auth behavior changed: updated compatibility contract
If breaking admin/control-plane API changes: updated migration guide
If breaking gRPC/proto wire changes: confirmed with explicit review

Follow-up for honua-io/honua-sdk-js#818

Two things remain on the SDK side before its scheduled lane reports executed, and neither can be fixed from this repo:

The harness has no client for this API. scripts/realtime-conformance-evidence.mjs observes passively — it opens each advertised transport and waits up to 15 s for a snapshot followed by a mutation, calling no write endpoint. It needs to lease a run, drive a mutation per transport, and release in its existing finally block.
Cross-transport determinism. reconcileExecutedTransportStates fails all executed transports unless eventCount, historySha256, and finalStateSha256 match exactly, and transports are opened sequentially. The workable shape is: lease → one insert before opening any transport → per transport, one touch of that record → cleanup in finally. touch exists for precisely this; an insert per transport cannot converge because the object ids differ.

The conformance source also needs geometry on every record: the current delta envelope drops a null geometry, and the SDK's honua-server decoder rejects an insert/update whose after-image lacks the member.

chatgpt-codex-connector · 2026-08-04T19:46:50Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

… baseline framing (#3038) Completes the server side of the SDK live-conformance contract. #3072 landed the protocol half (snapshot-then-delta, subscription-local sequence, immutable deployment revision); this lands REQ-005/REQ-006/NFR-001 plus the two wire details that kept the merged SDK harness from executing against it. Controlled-conformance mutation (REQ-005/REQ-006/NFR-001): - New Features/Streaming/Conformance slice: lease a run, apply bounded insert/update/touch/delete through the canonical edit pipeline, release in a finally block. Off by default; a deployment that has not deliberately provisioned a dedicated conformance source cannot be driven into a mutation by any caller, however authorized. - Every write targets the single service/layer named by configuration. No request parameter can redirect it, so no combination of inputs reaches a demo or user record. - Every controlled record carries a self-describing ownership marker (prefix, run id, absolute deadline) in a configured attribute. Update, touch, delete, and cleanup re-read that marker from the stored row, so two concurrent runs cannot claim or destroy each other's records even holding the same credential. - `touch` rewrites a record with its current values: state is unchanged but the canonical write path still publishes an event, so two subscriptions opened at different times observe an identical baseline and an identical mutation. That is what lets a cross-transport run compare normalized state across transports it can only open sequentially. - Mutations go through FeatureMutationEventService + IFeatureWriter with the transactional-outbox scope resolved first, so a conformance mutation is observable on the stream for the same reason a real edit is, not via a test-only side channel. - Bounds enforced regardless of the opt-in rate limiter: MaxConcurrentRuns (default 1, so a scheduled run's unfiltered baseline stays deterministic), MaxMutationsPerRun, MaxRecordsPerRun, RunTtl/MaxRunTtl, MaxSweepRecords. RateLimit attributes meter the routes when the limiter is enabled. - A PeriodicTimer sweeper deletes records whose marker deadline has passed, using only what is stored on the row — the case it exists for is exactly the one where the process that created them is gone. - Baseline digest: a sha256 over every record no run owns, returned at lease and at cleanup. Equal digests prove the run left the source as it found it. Admin POST /api/v1/admin/streaming/conformance/reset drops every lease and deletes every controlled record. - New ConformanceMutate authorization policy (admin baseline, separately named so it can be tightened to a dedicated grant without touching endpoint code) plus a per-run bearer token, so a valid credential alone is not enough to act as a particular run. - Fail closed on: disabled deployment (403), unresolvable source or missing deployment revision (503), mismatched expected revision/source or exhausted lease/budget (409), unknown run or foreign record (404 — indistinguishable, so the surface cannot confirm another run's records exist). Wire alignment with the merged SDK harness: - `mode=snapshot-then-delta` now selects a batched baseline: one `snapshot` frame carrying a `features` array and `replace: true`, consuming exactly one subscription-local sequence, so the first delta continues at sequence+1. `mode=snapshot` keeps the streamed begin/feature/end framing for large baselines. Boundary cursor, replacement-snapshot reasons, truncation reporting, and delta resumption are identical between them; only the framing differs. Baseline entries carry the delta envelope's exact identity (featureId string as `id`, serviceId as `sourceId`) and always write `geometry`/`properties`. - The immutable revision is now published as `serverRevision` alongside `deploymentRevision` on both /api/v1/streaming/features/capabilities and /api/v1/capabilities/manifest. A manifest that advertises the revision under a name no client reads is indistinguishable, to that client, from a deployment that has none. - Streaming capabilities advertise `snapshot-then-delta` in `modes` and carry a credential-free `conformance` block so an SDK can discover the contract's bounds anonymously. Tests: - New integration suite: lease/mutate/cleanup, ownership refusal across two concurrent runs, missing run token, exhausted lease and mutation budgets, revision and source-identity mismatch, unauthenticated refusal, admin reset, capability and manifest projections, and observation of a controlled insert on the live stream after a batched baseline. - New unit suite: lease bounds, token isolation, TTL expiry and reclamation, budget claim/release, and ownership-marker parsing (including rejection of values this server did not write). - Batched-baseline framing tests added to the snapshot suite for SSE and WebSocket, plus unknown-mode rejection. - feature-catalog.json, capability-matrix.v1.json, and admin-api.json regenerated or extended for the new routes.

+        var delta = await ReadSseEventAsync(reader, "feature-change", cts.Token);
+        delta.Should().NotBeNull("a controlled mutation goes through the canonical edit pipeline and is therefore streamed");
+        delta!.Value.GetProperty("objectId").GetInt64().Should().Be(objectId);
+        delta.Value.GetProperty("operation").GetString().Should().Be("insert");


+        delta.Should().NotBeNull();
+        delta!.Value.GetProperty("sequence").GetInt64().Should().Be(1,
+            "the first delta continues the batched baseline's single sequence");
+        delta.Value.GetProperty("cursor").GetInt64().Should().BeGreaterThan(baselineCursor);


+        foreach (var operation in All)
+        {
+            if (trimmed.Equals(operation, StringComparison.OrdinalIgnoreCase))
+            {
+                return operation;
+            }
+        }


+            foreach (var candidate in _runs.Values)
+            {
+                if (candidate.IsExpired(now))
+                {
+                    _runs.TryRemove(candidate.RunId, out _);
+                }
+            }


mikemcdougall force-pushed the feat/3038-conformance-controlled-mutation branch from c91f0c4 to 6fc339c Compare August 4, 2026 19:58

github-code-quality Bot found potential problems Aug 4, 2026

View reviewed changes

mikemcdougall mentioned this pull request Aug 4, 2026

test(realtime): drive the live conformance lane through controlled mutation honua-io/honua-sdk-js#1058

Merged

Navigation Menu

Uh oh!

feat(streaming): controlled-conformance mutation workflow and batched baseline framing - #3154

feat(streaming): controlled-conformance mutation workflow and batched baseline framing#3154
mikemcdougall wants to merge 1 commit into
trunkfrom
feat/3038-conformance-controlled-mutation

mikemcdougall commented Aug 4, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

mikemcdougall commented Aug 4, 2026

Pull Request

Issue Link

Summary

Controlled-conformance mutation (REQ-005/REQ-006/NFR-001)

Wire alignment with the merged SDK harness

Changes Made

Testing

Gate Impact

Docs or Contract Impact

Release/Deploy Impact

Breaking Changes

Pre-PR Checklist

Follow-up for honua-io/honua-sdk-js#818

Uh oh!

chatgpt-codex-connector Bot commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant