Skip to content

v0.2.2 (sync from internal development)#1

Open
dbotwinick wants to merge 112 commits into
mainfrom
develop
Open

v0.2.2 (sync from internal development)#1
dbotwinick wants to merge 112 commits into
mainfrom
develop

Conversation

@dbotwinick

Copy link
Copy Markdown
Contributor

This pull request introduces a new release (v0.2.2) with significant feature additions, improvements, and bug fixes across the gateway, SDKs, and supporting infrastructure. The changes also enhance documentation, update CI workflows, and clarify project structure. Below are the most important changes grouped by theme:

Major Features and Improvements:

  • Added a user-broadcast topic (uu::{user_id}) for platform-to-user messaging, with helpers and documentation in the SDKs, and enforced sender-type restrictions. [1] [2] [3]
  • Enabled OpenTelemetry OTLP metrics export (in addition to traces) in both gateway and aetherlite, gated by the standard environment variable. [1] [2]
  • Introduced new service modes and endpoints, including tenant-relay/aggregator tunnel modes, a /auth/verify-optional endpoint, and the ability for services to opt out of pool-task consumer routing.
  • Implemented background cleanup sweeps for stale tasks and audit retention, improving resource management and reliability. [1] [2]
  • Improved audit logging, event handling, and message delivery semantics (e.g., resume-or-tail for user/progress subscriptions, durable consumer offsets, and outcome logging).

Bug Fixes:

  • Fixed SDK metric/event topic helpers to use the correct :: separator, resolving silent message drops. [1] [2]
  • Resolved issues with agent metric publishing, event dispatch deadlocks, and consumer-offset durability in Badger router.

Documentation and Project Structure:

  • Updated CHANGELOG.md for the v0.2.2 release, summarizing all new features, changes, and fixes.
  • Expanded and clarified CLAUDE.md and README.md to document new binaries, topic schema, observability, cleanup, and configuration flags. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

Continuous Integration and Testing:

  • Updated CI workflows to use Go version 1.25.12 for all jobs, improving consistency and security. [1] [2] [3] [4] [5] [6]
  • Serialized integration and e2e tests to avoid resource contention and flakiness on CI runners. [1] [2]
  • Added a non-blocking, informational govulncheck scan for the Go SDK to surface known and new vulnerabilities without failing the build.

These changes collectively improve platform robustness, observability, developer experience, and documentation clarity.

dbotwinick added 30 commits May 23, 2026 21:58
…nd add regression tests

- Updated `buildJSKey` and `buildJSPrefix` functions to honor `ScopeSpec.Sharing` and `spec.Identity`, ensuring correct key layouts for exclusive and shared scopes.
- Added comprehensive regression tests to verify cross-agent scope behavior, preventing unintended isolation or leakage in shared scopes.
Lifts retry handling out of every worker into a platform primitive. Adds
a proto `RetryPolicy` (max_attempts, backoff, schedule, jitter,
retryable status codes, honor_retry_after) carried on
`CreateTaskRequest`. Tasks persist the policy; the store's `FailTask`
computes `next_retry_at` from it, and the gateway's reschedule path
consults it when re-pending.

Service principals now register in `implementationIndex` alongside
agents (workspace-less `:impl` key). `findWorkerByImplementation` falls
back to that key when the workspace-scoped lookup misses, so pool tasks
targeted at services (e.g. webhook delivery) find a healthy instance
without per-workspace registration. Cross-workspace by design — ACL
gates at task creation, not at delivery.

Workflow Engine retires its in-DAG immediate-re-dispatch path: WE
translates a step's `RetryConfig` into a proto `RetryPolicy` and
attaches it to the created task. The task store handles backoff; WE
only reacts to terminal status transitions.

Changes:
- api/proto/aether.proto: BackoffStrategy enum, RetryPolicy message,
  CreateTaskRequest.retry_policy field (regenerated bindings)
- pkg/tasks: RetryPolicy struct, ComputeNextRetryAt, persistence,
  conformance test for cross-workspace pool claims
- internal/gateway: services in implementationIndex, retry_policy proto
  conversion, rescheduleFn honoring policy
- internal/orchestration: RetryPolicy threaded through task_assignment;
  cross-workspace findWorker fallback
- internal/workflow: dag retires broken immediate-re-dispatch;
  executor translates RetryConfig -> RetryPolicy
- sdk/go/aether: CreateTaskOptions.RetryPolicy
- migrations/025 (postgres) + sqlite_tasks/003 (sqlite): retry_policy_json column
…e log spam and improve validation efficiency
…CompareAndDelete) for distributed coordination
Adds SetAdd (add member, returning newly-added + cardinality) and SetCard to
the KVReadWriter interface, implemented natively in Redis (SADD/SCARD via Lua),
Badger (txn read-modify-write), and NATS JetStream (revision-guarded CAS), with
a shared JSON set codec for the two software-emulated backends. The unique
caller observing added && cardinality==N is the single fan-in firer;
added==false marks a duplicate arrival for at-most-once dedup ledgers. Includes
Badger and embedded-NATS tests.
Adds the Join type and eight CRUD/sweep store methods (EnsureJoin, GetJoin,
UpdateJoinArrived, SetJoinExpected, SetJoinDirty, MarkJoinTerminal,
GetDueJoinDeadlines, ListJoins) in lockstep across the engine-internal
WorkflowStore mirror, the canonical storage/workflow.Store interface, the legacy
and native-sqlite implementations, the type-alias bridge, and both migration
trees. The SQL row is the durable mirror, observability surface, and deadline
driver for join instances whose authoritative arrival counter lives in KV.
Adds a declarative `type: join` rule destination handled by a new JoinEngine:
count-mode barrier (atomic arrival counter plus an exactly-once fire-marker
gate), dynamic-N arming via a separate arm_on_event arrival that supplies
expected_count without itself counting as a member, coalesce/debounce mode
(the server-side kb_update pattern), per-id dedup, and emit_event chaining.
Wired into the Router; built on coord.Counter/Locker abstractions so it is
unit-tested with in-memory fakes (no gateway). Join keys live under the reserved
_sys/coord/ namespace, portable across Redis/Badger/JetStream.
…elds

Regenerates Go/Python/TS bindings for: KVOperation SET_ADD/SET_CARD; WorkflowOperation
LIST_JOINS/GET_JOIN/CANCEL_JOIN; and CreateTaskRequest/TaskInfo/TaskFilter
idempotency_key + correlation_id + root_task_id. All additive (new enum values
and optional fields); existing wire semantics unchanged.
Persists on_complete/on_timeout/on_partial_failure on the workflow_joins row so
the leader-gated scheduler sweep can fire a join's timeout action with no live
event. Adds JoinEngine.HandleDeadline (abort vs proceed/proceed_degraded policy,
exactly-once via the fire-marker gate) and wires GetDueJoinDeadlines into the
scheduler poll loop. A never-completing barrier is now GC'd at its deadline
rather than leaking.
Exposes the atomic set primitive end-to-end: gateway KV handler cases for
SET_ADD/SET_CARD (with audit op constants), Go SDK SetAddSync/SetCardSync, and
the JoinModeSet path in the engine — members are added to a KV set (inherently
deduped) and the join fires when cardinality reaches the expected_set size, with
the same fire-marker exactly-once gate. Also backfills SetAdd/SetCard on the
gateway test mock.
Joins now stamp a stable idempotency_key (join:name:ws:corr, identical across
the on_complete and on_timeout fire paths) on their downstream create_task. The
gateway dedups creation on that key via a SetNX ledger under _sys/idem/task/:
the first create proceeds and records its task_id; duplicates (retries, engine
restart, or a timeout sweep racing a completion) create no second task and echo
the original task_id when a response is expected. Fail-open on KV outage.
Adds LIST_JOINS/GET_JOIN/CANCEL_JOIN WorkflowOperation handlers and matching
admin REST endpoints (GET /joins, GET /joins/{name}/{corr}, POST
/joins/{name}/{corr}/cancel), exposing a joinView surface (name, workspace,
correlation_key, mode, arrived, expected, status, deadline) so operators can
list/inspect in-flight barriers and GC a wedged one.
Covers architecture, the event/rule model, all destination kinds, joins in
depth (count/coalesce/set, correlation contract, dynamic-N arming, dedup,
deadline sweep, exactly-once via fire-marker + idempotent create_task, KV keys,
storage, observability), the scheduler, DAG/state-machine engines, admin API,
design decisions and rationale, deferred work (feed B, correlation propagation,
DAG AND-join), and a proto + source map.
drain saved the consumer offset per live message, but replay (the reconnect
catch-up path) did not — so a named consumer that caught up via replay never
committed its progress and re-replayed the same historical backlog on every
reconnect. This is acute for messages that ONLY ever reach a consumer via
replay: Publish drops from the live fan-out channel (non-blocking select/
default) when the subscriber channel is full but still persists to badger, so
a slow consumer's messages are delivered only via replay. Without an offset
save they re-deliver every reconnect and, at the gateway, re-shed under
backpressure indefinitely (observed: user-window us:: sessions re-shedding
~1900 messages on each 2h connection recycle).

After a successful replay, save the offset to the highest replayed sequence
for named consumers, so the next reconnect resumes from head. Mirrors drain's
saveOffset; batched to the end of the synchronous replay (single write).
Lite/badger only — the RabbitMQ Streams router auto-commits on read.
… contention

A concurrent 10x window refresh surfaced two lite-mode contention bugs (both
absent in the Postgres/full deployment):

1. KV ratelimit counter had no retry. addDelta (kv/badger_store.go) did a single
   db.Update read-modify-write and returned "failed to modify counter:
   Transaction Conflict" on badger.ErrConflict — while every OTHER badger KV op
   retries. The per-user ratelimit key is hot (every request increments it), so
   concurrent refreshes guaranteed conflicts and hard failures. Retry the pure
   read-modify-write on ErrConflict (casMaxAttempts), matching addDeltaGuarded.

2. acl.db writes weren't serialized → SQLITE_BUSY ("database is locked") on
   concurrent authority-grant INSERTs. aclsqlite.New never called
   SetMaxOpenConns(1) (unlike the tasks/registry/workflow sqlite stores), so the
   acl.db handle used a multi-connection pool; and busy_timeout is a
   per-connection pragma set via ExecContext, so extra pool connections opened
   under load had no busy_timeout and hit immediate SQLITE_BUSY. Fix both:
   - aclsqlite.New: SetMaxOpenConns(1) on the shared acl.db handle (also used by
     the legacy internal/acl.Service authority-grant path).
   - openSQLiteWithDriver: SetMaxOpenConns(1) on every lite SQLite handle so the
     busy_timeout pragma always applies and writers serialize instead of racing.

Adds TestBadgerIncrement_ConcurrentSameKey (100 concurrent increments, -race).
saveOffset did a single db.Update read-modify-write and returned on
badger.ErrConflict with no retry, unlike appendMessage (which retries up to
appendMessageMaxRetries). The offset key is committed from the consumer's drain
goroutine per live message and once from the replay catch-up path; under a
connection storm / heavy concurrent publish load the optimistic txn conflicts,
the save silently fails (logged, non-fatal), the offset stalls, and the named
consumer re-replays + re-sheds the same backlog on the next reconnect.

Retry on ErrConflict with the same bounded, no-backoff loop appendMessage uses
(sub-millisecond txns). Test drives 100 concurrent saves to one offset key and
asserts no conflict error escapes + the key commits to a valid value (-race).
User-window shared/broadcast subscriptions used anonymous router.Subscribe,
which on the badger router replays the topic from sequence 0 with NO offset
tracking on every connect (368b72e's offset-persist covers named consumers
only), and badger has no retention — so these re-dumped their full retained
history on every (re)connect and re-shed it under gateway backpressure
(priority-2 PriorityRequest bursts co-timed with session connect), forever.

Add a resume-or-tail start policy: a named consumer resumes from its committed
offset, or — when none exists yet (cold) — starts at the tail instead of
replaying from seq 0. First connect gets no history dump; a reconnect replays
only the gap since the last committed offset.

- router: new SubscribeExclusiveResumeOrTail on MessageRouter + all backends.
  Badger implements it (loadOffsetOK distinguishes missing vs 0); RabbitMQ and
  JetStream already resolve this way for named/durable consumers, so they
  delegate to their existing paths.
- gateway: per-window named consumer (consumer=identity, incl. window id) for
  the user broadcast lanes gu/uw/uu and the live task-message lane (which also
  fixes its latent replay-from-0 despite its "no replay needed" contract);
  the live lane now shares the backfill path's consumer name for clean resume.
- tests: badger resume-or-tail (cold→tail, reconnect→gap-only); updated the
  routing/connect/switch/task-lane assertions that pinned the old shared path.

Progress topics (pg::*, priority-4, helper shared with agents) still replay
from 0 and are a follow-up. Retention/trim of badger topic logs (the reason
first-replay bursts are large) is a separate change pending default params.
…er TTL

Follow-up refinements to the resume-or-tail change (8cf3080):

- Task-message lane must REPLAY, not resume. A per-turn lane is small and has to
  be re-sent in full on every (re)connect: a page reload keeps the window id but
  loses the tab's rendered state, so the whole in-flight turn must be replayed —
  resume-from-offset would skip the already-emitted tokens the reloaded tab needs
  to re-render. Revert the live path to anonymous full-replay Subscribe (it also
  commits no offset, so the connect-time backfill still cold-replays the turn).

- Progress topics (pg::{ws}, pg::us::{user}) now use per-client resume-or-tail
  named consumers instead of anonymous replay-from-0, so a (re)connect starts at
  the tail (a reload loads fresh progress) and a reconnect replays only the gap,
  rather than re-dumping the full retained progress history every connect.

- Badger message retention TTL (native per-entry expiry, default 24h, override
  via AETHER_MESSAGE_RETENTION_TTL; "0" = retain forever). Bounds otherwise-
  unbounded topic-log growth and caps first-replay/full-replay burst size; the
  sequence counter and consumer offsets carry no TTL, so expiry never rewinds
  numbering — a resuming consumer just finds fewer messages to replay.

Tests: badger retention-TTL (expiry + preserved sequence) and resume-or-tail;
reverted the task-lane assertion to shared and moved pg::us::* to the exclusive
path in the routing table.
…ult)

Per-window resume-or-tail consumers write an offset key per (topic, window id).
Window ids churn (new tab / cleared sessionStorage), so without expiry these
off:{topic}:{consumer} keys would accumulate unboundedly.

Give offset keys their own TTL, refreshed on every save: an active consumer
keeps rewriting its offset so it never expires while alive, while an orphaned
window's offset stops refreshing and is reaped by Badger's native GC — no sweep.

The offset TTL is decoupled from message retention and deliberately generous
(defaultOffsetRetentionTTL = 7d vs 24h for messages): offsets are tiny 8-byte
keys, and keeping them well past the message window lets a consumer that was
away for days still RESUME (replaying whatever messages remain) instead of
cold-starting. Override via SetOffsetRetentionTTL / AETHER_OFFSET_RETENTION_TTL
("0" = retain forever). The sequence counter still carries no TTL, so numbering
is never rewound.

Test: an orphaned offset expires after its TTL while a refreshed one survives
until it too goes stale (reaping is driven by staleness, not key identity).
…nt_if/kv_decrement_if

The KVOperation proto has no int_value field (the atomic-increment operand is
delta_value, field 11) — the same constructor already sets delta_value=int(delta)
correctly on the next line. The stray int_value=delta raised
'Protocol message KVOperation has no int_value field' at runtime, breaking every
kv_increment_if/kv_decrement_if call (e.g. the onboarding claim-check counter).
Remove the 4 redundant lines (client.py + client_async.py).
Vetted every concrete claim against source and fixed drift across the doc set.
Highlights:

- Topic separator: event./metric. (and illustrative us./ag./gu./uw./pg. scenario
  topics) → "::" everywhere; the gateway rejects the dot form. Added the uu::
  user-broadcast topic to the topic tables.
- Subscription model: documented the third mode, resume-or-tail
  (SubscribeExclusiveResumeOrTail) — user broadcast (gu/uw/uu) + progress (pg::)
  lanes use it; per-task chat lane (tk::…::msg) uses full replay; identity topics
  stay exclusive-durable. MessageEnvelope proto updated (metadata/workspace/
  on_behalf_subject) + OBO subject delivery-time propagation.
- Retention: documented AETHER_MESSAGE_RETENTION_TTL (24h) and
  AETHER_OFFSET_RETENTION_TTL (7d/168h) in README + environment.md.
- environment.md/aetherlite.md: added the AETHERLITE_CLUSTER_MODE block, audit
  rate/coalesce vars, missing CLI flags; corrected AETHER_FANIN_SHARDS (stub),
  per-domain SQLite DB split, dispatcher/store type names, version 0.2.1.
- monitoring.md: corrected Prometheus metric names + label values to match
  source, fixed ops-port vs admin-port ownership, added OTLP export
  (OTEL_EXPORTER_OTLP_ENDPOINT) + backend metrics sections.
- admin-ui.md: /api → /api/v1 route prefix, unversioned /health & /info, ops vs
  admin server split, added ACL grants/groups/roles + rate-limit endpoints,
  AETHER_ALLOW_DEV_MODE gotcha.
- audit_testing_guide.md: migration numbers, removed a fabricated startup banner,
  fixed the old_value/new_value contradiction, coalescing window, corrected the
  cleanup.Service retention model; flagged the unimplemented admin-action audits.
- CLAUDE.md/horizontal-scaling.md: Observability + Background-Cleanup sections,
  binary/port inventory, lock TTL 30s (not 60s / no LOCK_TTL_SECONDS), 8 principal
  types, DisconnectReaper.
- CHANGELOG.md: added the notable post-0.2.0 changes under [Unreleased].
v0.2.1 was released 2026-05-22 (tag f5e6f27); the CHANGELOG's [Unreleased]
section still carried a stale "v0.2.1 stamp, no tag published yet" note and
conflated the released 0.2.1 work with newer changes.

- Cut a dated `## [0.2.1] - 2026-05-22` section for the pre-release work that
  shipped in the 0.2.1 SDK/gateway bump (the entries written before the bump).
- Moved the post-May-22 work (uu:: topic, OTLP, resume-or-tail + badger
  retention TTLs, OBO propagation, audit coalescing, stale-task sweeps, the
  metric/event :: fix, offset durability, queue retirement, …) into a new
  `## [0.2.2] - Unreleased` section.
- quickstart: example gateway/aetherlite version banners 0.2.1 -> 0.2.2 to
  match the in-development version.

version.go (bumped to 0.2.2) left as a working-tree change for you to commit.
… bumps

Reconcile go.sum with the committed go.mod requires (unchanged by this commit):
- api/go.sum was stale after the protobuf v1.36.11 / go 1.25.12 bump (had
  grpc v1.80.0 / otel v1.39.0; go.mod requires grpc v1.81.1) — a standalone
  (non-workspace/CI) build would fail on the missing checksums.
- sdk/go + server: drop the leftover go-backpressure v0.1.0 entry (go.mod
  already requires v0.1.1).
go mod verify passes for all three modules; server builds.
A Service principal is workspace-less — its identity is sv::{impl}::{spec} and
the ServiceIdentity proto has no workspace field — but config validation (added
in 142b6dd) required service.workspace whenever terminator/relay/tenant_relay
was enabled. That contradicted the model and failed every relay/relaymux/reload/
tenant_relay/composite test (they set impl+spec, not workspace). Drop the
workspace requirement; keep implementation + specifier (the real identity).
…identities)

The gateway runtime built AgentOptions + NewAgentClient, so it connected as an
AGENT — which requires a workspace — even though the sidecar's identity is a
SERVICE (sv::{impl}::{spec}, workspace-less; the ServiceIdentity proto has no
workspace field). That masked ~20 relay/relaymux/reload/tenant_relay/composite
tests behind the earlier config-validation bug, and left the runtime unable to
form a workspace-less service connection at all.

Make the runtime identity-agnostic: gatewayRuntime.client, Client(), installOn,
RegisterHandlers, and serviceClientTransport now hold *aether.BaseClient (both
AgentClient and ServiceClient embed it; all handler/Send methods live there).
init() selects by workspace presence — empty => NewServiceClient (workspace-less
sv::{impl}::{spec}); set => NewAgentClient (ag::{ws}::{impl}::{spec}) — so both
identities are supported.

Tests: TestRunner_ServiceIdentityPath asserts the InitConnection is a
ServiceIdentity (no workspace); TestRunner_AgentIdentityPath asserts an
AgentIdentity carrying the workspace. Full package passes with -race.
… sleep

setupCluster3 waited for NATS routes to form then slept a fixed 500ms for the
JetStream meta-leader to elect before returning. Under -race that election
routinely takes well over a second, so the first replicated KV-stream creation
(CreateOrOpenRegistryBucket, Replicas=3) raced the meta-election and timed out
with 'open KV bucket aether_registry: context deadline exceeded' — a flake that
CI's integration job (go test -race -tags=integration, no -short) hits.

Replace the fixed sleep with waitForJetStreamReady, which polls each node's
js.AccountInfo until it succeeds (meta-leader elected + reachable), with a
generous 30s deadline for -race overhead. cluster/integration now passes under
-race (was deterministically failing under -race before).
…sed)

- terminator.go: check the error from the fire-and-forget runConnectionLoop
  goroutine (errcheck) — log it instead of discarding.
- config.go / priority_test.go: apply the De Morgan simplifications staticcheck
  QF1001 flagged (semantics unchanged).
- admin_acl.go: remove the unused isACLGroupNotFound/isACLRoleNotFound helpers
  (unused) and the now-orphaned "errors" import.

golangci-lint v2.11.4 (the CI-pinned version) now reports 0 issues.
…e 1.25.12 normalization)

1fb74bc normalized go.mod + CI to 1.25.12 but left server/Dockerfile at golang:1.25.10;
with GOTOOLCHAIN=local, go mod download failed 'requires go >= 1.25.12'.
…lake)

The tenant-relay and composite real-runner harnesses waited only 3s for the
runner to dial the in-process fake gateway. That is far more than enough on an
idle box, but under CI's 2-core runner running the full -race unit suite the
runner goroutine can be starved past a few seconds, so TestTenantRelay_Tunnel-
EOFReturnsNil flaked with 'timed out waiting for gateway stream' in the -race
test job (passes 5/5 in isolation). Widen the connect waits to 15s — a passing
test returns the instant the stream arrives, so this only bounds a genuinely
stuck run and has no cost for green tests.
…pals

The anonymous/no-cert credential gate (added in 224f047) rejects any anonymous
connection claiming an impersonable principal (Agent/Service/Task/User/Bridge).
That is correct for production, but it broke every e2e test and local -dev usage:
a sidecar connecting as a workspace-less Service to a -dev aetherlite (no
authenticator configured) is denied ('anonymous connection as Service requires
API key auth but no authenticator is configured'), so the sidecar never confirms.

Gate the requirement on dev mode: AuthHandler.devMode (from AETHER_DEV_MODE, set
by the -dev flag) admits such connections WITHOUT a credential, logging a loud
NOT-FOR-PRODUCTION warning at both credential gates. Production (dev mode off) is
unchanged and still enforces credentials — verified by the existing gateway auth
tests. The full -tags=e2e suite passes.

Fixes the e2e CI job (broken in the untested window since 224f047, 2026-06-19).
The integration-test and e2e jobs run resource-heavy tests concurrently on
GitHub's 2-core runners: embedded NATS clusters (JetStream Raft elections),
real sidecar runners, and shared-gateway tunnels. Concurrency starved timing-
sensitive setup and produced intermittent flakes (TestAggregator_CNValidation;
the TestE2E_Metrics_TunnelOps tunnel PEER_RESET) that pass locally / in isolation.

Add -p 1 (one package at a time, so the NATS-cluster and sidecar-tunnel packages
no longer overlap) and -parallel 1 (serialize the t.Parallel tunnel tests that
share one aetherlite subprocess). Slower jobs, but no cross-test contention.
govulncheck now reports 4 docker/docker advisories reachable via the docker
orchestrator (all Fixed in: N/A, docker <= v28.5.2 is the latest module
version): the two already-tracked (GO-2026-4887, GO-2026-4883) plus two newer
docker cp race conditions — GO-2026-5617 (bind-mount redirection) and
GO-2026-5668 (symlink-swap arbitrary empty file). Still no upstream fix to bump
to; mitigation unchanged.
The security job only scanned the server module, so the docker/docker cp-race
advisories reachable via the SDK's docker orchestrator (SECURITY.md Known Issues)
were never surfaced by CI. Add a continue-on-error govulncheck run in sdk/go so
those (and any genuinely new SDK vuln) appear in the CI log without turning the
build red on the four already-tracked, no-upstream-fix advisories.
TestClusterIntegration_BackupRoundtrip_PerDomain polled 15s for both domains'
.bin uploads then cancelled the coordinator; under CI's 2-core -race load the
second domain (settings_cluster) hadn't finished, so the cancel hit it mid-
upload ('upload manifest: context canceled') and the assertion failed. Widen the
poll deadline to 45s and the overall ctx to 120s. The poll breaks the instant
both domains are present, so a passing run pays nothing.
The cluster/integration suite has ~40 tight (<30s) positive-wait deadlines that
blow under CI's `-race` on 2-core runners (embedded-NATS Raft elections/backups/
handshakes run 2-10x slower there) — a different marginal test tipped over each
run (aggregator, then backup_roundtrip, ...). Serializing the job (-p 1) helped
but the per-test deadlines were still too tight under -race.

Add a `scaled(d)` helper that multiplies durations by 4 under the race detector
(timescale_race_test.go) and is the identity otherwise (timescale_norace_test.go),
and route the package's positive-wait deadlines through it: operation timeouts
(context.WithTimeout), poll deadlines (time.Now().Add), and completion waits
(time.After in a wait-for-success select). Negative-assertion windows (absence
checks) and poll/tick intervals are deliberately left unscaled, as are behavior
config values (leader TTLs, MinInterval).

Deadlines stay tight for fast local runs; under CI's -race they auto-widen.
Verified: gofmt/vet clean, compiles under -race, and phase5 + backup_roundtrip
pass both non-race and -race.
README:
- CLI Flags table: remove nonexistent --db-password; --config is
  required-unless---dev (not defaulted to configs/dev.yaml); document --dev
- Option C: run commands from server/ (cd server); gateway --lite/bare run
  fixes so `go run ./cmd/gateway --dev` works without a config file

Docs hygiene (server/docs/):
- delete proxy-cutover.md (MemoryLayer→Aether migration runbook, not aether
  reference), proxy-architecture-roadmap.md (Phase-7 planning artifact), and
  proxy-load-test-results.md (empty benchmark snapshot); clean the dangling
  Related-documents links in proxy.md / proxy-quickstart.md / proxy-sandbox.md
  and the two source-comment references

Stale content:
- lite mode is cmd/aetherlite-only; cmd/gateway --lite now fatals at startup
  (aetherlite.md, specification.md, horizontal-scaling.md)
- principal-type counts: SDK exposes all eight typed clients
  (specification.md six→eight + add Service/Bridge; workflow-engine.md seven→eight)
- aetherlite-clustering.md: drop leaked internal /home/drew/.claude/... path
ListRoleAssignments / ListGroupMembers / ListPrincipalRoles / ListPrincipalGroups
scanned granted_at straight into the RoleAssignment/GroupMember time.Time field.
SQLite stores timestamps as TEXT (writes go through formatTime), so the driver
returns a string and the scan fails:

  sql: Scan error on column index 3, name "granted_at": unsupported Scan,
  storing driver.Value type string into type *time.Time

— surfacing to the superadmin as "failed to list role assignments". The sibling
expires_at was already handled correctly (sql.NullString + assignExpiry), and the
working acl_rules read parses granted_at via a string too; the four group/role
list reads were the outliers. Add an assignGranted helper mirroring assignExpiry
and use it in all four.

These read-back paths had NO test coverage (the other tests exercise CheckAccess /
ExplainAccess, which read the in-memory enforcer, not the SQL rows), so the bug
shipped green. Add TestSQLiteGroupsRoles_ListReadsGrantedAt covering all four
list reads + asserting granted_at parses to a non-zero recent time.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant