Observer verification: fix coverage bookkeeping, restart sessions when scope widens - #380
Conversation
db563f7 to
4aa84ec
Compare
Preview:
|
|
Submitted 3 actionable inline review findings. |
|
Submitted 2 actionable inline review findings. |
This comment was marked as resolved.
This comment was marked as resolved.
kentonv
left a comment
There was a problem hiding this comment.
I started reviewing this but I see you're making other changes so decided to post what I have and come back later.
9e5ab69 to
a33a8e8
Compare
This comment was marked as resolved.
This comment was marked as resolved.
|
Submitted 3 actionable inline review findings. |
kentonv
left a comment
There was a problem hiding this comment.
Bugfix: Don't let a delayed abort swallow a concurrent access change.
This commit is suspicious. It seems to be complaining that a randomly-placed abort() could leave the DO in an inconsistent state, if something else is happening in the meantime that gets interrupted in the middle.
But if that's the case, then that something else is the buggy bit. Every piece of code should be tolerant of a random crash interrupting it. Usually, DO's atomicity guarantees take care of it: any writes performed without an intervening await are automatically coalesced into a transaction. But anything that does writes around an async operation needs to consider what happens if the async operation never returns (due to a crash).
Of course, adding and removing observers from a gatekeeper is inherently async, and we can't even wrap it in a transaction since the gatekeeper itself is modifying its own storage. (I do want to create cross-facet transactions at some point... but we don't have them yet.)
The rule to follow here is: Ensure the gatekeeper overestimates. That is:
- addObserver() must succeed before the observer is granted any access.
- remoteObserver() must not be called until the observer's access has been revoked.
Since it's an over-estimate, in the event of a badly-timed crash, a gatekeeper can end up having an observer registered who doesn't actually have access. That is OK: it only causes that observer to possibly show up in excludeObservers in the future. At that point, the overseer checks and discovers that the excluded observer doesn't have access in the first place, and can then remove them from the gatekeeper, thus bringing it into sync.
In any case, though, I don't quite understand the situation that the first paragraph of the commit description is worried about. If removeCollaborator() has already removed the sharing edge (and the observer has also been removed from the sharing table), then it doesn't really matter if we fail to finish removing the observer from all gatekeepers -- the gatekeepers can be left over-estimating.
|
|
Submitted 4 actionable inline review findings. |
|
|
|
|
Submitted 3 actionable inline review findings. |
|
|
docs/observers.md gains a "Restarting when verification scope widens" subsection: the four triggers, why the merge trigger compares scopes rather than firing on any promotion, why shrinking scope and role rises are deliberately not triggers, why addGatekeeper's publication order is load-bearing under the restart, and where the enforcement moment actually falls for each trigger. Step 3 gains the record prune, the scrub-and-restart failure path and the returning-observer rollback rule; edge cases 3 and 5 are rewritten around them, and Step 6's justification for an orphaned entry is corrected -- a registration is what admits an open, so a stale one grants nothing on its own. docs/sharing.md renames scheduleRevocationRestart and documents the abort's second purpose, whose trigger is a grant rather than a revocation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scheduleAccessRestart() syncs, waits ~100ms, then calls ctx.abort(), which ignores the output gate. The sync covered the write that triggered *this* restart, but the timer runs concurrently with everything else: a removeCollaborator() that has already written its sharing edge and is still awaiting tearDownLostObservers()/refreshAffectedCollaboratorListings() gets aborted with the revocation still buffered. The DO comes back with the collaborator still a collaborator, while the owner's UI says the removal succeeded. That is not a delayed revocation, it's a lost one, so unlike the abort delay itself it isn't inside any tolerance. So sync again immediately before the abort, with nothing awaited in between, and coalesce concurrent triggers onto one timer instead of racing several. This makes the storage write safe, which is the unbounded part. The awaited external effects can still be cut short: a skipped removeObserver() leaves an orphaned registration (harmless -- the gate is admission, and every open re-runs addObserver), and the listing refresh self-heals on reconnect. Also record, beside the existing residual note in docs/observers.md, that the tolerance for an access change taking effect is 5s -- which is what makes the abort delay an accepted residual rather than a defect, and this lost write not.
bindWorkpiece()'s guard asked whether the edge was permanent and its target account-requiring, but never whether that target was already in "use" scope. So binding a second name onto an already-bound connection severed every session on a shared workspace for a widening that didn't happen. Replace the condition with the before/after comparison mergeChatChanges() already uses. #accountRequiringUseScope() filters on `pending` and visibleBindings (subsuming the chatId check) and on gatekeeperVendorId (subsuming the creationSpec/vendorId check), so this is a smaller condition built from an existing helper, and both widening triggers now read identically.
two roles widen independently: a new connection enters every "build" collaborator's scope at once and no "use" collaborator's until a gadget binds it, and binding one enters "use" scope having been in "build" scope since it was created. So adding an unbound connection severed every session on a use-only workspace, and binding one severed every session on a build-only workspace, in both cases without changing anyone's verification requirements. Pass the affected role and skip when no collaborator holds it. The scrub trigger passes none: a failed re-verification isn't a widening and must sever regardless of role. Also record two claims where the next reader will look for them, both raised in review: - Why the pre-abort sync is sufficient rather than merely a narrower window. The input gate is closed for the duration of a storage operation, so nothing can interleave a mutation between that sync and the abort; the window it closes is the open one before it, across scheduler.wait(100). - Why the scrub trigger schedules its restart in the catch rather than at the scrub, leaving a delay the failing collaborator controls. Stalling the re-prompt preserves exactly the sessions that never re-opening would, so it is a way to decline to leave rather than a way in -- and scheduling at the scrub would cut off every repair before a human could answer it. Tests cover both directions of the role filter; covering one would leave half the condition unexercised.
A sync() followed by ctx.abort() only narrows the window it was meant to close: a request delivered after the sync resolves can still write a mutation that the abort then discards. Run the flush and the reset together inside blockConcurrencyWhile() instead, throwing out of the callback to reset the object -- nothing is delivered to the object for the duration of the block, so there is no moment between "everything is durable" and "the object is gone" at which anything can run. The two sync() calls collapse into the one inside the block, which covers every write issued before it (including a concurrent access mutation's). The scheduler.wait() stays outside, since blocking concurrency across the whole delay would stall unrelated requests. The rejection is swallowed: unlike ctx.abort(), this form rejects, and all three direct callers fire and forget. This retracts the input-gate argument the previous commit rested on: the gate is not what makes the barrier airtight. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A restart's whole effect is to sever live sessions, so the sharing table is the wrong thing to consult: a collaborator who isn't connected has no session to cut, and an entry for one bought a workspace-wide reset that reached only the owner. OverseerImpl now counts live sessions by the capability each holds, through a joinSession() pairing modelled on joinPresence(). Both client interfaces join synchronously in their constructor and leave in [Symbol.dispose], and #restartIfShared -- renamed #restartIfSessionsAffected, since it no longer asks about sharing -- gates on a live non-owner session of the affected role. Being synchronous is the point: the old form reached the sharing manager over RPC, so a widening could be missed outright when that lookup failed, and the restart was scheduled some unbounded time after the change. The counter is deliberately not derived from #presence, which looks like it already holds this: a session joins presence only once its fetchProfile() resolves, so a just-opened session is briefly invisible there -- fine for a roster, fail-open for an access decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`excludeObservers` blocked whenever a named observer was still authorized in the sharing graph, without asking whether that observer could actually see the connection naming them. A "use" collaborator's opens only verify them against gadget-bound connections, so unbinding one leaves their registration on it untouched -- their next open neither re-registers nor removes it -- and the gatekeeper goes on naming them forever, blocking observations from a connection they have no way to reach. `#enforceExcludeObservers` now takes the gatekeeper id its only caller already has, and blocks only when the observer is still authorized *and* that gatekeeper is still in their role's verification scope. Otherwise the observation proceeds and they are de-registered from that one gatekeeper; the record stays, since they are still a collaborator and a rebind must put them back in scope rather than start them from scratch. The scope test is fail-closed and narrow: out of scope means only "role is `use`, the connection requires an account, and no gadget binds it". It reuses #accountRequiringUseScope() rather than #inScopeGatekeepers, whose observerVendorId() throws on a legacy record with no creationSpec -- an unrelated legacy connection must not turn the observation path into an error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five fixes for the races the PR review surfaced, all in the same spirit: the session count must cover everything a collaborator holds a role's access through, and observer bookkeeping must only ever err toward over-registration (a spurious registration blocks fail-closed; a lost one fails open). - addGatekeeper publication window: the record must be durable before the scheduled reset, but the severed sessions stay live for the reset's ~100ms response-delivery delay and ids are guessable. When a restart was scheduled, the new id is marked in the in-memory #gatekeepersPendingRestart set -- in the same synchronous block as the put -- and getGatekeeperById/openSession refuse it with a retryable error until the reset destroys the mark with the sessions. - In-flight authorization: authorizeCollaborator holds a joinSession lease for the resolved role across ensureObserver, so an open parked on the configuration prompt or verifier RPCs counts as a session and a widening restarts it. receiveExternalMessage holds a "build" lease for a non-owner caller across the whole call, since it produces a reply from workspace data without ever constructing a counted interface. - Retained descendant capabilities: GadgetClientImpl, UseGadgetClientInterface, and GatekeeperClientImpl now count toward #hasCollaboratorSession for their own lifetime when minted into a collaborator session, since a client can dispose the parent interface while retaining a child stub. Owner mints and internal constructions don't count. - excludeObservers teardown: each observer is re-classified against current state adjacent to their own awaited removal. An observer put back in scope by a bind + fresh open mid-teardown blocks the observation instead of having their fresh registration deleted by the stale removal. - Returning-observer rollback: a failed open no longer rolls back any of a returning observer's registrations. The persisted observerId is shared with concurrent opens, so the rollback could delete a registration a concurrent successful open just made and persisted -- under-registering, the fail-open direction. Only a first-ever verification (whose observerId is private to the call) still rolls back. Also fixes the TS18047 null guard in observer-role-scope.test.ts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s in use scope. Closes the next round of review findings against the observer/session-restart work, one concern per fix: - Every "use"-scope widening (bindWorkpiece, merge promotion, and now enableHook) marks the widened connection ids in #gatekeepersPendingRestart, so the ~100ms pre-reset window can't hand a severed session a fresh binding loopback (shared helper #restartIfUseScopeWidened). - Enabled hooks are part of "use" verification scope (#useScopeGatekeeperIds): a hook is a live write channel into a gadget a "use" collaborator can open, so an unbound-but-hook-armed connection still blocks excluded observations, is verified on every "use" open, and enabling one is a widening. removeGatekeeper now synchronously deletes the connection's hook records (the authoritative kill) and fires the gatekeeper-side disables best-effort. - Subscriptions on both client interfaces hold a session lease for their own lifetime (#subscriptionLease), so a retained chat/action/console stream can't escape #hasCollaboratorSession after its interface is disposed. Also gives subscribeToChat's unsubscribe the disposed guard its siblings have. - receiveExternalMessage takes its "build" lease only after authorizeCollaborator admits the caller, so a denied stranger racing an addGatekeeper no longer causes a needless workspace reset. - removeCollaborator/revokeShareLink schedule the revocation restart in the same synchronous step as the sharing mutation, before the best-effort cross-DO cleanup that could previously delay it unboundedly. - addObserver/removeObserver RPCs are serialized per (observer, gatekeeper) (#withObserverGatekeeperLock), closing the disclosed in-flight remove-vs-add residual in the exclusion teardown. - The remaining client-reachable routes to a pending-restart connection are gated: the slash-command invoke and bindWithSuggestedName assert, while listSlashCommands and the ambient agent-catalog load silently omit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes for the re-review findings against 63fdaaf, one concern each: - startHook now asserts the connection is not quarantined pending a scope-widening restart: the enable that arms a hook is itself the widening that sets the mark, and the inbound delivery route was the one route that never checked it. - The hook state flips revalidate after their gatekeeper round trips instead of trusting the record captured before the await: enableHookRecord re-reads the hook and its connection, and when either was deleted mid-enable fires a best-effort compensating disable and throws rather than re-putting -- a resurrected enabled hook on a deleted connection kept delivering (startHook accepts via the denormalized vendorId) while being invisible to the widening detector. disableHook likewise re-reads rather than re-putting a deleted record as a zombie. - The raw gadget facet stub connectToGadget() returns now counts toward #hasCollaboratorSession for its own lifetime (getGadgetFacet's joinAs, released when the stub is disposed): enabling a hook widens "use" scope without aborting gadget facets, so a retained facet with every wrapper disposed made the widening silent -- no restart, no quarantine -- while the facet read the hook-written state indefinitely. - The chat-seed materialization in prepareChatBindings filters connections quarantined pending a restart, matching the ambient catalog fetch. The wider finding it came from -- an agent turn started through receiveExternalMessage outlives that RPC's session lease (and, via its persisted record, any reset) with no re-verification -- is deferred while nothing calls that endpoint, and documented as a KNOWN GAP at receiveExternalMessage with the required design spelled out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, recheck exclusions. Three fixes from review of the observer scope model: - #useScopeGatekeeperIds now closes transitively over agent spawner envs: a gadget-bound spawner hands a use collaborator's spawned agent every connection its config.env names (with the creator's authority), so those targets join use-scope verification, exclusion, and the widening diffs at every existing trigger. An unbound spawner stays unreachable, so no new trigger site is needed. - startHook now returns per-firing capabilities: a wrapper over the stored persistent callback plus an approval queue that both re-check the hook record on every call (requireLiveHook), implementing the session contract documented on Gatekeeper.bindHook. Previously the persistent callback itself was handed out, un-revocable across DO resets and other DOs, and the queue kept authorizing after a disable/delete shrank scope. startHook also re-reads the record after its admin-config await. - #enforceExcludeObservers re-classifies every named observer once more after the final awaited removal, whose window no loop-head re-check covered: a widening landing there could commit an observation against a stale out-of-scope classification and fail-open de-register an observer who was back in scope. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bonk's re-review of the wave-6 tree found two real P1s, both deferred by decision rather than disputed (neither is a regression this PR introduced, and neither is reachable by accident): - A binding loopback's props name the gatekeeper id, not the edge, and every call re-resolves a session revalidating nothing, so a retained loopback outlives unbindWorkpiece — the outOfScope de-register in #enforceExcludeObservers assumed unreachability that nothing enforces, and de-registering is fail-open for every later observation. Required fix recorded: #assertBindingEdgeLive in startGatekeeperSession. - The facet-stub lease covers only the stub the overseer minted; a gadget-minted child is independently owned and reachable only by the facet abort, which enableHookRecord — alone among the widenings — does not perform. Required fix recorded: unconditional bumpVersion([gadgetId]) in enableHookRecord. Also corrects the quarantine paragraph: a loopback is a per-call route, not something re-minted on facet reload. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s' hooks out of use scope. Two fail-closed scope-computation fixes from bonk's wave-8 review, both P2 availability/bookkeeping bugs in this PR's own code: - #inScopeGatekeepers now applies the use-scope filter before observerVendorId(), so an unrelated legacy connection (no creationSpec) outside a use collaborator's scope can't block their every open. An in-scope legacy record still throws, fail-closed, and build scope is unchanged (it is everything). - #useScopeGatekeeperIds no longer counts an enabled hook whose target gadget is still provisional to a chat: use collaborators can't open a pending gadget, so the hook's connection isn't reachable from their sessions. Promotion at merge is the widening moment, and mergeChanges' existing scope diff reports it then. An unresolvable target (no gadgetId and no default gadget, or a deleted record) stays in scope, fail-closed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…visional-hook scope exemption. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…enough. Undo the second sync and the blockConcurrencyWhile() barrier around the access restart (b6bbae7, 32df1e2). Neither closed a real gap: output gating already guarantees that a client never observes success for a write the abort discards, so a mutation that lands between the flush and the abort is simply one that never happened, from every observer's point of view. The extra machinery added complexity and documentation that described a hazard which does not exist. scheduleAccessRestart() is back to sync(), wait 100ms, abort(). The coalescing promise goes too: with ctx.abort() the first abort wins and later calls land on a dead object. Callers are unchanged (all three already fire and forget; deleteGadget() calling this from inside its own blockConcurrencyWhile() is the pre-existing shape on main). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…g them. The hardcoded `build > 0 || use > 0` would silently miss a new CollaboratorRole; iterate #liveSessions and skip only the owner's count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… together. The per-iteration re-classification and the final synchronous pass were guarding a window the design already tolerates: an observer put back in scope while a teardown is in flight is the same ~100ms tolerance the restart path accepts, not something worth a loop-head doctrine. Classify every named observer once, up front (so a blocked observation still leaves no teardown behind it), then start every removal synchronously and await them with one Promise.all. The per-(observer, gatekeeper) lock stays: it is what keeps a racing open's addObserver from being silently undone by an older removeObserver. The racing-registration test now expects the observation to be admitted and keeps asserting the remove/add ordering; the two tests that pinned the re-classification behaviour are gone, replaced by one covering two out-of-scope observers being de-registered together. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e scope. Re-apply the exemption 4e432f8 reverted: a hook waking a gadget that is still provisional to a chat stays out of "use" scope until promotion, which the merge's scope diff then reports. The revert was building complexity around a known, obscure bindHook attribution bug (its TODO stands) that will be fixed on its own; trusting `gadgetId` is fine. The legacy-connection widening from that commit is kept. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ely on. The spawner-env closure in #useScopeGatekeeperIds is complete because a spawned chat's agent has no requestConnection tool, and connection requests are created only by that tool -- so config.env is the ceiling on what a spawned chat can hold, not just the seed. Say so, and what must change if spawned agents ever gain the tool. makeHookFiringCallback guards the root callback only; capabilities a callback method returns are independent stubs the bindHook contract allows and are not re-checked. Say that this is deliberate: a guard against a stale firing by mistake, not a revocable membrane. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…m main. Main gave gadget records a `type` discriminant and agent steps `createdWorktrees`/`worktreeCommits`; the observer tests seeded neither, so their gadgets read as worktrees and their merges threw. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
d245e4c to
3601f93
Compare
|
Makes sense I should have caught this myself too thanks for the explanation, I dropped the commits in cb9a6bd. |
Same reply as this one #380 (comment), this is ok |
|
Validated sharing in previews with @ndisidore too |
Why:
Before this PR: We didn't sever existing observers, this was just actually documented in the original plan "existing observers see an incremental modal for just the new binding on their next open..."
The main gap we have right now is adding a binding does not restart live sessions, so an already-open collaborator is only verified against it at their next open.
What:
authorizeCollaborator() is now responsible for being the single gate for checking whether observers have access to underyling data. Everything routes through here, including
receiveExternalMessage()which previously only checked the role and never checked whether the observer had access to the underlying data.We also decided to abort the DO in order to revoke the RPC capabilities, we already use the same mechanism when revoking collaborators and we can just the same mechanism here. We made this a little less disruptive by checking graph first to see if there's any other collaborators before aborting the DO
Testing:
I added extensive integration tests in
observer-role-scope.test.ts,observer-reverification.test.tsandexternal-message-verification( even though it's actually used yet in any gatekeepers ) for testing the observer verification logic