Skip to content

fix: project agent mentions, mesh_demo flake, Builderlab sign-in timeout#2498

Open
Chessing234 wants to merge 4 commits into
block:mainfrom
Chessing234:fix/project-mentions-mesh-flake
Open

fix: project agent mentions, mesh_demo flake, Builderlab sign-in timeout#2498
Chessing234 wants to merge 4 commits into
block:mainfrom
Chessing234:fix/project-mentions-mesh-flake

Conversation

@Chessing234

@Chessing234 Chessing234 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Project issue/PR @mentions publish as kind:9 with channel h (and read buzz-channel) so buzz-acp receives them (#2462)
  • Serialize + clear Redis keys for mesh_demo echo tests under parallel cargo test (#2458)
  • Cap Builderlab “Waiting for your browser…” with timeout, retry, and back-to-relay recovery (#2484)

Test plan

  • Link a project to a channel (buzz-channel), @mention an agent on an issue — agent gets a turn
  • cargo test -p buzz-relay --lib (default parallelism) — mesh_demo tests stay green
  • Hosted community sign-in: kill/block Builderlab — UI times out with Retry / Back

Fixes #2462
Fixes #2458
Fixes #2484

Made with Cursor

Chessing234 and others added 3 commits July 23, 2026 14:05
Signed-off-by: Taksh <takshkothari09@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Taksh <takshkothari09@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Taksh <takshkothari09@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@brocoppler

Copy link
Copy Markdown

Scoped to the mesh_demo/#2458 part of this PR, against head 7b99a27.

The serialization doesn't address the mechanism behind #2458.

The Redis keys can't collide. Both tests generate fresh UUIDv4 community_id/session_id per run, and the lease keys are scoped buzz:{community}:tunnel:{session}:… — no concurrent test, mesh_demo or otherwise, can touch them. clear_keys releases a lease (RuntimeId([1; 32])) that is never acquired and DELs keys that cannot pre-exist; both the pre- and post-test calls are no-ops.

The mechanism is the 5s test lease TTL. redis_directory_if_available builds the directory with with_lease_ttl(pool, Duration::from_secs(5)). The forwarded-arm test acquires the owner lease, then does QUIC endpoint setup and task spawns before the forwarding join. Under full-suite parallelism (~760 tests competing for CPU) the wall clock between the two joins can exceed the TTL; the owner lease expires, directory.acquire on the forwarding side succeeds, join returns Owned, and the "forwarded" assert fails.

Deterministic repro on this branch — mutex held, clear_keys in place, run standalone so there is zero cross-test contention:

// inserted between the owner join assert and the forwarding join
tokio::time::sleep(Duration::from_secs(6)).await;
assertion `left == right` failed
  left: String("owned")
 right: "forwarded"
test result: FAILED. 0 passed; 1 failed; ... finished in 6.21s

The mutex removes the sibling test from the contention pool, which narrows the window; the other ~760 tests still widen it. The flake gets rarer instead of impossible, which is the worst outcome for a flake fix — it resurfaces after this lands as "fixed".

Separately, just ci rejects the lock as written: cargo clippy --workspace --all-targets -- -D warnings fails with two await_holding_lock errors (the std::sync::MutexGuards are held across the tests' .awaits).

Minimal fix: raise the test directory TTL. Neither test exercises expiry, the production default is 30s, and the passing path holds the lease well under a second — with_lease_ttl(pool, Duration::from_secs(60)) closes the window with no lock and no key clearing, and moots the clippy failure.

The 5s test TTL expired under full-suite load before the forwarded
join finished. Drop the mutex/clear_keys workaround (clippy
await_holding_lock) and use a 60s lease instead.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Taksh <takshkothari09@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@Chessing234
Chessing234 force-pushed the fix/project-mentions-mesh-flake branch from 09baef4 to 1a9b11f Compare July 24, 2026 07:15
@Chessing234

Copy link
Copy Markdown
Contributor Author

Raised the mesh_demo test lease TTL to 60s and dropped the mutex/clear_keys workaround — that was the real #2458 failure mode under full-suite load, and it also clears the await_holding_lock clippy hits.

@brocoppler

Copy link
Copy Markdown

Verified 1a9b11f with the same instrument as before. Two results — one closes this out, one is new.

The lease mechanism is fixed. Unmodified head, standalone: both tests pass (0.25s). The 6s-delay instrument can no longer flip owned/forwarded — with the 60s TTL that path is closed, and the mutex/clear_keys removal is the right shape.

The instrument found a second, pre-existing sensitivity the TTL does not touch. Inserting a delay between the owner join and the owner-task spawn now fails the forwarded arm with a different signature — 504 (echo timeout) instead of owned != forwarded:

head delay result
1a9b11f (60s TTL) none pass (0.25s)
1a9b11f 1s 504
1a9b11f 3s 504
1a9b11f 6s 504
7b99a27 (5s TTL + mutex) 1s 504

The last row is the important one: this reproduces on the pre-fix head, so #2458 has had two overlapping mechanisms all along — lease expiry (~5s threshold, fixed here) and something in the echo path with a ≤1s threshold that I could not match to any timeout constant in buzz-relay-mesh (the only 5s timeouts there are test-local). Under the same suite-load stalls that produced the original flake, this second mechanism can still fire, so the flake likely persists at some rate with a 504 signature.

Suggest rewording Fixes #2458Mitigates #2458 so the merge doesn't auto-close the issue — I've posted the mechanism-2 evidence there. The TTL change itself is correct and worth landing as-is.

@brocoppler

Copy link
Copy Markdown

Closing the loop on the "second, pre-existing ≤1s mechanism" from my verification comment: root-caused in #2745 — a cancel-safety bug in run_demo_echo (the drain tick cancels recv_validated during Redis fence validation, destroying the frame it had already read), not a transport threshold. The sleep-matrix numbers were sampling tick alignment; my interpretation was wrong, the fix is there.

#2745 deliberately doesn't touch this PR's TTL line — the two compose, and with both in, Fixes #2458 here is accurate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants