CBG-5950 guard DCP client metadata with per-vbucket locks

[torcolvin]

A vbucket's DCP metadata is written by the owning DCP worker (SetSnapshot, UpdateSeq, Persist), but also off the worker: the OpenStream callback calls SetFailoverEntries, openStream calls Rollback and GetMeta, and GetMetadata reads every vbucket. Nothing synchronized these, so the race detector flags SetFailoverEntries and UpdateSeq against Persist. CBG-1866 fixed the failover case by routing it through the worker, and CBG-3247 undid that to keep the callback from blocking on the worker channel.

Lock each vbucket in every accessor. Persist copies each vbucket under its lock and writes to the bucket outside it, and the constructors share newDCPMetadataBase.

Why mutexes are fine here:

• The locks are per vbucket, so workers never contend with each other. A vbucket's lock is only contended when a stream opens, rolls back or GetMetadata runs, which is rare next to mutations.
• Uncontended, the lock adds about 10ns to UpdateSeq (2.5ns to 12.5ns), which is small next to the mutation callback each event already runs. 32 goroutines across 1024 vbuckets average 5ns per call.
• No lock is held across I/O or a channel send, so the OpenStream callback still never waits on a worker queue, which keeps the goal of CBG-3247.

Add tests that reproduce both races; they fail under -race without the locks.

Pre-review checklist

• Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
• Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Integration Tests

• https://jenkins.sgwdev.com/job/SyncGatewayIntegration-Pipeline/0000/

[factory-droid]

Droid finished @torcolvin's task —— View job

---

LGTM — I did not find any high-confidence, actionable issues in the locking changes or the added race-repro tests.

[Copilot]

Copilot review overview

🟡 Changes recommended

Rollback holds its vbucket mutex during potentially blocking trace logging, which can stall worker operations.

Review effort: Balanced
Findings: 1

Open (1)

• Avoid trace logging while holding the vbucket mutex · New

What changed in this PR

Protects Sync Gateway’s DCP metadata from concurrent access using per-vbucket locks.

Changes:

• Synchronizes metadata access and shares constructor initialization.
• Copies metadata under locks before persistence.
• Adds concurrency regression tests.

File	Description
base/​dcp_client_metadata.go	Adds per-vbucket locking and synchronized persistence snapshots.
base/​dcp_client_metadata_test.go	Tests concurrent persistence, failover updates, and rollback.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.