Skip to content

docs: file BUG report for staging VPC connector saturation - #430

Open
ran-taig wants to merge 1 commit into
mainfrom
docs/bug-staging-vpc-connector-saturation
Open

ran-taig wants to merge 1 commit into
mainfrom
docs/bug-staging-vpc-connector-saturation

Conversation

@ran-taig

Copy link
Copy Markdown
Contributor

Summary

Files the RCA report for the staging VPC Serverless Connector saturation bug that silently disables Path C entity-aware contradiction detection on dev.

Grounded in timestamped Cloud Run + storage_client logs from the 2026-06-16 wet-test window:

  • Storage server p99 was healthy (15-141ms) during the failing minute.
  • 9 simultaneous POST /memories/entity-links calls from core-api all hit ConnectTimeout at TCP-handshake — they never reached the server.
  • memclaw-staging-vpc connector was e2-micro × min=2, shared by 10+ Cloud Run services with vpc-access-egress=all-traffic.
  • The 30s _CONTEXT_FETCH_TIMEOUT_SECONDS (CAURA-134) cannot be raised high enough to mask a connector that drops handshakes outright.

Mitigation status (verified before PR)

Infra has applied Option A: connector machine type upgraded e2-micro → e2-standard-4. Path C failure rate dropped from ~100% during bursts to ~10-15% (see PR description for daily trend). The 9-way fan-out shape from _fetch_entity_context is still observable in logs and remains a separate architectural concern (tracked elsewhere).

Test plan

  • Doc-only change — no code paths touched
  • Reviewers verify log-query examples reproduce against alpine-theory-469016-c8

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'ran-taig' is not a public member of the 'caura-ai' org

@ran-taig
ran-taig force-pushed the docs/bug-staging-vpc-connector-saturation branch from 8d2c518 to 496853f Compare September 24, 2026 10:05
@ran-taig
ran-taig requested a review from a team as a code owner September 24, 2026 10:05
@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'ran-taig' is not a public member of the 'caura-ai' org

@ran-taig
ran-taig force-pushed the docs/bug-staging-vpc-connector-saturation branch from 496853f to 6252abc Compare September 24, 2026 18:53
@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'ran-taig' is not a public member of the 'caura-ai' org

@ran-taig
ran-taig force-pushed the docs/bug-staging-vpc-connector-saturation branch from 6252abc to 5198e4a Compare September 24, 2026 22:16
@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'ran-taig' is not a public member of the 'caura-ai' org

@ran-taig
ran-taig force-pushed the docs/bug-staging-vpc-connector-saturation branch from 5198e4a to 02482ee Compare September 25, 2026 12:23
@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'ran-taig' is not a public member of the 'caura-ai' org

Captures the RCA chain for the Path C entity-aware judge being silently bypassed:
e2-micro VPC connector saturating under burst → ConnectTimeout cascade →
asyncio.gather exhausts the 30s _CONTEXT_FETCH_TIMEOUT_SECONDS budget → fallback
to base LLM judge. Storage-server-side latency is fine (15-141ms); the
bottleneck is connection establishment through the shared connector.

Grounded with timestamped log queries; includes two infra-side mitigations
(machine-type upgrade or egress-mode change) and a verification plan.

Signed-off-by: Ran Taig <ran@caura.ai>
@ran-taig
ran-taig force-pushed the docs/bug-staging-vpc-connector-saturation branch from 02482ee to 3325d9f Compare September 25, 2026 19:49
@github-actions

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'ran-taig' is not a public member of the 'caura-ai' org

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant