fix(lesson 5.6): Incidente 6 needed 2 bumps — 5min watch was falsely stable - #24
Merged
Merged
Conversation
…e revealed only in hours of real load Lesson V3 (PR #23) declared MemoryHigh bump 1500M→1800M resolved the issue based on 5min watch (memory 1.5G stable, high events frozen). It wasn't. Recurrence T+1h45min: canary FAIL(2) again at 18:17. Memory had crept back to 100% MemoryHigh (1.8G/1.8G) over 43min uptime. high_events grew 4554 → 21793 (+17K). Latency degraded 3s+ sequential, 8-13s concurrent. Cause: working set real grows with usage. Daily queries warm caches (chunks/embeddings/FTS5). 5min snapshot post-restart was falsely stable (restart cleared caches, queries hadn't run yet). Bump 2: MemoryHigh 1800M→2400M, MemoryMax 2500M→3000M. Result after 10min watch: memory 1.7G constant (real working set revealed), high events ZERO, latency 0.9-1.5s consistent, canary 2 consecutive OKs. Lesson refined: - Working set steady-state revealed in HOURS, not 5min - Run 24h+ under real load before sizing - MemoryHigh = peak * 1.4 (40% headroom, not 20%) - MemoryMax = MemoryHigh * 1.25 - Re-monitor 24h post-bump Anti-pattern: declaring resolved after 5min observation post-restart. Restart clears cache, queries haven't run yet, snapshot is falsely stable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #23 declared the memory bump 1500M→1800M resolved Incidente 6 based on 5min watch. It wasn't. Recurrence T+1h45min: same canary alert disparou novamente. Bump 2 (1800M→2400M) é o que realmente resolveu.
Root cause da false-positive
Working set real cresce com uso. Daily queries warm caches (chunks/embeddings/FTS5). 5min snapshot pós-restart era falsamente estável — restart cleared caches, queries hadn't run yet.
Métricas comparativas
/api/searchLição refinada (mais valiosa que a V3)
Working set steady-state revelado em horas, não em 5min snapshot. Padrão correto:
Anti-padrão: declarar resolvido após 5min observação — restart libera cache, queries não rodaram, snapshot falsamente estável.
Test plan
🤖 Generated with Claude Code