Skip to content

feat: nightly hardening - jittered retry backoff#32

Open
mouse-value-add wants to merge 1 commit into
brainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260523-retry-jitter
Open

feat: nightly hardening - jittered retry backoff#32
mouse-value-add wants to merge 1 commit into
brainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260523-retry-jitter

Conversation

@mouse-value-add

Copy link
Copy Markdown
Contributor

Problem

When many requests fail at once, all retries currently occur on identical exponential schedules. This can create synchronized retry spikes against providers and increase cascading failures.

Approach

  • Added optional to (default )
  • Applied bounded random jitter to each retry sleep ()
  • Clamped negative jitter input to for safety
  • Added focused async tests for jitter behavior and negative-input hardening

Verification

  • Ran: ============================= test session starts ==============================
    platform darwin -- Python 3.9.6, pytest-8.4.2, pluggy-1.6.0
    rootdir: /Users/mouse/.openclaw/workspace/nightly/PromptLens
    configfile: pyproject.toml
    plugins: anyio-4.12.1, asyncio-1.2.0, cov-7.1.0
    asyncio: mode=strict, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
    collected 2 items

tests/test_retry_backoff_jitter.py .. [100%]

================================ tests coverage ================================
_______________ coverage: platform darwin, python 3.9.6-final-0 ________________

Name Stmts Miss Cover

promptlens/init.py 3 0 100%
promptlens/main.py 3 3 0%
promptlens/cli.py 169 169 0%
promptlens/exporters/init.py 6 6 0%
promptlens/exporters/base.py 15 15 0%
promptlens/exporters/csv_exporter.py 26 26 0%
promptlens/exporters/html_exporter.py 38 38 0%
promptlens/exporters/json_exporter.py 16 16 0%
promptlens/exporters/markdown_exporter.py 77 77 0%
promptlens/judges/init.py 3 3 0%
promptlens/judges/base.py 15 15 0%
promptlens/judges/llm_judge.py 71 71 0%
promptlens/judges/parser.py 107 107 0%
promptlens/judges/prompts.py 30 30 0%
promptlens/loaders/init.py 4 4 0%
promptlens/loaders/base.py 14 14 0%
promptlens/loaders/json_loader.py 23 23 0%
promptlens/loaders/yaml_loader.py 34 34 0%
promptlens/models/init.py 4 4 0%
promptlens/models/config.py 41 41 0%
promptlens/models/result.py 60 60 0%
promptlens/models/test_case.py 25 25 0%
promptlens/models/tools.py 74 74 0%
promptlens/providers/init.py 3 3 0%
promptlens/providers/anthropic.py 52 52 0%
promptlens/providers/base.py 23 23 0%
promptlens/providers/factory.py 21 21 0%
promptlens/providers/google.py 47 47 0%
promptlens/providers/http.py 67 67 0%
promptlens/providers/openai.py 57 57 0%
promptlens/providers/you.py 57 57 0%
promptlens/runners/init.py 2 2 0%
promptlens/runners/runner.py 96 96 0%
promptlens/utils/init.py 1 0 100%
promptlens/utils/cost.py 15 15 0%
promptlens/utils/diff.py 25 25 0%
promptlens/utils/retry.py 25 4 84%
promptlens/utils/timing.py 24 24 0%

TOTAL 1373 1348 2%
Coverage HTML written to dir htmlcov
============================== 2 passed in 0.24s ===============================

  • Result:

Risks

  • Retry timing is now non-deterministic by default, which can affect tests that assume exact sleeps.
  • Mitigation: jitter remains small (10% by default), and tests can set deterministic random behavior via monkeypatch.

Rollback Plan

  • Revert this PR to restore fixed-delay retry behavior.
  • Alternatively, set at call sites to disable jitter without reverting code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant