Summary
The service has known performance hazards — blocking upstream calls inside async handlers and a single small Render instance — but no measurements: there is no load-testing tooling, no latency baseline, and nothing stopping a PR from silently doubling p95. This issue adds Locust scenarios for /chat and /zakat (with mocked upstreams so runs are cheap and deterministic), a documented latency/throughput baseline, a CI job that fails on regression beyond an agreed budget, and a findings report on what the first measurements reveal.
Current state
/chat (main.py) calls the synchronous chat.send_message(...) inside an async def handler, and /zakat (stellar.py, fetch_usdc_balance()) calls Horizon through the synchronous stellar_sdk.Server(...).accounts().call() — both serialize the event loop under concurrency. Fixing the blocking is a separate open issue; this issue's job is to measure it, quantify the improvement when that fix lands, and prevent regressions after.- There is no perf tooling:
requirements.txt contains only runtime deps; CI (.github/workflows/ci.yml) runs flake8 + py_compile only.
- Latency is already an acknowledged pain point:
.github/workflows/keep-render-awake.yml exists solely to keep the Render instance warm.
What to build
- Mock-upstream test mode — the non-negotiable design point: load tests must not burn Gemini quota or hammer public Horizon. Add an app-level switch (
MOCK_UPSTREAMS=1) that swaps the Gemini call and the Horizon fetch for local stubs with configurable artificial latency (e.g. MOCK_LLM_LATENCY_MS, default ~800 to mimic reality). Implement as a thin seam at the two call sites — clean enough that it doubles as the test seam other issues need. Document clearly that mock-mode numbers measure the service, not the upstreams.
- Locust scenarios —
loadtest/locustfile.py:
/chat: new-session single-turn, multi-turn on one chat_id (exercises the active_chats history growth), and a mixed profile with realistic think time./zakat: valid testnet key, invalid key (400 path), unknown-account (404 path) — error paths are part of the budget too.- Parameterized users/spawn-rate/duration via env so the same file serves local runs and CI.
- Baseline document —
docs/performance.md: methodology (hardware, mock latencies, versions), and a results table: p50/p95/p99 latency, throughput, error rate, at 1/10/50 concurrent users for each endpoint. Include the headline finding you will almost certainly observe: with N concurrent users and a blocking LLM call, /chat latency grows ~linearly with N (event-loop serialization) — quantified, with a pointer to the open event-loop-blocking issue.
- Performance budget in CI —
loadtest/budget.yaml (per endpoint: max p95, min RPS, max error rate at a fixed small load appropriate for a CI runner) and a perf job in .github/workflows/ci.yml: start uvicorn with MOCK_UPSTREAMS=1, run Locust headless (~60–90 s), parse the stats CSV, compare against the budget, fail on breach with a readable diff. Keep the job non-flaky: generous-but-meaningful margins, warmup excluded, and document how to update the budget when a legitimate perf change lands.
- Findings report — a short section in
docs/performance.md (or the PR description) listing concrete observed issues with numbers — e.g. event-loop serialization impact, unbounded active_chats memory growth across a long run (visible via RSS sampling during the soak), cold-start impact — each linked to the existing issue that owns the fix, or proposed as a new issue.
Acceptance criteria
Pointers
main.py (chat() — the blocking send_message call site; active_chats growth), stellar.py (fetch_usdc_balance — blocking Horizon call), .github/workflows/ci.yml, render.yaml.- Do not fix the event-loop blocking here — that's a separately assigned issue; your seam + baseline is what proves their fix works (before/after numbers are a great PR artifact for them).
- Locust's
--headless --csv output is the easiest thing to parse for the budget gate; k6 is acceptable if you prefer, but justify the deviation and keep the runner installable in CI without Docker.
- GitHub Actions runners are noisy neighbors — pin the budget to coarse thresholds (e.g. p95 < 2× expected) rather than tight ones, or the job will cry wolf and get ignored.
- PRs target
dev (see CONTRIBUTING.md).
Difficulty
Medium — no algorithmic depth, but doing load testing credibly (mock seams, non-flaky CI gating, honest methodology) requires solid engineering judgment.
🏆 GrantFox OSS — Official Campaign | FWC26. Apply for this issue through the GrantFox campaign page. The maintainer assigns one contributor before work starts; unassigned PRs may not be reviewed. PRs target the dev branch. Quality bar: CI must stay green.
💬 Questions or need help? Reach the maintainers and other contributors on the DeenBridge Telegram: https://t.me/+nst9lXNj1wc4ZDE0
Summary
The service has known performance hazards — blocking upstream calls inside async handlers and a single small Render instance — but no measurements: there is no load-testing tooling, no latency baseline, and nothing stopping a PR from silently doubling p95. This issue adds Locust scenarios for
/chatand/zakat(with mocked upstreams so runs are cheap and deterministic), a documented latency/throughput baseline, a CI job that fails on regression beyond an agreed budget, and a findings report on what the first measurements reveal.Current state
/chat(main.py) calls the synchronouschat.send_message(...)inside anasync defhandler, and/zakat(stellar.py,fetch_usdc_balance()) calls Horizon through the synchronousstellar_sdk.Server(...).accounts().call()— both serialize the event loop under concurrency. Fixing the blocking is a separate open issue; this issue's job is to measure it, quantify the improvement when that fix lands, and prevent regressions after.requirements.txtcontains only runtime deps; CI (.github/workflows/ci.yml) runs flake8 +py_compileonly..github/workflows/keep-render-awake.ymlexists solely to keep the Render instance warm.What to build
MOCK_UPSTREAMS=1) that swaps the Gemini call and the Horizon fetch for local stubs with configurable artificial latency (e.g.MOCK_LLM_LATENCY_MS, default ~800 to mimic reality). Implement as a thin seam at the two call sites — clean enough that it doubles as the test seam other issues need. Document clearly that mock-mode numbers measure the service, not the upstreams.loadtest/locustfile.py:/chat: new-session single-turn, multi-turn on onechat_id(exercises theactive_chatshistory growth), and a mixed profile with realistic think time./zakat: valid testnet key, invalid key (400 path), unknown-account (404 path) — error paths are part of the budget too.docs/performance.md: methodology (hardware, mock latencies, versions), and a results table: p50/p95/p99 latency, throughput, error rate, at 1/10/50 concurrent users for each endpoint. Include the headline finding you will almost certainly observe: with N concurrent users and a blocking LLM call,/chatlatency grows ~linearly with N (event-loop serialization) — quantified, with a pointer to the open event-loop-blocking issue.loadtest/budget.yaml(per endpoint: max p95, min RPS, max error rate at a fixed small load appropriate for a CI runner) and aperfjob in.github/workflows/ci.yml: start uvicorn withMOCK_UPSTREAMS=1, run Locust headless (~60–90 s), parse the stats CSV, compare against the budget, fail on breach with a readable diff. Keep the job non-flaky: generous-but-meaningful margins, warmup excluded, and document how to update the budget when a legitimate perf change lands.docs/performance.md(or the PR description) listing concrete observed issues with numbers — e.g. event-loop serialization impact, unboundedactive_chatsmemory growth across a long run (visible via RSS sampling during the soak), cold-start impact — each linked to the existing issue that owns the fix, or proposed as a new issue.Acceptance criteria
MOCK_UPSTREAMS=1runs the full app with stubbed Gemini/Horizon and configurable stub latency; no real upstream calls occur under load tests (verified by running with noGEMINI_API_KEY)./chat(single-turn, multi-turn, mixed) and/zakat(success + 400 + 404) and run locally with one documented command.docs/performance.mdcontains the methodology and a filled baseline table (p50/p95/p99, RPS, error rate at 3 concurrency levels per endpoint).perfjob that runs headless Locust against mock mode and fails whenbudget.yamlis breached; the job passes on currentdevwith the committed budget.budget.yaml).Pointers
main.py(chat()— the blockingsend_messagecall site;active_chatsgrowth),stellar.py(fetch_usdc_balance— blocking Horizon call),.github/workflows/ci.yml,render.yaml.--headless --csvoutput is the easiest thing to parse for the budget gate; k6 is acceptable if you prefer, but justify the deviation and keep the runner installable in CI without Docker.dev(seeCONTRIBUTING.md).Difficulty
Medium — no algorithmic depth, but doing load testing credibly (mock seams, non-flaky CI gating, honest methodology) requires solid engineering judgment.
🏆 GrantFox OSS — Official Campaign | FWC26. Apply for this issue through the GrantFox campaign page. The maintainer assigns one contributor before work starts; unassigned PRs may not be reviewed. PRs target the
devbranch. Quality bar: CI must stay green.💬 Questions or need help? Reach the maintainers and other contributors on the DeenBridge Telegram: https://t.me/+nst9lXNj1wc4ZDE0