Skip to content

docs: stale performance claims in BENCHMARK.md and USER_GUIDE.md #9

Description

@heynemann

Stale performance claims remaining in docs

While fixing the README/PERFORMANCE.md claims in #8, I found that BENCHMARK.md and USER_GUIDE.md carry the same class of stale numbers. These were left untouched to keep that PR reviewable; this issue tracks the follow-up.

BENCHMARK.md

  • Claimed throughput figures (e.g. "84.7M msg/s", "77.1M msg/s") predate the throughput-benchmark timing fix and were measured with a fixed 500ms sleep inside the timed window. True end-to-end fan-out measured after the fix: 9.9–10.7M deliveries/s (direct goroutines), 12.4–12.8M/s (worker pool), 1000 subscribers, Apple M4 Pro, Go 1.26.6.
  • "30–50x faster than MQTT" and "1,000–5,000x faster" wildcard claims are not reproducible. Measured vs MochiMQTT in-process routing: ~1.04x faster single publish, ~1.9x faster concurrent publish, 7–11x less memory per publish, 3.5x slower subscribe/unsubscribe churn.

USER_GUIDE.md

  • Repeats the "84.7M msg/s" and "10–13% faster" delivery-mode figures; the measured crossover is the reverse at 1000 subscribers (worker pool slightly faster than direct goroutines).
  • MaxConcurrentSubscriptions guidance ("optimal 5–300 worker pool / 1–750 direct", "up to 34x worse when set too high") is unverified in this session.

Suggested fix

Re-run the affected benchmarks and rewrite these files to cite the same measured ranges used in README/PERFORMANCE.md, or trim them down to qualitative guidance with a pointer to PERFORMANCE.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions