Skip to content

mev-protect: separate coverage from reliability in success #1094

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Flotapponnier merged 1 commit into from
Jul 10, 2026
Merged

mev-protect: separate coverage from reliability in success #1094

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions benchmarks/mev-protect-rpc.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ methodology:
- "Headline: median latency across the methods the provider served that tick, aggregated over 24h with quantile_over_time. A rejected method does not poison the latency; it lowers the coverage panel instead."
- "Coverage: mev_rpc_methods_supported counts the wallet methods served on the last tick (max 7). Flashbots rejects eth_call on the public endpoint; Blink blocks full-node reads like eth_getBlockByNumber, which keeps it off the general RPC benches but not off this one."
- "Out of scope, disclosed: inclusion rate, refund economics and sandwich protection efficacy are the write path and require funded transactions. See the arXiv study Private MEV Protection RPCs (2505.19708) for a one-off execution-quality comparison."
- "Failures (timeouts, 403, 429) increment mev_rpc_call_total{result} and count against the success rate; the gauge keeps its last value so the chart shows the outage in the success column rather than a fake zero."
- "Success rate counts transport failures only (timeouts, throttling, network); a method a gateway refuses by policy is a coverage gap shown in the coverage panel, not an outage. Failures increment mev_rpc_call_total{result}; the gauge keeps its last value so the chart shows the outage in the success column rather than a fake zero."

findings:
- "{{best_name}} leads at {{best_p50}} (p50, 24h) on the cross-region wallet call median over {{count}} measured gateways."
Expand Down Expand Up @@ -100,7 +100,7 @@ providers:
p90: quantile_over_time(0.90, mev_rpc_wallet_latency_milliseconds{provider="flashbots"}[24h])
p99: quantile_over_time(0.99, mev_rpc_wallet_latency_milliseconds{provider="flashbots"}[24h])
mean: avg_over_time(mev_rpc_wallet_latency_milliseconds{provider="flashbots"}[24h])
success: sum(increase(mev_rpc_call_total{provider="flashbots",result="ok"}[24h])) / clamp_min(sum(increase(mev_rpc_call_total{provider="flashbots"}[24h])), 1)
success: sum(increase(mev_rpc_call_total{provider="flashbots",result="ok"}[24h])) / clamp_min(sum(increase(mev_rpc_call_total{provider="flashbots",result!~"blocked|method_not_found"}[24h])), 1)
sample_size: sum(increase(mev_rpc_call_total{provider="flashbots"}[24h]))
series: avg(avg_over_time(mev_rpc_wallet_latency_milliseconds{provider="flashbots"}[1h]))
regions:
Expand All @@ -123,7 +123,7 @@ providers:
p90: quantile_over_time(0.90, mev_rpc_wallet_latency_milliseconds{provider="mevblocker"}[24h])
p99: quantile_over_time(0.99, mev_rpc_wallet_latency_milliseconds{provider="mevblocker"}[24h])
mean: avg_over_time(mev_rpc_wallet_latency_milliseconds{provider="mevblocker"}[24h])
success: sum(increase(mev_rpc_call_total{provider="mevblocker",result="ok"}[24h])) / clamp_min(sum(increase(mev_rpc_call_total{provider="mevblocker"}[24h])), 1)
success: sum(increase(mev_rpc_call_total{provider="mevblocker",result="ok"}[24h])) / clamp_min(sum(increase(mev_rpc_call_total{provider="mevblocker",result!~"blocked|method_not_found"}[24h])), 1)
sample_size: sum(increase(mev_rpc_call_total{provider="mevblocker"}[24h]))
series: avg(avg_over_time(mev_rpc_wallet_latency_milliseconds{provider="mevblocker"}[1h]))
regions:
Expand All @@ -146,7 +146,7 @@ providers:
p90: quantile_over_time(0.90, mev_rpc_wallet_latency_milliseconds{provider="blinklabs"}[24h])
p99: quantile_over_time(0.99, mev_rpc_wallet_latency_milliseconds{provider="blinklabs"}[24h])
mean: avg_over_time(mev_rpc_wallet_latency_milliseconds{provider="blinklabs"}[24h])
success: sum(increase(mev_rpc_call_total{provider="blinklabs",result="ok"}[24h])) / clamp_min(sum(increase(mev_rpc_call_total{provider="blinklabs"}[24h])), 1)
success: sum(increase(mev_rpc_call_total{provider="blinklabs",result="ok"}[24h])) / clamp_min(sum(increase(mev_rpc_call_total{provider="blinklabs",result!~"blocked|method_not_found"}[24h])), 1)
sample_size: sum(increase(mev_rpc_call_total{provider="blinklabs"}[24h]))
series: avg(avg_over_time(mev_rpc_wallet_latency_milliseconds{provider="blinklabs"}[1h]))
regions:
Expand Down
Loading