Operational event: degraded storage provider performance where some deal creation is exceeding monitoring timeouts (2026-07-09)

[BigLep]

Impact

Public service dashboards (e.g., https://probelab.io/filecoin/foc/, https://telemetry.betterstack.com/dashboards/Zz8k7L?rf=now-72h&rt=now from dealbot.filoz.org, https://spdash.ezpdpz.net/) show degraded performance for approved SPs with "data storage" checks and "data retention" checks.

Per https://probelab.io/filecoin/foc/providers/approved/?days=7&foc-range=on, the main SP driving these dashboards red is ezpdpz-main. Proving remains healthy on this provider; the delayed confirmations affect deal onboarding latency.

Data Storage

Data sets are still being created and pieces added successfully, but they are experiencing delay as [ezpdpz-main] is involved with onboarding data from Storacha. It's experiencing contention is in the wallet nonce lane at the message layer where dealbot jobs get queued behind the migration work missing dealbot's timeouts. Either dealbot's request's need special or (see filecoin-project/curio#1313) or dealbot's timeout parameters need an increase.

Data Retention

With the migration, messages generally land late but inside the window. We have found zero faults in the incidents. Our actual proving misses were with data set 1245 auto-termination (see filecoin-project/curio#1329 and FilOzone/pdp#283 where a removals flush wedged nextProvingPeriod) and a one-off proving wallet funding blip on ezpdpz-main.

Current status

• Because our service dashboards aren't giving a clear signal and are requiring more nuance at the moment, we are continuing to check in on individual SP quality of service directly.
• We are working through the large onboarding. Expected completion data of this data onboarding will be shared by EOD 2026-07-10.
• We are in progress on evaluating and correcting the issues above so our monitoring is more resilient to seasons of larger onboarding.

We will update this issue if we detect actual sustained failures in onboarding data and proving it.

Suggested user action

Users don't need to take any action currently. Their data is being stored, and retrievals continue to be served without issue (as evidence by high retrieval check success rate in the dashboards).

Timeline

• 2026-07-09T18:34:09Z: Event opened.

Public status notice

https://status.filecoin.cloud/incident/947574

Dashboards

• https://probelab.io/filecoin/foc/providers/approved/
• https://telemetry.betterstack.com/dashboards/Zz8k7L?rf=now-72h&rt=now

[ZenGround0]

Myself @LexLuthr and @snadrus have been focused on resolving this issue over the last few days. Thankfully the worst pieces of it are related to issues we understand well.

In response to this incident I went through all of our SP logs to classify proving errors. The findings are written up here. All proving errors are cases of a well documented and debugged curio proving error. This error has the nice property of not actually losing any data. So while the dashboard is indicating "data retention failures" there is actually no data lost, just proofs missed.

Furthermore we've been designing a solution to this for a while. Our latest iteration is @LexLuthr's PR up here which I am reviewing. This alongside the work tracked in curio #1359 to clean up the bad state in SP's databases should resolve 100% of the proving failures. At least the failures we know about so far.

Data onboarding latency is the other problem highlighted in this incident. There is a lot of discussion and some solutions proposed in this issue. However our takeaway is that this is fundamentally more of a shared hardware, metrics definition and user flow issue than a software issue. This will be trickier to resolve but the conversation on what exactly to do has at least gotten started.

[TippyFlitsUK]

Update 2026-07-21: two further degradation causes root-caused over the weekend (both distinct from curio#1313)

Dealbot deal success dipped on all healthy mainnet SPs (endorsed IDs 5 and 7 included) from 2026-07-19, with retrievals at 100% throughout. Full trace of every failure in a 25h window:

• 11 of 13: gas-squeeze commit lag on stock Curio. Data stored every time; addPieces confirmed 10-100 mins late, blowing dealbot's 300s window - all false negatives. Proven per-tx on-chain: infrafolio's wallet lane went silent for 99 minutes then drained 12 nonce-ordered messages (addPieces + proving) in a single block when base fee dipped. Sampled txs are type 0x0 legacy - the curio#1354 zero-headroom sender defects, live on external stock-Curio SPs. Base fee was held in the 1.5-2.5M band by the Storacha migration commits (~half of InvokeContract volume) plus f0870558's mass sector-termination campaign (~42-45% of ALL mainnet fees, still running as of 12:35 UTC today). SPs 5 and 7 also dropped a handful of proving periods in the window - proving shares the stranded wallet lane; no data loss.
• 2 of 13: eRPC-side (rate-limit + a bogus "provider not found in registry" read) - infra#325.

Escalation overnight (07-21): base fee plateaued at 2.0-2.2M for ~8h and an ExtendClaimTerms wave (~23% of fees, many wallets) stacked on top. Zero 10+ piece addPieces batches landed network-wide between 07:00 and 12:30+ UTC - including from my patched senders (259 + 85 messages backloaded with fully-priced caps). This phase is giant-slot inclusion competition, not cap stranding: fee bumping is not a lever; premiums, pausing commits, or waiting out the wave are.

Monitoring implications for this issue: the 300s confirmation window is far below worst-case inclusion during network gas events (hours, not minutes) - timeout semantics and replacement-message handling now under discussion with @rvagg, beck and @LexLuthr. Per agreement with Jennifer, confirmed network-side causes now get a status.filecoin.cloud notice plus a stand-down note in the SP channel (done for this event).

User action: none - data is stored and retrievals are unaffected.

[TippyFlitsUK]

Update 2026-07-24: congestion has escalated and is now externally driven; termination wave is a rolling series, not one miner

Base fee has ratcheted to double last week's band: sampled at 2.6-5.7M attoFIL across 22-24 Jul vs 1.5-2.5M during last week's events. Peak samples: 5.72M (23 Jul 13:30 UTC), 5.10M (24 Jul 13:25 UTC).

The external load composition (Filfox, trailing 24h):

• TerminateSectors: 48.9% of all mainnet fees (4,108 messages)
• SettleDealPaymentsExported: 22.6% (5,142 messages - two wallets settling against the market actor on an automated ~1 msg/min cadence)
• ExtendClaimTerms: 12.9%

The termination wave is a rolling series. f0870558 is nearly drained (~21k of its original ~161k sectors remain), but the same GLIF agent contract that owns it is now terminating its next miner, f01106668 (33k sectors), with a second terminator f02309036 (18k sectors, different owner contract) running in parallel at the same scripted cadence. At least ~52k sectors remain queued across the two active miners.

This week's congestion is external demand alone. My Storacha migration runs have been stopped for most of the last 48h - InvokeContract is down to ~9% of all fees - and base fee still ratcheted to its highest levels yet. Commits only land when base fee dips (166 batches landed 23 Jul evening during a dip to ~2.6M); zero addPieces batches of 10+ pieces landed network-wide between 03:00 and 13:00 UTC today, and a batch that does land costs ~3x last week's price.

Impact: unchanged in kind from the 21 Jul update - dealbot deal checks report false negatives under confirmation lag, data is stored, retrievals are unaffected. Migration ETAs are slipping with the stop/start pattern; revised ETAs will follow in the weekly summary.

User action: none required.

[ZenGround0]

An update on our development progress as it relates to the original issue and the above reports.

1. @LexLuthr has put up a fix to json rpc that should eliminate the main cause of false negative mpool push errors causing corrupt db metadata state leading to proving errors
2. We've merged curio #1369 which handles any remaining transient false negative errors that still show up through json rpc
3. We've merged curio #1301 which does automatic mpool replacement to flush out messages blocked by high base fee by adaptively increasing fee cap to get messages sent to chain in times of congestion. This is particularly relevant for @TippyFlitsUK's two latest reports in this issue

We are focused on the work that remains to squash proving errors:

1. #1359 for cleaning up corrupted metadata in dbs to stop existing failures
2. Make changes to curio and synapse sdk so that message replacement works smoothly with our apis

We're targeting a curio release sometime next week to get these fixes out to SPs