Content Routing resiliency

[BigLep]

Content Routing is a risk.
We don't want to have the situation where data isn't retrievable, and we point the finger at an underlying situation. That takes away from the narrative we're going for with FOC that "I store data with Filecoin and it works". Exposing/blaming the internal decisions we've made of using IPNI is similar to forcing a client to be educated on SPs.
I'm assuming we aren't building a new content routing solution between now and GA.
This s a placeholder with an estimate for any work we're going to do do monitor or communicate about the situation.

[BigLep]

2026-02-06 thoughts on the minimum we need here:

• Generally about any content routing system
  • We want to know if there's a problem with content routing before customers know
  • If there's a problem, we want to know who to escalate to, how to escalate to them, and to have assurance that we'll get help
• Ultimately we want to make sure content routing works for paid deals with approved SPs. Things that could get in the way:
  • Content routing system is down (should be caught by the above)
  • Individual SP is failing to make their content discoverable. This could be intentional or accidental.

How to determine if "IPNI is down"?

Specific things we can do:

1. Understand how IPNI monitors itself
   • There is https://probelab.io/ipfs/ipni/cid.contact/
   • What automated alerting do they have?
2. Look at signal from dealbot if we're seeing a lot more IPNI-related failures

• Note that dealbot right now is just querying filecoinpin.contact. It's not doing anything to check cid.contact. This could be added. Basically, first check filecoinpin.contact. If that looks good, then check cid.contact.

Alerting/escalation for IPNI system being down

Things we can do:

1. Meet with IPNI team and shipyard to understand how we page them if we need to.
2. Setup alarm if we see widespread consistent failure from dealbot with IPNI so we get a human in the loop.

Individual SPs aren't making their content discoverable

Things we can do:

• SP is intentionally not making their content discoverable
  • Dealbot will catch this if they are doing blanket IPNI disablement. Dealbot won't catch if the SP is being selective.
  • We could build a system that checks random pieces that have ipni indexing enabled to make sure their records show up in IPNI. I assume this is an improvement we would consider post GA.
• SP is unintentionally not making their content discoverable
  • We had a good example of this in https://filecoinproject.slack.com/archives/C07CGTXHHT4/p1770301489215909 is an example.
  • We could be querying https://filecoinpin.contact/providers/ to proactively look for errors that SPs may be having and to expose this on dealbot and alert
  • Dealbot will expose if there is issues with the pieces it has added. We can get slack notifications if we're seeing consistent failure for an approved SP.
  • We need to write up a runbook of what an SP can do to diagnose and fix the situation (i.e., make a runbook from https://filecoinproject.slack.com/archives/C07CGTXHHT4/p1770301489215909).

I know we need to commit what specifics we're going to do, but I think tackling most of the above is more like 3 days instead of 5. Does reducing the estimate seem good to you @rjan90 ?

[BigLep]

@jennijuju and I agreed to defer this to M4.2. Monitoring content routing and responding quickly to issues is certainly important before taking paying users money. We agreed it could be deferred because we believe some of the biggest/main initial users when we are "mainnet ready" with M4.1 are Akave and Storacha who don't need this functionality.

[BigLep]

@LexLuthr : I didn't get a chance to brainstorm a clear set of specifics for closing out this issue. I will do so tomorrow, but I also welcome your thoughts if you have any.

[LexLuthr]

Rough thoughts

These are rough thoughts on paper. Kind of like a conversation with myself.

IPNI protocol has 2 sides SP and Indexer
Dealbot is really not a good measure for checking indexer health.

SP

We check SP is in sync with IPNI or not. If not, we show them alert and healthy endpoint reports not healthy. That should prevent deal bot and MSP from making deals with SP. This is the best way to force SP to look into IPNI side of being an SP and report issues with sync. We can easily fix them because Curio's IPNI provider implementation has a bunch of tools to easily find and fix the problems.

IPNI Indexers

1. This is much more complicated and we should defer to IPNI teams judgment on this. cid.contact is not a single indexer. It is a bunch of indexers all ingesting data and then syncing with each other. I need to check if federated version is also in place or not. That complicates stuff further.
2. Let's say we have a partial outage and some instances are responsive and have partial data. What do we do then? At best, we can hope IPNI team fixes the outage quickly. FoC or FilOz cannot do much here. We can have all the alerts we want but as long as IPNI is a hardcoded path in retrievals, every outage will impact us.
3. In the next working group, I can ask IPNI team if we can come up with a singular /health endpoint for cid.contact. This might be the best option we have. Not sure what decision would a unhealthy cid.contact will trigger. Do we stop making deals?
4. If we really want to keep IPNI in the mix. My long term suggestion would be to run our own indexer and not by other team like filecoinpin.contact. It must be run by us with complete control over infra. We can learn how to run it most efficiently by working with IPNI team. Now comes the coup de grâce, we only sync PDP providers on this one. cid.contact can send in parallel query to this instance and we are likely to respond faster. With a limited number of provider to sync, we will have a much better perf than cid.contact and much lower infra requirements. Best part - cid.contact will become our backup.
5. If we decide to explore this option, I am willing to lead it. I will modify the indexer code for our custom version and maintain it. I will explore the cost analysis of running a indexer on Cloud/On-prem. We really should lean into cloud here for availability. We can offset the cost of running this indexer later into the pricing and protocol.