I have noticed that Quicknode sometimes gives different kind of spurious errors when hitting RPC endpoints:
- timeouts
- connection reset
The implementation of eth-client currently handles rate limiting. If the server returns a 429 HTTP Error (Too Many Requests) it uses the backoff crate to retry with an exponential backoff. So we should not be seeing these errors in the node. But we may see other kinds of errors that come from network issues of RPC node malfunction.
The archiver currently doesn't handle these errors, it would just crash.
The synchronizer follows a "bruteforce" approach to handle these situations: it just retries the http connection after a delay (with configurable number of retries and delay).
I think we should analyze what kind of errors we see on RPC http requests and based on those handle them accordingly instead of having a catch-all.
For example, if there's a spurious network failure, we can retry with exponential backoff.
If we encounter 404, it may mean the remote node doesn't have the data we're querying yet, so we can retry again periodically with a very long timeout.
If we encounter an error we haven't seen yet, perhaps it's better to crash and report it, so that we have better observability of a condition we didn't predict. By not crashing, the error could get hidden and we could have a performance degradation without being aware of it.
I have noticed that Quicknode sometimes gives different kind of spurious errors when hitting RPC endpoints:
The implementation of
eth-clientcurrently handles rate limiting. If the server returns a 429 HTTP Error (Too Many Requests) it uses the backoff crate to retry with an exponential backoff. So we should not be seeing these errors in the node. But we may see other kinds of errors that come from network issues of RPC node malfunction.The archiver currently doesn't handle these errors, it would just crash.
The synchronizer follows a "bruteforce" approach to handle these situations: it just retries the http connection after a delay (with configurable number of retries and delay).
I think we should analyze what kind of errors we see on RPC http requests and based on those handle them accordingly instead of having a catch-all.
For example, if there's a spurious network failure, we can retry with exponential backoff.
If we encounter 404, it may mean the remote node doesn't have the data we're querying yet, so we can retry again periodically with a very long timeout.
If we encounter an error we haven't seen yet, perhaps it's better to crash and report it, so that we have better observability of a condition we didn't predict. By not crashing, the error could get hidden and we could have a performance degradation without being aware of it.