Caching registry fetch errors forever leads to PODs ending needing re…#127
Caching registry fetch errors forever leads to PODs ending needing re…#127hennikul wants to merge 2 commits intoheetch:masterfrom
Conversation
…start to fix temporary errors like routing after redeploy. - Cache fetch errors for one minute to avoid overloading registry server. - Should fix context cancelled loops seen after redeploy.
| if es, ok := wType.avroType.(errorSchema); ok && (es.invalidAfter.IsZero() || time.Now().Before(es.invalidAfter)) { | ||
| return nil, es.err | ||
| } | ||
| } else { |
There was a problem hiding this comment.
Sorry if I'm missing something but aren't we discarding the "normal" case here where wType is not nil and does not contain a schema error?
Or is that case already covered by checking c.programs in the beginning of the func call?
There was a problem hiding this comment.
You are correct. The c.programs in the beginning is the normal case where a compiled decoder for the schema is ready.
When a complied decoder is not ready, we need to fetch it. This could fail due to network errors, busy servers etc. In these cases we need to retry the fetch later, if not the state will be bad until the program is restarted. This is where invalidAfter comes into play. The next time the schema is requested, we find wType, and this will be an error with invalidAfter, so we can decide if we are retry to retry or not. Some errors are permanent, for example if we have a response from the registry, but we cannot compile a decoder for it. In this case, invalidAfter will be IsZero(), and we will never retry. To fix this state, a new schema must be uploaded, or the program restarted.
I have added some comments to make this more clear.
|
ℹ️ Just FYI that #143 was merged, to solve the same issue, but without the time-limited error cache |
5 participants
…start to fix temporary errors like routing after redeploy.