We cache the github content manifest in KV and regen on requests for / with no-cache headers.
I noticed that some crawlers (hostile bots?) trigger the regen becuase they set no-cache headers.
A nice optimization is to track the git sha and only regen only when there's a new sha on the HEAD of the branch.
Long term the fix is to regen on a hook, or only requests with auth.
We cache the github content manifest in KV and regen on requests for
/with no-cache headers.I noticed that some crawlers (hostile bots?) trigger the regen becuase they set no-cache headers.
A nice optimization is to track the git sha and only regen only when there's a new sha on the HEAD of the branch.
Long term the fix is to regen on a hook, or only requests with auth.