Increase Default Sidecar Update Interval

[patrickmann]

Support issue Graylog2/support#384

What: The default update interval of 10s causes large deployments to crumble under their own rate. Increasing the default should prevent this in large deployments with no real impact on smaller deployments.

Why:
Mongo updating "Token Last Used" for sidecar token. For large number of sidecars (500+) the default update interval (10 seconds) is FAR too frequent. It overloads mongo. A customer added 1600 sidecars then saw this slowdown.

It's been set to 5m which appears to have completely resolved their issue.

This seems like a reasonable new default to use.

[bernd]

Settings like this one are always a trade-off. For smaller deployments, the 10-second delay is fine and results in less latency for config updates. If we set it to five minutes, other people will complain that it takes "ages" for a Sidecar to get the config update. A long interval would also result in a terrible first-time experience.

This sounds like a documentation issue to me. We probably need better documentation for scaling Sidecar deployments.

[tellistone]

Similar to issue for auto-scaling sidecar interval with sidecar count (https://github.com/Graylog2/graylog-plugin-enterprise/issues/6595)