DOCS: Does a check trigger the docker hub rate limits?

[scyto]

This might be a dumb question

I am about to implement shepherd in my swarm.
Currently it has 28 containers across 25 services.

Will each check every 5 mins trip the rate limit counter, or just pulls on an image change?

The images don't change frequently - often for weeks at a time.

[djmaze]

If you look at the definition, it seems yes, the rate limit will be affected unfortunately.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Vaults]

I can confirm every check is an addition to the rate limit. However, I do have a workaround which is quite some work but worth it:

• Configure a secure pull-through registry at a domain you control, example: https://domain.com
• Transfer all your services from using image <user>/<docker-image> to domain.com/<user>/<docker-image> or in the case when the image is official and has no user, like nginx for example from nginx to domain.com/library/nginx

With the amount that Shepherd pulls it is wise to configure your own private registry and build your own images anyway to prevent more pulls. Unfortunately all the apis that circumvent this and can be used just to check if there's a new image are paid.

After having implemented this workaround, imagine 6 services which all use the nginx docker image divided over three machines. A single shepherd update of that set of services alone can cost over a whopping 18 pulls. Barring crazy states where every service has wildly varying versions, the workaround brings it back down to a single pull, which doesn't count double, triple etc. when repeated if there's no upstream updates. So in that case, you would need 200 seperate image/tag combos to be updated upstream every 6 hours to hit the limit in which case I think it would be fair if one started thinking about payment..

[shizunge]

I have recently opened source of my service updating scripts:
https://github.com/shizunge/gantry

It should use less docker hub rate than Shepherd

Would you like to give it a try?

[madmurl0c]

@Vaults Are you sure that you have to rewrite all the images? Can't you just define registry-mirrors to the docker process?

I've been using a docker mirror for a while but it doesn't seem to be used by shepherd although I mounted /etc/docker/daemon.json:/etc/docker/daemon.json:ro.

The /etc/docker/daemon.json looks like this:

{
  "registry-mirrors": ["https://docker-mirror01.company.lan:5001"]
}

It works on the host but it doesn't work in the container. Is there any way to set the --registry-mirrors parameter or use $DOCKER_OPTS inside the image?

[moschlar]

@madmurl0c The problem is basically the docker manifest inspect call that shepherd issues beforehand to determine whether it need to trigger a service update at all. Since that call is just done "client side" by the docker cli in the container, it has no knowledge of the configured registry mirrors of the daemon on the host (though it could probably try to find that out...).

Will keep this open until we thought of a solution!

[mrambossek]

no solution yet? :(