It seems that on ECS Fargate/EFS cluster setup, the Directory watcher used in titanoboa.repo/RepoWatcherComponent does not get triggered every time a new workflow (aka job definition) revision is saved.
This leads to a situation where some nodes may have an obsolete view of repo:
e.g. node A sees that the latest :head revision of workflow X is 5
while node B sees that the latest :head revision of workflow X is 2
This is interesting since with EFS this DOES work on EC2/EFS setup. So it may be related to some idiosyncrasy of Fargate.
We can fix this by periodically refreshing the workflow repo atom.
It seems that on ECS Fargate/EFS cluster setup, the Directory watcher used in
titanoboa.repo/RepoWatcherComponentdoes not get triggered every time a new workflow (aka job definition) revision is saved.This leads to a situation where some nodes may have an obsolete view of repo:
e.g. node A sees that the latest
:headrevision of workflow X is5while node B sees that the latest
:headrevision of workflow X is2This is interesting since with EFS this DOES work on EC2/EFS setup. So it may be related to some idiosyncrasy of Fargate.
We can fix this by periodically refreshing the workflow repo atom.