[ci] Improve apt reliability

[tyler-yankee]

Is your feature request related to a problem? Please describe.

The occasional nightly job failure caused by apt download issues has become mostly expected at this point. There have been a few times in the last year+ where EC2's apt mirror is completely unavailable, causing substantial downtime for a large portion of our infrastructure.

The sources list for an unprovisioned Noble x86_64 image used by drake-ci currently only uses http://us-east-1.ec2.archive.ubuntu.com/ubuntu/.

Describe the solution you'd like

We should consider backing that with something like Canonical's http://archive.ubuntu.com/ubuntu/, or another mirror if folks have ideas/preference, to improve download reliability and avoid infrastructure meltdowns.

Describe alternatives you've considered

Alternatively, we could:

• reduce our footprint of unprovisioned jobs, but some amount of these will always be necessary.
• back up the debs we need on drake-mirror.csail.mit.edu. This seems like a lot of extra maintenance work and infra, doing so per architecture and version we support. The added maintenance also further locks us into Ubuntu's ecosystem and makes it harder to potentially support non-Ubuntu Linux in the long-term future.

Additional context

(1) Regarding implementation details, this note is atop the sources list file:

## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
##
## If you wish to make changes you can:
## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
##     or do the same in user-data
## b.) add supplemental sources in /etc/apt/sources.list.d
## c.) make changes to template file
##      /etc/cloud/templates/sources.list.ubuntu.deb822.tmpl
##

Likely this change would be baked into our unprovisioned images via ami_init_script.

(2) Once the initial change is in for drake-ci, we should consider extending it to drake's wheel builder for the apt downloads it manages via Docker separately.

(3) There is a side bug where when a job fails during apt downloads, the buildcop emails don't go out, due to the timing of how/when the RESULT file of the build is written.

[tyler-yankee]

cc: @BetsyMcPhail @jwnimmer-tri

[tyler-yankee]

Bumping to high priority as it's posing problems again in CI this week.

[tyler-yankee]

Beyond the configuration issue pointed out in (1) under "additional context" in the issue description, there are a few things to consider here:

• Simply adding a file /etc/apt/sources.list.d/foo.sources does not tell apt "fetch from foo if a download from ubuntu.sources (EC2) fails"; it's an alternative source. If anything it could lead to hash sum mismatch issues if the two mirrors diverge in packages provided, or etc.
• In theory on x86_64 we could use mirror://mirrors.ubuntu.com/mirrors.txt to tell apt to find other geographically feasible mirrors if one is currently unavailable. Problem is, this mirror apparently isn't available on aarch64, which our recent infrastructure issues have been; only http://ports.ubuntu.com/ubuntu-ports/, which, in and of itself, is different from x86's http://archive.ubuntu.com/ubuntu/.

[tyler-yankee]

Gemini seems to be pessimistic on the approach of adding sources to resolve the 4xx/5xx errors, and instead suggests:

Acquire::http::Timeout "10";
Acquire::https::Timeout "10";
Acquire::Retries "5";
Acquire::http::Pipeline-Depth "0";

in some /etc/apt/apt.conf.d/99network-config file.

[tyler-yankee]

Of course, if fallbacks aren't really an option, we could consider simply replacing the EC2 source with the Canonical source; who knows if it would be more reliable.