[Paulkm2006]

This PR implements a golang downloader, which retrieves all versions and download links from the golang official website https://go.dev/dl/ , and download them concurrently.

[Copilot]

Pull request overview

Adds a new golang.py mirror sync script intended to fetch Go release metadata from go.dev and download artifacts concurrently into a local working directory (tunasync-style).

Changes:

• Introduces RemoteSite + GoRelease modeling to enumerate Go release artifacts from go.dev.
• Adds a threaded download worker queue with optional include/exclude filtering and SHA256 verification.
• Implements a local cleanup step to delete files not present in the computed upstream file list.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[taoky]

只从 JSON 获取 stable (supported) release 是有必要的，因为对于空间紧张的镜像站来说，很多时候不希望同步的仓库大小是无限增长的。另外对于只获取 stable release 的情况，可能还需要移除上游已经去除的文件。

[Paulkm2006]

已修改。移除文件的逻辑位于 R317-R326

[Paulkm2006]

脚本已在 hust-mirrors 运行并通过验证。网页所展示的同步失败原因是华中科技大学校园网环境对 go.dev 的连通性不佳，可能导致间接性失败

[yaoge123]

One concern from the mirror-operator side (we run a /golang mirror at NJU): storing files as {version}/{filename} changes the on-disk layout away from dl.google.com/go/'s flat structure. Existing golang mirrors (TUNA included) serve flat paths like /golang/go1.21.0.linux-amd64.tar.gz, and switching layout would break every existing downstream URL. Could this keep the flat layout (filename only) instead? The retries/concurrency/sha256 checks look great otherwise.

[yaoge123]

Follow-up with evidence on the layout question — the PR code and your own production site disagree:

• The PR branch (4b04007) computes dst_file = working_dir / f"{version}/{filename}", i.e. on-disk go1.27.1/go1.27.1.linux-amd64.tar.gz (two-level).
• But https://mirrors.hust.edu.cn/golang/ serves flat paths: /golang/go1.27.1.linux-amd64.tar.gz → 200, while /golang/go1.27.1/go1.27.1.linux-amd64.tar.gz → 404. Same as dl.google.com/go/ (flat; version subdirs 404) and our NJU mirror (flat).

So it looks like HUST runs a different (flat-layout) iteration than what is pushed here. If your local version already stores flat, could you push it? That would resolve the compatibility concern entirely — with a flat layout matching dl.google.com/go/, prefix-substitution from the official URLs just works.

[yaoge123]

I ran the exact code from this PR (4b04007) in a clean container to verify the on-disk layout empirically:

$ python3 golang.py --working-dir /data --workers 2 --include linux-amd64
downloading https://go.dev/dl/go1.27.1.linux-amd64.tar.gz
downloading https://go.dev/dl/go1.26.8.linux-amd64.tar.gz
Sync completed!

$ find /data -type f
/data/go1.26.8/go1.26.8.linux-amd64.tar.gz
/data/go1.27.1/go1.27.1.linux-amd64.tar.gz

So the current PR writes go/{version}/{filename} (two levels), while upstream dl.google.com/go/ is flat:

• https://dl.google.com/go/go1.27.1.linux-amd64.tar.gz.sha256 → 200
• https://dl.google.com/go/go1.27.1/go1.27.1.linux-amd64.tar.gz → 404

A two-level mirror breaks URL compatibility with upstream (go.dev/dl/ redirect targets, gvm/asdf/goenv and hard-coded download URLs all expect the flat layout). Could you switch relative_path to just {filename}?

One more finding from the test: if a download fails, the worker thread only prints a traceback and swallows the exception, so the script still prints Sync completed! and exits 0 — tunasync would mark the job successful on a partial/empty mirror. It would be safer to track failures and exit non-zero.

[Paulkm2006]

Hi @yaoge123 ,

Sorry for the late response, and thank you for your valuable feedback! I'll implement a switch for flat directory structure (which will be the default structure) and better return value for downloading faults.

As for the HUST mirrors inconsistency, we initially adopted the scripts I proposed here, but our campus network started to fail on syncing from Google quite often this semester. Thus, we changed our upstream provider to NJU, which uses the flat directory structure.

[Paulkm2006]

Hi @yaoge123 ,
I've updated the script to use a flat layout by default. The previous version/filename layout is available with --layout nested.
Download failures now cause exit status 1 and skip cleanup, rather than reporting a successful sync. I also fixed worker shutdown and added validation for invalid worker counts.