From 7cdbf6b80b5c6bd3c80eed27a58183899079231c Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 02:48:30 +0100 Subject: [PATCH 1/6] docs(annotsv): count the 2,409 coding tandem repeats the pinned TRExplorer table holds Bead: pgp-5bja.22 --- docs/05-annotsv.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/05-annotsv.md b/docs/05-annotsv.md index 7c343a39..2d64489d 100644 --- a/docs/05-annotsv.md +++ b/docs/05-annotsv.md @@ -124,7 +124,7 @@ Some class 4 and 5 calls are common or unreliable for a reason AnnotSV does not - **The MHC** (`mhc`). An SV more than half inside the MHC, GRCh38 chr6:28,510,120-33,480,577. Its haplotypes differ by tens of kilobases between people, and gnomAD-SV marks every SV over half inside IGH or the MHC as low confidence (FILTER `IGH_MHC_OVERLAP`, Collins 2020, PMID 32461652). CYP21A2 lies in the MHC; short reads cannot call it here, and [step 35b](35-paralogs.md) reads it from PacBio HiFi reads. - **A pseudogene named by a disease gene's symbol** (`gene_alias`). AnnotSV joins its gene annotations by symbol. A few pseudogenes carry a disease gene's symbol as an HGNC alias or previous symbol (for example RHEBP1, whose HGNC alias is RHEB). When `Gene_name` lists that symbol and the SV reaches no transcript of the disease gene in AnnotSV's genes file, the table names the pseudogene instead, and the call moves to class 3 when no other gene is left. This needs the genes file, like the exon-edge check. -The regions are in `assets/sv_rule_regions.tsv`, one row each (`chrom`, `start` 0-based, `end`, `rule`, `action`, `name`, `source`): 2,349 coding tandem repeats, the six IG and TR loci, the MHC and 10 pseudogenes. `scripts/ci/build-sv-rule-regions.sh` writes it from four pinned downloads, each checked against its sha256: TRExplorer v2.1's GRCh38 tandem repeat catalog (every repeat of 50 bp or more on a MANE Select coding exon of GENCODE 50), GENCODE 50's IG and TR genes, the MHC row of the GRCh38 assembly regions report, and HGNC's complete set of 2026-10-06 (each pseudogene whose alias or previous symbol is the approved symbol of a gene with a GenCC curation). The file is about 250 KB and ships with the repository, so the step downloads nothing. Step 05 mounts it into the AnnotSV image and the Nextflow `ANNOTSV` gets it as an input. +The regions are in `assets/sv_rule_regions.tsv`, one row each (`chrom`, `start` 0-based, `end`, `rule`, `action`, `name`, `source`): 2,409 coding tandem repeats, the six IG and TR loci, the MHC and 10 pseudogenes. `scripts/ci/build-sv-rule-regions.sh` writes it from four pinned downloads, each checked against its sha256: TRExplorer v2.1's GRCh38 tandem repeat catalog (the file upstream rebuilt in place on 2026-10-30, sha256 `df9c23c4…`; every repeat of 50 bp or more on a MANE Select coding exon of GENCODE 50), GENCODE 50's IG and TR genes, the MHC row of the GRCh38 assembly regions report, and HGNC's complete set of 2026-10-06 (each pseudogene whose alias or previous symbol is the approved symbol of a gene with a GenCC curation). The file is about 250 KB and ships with the repository, so the step downloads nothing. The e2e case `followups-ci-3-sv-rule-regions` rebuilds it with `scripts/ci/build-sv-rule-regions.sh --check`, which fails when one byte differs, so a hand edit or a drifted pin cannot ship. Step 05 mounts it into the AnnotSV image and the Nextflow `ANNOTSV` gets it as an input. How many callers saw each SV comes from step 22's consensus (`sv_merged/_sv_consensus.vcf.gz`), which runs after this step, and beside `ANNOTSV` in Nextflow. So the reports (`bin/collect_summary.py`) join it when they are written: an SV matches the consensus record of the same type whose two breakpoints lie within 1,000 bp of its own, step 22's distance, and its support is that record's `SUPP` with the callers of `SUPP_VEC`. An SV in no consensus record was made by one caller and shows as "single caller". Without the consensus file the support is "not known", never 1. `annotsv_postfilter.py --consensus` adds the same `Support` and `Support_callers` columns to the table. From e59836779eb7fcdb0354087fce50958de18224ce Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 02:52:50 +0100 Subject: [PATCH 2/6] test(sv): rebuild assets/sv_rule_regions.tsv with --check in e2e, with a one-cell red control The new case followups-ci-3-sv-rule-regions runs scripts/ci/build-sv-rule-regions.sh --check on the committed table and on a copy with the MHC row's end moved by one; the copy must fail and show the row. The builder's four downloads join the catalogs data cache entry. The ci-e2e-2-locityper times row takes the 142 s the case takes with the jellyfish hash sized to the reference (was 2289 s with a fixed 3G). Bead: pgp-5bja.22 --- .github/workflows/e2e.yml | 6 +-- docs/testing.md | 6 ++- scripts/ci/e2e-case-paths.tsv | 1 + scripts/ci/e2e-case-times.tsv | 3 +- scripts/ci/e2e-run.sh | 9 +++-- tests/e2e/followups-ci-3-sv-rule-regions.sh | 44 +++++++++++++++++++++ 6 files changed, 59 insertions(+), 10 deletions(-) create mode 100755 tests/e2e/followups-ci-3-sv-rule-regions.sh diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 760b239d..bb9f8e28 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -637,7 +637,7 @@ jobs: grep -qx archaic-ancestry-1 "$list" && echo "data_archaic=true" >> "$GITHUB_OUTPUT" grep -qx bash-step-08-hla "$list" && echo "data_hla=true" >> "$GITHUB_OUTPUT" grep -qx prs-scores-and-run-options-1-strand-sex "$list" && echo "data_pgs=true" >> "$GITHUB_OUTPUT" - grep -qxE 'callability-1-catalog|cn-intervals-1-catalog' "$list" && echo "data_catalogs=true" >> "$GITHUB_OUTPUT" + grep -qxE 'callability-1-catalog|cn-intervals-1-catalog|followups-ci-3-sv-rule-regions' "$list" && echo "data_catalogs=true" >> "$GITHUB_OUTPUT" echo "list=${list}" >> "$GITHUB_OUTPUT" # Before the base outputs: the base leg pulled the same images. @@ -695,8 +695,8 @@ jobs: # four real PGS Catalog score files of prs-scores-and-run-options-1. # A third keyed the same way: the pinned downloads of the two table # builders (GENCODE's GTF from EBI among them) that - # callability-1-catalog and cn-intervals-1-catalog rebuild their - # tables from. + # callability-1-catalog, cn-intervals-1-catalog and + # followups-ci-3-sv-rule-regions rebuild their tables from. - name: Data cache keys id: data-keys run: | diff --git a/docs/testing.md b/docs/testing.md index d11a1ed9..a740697c 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -102,7 +102,7 @@ The base leg runs every base case whenever any case runs, because every shard st The images the cases use are cached in four GitHub Actions cache entries, one per group. `scripts/ci/e2e-run.sh --image-groups` names each image's group, taken from its repository name without the tag, so a version bump changes the key of one group only; a group whose key missed restores the newest older entry of that group and pulls only what changed. A group keeps its smallest images up to 1500 MB of zstd output, and the larger ones are pulled on every run. Four groups of 1500 MB and the six data entries below stay under the repository's 10 GB cache limit even while main replaces one group: 5.9 GB of images, 1.5 GB more for the group being replaced, and about 2.4 GB of data. Images pinned by digest are not cached, because `docker load` gives them back without the digest, and they are left out of the group's key, so a digest bump misses no entry. Only the base leg saves, and only on main (or in a run by hand with `save_images`, on any branch), after it pulled every image the cases use and got through its cases, and only the groups whose key missed: a cache saved by a pull request can be read by that pull request alone and would only push main's entries out. The job summary lists every image with its size and what the cache did with it: restored on an exact key hit, restored from an older entry, saved, or pulled. Every merge in the e2e scope starts a run on main, so the cache follows main's images. GitHub drops an entry no run has read for 7 days, so after a quiet week the next run on main fills the cache again. Until this layout the cache was one entry, and by October 2026 it had grown to about 11 GB, over the limit, so every run missed it. -Six downloads are cached the same way. Two come from EBI's 1000 Genomes server, which is slow and at times stalls: MoChA's chr20 panel with its manifest (0.64 GB, case `mocha-1`), keyed on `MOCHA_PANEL_MANIFEST_SHA256` and the chromosome, and the 1000 Genomes phase 3 sites VCF (1.36 GB, case `archaic-ancestry-1`), keyed on `ARCHAIC_1KG_SITES_MD5`. On a miss, each of those two is tried first at `E2E_DATA_MIRROR`, the GitHub release `e2e-data-1` that holds copies of the pinned files, and then at EBI; the pin checks both. The release also holds the four PGS Catalog score files, the HAPNEST panel and the table builders' downloads below, so an EBI outage costs none of them on a cold cache. Two are small and failed whole runs on 2026-11-04, because their case is in the base leg or a shard that cannot start without it: `hla.dat` of the two IPD-IMGT/HLA releases `bash-step-08-hla` installs, from raw.githubusercontent.com (which served it under 1 KB/s), with GENCODE's gene and exon files from EBI's FTP server (curl exit 7 and 28), and the four real PGS Catalog score files of `prs-scores-and-run-options-1-strand-sex`, from the same server. `scripts/ci/e2e-run.sh --data-key` names their keys: the releases and a hash of the code in `scripts/lib/common.sh` that installs them, and a hash of the score ids and their server. `scripts/ci/e2e-run.sh` hands each case its files just before it runs, at the paths `setup.sh` and `install_data_file` read them from, so they download nothing; the score files the case copies from the cache itself. On a miss it downloads them first, checked against their pins or the checksum files the servers publish beside them (`MD5SUMS`, `md5checksum.txt`, the `.md5` of each score), trying up to 6 times 30 s apart. The four score files are pinned by md5 in `PGS_MD5` of `e2e-run.sh` (the md5 the PGS Catalog served beside each on 2026-11-06), tried on the mirror first and then at EBI; a restored or downloaded score must match its pin as well as its `.md5`. A file gets at most `E2E_MIRROR_BUDGET` seconds on the mirror (900, or half the fill's budget when that is less), so a mirror that trickles leaves EBI its turn. One fill may take at most `E2E_DATA_BUDGET` seconds (2400 in `e2e.yml`): a source that trickles above `fetch`'s stall limit would otherwise run on to the leg's 240-minute limit, as EBI's 1000 Genomes server did twice on 2026-11-05. A fill still running then is stopped and its case fails as `FAIL (data fill over 2400s)` without running, since its own download would be as slow. A restored entry is checked again before use (`--data-check`): the IPD-IMGT/HLA and GENCODE entry against the sha256 list written when it was filled, the score files against their `.md5`; one that fails is downloaded again. `bash-step-08-hla` then runs with every HTTP and HTTPS request sent to a closed port, after a control shows that a request under that block fails at the closed port, so a warm cache cannot reach either host. Only the leg whose cases include the case restores and saves its entry, and it saves only on main or in a run by hand with `save_images`, as with the images. The job summary says whether each key hit, and the case's log says where its files came from. The fifth is the PGS Catalog's synthetic HAPNEST panel (0.27 GB, cases `prs-2-ancestry` and `prs-scores-and-run-options-2-panel-chrx`) from EBI's FTP server, which failed cases with curl exit 7 on 2026-11-04. It is keyed on the sha256 `tests/e2e/lib.sh` pins (`HAPNEST_SHA256`). The cases handle it themselves (`setup_ancestry_panel` in `lib.sh`): with the panel in the cache, or on a miss from `E2E_DATA_MIRROR`, they check the sha256 and run `setup.sh --ancestry-panel` with every HTTP and HTTPS request sent to a closed port, so an EBI outage cannot fail them; when neither has it, `setup.sh` downloads the panel from EBI as a user's run does. The first case puts a panel that did not come from the cache in the cache for the save step. A red control in `prs-2-ancestry` runs `setup.sh` with the network blocked and no panel, which must fail. The sixth, entry `catalogs` (0.08 GB), holds the pinned downloads of `scripts/ci/build-callability.sh` and `scripts/ci/build-cn-intervals.py`, whose cases `callability-1-catalog` and `cn-intervals-1-catalog` rebuild `assets/callability_genes.tsv` and the interval rows of `bin/cn_loci.tsv` and compare them with the committed files. Each builder lists its downloads with `--list`; the key is a hash of those lists, so a new pin is a new key. GENCODE's GTF among them comes from EBI's FTP server, which refused every connection for minutes on 2026-11-06 and failed `callability-1-catalog` on main (run 42259432681). The runner fills the entry as it does the others, mirror first and every file checked against its sha256, and each case points its builder's cache at its folder there; a builder that still has to download tries for about five minutes, 30 s apart. Every download, there and in `setup.sh`, gives up on a transfer that stalls (`FETCH_STALL`, see `fetch` in `scripts/lib/common.sh`), and the next try resumes it. +Six downloads are cached the same way. Two come from EBI's 1000 Genomes server, which is slow and at times stalls: MoChA's chr20 panel with its manifest (0.64 GB, case `mocha-1`), keyed on `MOCHA_PANEL_MANIFEST_SHA256` and the chromosome, and the 1000 Genomes phase 3 sites VCF (1.36 GB, case `archaic-ancestry-1`), keyed on `ARCHAIC_1KG_SITES_MD5`. On a miss, each of those two is tried first at `E2E_DATA_MIRROR`, the GitHub release `e2e-data-1` that holds copies of the pinned files, and then at EBI; the pin checks both. The release also holds the four PGS Catalog score files, the HAPNEST panel and the table builders' downloads below, so an EBI outage costs none of them on a cold cache. Two are small and failed whole runs on 2026-11-04, because their case is in the base leg or a shard that cannot start without it: `hla.dat` of the two IPD-IMGT/HLA releases `bash-step-08-hla` installs, from raw.githubusercontent.com (which served it under 1 KB/s), with GENCODE's gene and exon files from EBI's FTP server (curl exit 7 and 28), and the four real PGS Catalog score files of `prs-scores-and-run-options-1-strand-sex`, from the same server. `scripts/ci/e2e-run.sh --data-key` names their keys: the releases and a hash of the code in `scripts/lib/common.sh` that installs them, and a hash of the score ids and their server. `scripts/ci/e2e-run.sh` hands each case its files just before it runs, at the paths `setup.sh` and `install_data_file` read them from, so they download nothing; the score files the case copies from the cache itself. On a miss it downloads them first, checked against their pins or the checksum files the servers publish beside them (`MD5SUMS`, `md5checksum.txt`, the `.md5` of each score), trying up to 6 times 30 s apart. The four score files are pinned by md5 in `PGS_MD5` of `e2e-run.sh` (the md5 the PGS Catalog served beside each on 2026-11-06), tried on the mirror first and then at EBI; a restored or downloaded score must match its pin as well as its `.md5`. A file gets at most `E2E_MIRROR_BUDGET` seconds on the mirror (900, or half the fill's budget when that is less), so a mirror that trickles leaves EBI its turn. One fill may take at most `E2E_DATA_BUDGET` seconds (2400 in `e2e.yml`): a source that trickles above `fetch`'s stall limit would otherwise run on to the leg's 240-minute limit, as EBI's 1000 Genomes server did twice on 2026-11-05. A fill still running then is stopped and its case fails as `FAIL (data fill over 2400s)` without running, since its own download would be as slow. A restored entry is checked again before use (`--data-check`): the IPD-IMGT/HLA and GENCODE entry against the sha256 list written when it was filled, the score files against their `.md5`; one that fails is downloaded again. `bash-step-08-hla` then runs with every HTTP and HTTPS request sent to a closed port, after a control shows that a request under that block fails at the closed port, so a warm cache cannot reach either host. Only the leg whose cases include the case restores and saves its entry, and it saves only on main or in a run by hand with `save_images`, as with the images. The job summary says whether each key hit, and the case's log says where its files came from. The fifth is the PGS Catalog's synthetic HAPNEST panel (0.27 GB, cases `prs-2-ancestry` and `prs-scores-and-run-options-2-panel-chrx`) from EBI's FTP server, which failed cases with curl exit 7 on 2026-11-04. It is keyed on the sha256 `tests/e2e/lib.sh` pins (`HAPNEST_SHA256`). The cases handle it themselves (`setup_ancestry_panel` in `lib.sh`): with the panel in the cache, or on a miss from `E2E_DATA_MIRROR`, they check the sha256 and run `setup.sh --ancestry-panel` with every HTTP and HTTPS request sent to a closed port, so an EBI outage cannot fail them; when neither has it, `setup.sh` downloads the panel from EBI as a user's run does. The first case puts a panel that did not come from the cache in the cache for the save step. A red control in `prs-2-ancestry` runs `setup.sh` with the network blocked and no panel, which must fail. The sixth, entry `catalogs`, holds the pinned downloads of `scripts/ci/build-callability.sh`, `scripts/ci/build-cn-intervals.py` and `scripts/ci/build-sv-rule-regions.sh`, whose cases `callability-1-catalog`, `cn-intervals-1-catalog` and `followups-ci-3-sv-rule-regions` rebuild `assets/callability_genes.tsv`, the interval rows of `bin/cn_loci.tsv` and `assets/sv_rule_regions.tsv` and compare them with the committed files. Each builder lists its downloads with `--list`; the key is a hash of those lists, so a new pin is a new key. GENCODE's GTF among them comes from EBI's FTP server, which refused every connection for minutes on 2026-11-06 and failed `callability-1-catalog` on main (run 42259432681). The runner fills the entry as it does the others, mirror first and every file checked against its sha256, and each case points its builder's cache at its folder there; a builder that still has to download tries for about five minutes, 30 s apart. Every download, there and in `setup.sh`, gives up on a transfer that stalls (`FETCH_STALL`, see `fetch` in `scripts/lib/common.sh`), and the next try resumes it. How long it took in one job, before the split: a passing e2e job took about 105 to 205 minutes with 84 cases, and 230 minutes of cases with 130 (run 41390434697, 2026-11-02). The runner decides much of it, not the cache: the same CPU model took 106 minutes in one run and 166 in another. In October 2026, with 84 cases, the same code took 106 and 166 minutes on an AMD EPYC 9V45 (runs 39197502832 and 39075176392, the second of which also saved the cache), 112 minutes on an Intel Xeon 6973P (run 39088086725), 156 and 191 minutes on an AMD EPYC 7763 (runs 39075764278 and 39075201465) and 205 minutes on an AMD EPYC 9V74 (run 39197512594, which saved three groups). The image cache saves about 7 minutes of that: pulling every image took 9 minutes, restoring and loading the four groups took 2.5 (run 39075201465 restored 39 images in 118 seconds), and saving them all took 12. Each leg has a limit of 240 minutes (355 for a traced shard leg); the base leg waits up to 75 minutes for a fixture a pull request bumps. Each leg's first steps print the runner's CPU model and CPU count to the log and the job summary, so a later timing can name the hardware. @@ -121,7 +121,9 @@ What it covers: - **Index times** (`run-all-sample-qc-and-stale-index-1-index-times.sh`): every index the direct `nextflow run` and the launcher published is not older than its data file, and samtools and bcftools print no stale-index warning on the BAM and the VCF. Then a short VCF-only run with three planted pairs checks `main.nf`'s completion handler: it sets the index times of the run's own files after the last copy, and leaves an index older than the run as it was. A copied pair with its index set an hour older is the negative control. - **The Nextflow report's tables** (`paralogs-report-section-1-nf-report-tables.sh`): `HTML_REPORT` runs for real on the CYP2D6 depth check, the cDNA read check and the phased compound-het pairs that earlier cases wrote, and its reports must show the depth-check line, the cDNA column with its flagged count and the pairs by phase; the same run without the three tables must show none of them. A stub run of every tool then checks that `main.nf` passes the cDNA table to each `HTML_REPORT` task that got ClinVar hits, against a copy of the pipeline with that join removed. -- **Locityper, step 53** (`ci-e2e-2-locityper.sh`): `setup.sh --locityper-data` builds the database for the HLA-DQB1 row of `assets/locityper_loci.bed` from a chr6 and chr20 reference cut from the fixture's, and `scripts/53-locityper.sh` genotypes the fixture's reads aligned to it, with chr20 as the background region. The fixture has no reads over FCGR or KIR, and HLA-DRB1 is left out for time: with it the case took 59 minutes, 35 of them jellyfish's k-mer count and 23 `locityper augment`. With HLA-DQB1 alone `augment` takes about a minute, but jellyfish, which does not depend on the loci, still takes 36 (38 minutes in all on an AMD EPYC 9V74), so the case raises its own limit to 90 minutes (`# case-timeout: 5400`; every other case keeps 60). The locus must come out called, with two panel haplotypes, a quality, reads and the authors' filter, in the columns `bin/locityper_report.py` writes. Red controls: the step stops without the database, without the BAM's index, and with a database whose `DONE` names another reference. +- **Locityper, step 53** (`ci-e2e-2-locityper.sh`): `setup.sh --locityper-data` builds the database for the HLA-DQB1 row of `assets/locityper_loci.bed` from a chr6 and chr20 reference cut from the fixture's, and `scripts/53-locityper.sh` genotypes the fixture's reads aligned to it, with chr20 as the background region. The fixture has no reads over FCGR or KIR, and HLA-DRB1 is left out for time: with it the case took 59 minutes, 35 of them jellyfish's k-mer count and 23 `locityper augment`. With HLA-DQB1 alone `augment` takes about a minute. Jellyfish does not depend on the loci: with its hash fixed at 3G entries it took 36 minutes (2,289 s for the case, fork run 42325306077), and with the hash sized to the reference it takes 36 seconds (142 s for the case, fork run 42421585308). The case keeps a limit of 90 minutes (`# case-timeout: 5400`; every other case keeps 60) as room for a slow runner. The locus must come out called, with two panel haplotypes, a quality, reads and the authors' filter, in the columns `bin/locityper_report.py` writes. Red controls: the step stops without the database, without the BAM's index, and with a database whose `DONE` names another reference. + +- **The SV rule regions** (`followups-ci-3-sv-rule-regions.sh`): `scripts/ci/build-sv-rule-regions.sh --check` rebuilds `assets/sv_rule_regions.tsv` from its four pinned downloads and must find the committed file byte for byte, with its row counts (2,409 coding tandem repeats). Red control: `--check` on a copy with the MHC row's end moved by one must exit 1 and show that row. The downloads come from the `catalogs` data cache entry. - **The 1.0 output contract** (`output-contract-1-hg002.sh`): every stable file of [`assets/output_contract.tsv`](https://github.com/GeiserX/OwnGenome/blob/main/assets/output_contract.tsv) whose `e2e` column names a leg is in what the base cases' bash steps and case `60-nextflow` wrote, under its contracted name, with its required columns, VCF fields and JSON keys. Its red control drops a column and renames a file on a copy and must fail on both. `tests/test_output_contract.py` checks the same contract against the code with no run ([What 1.0 keeps stable](output-contract.md)). diff --git a/scripts/ci/e2e-case-paths.tsv b/scripts/ci/e2e-case-paths.tsv index c302ec67..39e7ac75 100644 --- a/scripts/ci/e2e-case-paths.tsv +++ b/scripts/ci/e2e-case-paths.tsv @@ -89,6 +89,7 @@ expansions-and-rfc1-1-motif assets/str_extra_loci.json assets/stub/db/catalog.js families-file-1 assets/acmg_sf.tsv assets/callability_genes.tsv assets/carrier_genes.tsv assets/construct_variants.tsv assets/deep_intronic_sites.tsv assets/families.example.tsv assets/mito_confirmed.tsv assets/par_grch38.bed assets/pedigree.example.tsv assets/pgx_genes.tsv assets/stub/db/hla.dat assets/stub/db/kir.dat assets/stub/pedigree.tsv assets/stub/sample.bam assets/stub/sample.bam.bai assets/stub/sample.vcf.gz bin/cdna_scan.py bin/clinvar_hits.awk bin/clinvar_join_adjacent.awk bin/clip_check.py bin/cn_loci.tsv bin/collect_summary.py bin/copy_number.py bin/couple_report.py bin/cyp2d6_depth_check.py bin/family_report.py bin/family_segments.py bin/hba_copy_number.py bin/ibd_report.py bin/lpa_kiv2.py bin/pgx_outside_calls.py bin/render_report.py bin/results_index.py modules/local/align_minimap2/* modules/local/clinvar_screen/* modules/local/couple/* modules/local/deepvariant/* modules/local/fastp/* modules/local/hla_typing/* modules/local/html_report/* modules/local/t1k_build/* modules/local/vcf_precheck/* scripts/01b-fastp-qc.sh scripts/06-clinvar-screen.sh scripts/08-hla-typing.sh scripts/24-html-report.sh scripts/32-pypgx.sh scripts/36-pgx-consensus.sh scripts/38-family.sh scripts/39-couple.sh scripts/ci/check-images.sh scripts/ci/fake-docker/lib.sh scripts/validate-setup.sh tests/fixtures/families/stub_families.tsv family-joint-1 assets/construct_variants.tsv assets/deep_intronic_sites.tsv assets/families.example.tsv assets/par_grch38.bed assets/pedigree.example.tsv assets/stub/pedigree.tsv assets/stub/sample.bam assets/stub/sample.bam.bai bin/cdna_scan.py bin/clinvar_hits.awk bin/clinvar_join_adjacent.awk bin/clip_check.py bin/collect_summary.py bin/family_report.py bin/family_segments.py bin/ibd_report.py bin/mosaic_sex_loss.py modules/local/align_minimap2/* modules/local/clinvar_screen/* modules/local/deepvariant/* modules/local/fastp/* modules/local/roh/* modules/local/somalier/* modules/local/vcf_precheck/* scripts/01b-fastp-qc.sh scripts/06-clinvar-screen.sh scripts/11-roh-analysis.sh scripts/33-sample-qc.sh scripts/38-family.sh scripts/ci/check-images.sh scripts/ci/fake-docker/lib.sh scripts/validate-setup.sh fixitfelix-chr21-1 assets/callability_genes.tsv scripts/16b-mosdepth.sh scripts/ci/fake-docker/lib.sh scripts/setup.sh +followups-ci-3-sv-rule-regions assets/sv_rule_regions.tsv scripts/ci/build-sv-rule-regions.sh scripts/ci/fake-docker/lib.sh followups-clinvar-traits-1-rules-html assets/genotype_rules.tsv assets/traits.tsv bin/render_report.py bin/traits_report.py scripts/42-traits.sh scripts/ci/check-rule-alleles.sh followups-cn-1-couple-report assets/carrier_genes.tsv assets/par_grch38.bed assets/stub/pedigree.tsv bin/collect_summary.py bin/couple_report.py bin/family_report.py bin/render_report.py bin/results_index.py modules/local/couple/* modules/local/html_report/* scripts/39-couple.sh scripts/ci/fake-docker/lib.sh genome-lookup-1 scripts/ci/fake-docker/lib.sh scripts/owngenome-lookup.sh diff --git a/scripts/ci/e2e-case-times.tsv b/scripts/ci/e2e-case-times.tsv index 82e0a424..9e1f0b0e 100644 --- a/scripts/ci/e2e-case-times.tsv +++ b/scripts/ci/e2e-case-times.tsv @@ -64,7 +64,7 @@ bumps-drop-in-baseline shard 4 bash-step-11-roh,bash-step-14-imputation,bash-ste callability-1-catalog shard 74 - callability-2-genes shard 32 - chip-clonal-1 shard 277 - -ci-e2e-2-locityper shard 2289 - +ci-e2e-2-locityper shard 142 - clinical-filter-mane-and-quality-1-tiers shard 14 - clinical-filter-mane-and-quality-2-vep-norm shard 2 - clinical-filter-mane-and-quality-3-modules shard 8 clinical-filter-mane-and-quality-1-tiers @@ -88,6 +88,7 @@ expansions-and-rfc1-1-motif shard 9 - families-file-1 shard 76 - family-joint-1 shard 515 - fixitfelix-chr21-1 shard 98 - +followups-ci-3-sv-rule-regions shard 120 - followups-clinvar-traits-1-rules-html shard 2 - followups-cn-1-couple-report shard 69 - genome-lookup-1 shard 7 - diff --git a/scripts/ci/e2e-run.sh b/scripts/ci/e2e-run.sh index 2546b6aa..89fc0dc2 100755 --- a/scripts/ci/e2e-run.sh +++ b/scripts/ci/e2e-run.sh @@ -192,11 +192,12 @@ PGS000066 b34f31b8b3a7dbbbb58882a70e14649d"} # compare, as "FOLDER COMMAND...": each lists its pinned downloads with # --list ("NAME URL SHA256"), and entry catalogs holds them in FOLDER under # NAME, where the builder's cache reads them (callability-1-catalog, -# cn-intervals-1-catalog). GENCODE's GTFs come from +# cn-intervals-1-catalog, followups-ci-3-sv-rule-regions). GENCODE's GTFs come from # EBI's FTP server, which refused every connection on 2026-11-06 (run # 42259432681). CATALOGS="callability bash ${REPO}/scripts/ci/build-callability.sh -cn-intervals python3 ${REPO}/scripts/ci/build-cn-intervals.py" +cn-intervals python3 ${REPO}/scripts/ci/build-cn-intervals.py +sv-rule-regions bash ${REPO}/scripts/ci/build-sv-rule-regions.sh" # catalog_files: "FOLDER NAME URL SHA256" of every download of CATALOGS. catalog_files() { local dir interp builder out @@ -718,7 +719,7 @@ trap 'kill "$LOW_PID" 2> /dev/null || true' EXIT # finds them and downloads nothing. # prs-scores-and-run-options-1: the four real PGS Catalog score files # (ftp.ebi.ac.uk); the case copies them from E2E_DATA_CACHE/pgs. -# callability-1-catalog, cn-intervals-1-catalog: the pinned downloads of their builders (CATALOGS above); each case +# callability-1-catalog, cn-intervals-1-catalog, followups-ci-3-sv-rule-regions: the pinned downloads of their builders (CATALOGS above); each case # points its builder's cache at E2E_DATA_CACHE/catalogs/. data_name() { # shellcheck source=../../versions.env @@ -727,7 +728,7 @@ data_name() { archaic-ancestry-1.sh) echo "archaic ${GENOME_DIR}/reference/archaic/$(. "${REPO}/versions.env" && echo "$ARCHAIC_DATA_VERSION")/download" ;; bash-step-08-hla.sh) echo "hla ${GENOME_DIR} offline" ;; prs-scores-and-run-options-1-strand-sex.sh) echo "pgs -" ;; - callability-1-catalog.sh|cn-intervals-1-catalog.sh) echo "catalogs -" ;; + callability-1-catalog.sh|cn-intervals-1-catalog.sh|followups-ci-3-sv-rule-regions.sh) echo "catalogs -" ;; esac } # Every HTTP(S) request of a process run under it goes to a closed port. diff --git a/tests/e2e/followups-ci-3-sv-rule-regions.sh b/tests/e2e/followups-ci-3-sv-rule-regions.sh new file mode 100755 index 00000000..2a244f18 --- /dev/null +++ b/tests/e2e/followups-ci-3-sv-rule-regions.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# assets/sv_rule_regions.tsv is what scripts/ci/build-sv-rule-regions.sh +# writes from its four pinned downloads, byte for byte: the builder's --check +# mode rebuilds it and compares. A hand edit, a pin that moved without a +# rebuild, or a builder change that moves a row fails here. Control: --check +# on a copy of the committed file with one cell changed must exit 1 and show +# that row. The downloads come from the e2e data cache when the runner filled +# it (entry catalogs, mirror first; scripts/ci/e2e-run.sh), else from their +# sources. +. "$(dirname "$0")/lib.sh" + +COMMITTED="${REPO}/assets/sv_rule_regions.tsv" +CACHE="${CASE_TMP}/downloads" +[ -z "${E2E_DATA_CACHE:-}" ] || CACHE="${E2E_DATA_CACHE}/catalogs/sv-rule-regions" + +echo "+ scripts/ci/build-sv-rule-regions.sh --check assets/sv_rule_regions.tsv" +SV_RULE_CACHE="$CACHE" bash "${REPO}/scripts/ci/build-sv-rule-regions.sh" --check "$COMMITTED" > "$STEP_LOG" 2>&1 +rc=$? +cat "$STEP_LOG" +check_eq "--check on the committed table exits 0" "$rc" 0 +check "and says it is what the pinned downloads give" grep -q "is what the pinned downloads give, byte for byte" "$STEP_LOG" +check "with the builder's row counts" grep -q "^Rows: gene_alias 10, ig_tr_locus 6, mhc 1, tandem_repeat 2409$" "$STEP_LOG" +echo "${CASE_NAME}: $(grep -c '^Downloading ' "$STEP_LOG") of 4 downloads not found in ${CACHE}" >> "$E2E_NOTES" + +# The committed table's own counts, which docs/05-annotsv.md states. +check_eq "the committed table has 2409 tandem_repeat rows" \ + "$(awk -F'\t' '!/^#/ && $4 == "tandem_repeat"' "$COMMITTED" | wc -l | tr -d ' ')" 2409 + +# --- control: one cell changed --------------------------------------------------- +EDITED="${CASE_TMP}/edited.tsv" +awk -F'\t' -v OFS='\t' '!/^#/ && $4 == "mhc" {$3 = $3 + 1} 1' "$COMMITTED" > "$EDITED" +check_eq "control: the copy differs from the committed table in one line" \ + "$(diff "$COMMITTED" "$EDITED" | grep -c '^>')" 1 +echo "+ scripts/ci/build-sv-rule-regions.sh --check on a copy with the MHC row's end moved by 1" +SV_RULE_CACHE="$CACHE" bash "${REPO}/scripts/ci/build-sv-rule-regions.sh" --check "$EDITED" > "${CASE_TMP}/edited.log" 2>&1 +rc=$? +cat "${CASE_TMP}/edited.log" +check_eq "control: --check on the edited copy exits 1" "$rc" 1 +check "control: and says to rebuild it" grep -q "is not what the pinned downloads give; rebuild it" "${CASE_TMP}/edited.log" +check "control: and shows the MHC row that differs" grep -qP '^< chr6\t\d+\t\d+\tmhc\t' "${CASE_TMP}/edited.log" +check_eq "control: every download came from the cache the first run filled" \ + "$(grep -c '^Downloading ' "${CASE_TMP}/edited.log")" 0 + +finish From ee22ce4c99ee18ee36d52008e0d568e79d1edcfb Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 02:52:52 +0100 Subject: [PATCH 3/6] docs(locityper): give the jellyfish count's time before and after sizing its hash Bead: pgp-5bja.22 --- docs/53-locityper.md | 2 +- tests/e2e/ci-e2e-2-locityper.sh | 14 ++++++++------ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/53-locityper.md b/docs/53-locityper.md index b727da06..7a38b361 100644 --- a/docs/53-locityper.md +++ b/docs/53-locityper.md @@ -32,7 +32,7 @@ Pinned in `versions.env`; [Image versions](versions.md) lists the current tags. The panel is the HPRC release 2 Minigraph-Cactus pangenome against GRCh38, as a VCF (`LOCITYPER_PANEL_URL`, `LOCITYPER_PANEL_VERSION` in `versions.env`). It holds the two haplotypes of each of its 232 samples. The full VCF is 6.7 GB; `setup.sh` reads only the loci and 250 kb either side, over https, through the VCF's `.tbi`, whose sha256 `LOCITYPER_PANEL_TBI_SHA256` pins. Nothing of it is committed. The pin covers the index, not the records: they are trusted as the HPRC's S3 bucket serves them over https, as for the other data `setup.sh` reads in place. A changed file with the same index would build without an error. -Then it follows Locityper's documentation (https://locityper.vercel.app, "Preparing targets"): jellyfish counts the 25-mers of `REF_FASTA` (its hash table sized to the reference's length from the `.fai`, at least 10 million entries, so a small reference does not reserve the memory all of GRCh38 needs), `locityper target` cuts the panel's haplotypes at each locus, and `locityper augment` aligns them to each other, which the weighted distance needs. A new reference, image, panel or locus list builds again. `LOCITYPER_DATA` puts the database elsewhere and `LOCITYPER_LOCI` replaces the locus list. +Then it follows Locityper's documentation (https://locityper.vercel.app, "Preparing targets"): jellyfish counts the 25-mers of `REF_FASTA` (its hash table sized to the reference's length from the `.fai`, at least 10 million entries, so a small reference does not reserve the memory all of GRCh38 needs: on the 235 Mb reference of the e2e case `ci-e2e-2-locityper` the count took 36 minutes with a table fixed at 3G entries and takes 36 seconds now, and the case went from 2,289 to 142 seconds), `locityper target` cuts the panel's haplotypes at each locus, and `locityper augment` aligns them to each other, which the weighted distance needs. A new reference, image, panel or locus list builds again. `LOCITYPER_DATA` puts the database elsewhere and `LOCITYPER_LOCI` replaces the locus list. ## The loci diff --git a/tests/e2e/ci-e2e-2-locityper.sh b/tests/e2e/ci-e2e-2-locityper.sh index 41722bca..89185a9f 100755 --- a/tests/e2e/ci-e2e-2-locityper.sh +++ b/tests/e2e/ci-e2e-2-locityper.sh @@ -8,12 +8,14 @@ # The fixture holds reads over the MHC but none over FCGR or KIR, so the # database is built for the HLA-DQB1 row of assets/locityper_loci.bed # (LOCITYPER_LOCI). HLA-DRB1, the other MHC row, is left out for time: with -# both, jellyfish took 35 minutes and the pairwise alignments of -# `locityper augment` 23 more (fork run 42301973505). With HLA-DQB1 alone -# augment takes about a minute, but jellyfish, whose time does not depend on -# the loci, still takes 36 (38 minutes in all, fork run 42325306077), so the -# case keeps a limit of 90 minutes instead of the runner's 60 (the -# case-timeout line below). It runs in a GENOME_DIR of its own with a +# both, the pairwise alignments of `locityper augment` took 23 minutes (fork +# run 42301973505); with HLA-DQB1 alone augment takes about a minute. +# Jellyfish's time does not depend on the loci. With a hash fixed at 3G +# entries it took 36 minutes and the case 38 (2289 s, fork run 42325306077); +# with the hash sized to the reference it takes 36 s and the case 142 s +# (fork run 42421585308). The case keeps a limit of 90 minutes instead of +# the runner's 60 (the case-timeout line below), room for a slow runner. +# It runs in a GENOME_DIR of its own with a # reference of chr6 and chr20 cut from the fixture's, so jellyfish # counts 235 Mb instead of the fixture's 1.8 Gb; the fixture's reads over the # MHC and chr20:10.0-10.5 Mb are aligned to it, since Locityper needs the From 4c6b4b05e36c3b6fc11e5f6d6c0e1b6002961412 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 05:40:56 +0100 Subject: [PATCH 4/6] ci(e2e): time the SV rule regions case and the Locityper case from their runs followups-ci-3-sv-rule-regions took 44 s in fork run 42436219592; ci-e2e-2-locityper took 142 s and 174 s (fork runs 42421585308 and 42436219592), median 158. Bead: pgp-5bja.22 --- scripts/ci/e2e-case-times.tsv | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/ci/e2e-case-times.tsv b/scripts/ci/e2e-case-times.tsv index 9e1f0b0e..ca18bff5 100644 --- a/scripts/ci/e2e-case-times.tsv +++ b/scripts/ci/e2e-case-times.tsv @@ -64,7 +64,7 @@ bumps-drop-in-baseline shard 4 bash-step-11-roh,bash-step-14-imputation,bash-ste callability-1-catalog shard 74 - callability-2-genes shard 32 - chip-clonal-1 shard 277 - -ci-e2e-2-locityper shard 142 - +ci-e2e-2-locityper shard 158 - clinical-filter-mane-and-quality-1-tiers shard 14 - clinical-filter-mane-and-quality-2-vep-norm shard 2 - clinical-filter-mane-and-quality-3-modules shard 8 clinical-filter-mane-and-quality-1-tiers @@ -88,7 +88,7 @@ expansions-and-rfc1-1-motif shard 9 - families-file-1 shard 76 - family-joint-1 shard 515 - fixitfelix-chr21-1 shard 98 - -followups-ci-3-sv-rule-regions shard 120 - +followups-ci-3-sv-rule-regions shard 44 - followups-clinvar-traits-1-rules-html shard 2 - followups-cn-1-couple-report shard 69 - genome-lookup-1 shard 7 - From ab829b46a3b43f409adfa1ee1f3f90c019bf0d4a Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 07:47:24 +0100 Subject: [PATCH 5/6] test(e2e): count the third table builder's downloads and assert cache reuse only where it can fail The catalogs comments said two builders and two cases; there are three. tests/test_e2e_data_cache.sh now counts the SV rule regions builder's four pinned downloads too (15, was 11). The SV rule regions case checks that its second run downloaded nothing only without the e2e data cache: with it the runner fills the cache first, so that check could not go red. --- .github/workflows/e2e.yml | 2 +- scripts/ci/e2e-run.sh | 2 +- tests/e2e/followups-ci-3-sv-rule-regions.sh | 9 +++++++-- tests/test_e2e_data_cache.sh | 8 ++++---- 4 files changed, 13 insertions(+), 8 deletions(-) diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index bb9f8e28..84345ec6 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -693,7 +693,7 @@ jobs: # of two IPD-IMGT/HLA releases and GENCODE's genes and exons # (bash-step-08-hla, which then runs with the network blocked), and the # four real PGS Catalog score files of prs-scores-and-run-options-1. - # A third keyed the same way: the pinned downloads of the two table + # A third keyed the same way: the pinned downloads of the three table # builders (GENCODE's GTF from EBI among them) that # callability-1-catalog, cn-intervals-1-catalog and # followups-ci-3-sv-rule-regions rebuild their tables from. diff --git a/scripts/ci/e2e-run.sh b/scripts/ci/e2e-run.sh index 89fc0dc2..8d4bfe1c 100755 --- a/scripts/ci/e2e-run.sh +++ b/scripts/ci/e2e-run.sh @@ -188,7 +188,7 @@ PGS_MD5=${E2E_PGS_MD5-"PGS002257 bb1b2b11a499c5930e8c835647e48296 PGS000065 90ac7aabbbde9bdaa681dbdb66c0ebd8 PGS000064 43b39c4450f423f0772bbd8779dd640f PGS000066 b34f31b8b3a7dbbbb58882a70e14649d"} -# CATALOGS: the builders of the committed tables two cases rebuild and +# CATALOGS: the builders of the committed tables three cases rebuild and # compare, as "FOLDER COMMAND...": each lists its pinned downloads with # --list ("NAME URL SHA256"), and entry catalogs holds them in FOLDER under # NAME, where the builder's cache reads them (callability-1-catalog, diff --git a/tests/e2e/followups-ci-3-sv-rule-regions.sh b/tests/e2e/followups-ci-3-sv-rule-regions.sh index 2a244f18..60179eab 100755 --- a/tests/e2e/followups-ci-3-sv-rule-regions.sh +++ b/tests/e2e/followups-ci-3-sv-rule-regions.sh @@ -38,7 +38,12 @@ cat "${CASE_TMP}/edited.log" check_eq "control: --check on the edited copy exits 1" "$rc" 1 check "control: and says to rebuild it" grep -q "is not what the pinned downloads give; rebuild it" "${CASE_TMP}/edited.log" check "control: and shows the MHC row that differs" grep -qP '^< chr6\t\d+\t\d+\tmhc\t' "${CASE_TMP}/edited.log" -check_eq "control: every download came from the cache the first run filled" \ - "$(grep -c '^Downloading ' "${CASE_TMP}/edited.log")" 0 +# Cache reuse: only without the e2e data cache, where the first run filled +# CACHE itself. With it the runner filled CACHE before the case, so the first +# run already downloads nothing and this could not go red. +if [ -z "${E2E_DATA_CACHE:-}" ]; then + check_eq "control: the second run downloaded nothing, it read the first run's cache" \ + "$(grep -c '^Downloading ' "${CASE_TMP}/edited.log")" 0 +fi finish diff --git a/tests/test_e2e_data_cache.sh b/tests/test_e2e_data_cache.sh index 93348564..5d0063df 100755 --- a/tests/test_e2e_data_cache.sh +++ b/tests/test_e2e_data_cache.sh @@ -70,12 +70,12 @@ else echo "FAIL: the hla key does not match its inputs (${other:-?} ${pin:-?} ${ if "$RUN" --data-key pgs | grep -qE '^e2e-data-pgs-[0-9a-f]{16}$'; then echo "PASS: the pgs key is a hash" else echo "FAIL: the pgs key"; failed=1; fi -# catalogs: the downloads of the two table builders, each in its folder. +# catalogs: the downloads of the three table builders, each in its folder. if "$RUN" --data-key catalogs | grep -qE '^e2e-data-catalogs-[0-9a-f]{16}$'; then echo "PASS: the catalogs key is a hash" else echo "FAIL: the catalogs key"; failed=1; fi -n=$( { bash "${REPO}/scripts/ci/build-callability.sh" --list; python3 "${REPO}/scripts/ci/build-cn-intervals.py" --list; } | awk 'NF == 3 && $2 ~ /^https:/ && $3 ~ /^[0-9a-f]{64}$/' | wc -l | tr -d ' ') -if [ "$n" = 11 ]; then echo "PASS: the two builders list 11 pinned downloads (7 and 4)" -else echo "FAIL: the builders list ${n} pinned downloads, want 11"; failed=1; fi +n=$( { bash "${REPO}/scripts/ci/build-callability.sh" --list; python3 "${REPO}/scripts/ci/build-cn-intervals.py" --list; bash "${REPO}/scripts/ci/build-sv-rule-regions.sh" --list; } | awk 'NF == 3 && $2 ~ /^https:/ && $3 ~ /^[0-9a-f]{64}$/' | wc -l | tr -d ' ') +if [ "$n" = 15 ]; then echo "PASS: the three builders list 15 pinned downloads (7, 4 and 4)" +else echo "FAIL: the builders list ${n} pinned downloads, want 15"; failed=1; fi mkdir -p "${T}/catalogs/callability" expect "catalogs: an entry with no file fails" 1 catalogs bash "${REPO}/scripts/ci/build-callability.sh" --list | head -n 1 | { read -r name _ _; echo planted > "${T}/catalogs/callability/${name}"; } From 1693ae5c2faabb8a2ca17d06a74273de82ff670f Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 8 Nov 2026 10:05:42 +0100 Subject: [PATCH 6/6] test(e2e): check every row the table builders list, and say the TRExplorer catalog is not on the mirror A malformed --list row would pass the count of well-formed rows and reach the catalogs fill; the test now wants every row well formed. docs/testing.md said the mirror release holds every table builder download; the TRExplorer catalog of build-sv-rule-regions.sh comes from its own GitHub release. --- docs/testing.md | 2 +- tests/test_e2e_data_cache.sh | 9 ++++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/testing.md b/docs/testing.md index a740697c..c01c54c8 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -102,7 +102,7 @@ The base leg runs every base case whenever any case runs, because every shard st The images the cases use are cached in four GitHub Actions cache entries, one per group. `scripts/ci/e2e-run.sh --image-groups` names each image's group, taken from its repository name without the tag, so a version bump changes the key of one group only; a group whose key missed restores the newest older entry of that group and pulls only what changed. A group keeps its smallest images up to 1500 MB of zstd output, and the larger ones are pulled on every run. Four groups of 1500 MB and the six data entries below stay under the repository's 10 GB cache limit even while main replaces one group: 5.9 GB of images, 1.5 GB more for the group being replaced, and about 2.4 GB of data. Images pinned by digest are not cached, because `docker load` gives them back without the digest, and they are left out of the group's key, so a digest bump misses no entry. Only the base leg saves, and only on main (or in a run by hand with `save_images`, on any branch), after it pulled every image the cases use and got through its cases, and only the groups whose key missed: a cache saved by a pull request can be read by that pull request alone and would only push main's entries out. The job summary lists every image with its size and what the cache did with it: restored on an exact key hit, restored from an older entry, saved, or pulled. Every merge in the e2e scope starts a run on main, so the cache follows main's images. GitHub drops an entry no run has read for 7 days, so after a quiet week the next run on main fills the cache again. Until this layout the cache was one entry, and by October 2026 it had grown to about 11 GB, over the limit, so every run missed it. -Six downloads are cached the same way. Two come from EBI's 1000 Genomes server, which is slow and at times stalls: MoChA's chr20 panel with its manifest (0.64 GB, case `mocha-1`), keyed on `MOCHA_PANEL_MANIFEST_SHA256` and the chromosome, and the 1000 Genomes phase 3 sites VCF (1.36 GB, case `archaic-ancestry-1`), keyed on `ARCHAIC_1KG_SITES_MD5`. On a miss, each of those two is tried first at `E2E_DATA_MIRROR`, the GitHub release `e2e-data-1` that holds copies of the pinned files, and then at EBI; the pin checks both. The release also holds the four PGS Catalog score files, the HAPNEST panel and the table builders' downloads below, so an EBI outage costs none of them on a cold cache. Two are small and failed whole runs on 2026-11-04, because their case is in the base leg or a shard that cannot start without it: `hla.dat` of the two IPD-IMGT/HLA releases `bash-step-08-hla` installs, from raw.githubusercontent.com (which served it under 1 KB/s), with GENCODE's gene and exon files from EBI's FTP server (curl exit 7 and 28), and the four real PGS Catalog score files of `prs-scores-and-run-options-1-strand-sex`, from the same server. `scripts/ci/e2e-run.sh --data-key` names their keys: the releases and a hash of the code in `scripts/lib/common.sh` that installs them, and a hash of the score ids and their server. `scripts/ci/e2e-run.sh` hands each case its files just before it runs, at the paths `setup.sh` and `install_data_file` read them from, so they download nothing; the score files the case copies from the cache itself. On a miss it downloads them first, checked against their pins or the checksum files the servers publish beside them (`MD5SUMS`, `md5checksum.txt`, the `.md5` of each score), trying up to 6 times 30 s apart. The four score files are pinned by md5 in `PGS_MD5` of `e2e-run.sh` (the md5 the PGS Catalog served beside each on 2026-11-06), tried on the mirror first and then at EBI; a restored or downloaded score must match its pin as well as its `.md5`. A file gets at most `E2E_MIRROR_BUDGET` seconds on the mirror (900, or half the fill's budget when that is less), so a mirror that trickles leaves EBI its turn. One fill may take at most `E2E_DATA_BUDGET` seconds (2400 in `e2e.yml`): a source that trickles above `fetch`'s stall limit would otherwise run on to the leg's 240-minute limit, as EBI's 1000 Genomes server did twice on 2026-11-05. A fill still running then is stopped and its case fails as `FAIL (data fill over 2400s)` without running, since its own download would be as slow. A restored entry is checked again before use (`--data-check`): the IPD-IMGT/HLA and GENCODE entry against the sha256 list written when it was filled, the score files against their `.md5`; one that fails is downloaded again. `bash-step-08-hla` then runs with every HTTP and HTTPS request sent to a closed port, after a control shows that a request under that block fails at the closed port, so a warm cache cannot reach either host. Only the leg whose cases include the case restores and saves its entry, and it saves only on main or in a run by hand with `save_images`, as with the images. The job summary says whether each key hit, and the case's log says where its files came from. The fifth is the PGS Catalog's synthetic HAPNEST panel (0.27 GB, cases `prs-2-ancestry` and `prs-scores-and-run-options-2-panel-chrx`) from EBI's FTP server, which failed cases with curl exit 7 on 2026-11-04. It is keyed on the sha256 `tests/e2e/lib.sh` pins (`HAPNEST_SHA256`). The cases handle it themselves (`setup_ancestry_panel` in `lib.sh`): with the panel in the cache, or on a miss from `E2E_DATA_MIRROR`, they check the sha256 and run `setup.sh --ancestry-panel` with every HTTP and HTTPS request sent to a closed port, so an EBI outage cannot fail them; when neither has it, `setup.sh` downloads the panel from EBI as a user's run does. The first case puts a panel that did not come from the cache in the cache for the save step. A red control in `prs-2-ancestry` runs `setup.sh` with the network blocked and no panel, which must fail. The sixth, entry `catalogs`, holds the pinned downloads of `scripts/ci/build-callability.sh`, `scripts/ci/build-cn-intervals.py` and `scripts/ci/build-sv-rule-regions.sh`, whose cases `callability-1-catalog`, `cn-intervals-1-catalog` and `followups-ci-3-sv-rule-regions` rebuild `assets/callability_genes.tsv`, the interval rows of `bin/cn_loci.tsv` and `assets/sv_rule_regions.tsv` and compare them with the committed files. Each builder lists its downloads with `--list`; the key is a hash of those lists, so a new pin is a new key. GENCODE's GTF among them comes from EBI's FTP server, which refused every connection for minutes on 2026-11-06 and failed `callability-1-catalog` on main (run 42259432681). The runner fills the entry as it does the others, mirror first and every file checked against its sha256, and each case points its builder's cache at its folder there; a builder that still has to download tries for about five minutes, 30 s apart. Every download, there and in `setup.sh`, gives up on a transfer that stalls (`FETCH_STALL`, see `fetch` in `scripts/lib/common.sh`), and the next try resumes it. +Six downloads are cached the same way. Two come from EBI's 1000 Genomes server, which is slow and at times stalls: MoChA's chr20 panel with its manifest (0.64 GB, case `mocha-1`), keyed on `MOCHA_PANEL_MANIFEST_SHA256` and the chromosome, and the 1000 Genomes phase 3 sites VCF (1.36 GB, case `archaic-ancestry-1`), keyed on `ARCHAIC_1KG_SITES_MD5`. On a miss, each of those two is tried first at `E2E_DATA_MIRROR`, the GitHub release `e2e-data-1` that holds copies of the pinned files, and then at EBI; the pin checks both. The release also holds the four PGS Catalog score files, the HAPNEST panel and the table builders' downloads below, so an EBI outage costs none of them on a cold cache. The one builder download it lacks is the TRExplorer catalog of `build-sv-rule-regions.sh`, which is not on EBI: on a miss it comes from its own GitHub release. Two are small and failed whole runs on 2026-11-04, because their case is in the base leg or a shard that cannot start without it: `hla.dat` of the two IPD-IMGT/HLA releases `bash-step-08-hla` installs, from raw.githubusercontent.com (which served it under 1 KB/s), with GENCODE's gene and exon files from EBI's FTP server (curl exit 7 and 28), and the four real PGS Catalog score files of `prs-scores-and-run-options-1-strand-sex`, from the same server. `scripts/ci/e2e-run.sh --data-key` names their keys: the releases and a hash of the code in `scripts/lib/common.sh` that installs them, and a hash of the score ids and their server. `scripts/ci/e2e-run.sh` hands each case its files just before it runs, at the paths `setup.sh` and `install_data_file` read them from, so they download nothing; the score files the case copies from the cache itself. On a miss it downloads them first, checked against their pins or the checksum files the servers publish beside them (`MD5SUMS`, `md5checksum.txt`, the `.md5` of each score), trying up to 6 times 30 s apart. The four score files are pinned by md5 in `PGS_MD5` of `e2e-run.sh` (the md5 the PGS Catalog served beside each on 2026-11-06), tried on the mirror first and then at EBI; a restored or downloaded score must match its pin as well as its `.md5`. A file gets at most `E2E_MIRROR_BUDGET` seconds on the mirror (900, or half the fill's budget when that is less), so a mirror that trickles leaves EBI its turn. One fill may take at most `E2E_DATA_BUDGET` seconds (2400 in `e2e.yml`): a source that trickles above `fetch`'s stall limit would otherwise run on to the leg's 240-minute limit, as EBI's 1000 Genomes server did twice on 2026-11-05. A fill still running then is stopped and its case fails as `FAIL (data fill over 2400s)` without running, since its own download would be as slow. A restored entry is checked again before use (`--data-check`): the IPD-IMGT/HLA and GENCODE entry against the sha256 list written when it was filled, the score files against their `.md5`; one that fails is downloaded again. `bash-step-08-hla` then runs with every HTTP and HTTPS request sent to a closed port, after a control shows that a request under that block fails at the closed port, so a warm cache cannot reach either host. Only the leg whose cases include the case restores and saves its entry, and it saves only on main or in a run by hand with `save_images`, as with the images. The job summary says whether each key hit, and the case's log says where its files came from. The fifth is the PGS Catalog's synthetic HAPNEST panel (0.27 GB, cases `prs-2-ancestry` and `prs-scores-and-run-options-2-panel-chrx`) from EBI's FTP server, which failed cases with curl exit 7 on 2026-11-04. It is keyed on the sha256 `tests/e2e/lib.sh` pins (`HAPNEST_SHA256`). The cases handle it themselves (`setup_ancestry_panel` in `lib.sh`): with the panel in the cache, or on a miss from `E2E_DATA_MIRROR`, they check the sha256 and run `setup.sh --ancestry-panel` with every HTTP and HTTPS request sent to a closed port, so an EBI outage cannot fail them; when neither has it, `setup.sh` downloads the panel from EBI as a user's run does. The first case puts a panel that did not come from the cache in the cache for the save step. A red control in `prs-2-ancestry` runs `setup.sh` with the network blocked and no panel, which must fail. The sixth, entry `catalogs`, holds the pinned downloads of `scripts/ci/build-callability.sh`, `scripts/ci/build-cn-intervals.py` and `scripts/ci/build-sv-rule-regions.sh`, whose cases `callability-1-catalog`, `cn-intervals-1-catalog` and `followups-ci-3-sv-rule-regions` rebuild `assets/callability_genes.tsv`, the interval rows of `bin/cn_loci.tsv` and `assets/sv_rule_regions.tsv` and compare them with the committed files. Each builder lists its downloads with `--list`; the key is a hash of those lists, so a new pin is a new key. GENCODE's GTF among them comes from EBI's FTP server, which refused every connection for minutes on 2026-11-06 and failed `callability-1-catalog` on main (run 42259432681). The runner fills the entry as it does the others, mirror first and every file checked against its sha256, and each case points its builder's cache at its folder there; a builder that still has to download tries for about five minutes, 30 s apart. Every download, there and in `setup.sh`, gives up on a transfer that stalls (`FETCH_STALL`, see `fetch` in `scripts/lib/common.sh`), and the next try resumes it. How long it took in one job, before the split: a passing e2e job took about 105 to 205 minutes with 84 cases, and 230 minutes of cases with 130 (run 41390434697, 2026-11-02). The runner decides much of it, not the cache: the same CPU model took 106 minutes in one run and 166 in another. In October 2026, with 84 cases, the same code took 106 and 166 minutes on an AMD EPYC 9V45 (runs 39197502832 and 39075176392, the second of which also saved the cache), 112 minutes on an Intel Xeon 6973P (run 39088086725), 156 and 191 minutes on an AMD EPYC 7763 (runs 39075764278 and 39075201465) and 205 minutes on an AMD EPYC 9V74 (run 39197512594, which saved three groups). The image cache saves about 7 minutes of that: pulling every image took 9 minutes, restoring and loading the four groups took 2.5 (run 39075201465 restored 39 images in 118 seconds), and saving them all took 12. Each leg has a limit of 240 minutes (355 for a traced shard leg); the base leg waits up to 75 minutes for a fixture a pull request bumps. Each leg's first steps print the runner's CPU model and CPU count to the log and the job summary, so a later timing can name the hardware. diff --git a/tests/test_e2e_data_cache.sh b/tests/test_e2e_data_cache.sh index 5d0063df..bfaf97a2 100755 --- a/tests/test_e2e_data_cache.sh +++ b/tests/test_e2e_data_cache.sh @@ -73,9 +73,12 @@ else echo "FAIL: the pgs key"; failed=1; fi # catalogs: the downloads of the three table builders, each in its folder. if "$RUN" --data-key catalogs | grep -qE '^e2e-data-catalogs-[0-9a-f]{16}$'; then echo "PASS: the catalogs key is a hash" else echo "FAIL: the catalogs key"; failed=1; fi -n=$( { bash "${REPO}/scripts/ci/build-callability.sh" --list; python3 "${REPO}/scripts/ci/build-cn-intervals.py" --list; bash "${REPO}/scripts/ci/build-sv-rule-regions.sh" --list; } | awk 'NF == 3 && $2 ~ /^https:/ && $3 ~ /^[0-9a-f]{64}$/' | wc -l | tr -d ' ') -if [ "$n" = 15 ]; then echo "PASS: the three builders list 15 pinned downloads (7, 4 and 4)" -else echo "FAIL: the builders list ${n} pinned downloads, want 15"; failed=1; fi +# Every row must be "NAME URL SHA256": a malformed row would reach the fill. +list=$( { bash "${REPO}/scripts/ci/build-callability.sh" --list; python3 "${REPO}/scripts/ci/build-cn-intervals.py" --list; bash "${REPO}/scripts/ci/build-sv-rule-regions.sh" --list; } ) +n=$(awk 'NF == 3 && $2 ~ /^https:/ && $3 ~ /^[0-9a-f]{64}$/' <<< "$list" | wc -l | tr -d ' ') +all=$(wc -l <<< "$list" | tr -d ' ') +if [ "$n" = 15 ] && [ "$all" = 15 ]; then echo "PASS: the three builders list 15 pinned downloads (7, 4 and 4), every row well formed" +else echo "FAIL: the builders list ${all} rows, ${n} of them well formed; want 15 and 15"; failed=1; fi mkdir -p "${T}/catalogs/callability" expect "catalogs: an entry with no file fails" 1 catalogs bash "${REPO}/scripts/ci/build-callability.sh" --list | head -n 1 | { read -r name _ _; echo planted > "${T}/catalogs/callability/${name}"; }