Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 115 additions & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@ name: Nightly

# The slow checks pull requests don't wait for: every platform's tests and the release build of
# the apps, once a night on main. A night with no new commits since the last successful run is
# skipped. A failure opens an issue (or comments on the open one); the next success closes it.
# skipped for the build. Generated overlays run every night with a fresh seed: each new minimized
# disagreement opens its own issue, and an overlay run that fails without one is reported like a
# build failure. A build failure opens an issue (or comments on the open one); the next success
# closes it.
#
# gh workflow run nightly.yml -R opengeos/RustGIS --ref main

Expand Down Expand Up @@ -127,6 +130,117 @@ jobs:
cargo test --locked -p rustgis-data --test extent_query_properties
PROPERTY_CASES=20000 cargo test --locked -p rustgis-data --test date_properties

# Use a fresh seed even when the commit is the same as last night.
overlay:
name: Generated polygon overlays
if: github.repository == 'opengeos/RustGIS' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 20
outputs:
findings: ${{ steps.findings.outputs.artifact-id }}
env:
CARGO_PROFILE_DEV_DEBUG: line-tables-only
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec # stable
- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
- name: Compare overlays and minimize disagreements
id: compare
env:
PROPERTY_CASES: "256"
run: |
set -euo pipefail
cargo test --locked -p rustgis-analysis --test reference --no-run --message-format=json > "$RUNNER_TEMP/reference-build.jsonl"
runner="$(jq -er 'select(.reason == "compiler-artifact" and .target.name == "reference" and .executable != null) | .executable' "$RUNNER_TEMP/reference-build.jsonl")"
PROPERTY_SEED="$(python3 -c 'import secrets; print(secrets.randbits(64))')"
export PROPERTY_SEED
time uv run crates/analysis/tests/reference/generate.py --overlay "$runner" --findings "$RUNNER_TEMP/overlay-findings"
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
id: findings
if: failure() && steps.compare.outcome == 'failure'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings are lost when the compare step is cancelled (confidence: medium). failure() is false when the job is cancelled, and the job's 20-minute timeout-minutes cancels it. The step outcome is then cancelled, not failure, so this upload is skipped. overlay-report sees no artifact, takes the "failed without findings" branch, and opens an issue saying "the comparison itself broke", even though minimized reproducers were already written. The same condition at test-rust.yml:144 has the same gap if the job is cancelled. Use always() so the upload runs whenever the compare step didn't succeed:

Suggested change
if: failure() && steps.compare.outcome == 'failure'
if: always() && steps.compare.outcome != 'success'

if-no-files-found: ignore keeps the step harmless when nothing was written.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. When the job times out, the reproducers aren't attached, but the failure isn't lost: overlay-report still runs on the cancelled result and opens or updates the failure issue with the run link, and the log prints the seed, so PROPERTY_SEED=<seed> replays it. The PR's test-rust.yml step has no timeout of its own, and a cancelled PR run is rerun anyway. If timeouts turn out to happen in practice, uploading on always() is a small follow-up.

with:
name: overlay-findings
path: ${{ runner.temp }}/overlay-findings
if-no-files-found: ignore

overlay-report:
name: Report generated overlays
needs: overlay
# Only the reporting job has issue write access. Manual runs keep findings as artifacts.
# A job that ran out of time ends cancelled, but the run isn't: that is a failure too.
if: ${{ !cancelled() && github.event_name == 'schedule' && needs.overlay.result != 'skipped' }}
runs-on: ubuntu-latest
permissions:
actions: read
issues: write
steps:
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8
if: needs.overlay.outputs.findings != ''
with:
name: overlay-findings
path: findings
- name: Open issues for new minimized disagreements
if: needs.overlay.outputs.findings != ''
env:
GH_TOKEN: ${{ github.token }}
RUN: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
# Trust only markers in this workflow's own issues, as "<state> <marker>" lines.
gh api --paginate "repos/$GITHUB_REPOSITORY/issues?state=all&per_page=100" \
| jq -r '.[] | select(.pull_request | not) | select(.user.login == "github-actions[bot]")
| .state as $s | (.body // "") | split("\n")[] | select(startswith("<!-- overlay-finding:")) | "\($s) \(.)"' > reported
for file in findings/*.json; do
tool="$(jq -er '.case.tools[0]' "$file")"
kind="$(jq -er '.kind' "$file")"
fingerprint="$(jq -er '.fingerprint' "$file")"
marker="<!-- overlay-finding:$fingerprint -->"
# A reduced input is reported once. The grouped Clip seam class is reported again
# once its issue is closed, so a fix that doesn't hold shows up.
if grep -Fxq -- "open $marker" reported; then continue; fi
if [ "$fingerprint" != clip-mask-seam ] && grep -Fxq -- "closed $marker" reported; then continue; fi
{
echo "The [nightly run]($RUN) on $GITHUB_SHA found a generated $tool disagreement."
echo
jq -r '"Seed `PROPERTY_SEED=\(.seed)`, case index `\(.index)`. \(.detail)"' "$file"
echo
echo 'Save the minimized case below as JSON and pass it to the overlay replay command in crates/analysis/tests/reference/README.md.'
echo
echo '```json'
cat "$file"
echo '```'
echo "$marker"
} > issue.md
gh issue create -R "$GITHUB_REPOSITORY" --title "Generated overlay disagrees with GEOS ($tool, $kind)" --body-file issue.md
echo "open $marker" >> reported
done
- name: Report a check that failed without findings
env:
GH_TOKEN: ${{ github.token }}
RESULT: ${{ needs.overlay.result }}
FINDINGS: ${{ needs.overlay.outputs.findings }}
RUN: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
title="Nightly generated overlay check failed"
open="$(gh api --paginate "repos/$GITHUB_REPOSITORY/issues?state=open&per_page=100" \
| jq -sr --arg t "$title" '[.[][] | select(.pull_request | not) | select(.user.login == "github-actions[bot]") | select(.title == $t)][0].number // ""')"
if [ "$RESULT" = success ]; then
if [ -n "$open" ]; then
gh issue close "$open" -R "$GITHUB_REPOSITORY" --comment "Fixed: the nightly overlay check on $GITHUB_SHA passed ($RUN)."
fi
elif [ -z "$FINDINGS" ]; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A harness abort after some findings opens no issue (confidence: medium). This branch runs only when FINDINGS is empty. If overlay.py writes findings for one seed and then aborts (a subprocess.TimeoutExpired from the 60 s batch timeout, a RuntimeError from the adapter, or the ValueError for invalid generated input on a later seed), the run is red, the remaining seeds were never checked, and no issue says so. Only the finding issues appear. Track completion separately from findings, for example by having overlay.py write a completion marker into the findings directory on a clean finish, and open or comment on the failure issue whenever RESULT is not success and that marker is absent.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. That case is still reported: the findings it wrote each open their own issue with the seed, and the run is red. What's missed is only a separate note that the later seeds didn't run, and the next night's fresh seed covers them. A completion marker would add a second protocol between overlay.py and the workflow to cover a case that's already visible.

if [ -n "$open" ]; then
gh issue comment "$open" -R "$GITHUB_REPOSITORY" --body "Still failing on $GITHUB_SHA: $RUN"
else
gh issue create -R "$GITHUB_REPOSITORY" --title "$title" \
--body "The generated overlay check on $GITHUB_SHA failed without a minimized finding, so the comparison itself broke: $RUN. This issue closes itself when the check passes."
fi
fi

report:
name: Report
needs: [changed, rust, workflows]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Harness failures open no issue (confidence: medium). overlay-report runs only when overlay uploaded findings, and report does not list overlay in needs. So if the job fails without findings (adapter panic, a RuntimeError, the 60 s subprocess timeout, or the ValueError for an "invalid generated input" that aborts the whole seed in overlay.py), the run is red with no issue. The header comment's "a failure opens an issue" no longer holds for this job. Either add overlay to a reporting path or say in the header that its failures are only visible on the run page.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e8347f7: overlay-report now runs whenever the scheduled overlay job succeeds or fails. A failure with no minimized finding (adapter error, timeout, invalid generated input) opens or comments on a "Nightly generated overlay check failed" issue, which closes itself when the check passes. The header comment says so.

Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/test-rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -126,3 +126,23 @@ jobs:
- name: Build release apps
if: inputs.release-build
run: cargo build --release --locked -p rustgis -p rustgis-cli
# Last, so a disagreement here doesn't skip the doctests, layering check or release build.
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
if: matrix.os == 'ubuntu-latest' && github.workflow != 'Nightly' && (inputs.packages == '--workspace' || contains(format('{0} ', inputs.packages), '-p rustgis-analysis '))
- name: Compare generated polygon overlays with GEOS
if: matrix.os == 'ubuntu-latest' && github.workflow != 'Nightly' && (inputs.packages == '--workspace' || contains(format('{0} ', inputs.packages), '-p rustgis-analysis '))
id: overlay
env:
PACKAGES: ${{ inputs.packages }}
run: |
set -euo pipefail
# Use the test binary already built by nextest with these workspace features.
cargo nextest list $PACKAGES --locked --list-type binaries-only --message-format json > "$RUNNER_TEMP/test-binaries.json"
runner="$(jq -er '."rust-binaries"["rustgis-analysis::reference"]["binary-path"]' "$RUNNER_TEMP/test-binaries.json")"
time uv run crates/analysis/tests/reference/generate.py --overlay "$runner" --findings "$RUNNER_TEMP/overlay-findings"
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
if: failure() && steps.overlay.outcome == 'failure'
with:
name: overlay-findings
path: ${{ runner.temp }}/overlay-findings
if-no-files-found: ignore
9 changes: 6 additions & 3 deletions crates/analysis/src/tools.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2479,9 +2479,12 @@ fn overlay_on(inp: &dyn Inputs, p: &Json, op: Overlay, threads: Option<usize>) -
let rest = if inputs.is_empty() {
from_multipolygon(&z.poly)
} else {
let polys: Vec<MultiPolygon<f64>> =
inputs.iter().filter_map(|r| ds.features.get(*r)?.geometry.as_ref()).filter_map(to_multipolygon).collect();
from_multipolygon(&z.poly.difference(&union_all(polys.iter().flat_map(|m| m.0.iter()))))
let polygons =
inputs.iter().filter_map(|r| ds.features.get(*r)?.geometry.as_ref()).filter_map(to_multipolygon).flat_map(|m| m.0);
// A redundant union rounds a lone polygon's vertices. That can open a
// passage where it touches the zone boundary, joining separate pieces.
let cover = mask_of(MultiPolygon::new(polygons.collect()));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single covering polygons are no longer repaired (confidence: low-medium, a question). Before this change, one covering polygon went through union_all, which also repaired an invalid input (bowtie, self-touching ring), a common case in real data. mask_of now only orients a single polygon, so it reaches difference as is. The zone side already works this way, but this path had the repair. Please confirm that's intended for invalid input. The Union fix is also covered only by overlay-regressions/union-boundary-touch.json, which runs in the Ubuntu-only Python step. A Rust test in cargo test on this polygon would cover every platform.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intended. The union_all over a lone polygon is what rounded its vertices and joined pieces that should stay apart, the bug this PR fixes. mask_of handles a single polygon the same way Clip and Erase already treat their masks, and the zone side does the same, so Union's leftover pieces now follow the same rule as the other overlay tools. Repairing invalid input is a separate question for all of them, not something only this path did on purpose.

On coverage: the reduced Union case runs in the PR's Ubuntu test job, the same job that runs the rest of the reference comparison, so the fix fails CI if it regresses.

from_multipolygon(&z.poly.difference(&cover))
};
Ok::<_, String>(rest)
},
Expand Down
75 changes: 75 additions & 0 deletions crates/analysis/tests/reference.rs
Original file line number Diff line number Diff line change
Expand Up @@ -347,3 +347,78 @@ fn tools_agree_with_the_reference() {
let n = cases["cases"].as_array().map_or(0, Vec::len);
assert!(report.is_empty(), "{} of {n} cases differ from the reference:\n{}", report.len(), report.join("\n"));
}

/// The generated oracle uses the exact projected inputs the tool sees. Return each polygon's
/// rings in that plane for Shapely, including the actual output after RustGIS's round trip.
fn overlay_coordinates(g: &Geometry) -> Json {
let Some(polygons) = to_multipolygon(g) else { return serde_json::json!("non-polygon") };
let rings: Vec<_> = polygons
.0
.iter()
.map(|p| {
std::iter::once(p.exterior())
.chain(p.interiors())
.map(|ring| ring.0.iter().map(|c| [c.x, c.y]).collect::<Vec<_>>())
.collect::<Vec<_>>()
})
.collect();
serde_json::json!(rings)
}

/// A batch from generate.py. Test-harness output and tool diagnostics stay off the reply file.
#[test]
#[ignore = "driven by generate.py --overlay with a batch and reply path"]
fn generated_overlay_batch() {
let batch: Json = serde_json::from_reader(std::io::stdin().lock()).expect("overlay batch");
let mut replies = Vec::new();
for case in batch.as_array().expect("list of overlay cases") {
let layers = Layers(
["a", "b"]
.into_iter()
.map(|name| {
let features: Vec<_> = case[name]
.as_array()
.expect("input features")
.iter()
.enumerate()
.map(|(i, g)| serde_json::json!({"type": "Feature", "geometry": g, "properties": {"id": i.to_string()}}))
.collect();
let collection = serde_json::json!({"type": "FeatureCollection", "features": features});
let ds = rustgis_data::io::read_geojson(&collection.to_string(), name).expect("generated polygon layer").0;
(name.to_string(), ds)
})
.collect(),
);
let projected = |name: &str| {
layers.0[name].features.iter().map(|f| overlay_coordinates(f.geometry.as_ref().expect("generated polygon"))).collect::<Vec<_>>()
};
let mut reply = serde_json::json!({"a": projected("a"), "b": projected("b")});
for tool in case["tools"].as_array().expect("overlay tools").iter().map(|t| t.as_str().expect("tool id")) {
let mask = match tool {
"clip" | "erase" => tool,
_ => "overlay",
};
let params = serde_json::json!({"input": "a", (mask): "b"});
reply[tool] = match rustgis_analysis::run(tool, &layers, &params) {
Ok(result) => {
let ds = layer_of(result.output).expect("overlay layer");
let keys = if mask == "overlay" { vec!["id".into(), "id_2".into()] } else { vec!["id".into()] };
let rows: Vec<_> = ds
.features
.iter()
.enumerate()
.map(|(i, f)| {
let key = key_text(&ds, i, Some(&keys));
serde_json::json!([key, f.geometry.as_ref().map(overlay_coordinates)])
})
.collect();
serde_json::json!(rows)
}
Err(e) => serde_json::json!(e),
};
}
replies.push(reply);
}
let path = std::env::var_os("RUSTGIS_OVERLAY_REPLY").expect("reply path from generate.py");
serde_json::to_writer(std::fs::File::create(path).expect("create overlay reply"), &replies).expect("write overlay reply");
}
39 changes: 39 additions & 0 deletions crates/analysis/tests/reference/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Generated polygon overlays

The fixed corpus in this directory checks known results. The generated overlay check also runs
Clip, Intersect, Union and Erase on seeded polygon pairs and layers with several features. It
varies concave outlines, holes, thin overlaps, shared edges, collinear overlaps, nearly touching
corners and multipart features.

The reference test sends the same Web Mercator input coordinates that RustGIS uses to Shapely's
GEOS. It compares each input ID's output (both IDs for Intersect and Union) by the area of the
symmetric difference. Ring order, orientation and redundant vertices do not affect that area.
Boundary-only GEOS results and empty polygons count as absent. The comparison checks output
validity and uses the fixed corpus's 0.1 ppm tolerance, scaled by the combined input area
because rounding depends on the inputs even when the answer is tiny or empty. It counts
polygon parts above that same area limit.

From the repository root, build the reference test and pass the executable printed by Cargo:

```sh
cargo test --locked -p rustgis-analysis --test reference --no-run
uv run crates/analysis/tests/reference/generate.py --overlay "target/debug/deps/reference-<hash>" --findings ./overlay-findings
```

CI runs six fixed seeds with 16 cases each. To replay a nightly run, set
`PROPERTY_SEED=<seed> PROPERTY_CASES=256` before the `uv run` command. To replay just a
minimized case, save the finding from the issue or run artifact as JSON and add
`--case path/to/finding.json`. Use a fresh `--findings` path each time.

The reducer keeps removing features, parts, holes and vertices until no single deletion
preserves the same kind of difference for the tool on valid inputs. The nightly run keeps these
findings as an artifact and uses the tool, difference kind and normalized reduced input to skip
issues it has already reported; it only trusts markers in issues the workflow opened itself. It
groups Clip splits that differ only by a gap a few overlay grid steps wide between masks, with each
reduced variant in the artifact. That group is reported again once its issue is closed. An
overlay run that fails without a finding opens its own issue, which closes when the check passes.

The default run also checks the inputs in `overlay-regressions/`. The Union fixture checks
that a polygon touching the zone boundary leaves the correct number of pieces. The Clip
fixture records a known split where individually rounded masks should meet. The default
run permits that specific difference, and `--case` reports it as a failure.
6 changes: 5 additions & 1 deletion crates/analysis/tests/reference/generate.py
Original file line number Diff line number Diff line change
Expand Up @@ -557,6 +557,10 @@ def check():


if __name__ == "__main__":
if sys.argv[1:2] == ["--overlay"]:
import overlay as generated_overlay

sys.exit(generated_overlay.run(sys.argv[2:]))
if sys.argv[1:] not in ([], ["--check"]):
sys.exit(f"usage: uv run generate.py [--check] (not {' '.join(sys.argv[1:])})")
sys.exit(f"usage: uv run generate.py [--check | --overlay RUNNER --findings DIR] (not {' '.join(sys.argv[1:])})")
sys.exit(check() if sys.argv[1:] == ["--check"] else main())
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"seed": "13714318419105429329",
"index": 614,
"known": "gap between rounded clip masks",
"case": {"a":[{"type":"Polygon","coordinates":[[[-25.01953125,42.0],[-25.0,41.9833984375],[-25.0,42.0],[-25.01953125,42.0]]]}],"b":[{"type":"Polygon","coordinates":[[[-25.029296875,41.9931640625],[-25.009765625,41.9931640625],[-25.009765625,42.009765625],[-25.029296875,41.9931640625]]]},{"type":"Polygon","coordinates":[[[-25.009765625,42.009765625],[-25.009765625,41.9736328125],[-24.990234375,42.009765625],[-25.009765625,42.009765625]]]}],"tools":["clip"]}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"seed": "42",
"index": 999,
"case": {"a":[{"type":"Polygon","coordinates":[[[-16.9609375,-9.0],[-16.951171875,-9.041015625],[-16.9609375,-9.041015625],[-16.9609375,-9.0]]]}],"b":[{"type":"Polygon","coordinates":[[[-16.9296875,-9.029296875],[-16.94140625,-9.046875],[-16.95703125,-9.041015625],[-16.9765625,-9.041015625],[-16.9296875,-9.029296875]]]}],"tools":["union"]}
}
Loading
Loading