Repository navigation
test(analysis): compare seeded polygon overlays with GEOS #2386
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,7 +2,10 @@ name: Nightly | |
|
|
||
| # The slow checks pull requests don't wait for: every platform's tests and the release build of | ||
| # the apps, once a night on main. A night with no new commits since the last successful run is | ||
| # skipped. A failure opens an issue (or comments on the open one); the next success closes it. | ||
| # skipped for the build. Generated overlays run every night with a fresh seed: each new minimized | ||
| # disagreement opens its own issue, and an overlay run that fails without one is reported like a | ||
| # build failure. A build failure opens an issue (or comments on the open one); the next success | ||
| # closes it. | ||
| # | ||
| # gh workflow run nightly.yml -R opengeos/RustGIS --ref main | ||
|
|
||
|
|
@@ -127,6 +130,117 @@ jobs: | |
| cargo test --locked -p rustgis-data --test extent_query_properties | ||
| PROPERTY_CASES=20000 cargo test --locked -p rustgis-data --test date_properties | ||
|
|
||
| # Use a fresh seed even when the commit is the same as last night. | ||
| overlay: | ||
| name: Generated polygon overlays | ||
| if: github.repository == 'opengeos/RustGIS' && github.ref == 'refs/heads/main' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
| outputs: | ||
| findings: ${{ steps.findings.outputs.artifact-id }} | ||
| env: | ||
| CARGO_PROFILE_DEV_DEBUG: line-tables-only | ||
| steps: | ||
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec # stable | ||
| - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2 | ||
| - uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3 | ||
| - name: Compare overlays and minimize disagreements | ||
| id: compare | ||
| env: | ||
| PROPERTY_CASES: "256" | ||
| run: | | ||
| set -euo pipefail | ||
| cargo test --locked -p rustgis-analysis --test reference --no-run --message-format=json > "$RUNNER_TEMP/reference-build.jsonl" | ||
| runner="$(jq -er 'select(.reason == "compiler-artifact" and .target.name == "reference" and .executable != null) | .executable' "$RUNNER_TEMP/reference-build.jsonl")" | ||
| PROPERTY_SEED="$(python3 -c 'import secrets; print(secrets.randbits(64))')" | ||
| export PROPERTY_SEED | ||
| time uv run crates/analysis/tests/reference/generate.py --overlay "$runner" --findings "$RUNNER_TEMP/overlay-findings" | ||
| - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7 | ||
| id: findings | ||
| if: failure() && steps.compare.outcome == 'failure' | ||
| with: | ||
| name: overlay-findings | ||
| path: ${{ runner.temp }}/overlay-findings | ||
| if-no-files-found: ignore | ||
|
|
||
| overlay-report: | ||
| name: Report generated overlays | ||
| needs: overlay | ||
| # Only the reporting job has issue write access. Manual runs keep findings as artifacts. | ||
| # A job that ran out of time ends cancelled, but the run isn't: that is a failure too. | ||
| if: ${{ !cancelled() && github.event_name == 'schedule' && needs.overlay.result != 'skipped' }} | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| actions: read | ||
| issues: write | ||
| steps: | ||
| - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8 | ||
| if: needs.overlay.outputs.findings != '' | ||
| with: | ||
| name: overlay-findings | ||
| path: findings | ||
| - name: Open issues for new minimized disagreements | ||
| if: needs.overlay.outputs.findings != '' | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| RUN: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} | ||
| run: | | ||
| set -euo pipefail | ||
| # Trust only markers in this workflow's own issues, as "<state> <marker>" lines. | ||
| gh api --paginate "repos/$GITHUB_REPOSITORY/issues?state=all&per_page=100" \ | ||
| | jq -r '.[] | select(.pull_request | not) | select(.user.login == "github-actions[bot]") | ||
| | .state as $s | (.body // "") | split("\n")[] | select(startswith("<!-- overlay-finding:")) | "\($s) \(.)"' > reported | ||
| for file in findings/*.json; do | ||
| tool="$(jq -er '.case.tools[0]' "$file")" | ||
| kind="$(jq -er '.kind' "$file")" | ||
| fingerprint="$(jq -er '.fingerprint' "$file")" | ||
| marker="<!-- overlay-finding:$fingerprint -->" | ||
| # A reduced input is reported once. The grouped Clip seam class is reported again | ||
| # once its issue is closed, so a fix that doesn't hold shows up. | ||
| if grep -Fxq -- "open $marker" reported; then continue; fi | ||
| if [ "$fingerprint" != clip-mask-seam ] && grep -Fxq -- "closed $marker" reported; then continue; fi | ||
| { | ||
| echo "The [nightly run]($RUN) on $GITHUB_SHA found a generated $tool disagreement." | ||
| echo | ||
| jq -r '"Seed `PROPERTY_SEED=\(.seed)`, case index `\(.index)`. \(.detail)"' "$file" | ||
| echo | ||
| echo 'Save the minimized case below as JSON and pass it to the overlay replay command in crates/analysis/tests/reference/README.md.' | ||
| echo | ||
| echo '```json' | ||
| cat "$file" | ||
| echo '```' | ||
| echo "$marker" | ||
| } > issue.md | ||
| gh issue create -R "$GITHUB_REPOSITORY" --title "Generated overlay disagrees with GEOS ($tool, $kind)" --body-file issue.md | ||
| echo "open $marker" >> reported | ||
| done | ||
| - name: Report a check that failed without findings | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| RESULT: ${{ needs.overlay.result }} | ||
| FINDINGS: ${{ needs.overlay.outputs.findings }} | ||
| RUN: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} | ||
| run: | | ||
| set -euo pipefail | ||
| title="Nightly generated overlay check failed" | ||
| open="$(gh api --paginate "repos/$GITHUB_REPOSITORY/issues?state=open&per_page=100" \ | ||
| | jq -sr --arg t "$title" '[.[][] | select(.pull_request | not) | select(.user.login == "github-actions[bot]") | select(.title == $t)][0].number // ""')" | ||
| if [ "$RESULT" = success ]; then | ||
| if [ -n "$open" ]; then | ||
| gh issue close "$open" -R "$GITHUB_REPOSITORY" --comment "Fixed: the nightly overlay check on $GITHUB_SHA passed ($RUN)." | ||
| fi | ||
| elif [ -z "$FINDINGS" ]; then | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A harness abort after some findings opens no issue (confidence: medium). This branch runs only when
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Leaving this as is. That case is still reported: the findings it wrote each open their own issue with the seed, and the run is red. What's missed is only a separate note that the later seeds didn't run, and the next night's fresh seed covers them. A completion marker would add a second protocol between |
||
| if [ -n "$open" ]; then | ||
| gh issue comment "$open" -R "$GITHUB_REPOSITORY" --body "Still failing on $GITHUB_SHA: $RUN" | ||
| else | ||
| gh issue create -R "$GITHUB_REPOSITORY" --title "$title" \ | ||
| --body "The generated overlay check on $GITHUB_SHA failed without a minimized finding, so the comparison itself broke: $RUN. This issue closes itself when the check passes." | ||
| fi | ||
| fi | ||
|
|
||
| report: | ||
| name: Report | ||
| needs: [changed, rust, workflows] | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Harness failures open no issue (confidence: medium).
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in e8347f7: |
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2479,9 +2479,12 @@ fn overlay_on(inp: &dyn Inputs, p: &Json, op: Overlay, threads: Option<usize>) - | |
| let rest = if inputs.is_empty() { | ||
| from_multipolygon(&z.poly) | ||
| } else { | ||
| let polys: Vec<MultiPolygon<f64>> = | ||
| inputs.iter().filter_map(|r| ds.features.get(*r)?.geometry.as_ref()).filter_map(to_multipolygon).collect(); | ||
| from_multipolygon(&z.poly.difference(&union_all(polys.iter().flat_map(|m| m.0.iter())))) | ||
| let polygons = | ||
| inputs.iter().filter_map(|r| ds.features.get(*r)?.geometry.as_ref()).filter_map(to_multipolygon).flat_map(|m| m.0); | ||
| // A redundant union rounds a lone polygon's vertices. That can open a | ||
| // passage where it touches the zone boundary, joining separate pieces. | ||
| let cover = mask_of(MultiPolygon::new(polygons.collect())); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Single covering polygons are no longer repaired (confidence: low-medium, a question). Before this change, one covering polygon went through
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Intended. The On coverage: the reduced Union case runs in the PR's Ubuntu test job, the same job that runs the rest of the reference comparison, so the fix fails CI if it regresses. |
||
| from_multipolygon(&z.poly.difference(&cover)) | ||
| }; | ||
| Ok::<_, String>(rest) | ||
| }, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| # Generated polygon overlays | ||
|
|
||
| The fixed corpus in this directory checks known results. The generated overlay check also runs | ||
| Clip, Intersect, Union and Erase on seeded polygon pairs and layers with several features. It | ||
| varies concave outlines, holes, thin overlaps, shared edges, collinear overlaps, nearly touching | ||
| corners and multipart features. | ||
|
|
||
| The reference test sends the same Web Mercator input coordinates that RustGIS uses to Shapely's | ||
| GEOS. It compares each input ID's output (both IDs for Intersect and Union) by the area of the | ||
| symmetric difference. Ring order, orientation and redundant vertices do not affect that area. | ||
| Boundary-only GEOS results and empty polygons count as absent. The comparison checks output | ||
| validity and uses the fixed corpus's 0.1 ppm tolerance, scaled by the combined input area | ||
| because rounding depends on the inputs even when the answer is tiny or empty. It counts | ||
| polygon parts above that same area limit. | ||
|
|
||
| From the repository root, build the reference test and pass the executable printed by Cargo: | ||
|
|
||
| ```sh | ||
| cargo test --locked -p rustgis-analysis --test reference --no-run | ||
| uv run crates/analysis/tests/reference/generate.py --overlay "target/debug/deps/reference-<hash>" --findings ./overlay-findings | ||
| ``` | ||
|
|
||
| CI runs six fixed seeds with 16 cases each. To replay a nightly run, set | ||
| `PROPERTY_SEED=<seed> PROPERTY_CASES=256` before the `uv run` command. To replay just a | ||
| minimized case, save the finding from the issue or run artifact as JSON and add | ||
| `--case path/to/finding.json`. Use a fresh `--findings` path each time. | ||
|
|
||
| The reducer keeps removing features, parts, holes and vertices until no single deletion | ||
| preserves the same kind of difference for the tool on valid inputs. The nightly run keeps these | ||
| findings as an artifact and uses the tool, difference kind and normalized reduced input to skip | ||
| issues it has already reported; it only trusts markers in issues the workflow opened itself. It | ||
| groups Clip splits that differ only by a gap a few overlay grid steps wide between masks, with each | ||
| reduced variant in the artifact. That group is reported again once its issue is closed. An | ||
| overlay run that fails without a finding opens its own issue, which closes when the check passes. | ||
|
|
||
| The default run also checks the inputs in `overlay-regressions/`. The Union fixture checks | ||
| that a polygon touching the zone boundary leaves the correct number of pieces. The Clip | ||
| fixture records a known split where individually rounded masks should meet. The default | ||
| run permits that specific difference, and `--case` reports it as a failure. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| { | ||
| "seed": "13714318419105429329", | ||
| "index": 614, | ||
| "known": "gap between rounded clip masks", | ||
| "case": {"a":[{"type":"Polygon","coordinates":[[[-25.01953125,42.0],[-25.0,41.9833984375],[-25.0,42.0],[-25.01953125,42.0]]]}],"b":[{"type":"Polygon","coordinates":[[[-25.029296875,41.9931640625],[-25.009765625,41.9931640625],[-25.009765625,42.009765625],[-25.029296875,41.9931640625]]]},{"type":"Polygon","coordinates":[[[-25.009765625,42.009765625],[-25.009765625,41.9736328125],[-24.990234375,42.009765625],[-25.009765625,42.009765625]]]}],"tools":["clip"]} | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| { | ||
| "seed": "42", | ||
| "index": 999, | ||
| "case": {"a":[{"type":"Polygon","coordinates":[[[-16.9609375,-9.0],[-16.951171875,-9.041015625],[-16.9609375,-9.041015625],[-16.9609375,-9.0]]]}],"b":[{"type":"Polygon","coordinates":[[[-16.9296875,-9.029296875],[-16.94140625,-9.046875],[-16.95703125,-9.041015625],[-16.9765625,-9.041015625],[-16.9296875,-9.029296875]]]}],"tools":["union"]} | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Findings are lost when the compare step is cancelled (confidence: medium).
failure()is false when the job is cancelled, and the job's 20-minutetimeout-minutescancels it. The step outcome is thencancelled, notfailure, so this upload is skipped.overlay-reportsees no artifact, takes the "failed without findings" branch, and opens an issue saying "the comparison itself broke", even though minimized reproducers were already written. The same condition attest-rust.yml:144has the same gap if the job is cancelled. Usealways()so the upload runs whenever the compare step didn't succeed:if-no-files-found: ignorekeeps the step harmless when nothing was written.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Leaving this as is. When the job times out, the reproducers aren't attached, but the failure isn't lost:
overlay-reportstill runs on the cancelled result and opens or updates the failure issue with the run link, and the log prints the seed, soPROPERTY_SEED=<seed>replays it. The PR'stest-rust.ymlstep has no timeout of its own, and a cancelled PR run is rerun anyway. If timeouts turn out to happen in practice, uploading onalways()is a small follow-up.