Skip to content

Run and report withResource nesting lint with Jython [reduced-it] #15879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gerashegalov wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Run and report withResource nesting lint with Jython [reduced-it] #15879

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
cad0db9
Add advisory NVIDIA deprecation audit
gerashegalov Sep 1, 2026
4c28634
Fail optional audit when deprecations are found
gerashegalov Sep 1, 2026
910bd0d
Run deprecation checks with Jython
gerashegalov Sep 1, 2026
46edaea
Launch Jython from the plugin classpath
gerashegalov Sep 1, 2026
7781efe
Keep deprecation audit on root POM
gerashegalov Sep 1, 2026
e3b5787
Run withResource nesting lint with Jython
gerashegalov Sep 2, 2026
ffb3abc
Merge remote-tracking branch 'origin/main' into codex/with-resource-j…
gerashegalov Sep 2, 2026
f298d29
Publish withResource nesting audit report
gerashegalov Sep 2, 2026
761ca16
Address withResource audit review feedback
gerashegalov Sep 9, 2026
b1735b6
Fix withResource audit reporting
gerashegalov Sep 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/workflows/mvn-verify-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,3 +430,29 @@ jobs:
fi
}
done

with-resource-nesting-audit:
name: withResource nesting audit
runs-on: ubuntu-latest
steps:
- uses: NVIDIA/spark-rapids-common/checkout@main

- name: Report withResource nesting violations
run: |
audit_status=0
mvn --batch-mode -N antrun:run@with-resource-nesting-audit \

@wjxiz1992 wjxiz1992 Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Because this runs the script through Maven/Ant, every workflow command is prefixed with [INFO] [java] in the current audit log. GitHub therefore does not parse the emitted ::warning/::error lines: check run 102566070742 reports annotations_count=1, and that lone annotation is only the runner's Node.js deprecation warning, even though 59 scopes produced 50 source-warning commands in the log. Could we emit these commands directly to the runner, or strip the Maven prefix while preserving pipefail, so new violations receive the promised source annotations?

-DwithResource.audit.rawReport="$RUNNER_TEMP/with-resource-nesting-audit.json" \
-DwithResource.audit.summary="$GITHUB_STEP_SUMMARY" \
-DwithResource.audit.annotations="$RUNNER_TEMP/with-resource-nesting-annotations.txt" || audit_status=$?
if [[ -f "$RUNNER_TEMP/with-resource-nesting-annotations.txt" ]]; then
cat "$RUNNER_TEMP/with-resource-nesting-annotations.txt"
fi
exit "$audit_status"

- name: Upload withResource nesting report
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: with-resource-nesting-audit
path: ${{ runner.temp }}/with-resource-nesting-audit.json
if-no-files-found: ignore
53 changes: 35 additions & 18 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1153,6 +1153,9 @@
<scala.plugin.version>4.9.10</scala.plugin.version>
<maven.install.plugin.version>3.1.1</maven.install.plugin.version>
<maven.jar.plugin.version>3.3.0</maven.jar.plugin.version>
<withResource.audit.rawReport>${project.build.directory}/with-resource-nesting-audit.json</withResource.audit.rawReport>
<withResource.audit.summary>${project.build.directory}/with-resource-nesting-summary.md</withResource.audit.summary>
<withResource.audit.annotations>${project.build.directory}/with-resource-nesting-annotations.txt</withResource.audit.annotations>
<scalatest-maven-plugin.version>2.0.2</scalatest-maven-plugin.version>
<guava.cdh.version>30.0-jre</guava.cdh.version>
<arrow.cdh.version>2.0.0</arrow.cdh.version>
Expand Down Expand Up @@ -1993,6 +1996,38 @@ This will force full Scala code rebuild in downstream modules.
<!-- parent-pom only executions -->
<inherited>false</inherited>
<executions>
<!-- Run this repository-wide audit only from the canonical root reactor.
Encode option hyphens so the generated Scala 2.13 comment stays valid. -->
<!-- #if scala-2.12 -->
<execution>
<id>with-resource-nesting-audit</id>
<phase>verify</phase>
<goals><goal>run</goal></goals>
<configuration>
<target>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}">
<arg value="-B"/>
<arg value="scripts/tests/test_check_with_resource_nesting.py"/>
</java>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}"
output="${withResource.audit.annotations}">
<arg value="-B"/>
<arg value="scripts/check_with_resource_nesting.py"/>
<arg value="&#45;&#45;root"/>
<arg value="${project.basedir}"/>
<arg value="&#45;&#45;raw-report"/>
<arg value="${withResource.audit.rawReport}"/>
<arg value="&#45;&#45;summary"/>
<arg value="${withResource.audit.summary}"/>
</java>
</target>
</configuration>
</execution>
<!-- #endif scala-2.12 -->
<execution>
<!--
This is an alternative implementation of the scalastyle check invocation,
Expand Down Expand Up @@ -2030,24 +2065,6 @@ This will force full Scala code rebuild in downstream modules.
<arg line="--xmlEncoding ${project.reporting.outputEncoding}"/>
<arg line="${scalastyle.dirs}"/>
</java>
<exec executable="python3" failonerror="true"
dir="${spark.rapids.source.basedir}">
<arg value="-m"/>
<arg value="unittest"/>
<arg value="discover"/>
<arg value="-s"/>
<arg value="scripts/tests"/>
<arg value="-p"/>
<arg value="test_check_with_resource_nesting.py"/>
</exec>
<!-- Existing nesting exceptions are tracked by
https://github.com/NVIDIA/cudf-spark/issues/11713. -->
<exec executable="python3" failonerror="true"
dir="${spark.rapids.source.basedir}">
<arg value="scripts/check_with_resource_nesting.py"/>
<arg value="--root"/>
<arg value="${spark.rapids.source.basedir}"/>
</exec>
</target>
</configuration>
</execution>
Expand Down
53 changes: 35 additions & 18 deletions scala2.13/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1153,6 +1153,9 @@
<scala.plugin.version>4.9.10</scala.plugin.version>
<maven.install.plugin.version>3.1.1</maven.install.plugin.version>
<maven.jar.plugin.version>3.3.0</maven.jar.plugin.version>
<withResource.audit.rawReport>${project.build.directory}/with-resource-nesting-audit.json</withResource.audit.rawReport>
<withResource.audit.summary>${project.build.directory}/with-resource-nesting-summary.md</withResource.audit.summary>
<withResource.audit.annotations>${project.build.directory}/with-resource-nesting-annotations.txt</withResource.audit.annotations>
<scalatest-maven-plugin.version>2.0.2</scalatest-maven-plugin.version>
<guava.cdh.version>30.0-jre</guava.cdh.version>
<arrow.cdh.version>2.0.0</arrow.cdh.version>
Expand Down Expand Up @@ -1993,6 +1996,38 @@ This will force full Scala code rebuild in downstream modules.
<!-- parent-pom only executions -->
<inherited>false</inherited>
<executions>
<!-- Run this repository-wide audit only from the canonical root reactor.
Encode option hyphens so the generated Scala 2.13 comment stays valid. -->
<!-- #if scala-2.12 --><!--
<execution>
<id>with-resource-nesting-audit</id>
<phase>verify</phase>
<goals><goal>run</goal></goals>
<configuration>
<target>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}">
<arg value="-B"/>
<arg value="scripts/tests/test_check_with_resource_nesting.py"/>
</java>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}"
output="${withResource.audit.annotations}">
<arg value="-B"/>
<arg value="scripts/check_with_resource_nesting.py"/>
<arg value="&#45;&#45;root"/>
<arg value="${project.basedir}"/>
<arg value="&#45;&#45;raw-report"/>
<arg value="${withResource.audit.rawReport}"/>
<arg value="&#45;&#45;summary"/>
<arg value="${withResource.audit.summary}"/>
</java>
</target>
</configuration>
</execution>
--><!-- #endif scala-2.12 -->
<execution>
<!--
This is an alternative implementation of the scalastyle check invocation,
Expand Down Expand Up @@ -2030,24 +2065,6 @@ This will force full Scala code rebuild in downstream modules.
<arg line="--xmlEncoding ${project.reporting.outputEncoding}"/>
<arg line="${scalastyle.dirs}"/>
</java>
<exec executable="python3" failonerror="true"
dir="${spark.rapids.source.basedir}">
<arg value="-m"/>
<arg value="unittest"/>
<arg value="discover"/>
<arg value="-s"/>
<arg value="scripts/tests"/>
<arg value="-p"/>
<arg value="test_check_with_resource_nesting.py"/>
</exec>
<!-- Existing nesting exceptions are tracked by
https://github.com/NVIDIA/cudf-spark/issues/11713. -->
<exec executable="python3" failonerror="true"
dir="${spark.rapids.source.basedir}">
<arg value="scripts/check_with_resource_nesting.py"/>
<arg value="--root"/>
<arg value="${spark.rapids.source.basedir}"/>
</exec>
</target>
</configuration>
</execution>
Expand Down
9 changes: 8 additions & 1 deletion scripts/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,11 @@ Only hoist values that own their data. Views returned by methods such as `bitCas
`getChildColumnView`, `replaceListChild`, and `splitAsViews` must not outlive their owning parent.
Copy or convert a view to an owning resource before closing its parent.

The check and its unit tests run with the all-modules Scalastyle execution during `mvn verify`.
The script remains directly runnable with Python 3, but is also compatible with Jython 2.7. Maven
uses its managed Jython dependency to run the check and unit tests during root `mvn verify`; the
generated Scala 2.13 reactor does not repeat this repository-wide check.

The pull-request audit publishes every deep scope in the job summary and a JSON artifact. Existing
baseline entries are reported as debt but do not fail the check. New violations, stale baseline
entries, invalid exemptions, or report-generation errors fail it. GitHub source annotations show
the first 50 entries, with the complete set retained in the summary and artifact.
Loading
Loading