Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions examples/UDF-Examples/RAPIDS-accelerated-UDFs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,9 @@ mvn clean package -Pudf-native-examples
The build will automatically:
- Extract `libcudf.so` from older rapids-4-spark jars or reconstruct it from
the chunk-manifest representation used by newer jars
- Read the embedded `spark-rapids-jni` and `cudf-java` version metadata from the jar
- Download the matching `spark-rapids-jni` `cudf-pins` files
- Read the embedded `cudf-spark-jni` and `cudf-java` version metadata from
the jar
- Download the matching `cudf-spark-jni` `cudf-pins` files
- Clone the cuDF repository at the revision recorded in the jar
- Configure rapids-cmake with the matching `rapids-cmake` sha and package override file
- Build only your UDF native code against the prebuilt library
Expand All @@ -165,7 +166,7 @@ library.

When changing the rapids-4-spark jar version, rebuild the native UDFs with
matching cuDF/RMM/CCCL headers and libraries. In prebuilt mode, the Maven build
derives those native dependency pins from the `spark-rapids-jni` revision
derives those native dependency pins from the `cudf-spark-jni` revision
recorded in the jar. After changing jar versions, remove `target/native-deps`,
`target/cudf-repo`, `target/cudf-pins`, and `target/cpp-build` before
rebuilding so stale headers, pins, or libraries are not reused.
Expand Down
2 changes: 1 addition & 1 deletion examples/UDF-Examples/RAPIDS-accelerated-UDFs/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -402,7 +402,7 @@

<echo message="✓ libcudf.so extracted to: ${project.build.directory}/native-deps"/>

<!-- Resolve matching cuDF/RMM/CCCL pins from the spark-rapids-jni
<!-- Resolve matching cuDF/RMM/CCCL pins from the cudf-spark-jni
revision embedded in the rapids-4-spark jar. -->
<chmod file="${basedir}/resolve-jni-cudf-pins.sh" perm="755"/>
<exec executable="${basedir}/resolve-jni-cudf-pins.sh" failonerror="true">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,15 +87,15 @@ download_file() {
fi
}

JNI_INFO="spark-rapids-jni-version-info.properties"
JNI_INFO="cudf-spark-jni-version-info.properties"
Comment thread
NvTimLiu marked this conversation as resolved.
CUDF_INFO="cudf-java-version-info.properties"

JNI_REVISION="$(read_property_from_jar "$JNI_INFO" revision || true)"
JNI_URL="$(read_property_from_jar "$JNI_INFO" url || true)"
CUDF_REVISION="$(read_property_from_jar "$CUDF_INFO" revision || true)"

if [ -z "$JNI_REVISION" ] || [ -z "$JNI_URL" ]; then
echo "ERROR: Failed to read spark-rapids-jni revision/url from $JAR_PATH" >&2
echo "ERROR: Failed to read cudf-spark-jni revision/url from $JAR_PATH" >&2
echo "Expected $JNI_INFO in the jar." >&2
exit 1
fi
Expand All @@ -108,7 +108,7 @@ fi

RAW_BASE="$(normalize_github_raw_base "$JNI_URL")"
if [ -z "$RAW_BASE" ]; then
echo "ERROR: Unsupported spark-rapids-jni URL for automatic pin lookup: $JNI_URL" >&2
echo "ERROR: Unsupported cudf-spark-jni URL for automatic pin lookup: $JNI_URL" >&2
echo "Only github.com URLs can be resolved automatically." >&2
exit 1
fi
Expand Down Expand Up @@ -161,14 +161,14 @@ download_file "https://raw.githubusercontent.com/rapidsai/rapids-cmake/$RAPIDS_C

cat > "$PROPERTIES_FILE" <<EOF
jar.cudf.revision=$CUDF_REVISION
jar.spark.rapids.jni.revision=$JNI_REVISION
jar.cudf.spark.jni.revision=$JNI_REVISION
jar.rapids.cmake.sha=$RAPIDS_CMAKE_SHA
jar.rapids.cmake.file=$RAPIDS_CMAKE_FILE
jar.cudf.pins.file=$VERSIONS_FILE
EOF

echo "Resolved native dependency pins from rapids-4-spark jar:"
echo " spark-rapids-jni revision: $JNI_REVISION"
echo " cudf-spark-jni revision: $JNI_REVISION"
echo " cuDF revision: $CUDF_REVISION"
echo " rapids-cmake sha: $RAPIDS_CMAKE_SHA"
echo " rapids-cmake entrypoint: $RAPIDS_CMAKE_FILE"
Expand Down