Backport oneDNN commit to enable stack unwind in version 2.7.3 built by Tensorflow #9721

gartung · 2025-02-25T15:39:47Z

Description

Tensorflow builds OneDNN v2.7.3.
Later versions on OneDNN have a commit that is described as "stack magic".
uxlfoundation/oneDNN@2ae2450

cmsbuild · 2025-02-25T15:40:06Z

A new Pull Request was created by @gartung for branch IB/CMSSW_15_1_X/master.

@cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2025-02-25T15:40:07Z

cms-bot internal usage

tensorflow-sources.file

cmsbuild · 2025-02-25T21:24:56Z

Pull request #9721 was updated.

cmsbuild · 2025-02-25T21:28:45Z

Pull request #9721 was updated.

gartung · 2025-02-25T21:31:34Z

@smuzaffar is the build of the first commit still be published on cvmfs?

gartung · 2025-02-25T21:36:24Z

@smuzaffar is the build of the first commit still be published on cvmfs?

I found it
/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9721/44654

cmsbuild · 2025-02-26T05:56:28Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f04dad/44666/summary.html
COMMIT: 7aa07e2
CMSSW: CMSSW_15_1_X_2025-02-25-1200/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9721/44666/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f04dad/44666/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f04dad/44666/git-merge-result

Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920280
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-02-26T16:57:49Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f04dad/44693/summary.html
COMMIT: 7aa07e2
CMSSW: CMSSW_15_1_X_2025-02-26-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9721/44693/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

TF_SYSTEM_LIBS=absl_py,astor_archive,boringssl,com_github_grpc_grpc,com_google_protobuf,curl,cython,eigen_archive,flatbuffers,functools32_archive,gast_archive,gif,libjpeg_turbo,opt_einsum_archive,org_python_pypi_backports_weakref,org_sqlite,pasta,png,pybind11,six_archive,termcolor_archive,typing_extensions_archive,wrapt,zlib \
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/external/mkl_dnn_v1/_objs/mkl_dnn/primitive.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/mkl_dnn_v1/_objs/mkl_dnn/primitive.pic.o' -fPIC -iquote external/mkl_dnn_v1 -iquote bazel-out/k8-opt/bin/external/mkl_dnn_v1 -isystem external/mkl_dnn_v1/include -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/include -isystem external/mkl_dnn_v1/src -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src -isystem external/mkl_dnn_v1/src/common -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/common -isystem external/mkl_dnn_v1/src/common/ittnotify -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/common/ittnotify -isystem external/mkl_dnn_v1/src/cpu -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/cpu -isystem external/mkl_dnn_v1/src/cpu/gemm -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/cpu/gemm -isystem external/mkl_dnn_v1/src/cpu/x64/xbyak -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/cpu/x64/xbyak -Wno-all -Wno-extra -Wno-deprecated -Wno-deprecated-declarations -Wno-ignored-attributes -Wno-array-bounds -Wunused-result '-Werror=unused-result' -Wswitch '-Werror=switch' '-Wno-error=unused-but-set-variable' -DAUTOLOAD_DYNAMIC_KERNELS '-march=x86-64-v3' -Wno-sign-compare '-std=c++17' '-std=c++17' -fexceptions -UUSE_MKL -UUSE_CBLAS -DDNNL_ENABLE_MAX_CPU_ISA -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/mkl_dnn_v1/src/common/primitive.cpp -o bazel-out/k8-opt/bin/external/mkl_dnn_v1/_objs/mkl_dnn/primitive.pic.o)
# Configuration: a73d95db8dde0652880ed8f17425659a0036f75659d6822aea6ed5802a31e85e
# Execution platform: @local_execution_config_platform//:platform
In file included from external/mkl_dnn_v1/src/common/primitive.cpp:33:
external/mkl_dnn_v1/src/common/stack_checker.hpp:24:2: error: #error "Stack checker requires using concurrent scratchpad"
24 | #error "Stack checker requires using concurrent scratchpad"
|  ^~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 592.599s, Critical Path: 187.81s
INFO: 5848 processes: 1853 internal, 3995 local.

cmsbuild · 2025-03-04T19:23:27Z

Pull request #9721 was updated.

gartung · 2025-03-04T19:25:06Z

please test

cmsbuild · 2025-03-05T00:48:29Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f04dad/44807/summary.html
COMMIT: 099f812
CMSSW: CMSSW_15_1_X_2025-03-04-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9721/44807/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Update tag for RecoBTag-Combined to V01-27-00 #9727

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f04dad/44807/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f04dad/44807/git-merge-result

Comparison Summary

Summary:

You potentially added 15 lines to the logs
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920287
DQMHistoTests: Total failures: 5
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920262
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.010000000000000002 KiB( 48 files compared)
DQMHistoSizes: changed ( 145.014,... ): -0.001 KiB HLT/Filters
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-03-05T14:23:12Z

Pull request #9721 was updated.

gartung · 2025-03-05T14:23:33Z

please test

cmsbuild · 2025-03-05T19:48:21Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f04dad/44820/summary.html
COMMIT: d27fec0
CMSSW: CMSSW_15_1_X_2025-03-05-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9721/44820/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 21 lines to the logs
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 63
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920217
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-03-07T20:34:56Z

Pull request #9721 was updated.

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels Feb 25, 2025

cmsbuild added tests-started requires-external and removed tests-pending labels Feb 25, 2025

gartung linked an issue Feb 25, 2025 that may be closed by this pull request

Enable OneDNN stack checker in Tensorfow cms-sw/framework-team#1268

Closed

smuzaffar reviewed Feb 25, 2025

View reviewed changes

tensorflow-sources.file Outdated Show resolved Hide resolved

smuzaffar reviewed Feb 25, 2025

View reviewed changes

tensorflow-sources.file Outdated Show resolved Hide resolved

cmsbuild added tests-pending and removed tests-started requires-external labels Feb 25, 2025

cmsbuild added tests-started requires-external and removed tests-pending labels Feb 25, 2025

cmsbuild added tests-approved tests-started and removed tests-started tests-approved labels Feb 26, 2025

cmsbuild added tests-rejected and removed tests-started labels Feb 26, 2025

cmsbuild added the tests-started label Mar 4, 2025

cmsbuild added tests-pending and removed tests-started labels Mar 4, 2025

cmsbuild added tests-started and removed tests-pending labels Mar 4, 2025

gartung changed the title ~~Enable OneDNN memory debug in Tensorflow~~ Backport oneDNN commit to enable stack unwind in version 2.7.3 built by Tensorflow Mar 4, 2025

cmsbuild added tests-approved and removed tests-started labels Mar 5, 2025

gartung linked an issue Mar 5, 2025 that may be closed by this pull request

Backport oneDNN commit to enable stack unwind in version 2.7.3 built by Tensorflow cms-sw/framework-team#1280

Closed

gartung force-pushed the IB/CMSSW_15_1_X/master branch from 099f812 to d27fec0 Compare March 5, 2025 14:22

cmsbuild added tests-pending and removed tests-approved labels Mar 5, 2025

cmsbuild added tests-started and removed tests-pending labels Mar 5, 2025

cmsbuild added tests-approved and removed tests-started labels Mar 5, 2025

gartung closed this Mar 7, 2025

gartung reopened this Mar 7, 2025

Build v2.12.0 with patch for stack unwind

8dd7015

gartung force-pushed the IB/CMSSW_15_1_X/master branch from d27fec0 to 8dd7015 Compare March 7, 2025 20:34

cmsbuild added tests-pending and removed tests-approved labels Mar 7, 2025

gartung closed this Mar 7, 2025

Backport oneDNN commit to enable stack unwind in version 2.7.3 built by Tensorflow #9721

Backport oneDNN commit to enable stack unwind in version 2.7.3 built by Tensorflow #9721

Uh oh!

Conversation

gartung commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

cmsbuild commented Feb 25, 2025

Uh oh!

cmsbuild commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmsbuild commented Feb 25, 2025

Uh oh!

cmsbuild commented Feb 25, 2025

Uh oh!

gartung commented Feb 25, 2025

Uh oh!

gartung commented Feb 25, 2025

Uh oh!

cmsbuild commented Feb 26, 2025

Comparison Summary

Uh oh!

cmsbuild commented Feb 26, 2025

External Build

Uh oh!

cmsbuild commented Mar 4, 2025

Uh oh!

gartung commented Mar 4, 2025

Uh oh!

cmsbuild commented Mar 5, 2025

Comparison Summary

Uh oh!

cmsbuild commented Mar 5, 2025

Uh oh!

gartung commented Mar 5, 2025

Uh oh!

cmsbuild commented Mar 5, 2025

Comparison Summary

Uh oh!

cmsbuild commented Mar 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gartung commented Feb 25, 2025 •

edited

Loading

cmsbuild commented Feb 25, 2025 •

edited

Loading