[do not merge]: Testing force re-build of all packages by smuzaffar · Pull Request #210 · cms-sw/pkgtools

smuzaffar · 2024-06-13T07:20:45Z

Do not merge

cmsbuild · 2024-06-13T07:21:05Z

A new Pull Request was created by @smuzaffar for branch V00-34-XX.

@cmsbuild, @smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2024-06-13T07:21:05Z

cms-bot internal usage

smuzaffar · 2024-06-13T07:21:26Z

please test

cmsbuild · 2024-06-15T03:38:09Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/39889/summary.html
COMMIT: aaddec3
CMSSW: CMSSW_14_1_X_2024-06-13-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/39889/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 16 lines from the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3345018
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3344995
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

smuzaffar · 2024-07-16T07:43:31Z

please test

cmsbuild · 2024-07-17T03:23:11Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/40418/summary.html
COMMIT: aaddec3
CMSSW: CMSSW_14_1_X_2024-07-15-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/pkgtools/210/40418/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test test-particleLevel_fromMiniAod had ERRORS

Comparison Summary

Summary:

You potentially removed 18 lines from the logs
Reco comparison results: 8772 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3345094
DQMHistoTests: Total failures: 14313
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3330761
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: found differences in 2 / 46 workflows

smuzaffar · 2025-01-29T10:16:06Z

test parameters:

set_env = NO_PACKAGE_STORE

smuzaffar · 2025-01-29T10:19:12Z

please test

cmsbuild · 2025-01-29T22:20:57Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/44019/summary.html
COMMIT: aaddec3
CMSSW: CMSSW_15_0_X_2025-01-28-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/pkgtools/210/44019/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/44019/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/44019/git-merge-result

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 3 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4016770
DQMHistoTests: Total failures: 44
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4016706
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 218 log files, 189 edm output root files, 50 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-11-24T23:08:07Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49645/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_ROOT6_X_2025-11-23-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49645/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@akritkbehera [ROOT6] Updated root to tip of branch master cmsdist#10219

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49645/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49645/git-merge-result

Failed Unit Tests

I found 3 errors in the following unit tests:

---> test testTorchSimpleDnn had ERRORS
---> test test_DavixReadV_http had ERRORS
---> test test_DavixRead_http had ERRORS

Comparison Summary

Summary:

You potentially added 209 lines to the logs
Reco comparison results: 5 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3905633
DQMHistoTests: Total failures: 12761
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3892852
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-11-25T04:43:04Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49644/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_X_2025-11-24-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49644/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 2 errors in the following unit tests:

---> test test_DavixReadV_http had ERRORS
---> test test_DavixRead_http had ERRORS

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 10 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3905633
DQMHistoTests: Total failures: 21
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3905592
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

smuzaffar · 2025-11-25T09:07:49Z

please test with cms-sw/cms-bot#2623 for CMSSW_16_0_ROOT6_X/el8_aarch64_gcc13

cmsbuild · 2025-11-25T10:14:26Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49659/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_DEVEL_X_2025-11-23-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49659/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 2 errors in the following unit tests:

---> test test_DavixReadV_http had ERRORS
---> test test_DavixRead_http had ERRORS

Comparison Summary

Summary:

You potentially added 183 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 4 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3905633
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3905606
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-11-26T05:31:04Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49661/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_ROOT6_X_2025-11-24-2300/el8_aarch64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49661/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 4 errors in the following unit tests:

---> test TestDQMOnlineClient-ecalgpu_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-hcalgpu_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-pfgpu_dqm_sourceclient had ERRORS
and more ...

smuzaffar · 2025-12-12T10:16:50Z

please test with cms-sw/cms-bot#2623 for CMSSW_16_0_ROOT6_X/el8_aarch64_gcc13

cmsbuild · 2025-12-13T00:22:33Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49923/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_ROOT6_X_2025-12-11-2300/el8_aarch64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49923/install.sh to create a dev area with all the needed externals and cmssw changes.

smuzaffar · 2026-01-20T11:38:16Z

please test with #233 for CMSSW_16_1_ROOT6_X/el8_aarch64_gcc13

cmsbuild · 2026-01-23T11:11:33Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/50826/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_1_ROOT6_X_2026-01-22-2300/el8_aarch64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/50826/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

Requested to quit.
* The action "build-build-external+py3-torch+2.6.0-37e84154f8b03d95a4d532166b4bd23c" was not completed successfully because Failed to build py3-torch. Log file in /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/py3-torch/2.6.0-37e84154f8b03d95a4d532166b4bd23c/log. Final lines of the log file:
full command: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/python3/3.9.14-5875dbb11c59ccb3958b0204933ce45a/bin/python3 /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/external/py3-pip/25.1.1-9a76ae404882ba6b1e7abaf100b3c6af/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/py3-torch/2.6.0-37e84154f8b03d95a4d532166b4bd23c/cmsdist-tmp/tmpn77ldvay
cwd: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_aarch64_gcc13/external/py3-torch/2.6.0-37e84154f8b03d95a4d532166b4bd23c/cmsdist-tmp/pip-req-build-sustwpsn
Preparing metadata (pyproject.toml): finished with status 'error'
error: metadata-generation-failed

 Encountered error while generating package metadata.
> See above for output.

note: This is an issue with the package mentioned above, not pip.

cmsbuild · 2026-01-24T13:18:56Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/50868/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_1_ROOT6_X_2026-01-22-2300/el8_aarch64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/50868/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

16834.0

----- Begin Fatal Exception 24-Jan-2026 13:40:38 CET-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin PPSSimTrackProducer
   Additional Info:
      [a] Fatal Root Error: @SUB=TCling::LoadPCM
ROOT PCM /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/lcg/root/6.39.1-3c8ac7fcf98f2401b6ca95fc6e2dd551/lib/libROOTHist_rdict.pcm file does not exist

----- End Fatal Exception -------------------------------------------------

14234.0

----- Begin Fatal Exception 24-Jan-2026 13:40:32 CET-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin PPSSimTrackProducer
   Additional Info:
      [a] Fatal Root Error: @SUB=TCling::LoadPCM
ROOT PCM /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/lcg/root/6.39.1-3c8ac7fcf98f2401b6ca95fc6e2dd551/lib/libROOTHist_rdict.pcm file does not exist

----- End Fatal Exception -------------------------------------------------

11634.0

----- Begin Fatal Exception 24-Jan-2026 13:39:43 CET-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Constructing the EventProcessor
   [1] While attempting to load plugin PPSSimTrackProducer
   Additional Info:
      [a] Fatal Root Error: @SUB=TCling::LoadPCM
ROOT PCM /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_aarch64_gcc13/lcg/root/6.39.1-3c8ac7fcf98f2401b6ca95fc6e2dd551/lib/libROOTHist_rdict.pcm file does not exist

----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

smuzaffar · 2026-01-26T12:23:05Z

please test with #233 for CMSSW_16_1_X

smuzaffar · 2026-01-26T22:26:50Z

please test with #233 for CMSSW_16_1_X

smuzaffar · 2026-01-27T10:31:21Z

please test with #233 for CMSSW_16_1_X

cmsbuild · 2026-01-27T10:35:08Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/50923/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_1_X_2026-01-26-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/pkgtools/210/50923/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation warning when building: See details on the summary page.

smuzaffar · 2026-01-27T10:59:19Z

please test with #233 for CMSSW_16_1_X

cmsbuild · 2026-01-27T10:59:27Z

Pull request #210 was updated.

cmsbuild · 2026-01-27T23:44:03Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/50926/summary.html
COMMIT: 3ec43a7
CMSSW: CMSSW_16_1_X_2026-01-26-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/pkgtools/210/50926/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@quinnanm Update AXOL1TL to v6.0.0 cmsdist#10298

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/50926/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/50926/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 46
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025470
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 193 edm output root files, 52 DQM output files
TriggerResults: no differences found

[do not merge]: Testing force re-build of all packages

aaddec3

cmsbuild added orp-pending pending-signatures externals-pending tests-pending labels Jun 13, 2024

cmsbuild added tests-started requires-external and removed tests-pending labels Jun 13, 2024

cmsbuild added tests-approved and removed tests-started labels Jun 15, 2024

cmsbuild added tests-started and removed tests-approved labels Jul 16, 2024

cmsbuild added tests-rejected and removed tests-started labels Jul 17, 2024

smuzaffar closed this Sep 22, 2024

smuzaffar deleted the test-rebuild branch September 22, 2024 16:17

smuzaffar restored the test-rebuild branch January 29, 2025 09:43

smuzaffar reopened this Jan 29, 2025

cmsbuild removed the requires-external label Jan 29, 2025

cmsbuild added tests-started and removed tests-rejected labels Jan 29, 2025

smuzaffar mentioned this pull request Nov 24, 2025

protection against missing build type in stats #229

Merged

cmsbuild added tests-rejected and removed tests-started labels Nov 25, 2025

smuzaffar mentioned this pull request Dec 12, 2025

improve cpu utilization monitoring #231

Merged

cmsbuild added tests-started and removed tests-rejected labels Jan 26, 2026

cmsbuild added tests-rejected and removed tests-started labels Jan 27, 2026

Update comment from 'test2' to 'test3'

3ec43a7

cmsbuild added tests-started and removed tests-rejected labels Jan 27, 2026

cmsbuild added tests-approved and removed tests-started labels Jan 27, 2026

Conversation

smuzaffar commented Jun 13, 2024

Uh oh!

cmsbuild commented Jun 13, 2024

Uh oh!

cmsbuild commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Jun 13, 2024

Uh oh!

cmsbuild commented Jun 15, 2024

Comparison Summary

Uh oh!

smuzaffar commented Jul 16, 2024

Uh oh!

cmsbuild commented Jul 17, 2024

Unit Tests

Comparison Summary

Uh oh!

smuzaffar commented Jan 29, 2025

Uh oh!

smuzaffar commented Jan 29, 2025

Uh oh!

cmsbuild commented Jan 29, 2025

Comparison Summary

Uh oh!

cmsbuild commented Nov 24, 2025

Failed Unit Tests

Comparison Summary

Uh oh!

cmsbuild commented Nov 25, 2025

Failed Unit Tests

Comparison Summary

Uh oh!

smuzaffar commented Nov 25, 2025

Uh oh!

cmsbuild commented Nov 25, 2025

Failed Unit Tests

Comparison Summary

Uh oh!

cmsbuild commented Nov 26, 2025

Failed Unit Tests

Uh oh!

smuzaffar commented Dec 12, 2025

Uh oh!

cmsbuild commented Dec 13, 2025

Uh oh!

smuzaffar commented Jan 20, 2026

Uh oh!

cmsbuild commented Jan 23, 2026

Failed External Build

Uh oh!

cmsbuild commented Jan 24, 2026

Failed RelVals

Uh oh!

smuzaffar commented Jan 26, 2026

Uh oh!

smuzaffar commented Jan 26, 2026

Uh oh!

smuzaffar commented Jan 27, 2026

Uh oh!

cmsbuild commented Jan 27, 2026

Failed External Build

Uh oh!

smuzaffar commented Jan 27, 2026

Uh oh!

cmsbuild commented Jan 27, 2026

Uh oh!

cmsbuild commented Jan 27, 2026

Comparison Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cmsbuild commented Jun 13, 2024 •

edited

Loading