Skip to content

Add step to update spack-packages repo during snapshot - #1117

Open
kwryankrattiger wants to merge 11 commits into
spack:mainfrom
kwryankrattiger:split_repo_snapshots
Open

Add step to update spack-packages repo during snapshot#1117
kwryankrattiger wants to merge 11 commits into
spack:mainfrom
kwryankrattiger:split_repo_snapshots

Conversation

@kwryankrattiger

Copy link
Copy Markdown
Collaborator

Add spack-package update to snapshot cron.

@kwryankrattiger

Copy link
Copy Markdown
Collaborator Author

I am not sure if this is the correct place to do this, but we can relocate the logic as needed.

A few questions I had while looking at this.

  • Will spack core still get weekly tags
  • If there are weekly tags, will there be a gitlab status to check or will we move the repo testing into github with a very very small stack to build on PRs
  • If no weekly tags, should this cron just grab the latest head of develop for the update?
  • Should we run the spack/spack-packages:develop snapshot on acceptance of the updated spack ref, or should it run on the spack ref. of the previous week that the packages had been developed on.

old_date = py_gh_repo.get_commit(old_sha).commit.committer.date
new_date = py_gh_repo.get_commit(tag.sha).commit.committer.date
# The if new sha is older than the old sha, don't update
if old_date > new_date:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is safe since all commits going into develop are going to be merge commits.


spackages_repo = py_github.get_repo("spack/spack-packages", lazy=True)
# Create the PR
pr = spackages_repo.create_pull(

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this get an urgent tag?

)
last_commit = pr.get_commits()[pr.commits - 1]
# Trigger a rebuild everything on the PR to be reviewed by maintainers
pr.create_comment("@spackbot rebuild everything", last_commit, "", 0)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should work with the urgent tag, but I am not sure about timing of the events sent to spackbot and the periodicity of the sync script.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a label to denote a different branch name scheme to use for this type of CI.

@kwryankrattiger
kwryankrattiger force-pushed the split_repo_snapshots branch 2 times, most recently from 45c292c to 98f5210 Compare October 11, 2025 21:19
Buildcache-tools image provides tools for multiple build cache
operations.

validate-index, publish (to top level), snapshot
@kwryankrattiger
kwryankrattiger marked this pull request as ready for review November 13, 2025 19:54
type=int,
default=2,
default=3,
help=("Target layout version to publish (either 2 or 3, defaults to 2)"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
help=("Target layout version to publish (either 2 or 3, defaults to 2)"),
help=("Target layout version to publish (either 2 or 3, defaults to 3)"),

if isinstance(project, str):
project = GL.projects.get(project, lazy=True)

pipeline = project.pipelines.list(get_all=False, per_page=1, ref=branch, status="success")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we're relying on default behavior here, which is to sort by pipeline ID, descending.

I think this is the right thing to do; the only other applicable option would be updated_at.

https://docs.gitlab.com/api/pipelines/

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I don't think there is really another option here. updated_at is too specific for this search, I want it to find simething, even if it is older than N weeks.

Comment thread images/buildcache-tools/pkg/snapshot.py Outdated
LOGGER.info(f"Creating snapshot for: {t.name} from {branch} using pipeline {pipeline[0].id}")

# Assuming all snapshots are v3 only now
include_stacks = ["windows-vis"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this leftover from local debugging?

create_develop_snapshot_tag(args.project)

# Iterate all of the project tags and attempt to create a
# snaptshot if it is needed

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# snaptshot if it is needed
# snapshot if it is needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants