Use cases for BlueAPI Deployment PVC

As a support engineer wanting to manage my deployments I would like to do the following (in order of importance):

Roll back to a known "good" PVC
Have BlueAPI make a new PVC pulling a clean main (or a release tag) of my repos, the current PVC should be kept around
Remotely see if my current PVC has uncommited changes
Read the code in an old PVC without changing live
Delete an old PVC
See when a PVC was made/last used/last edited
Delete based on some criteria e.g. all that haven't been used for X time

Acceptance Criteria

All the above are documented on how to do them
If they are hard or require more than 2 or 3 steps there are issues to make them easier

@DominicOram

Hi @DominicOram Thanks for the request. Could you please clarify in this ticket a bit more - what is this PVC for? is it for

storage for user scripts?
storage for visit experiment data?
storage for workspace of application cache?
or anything else?
we may need to develop policy for each of these storage type if not already available.
I am not yet a Cloud software developer, and not sure PVC can or cannot be rolled back or forth independent of pod.

@hyperrealist

@hyperrealist @gilesknap please can you think about whether we have the same requirements for FastCS / epics-containers dev PVCs

Could you please clarify in this ticket a bit more - what is this PVC for?

This is for the PVC that BlueAPI currently uses for deployments i.e. what mounts village-bluesky plan repositories and the python environment that runs them (so dodal + ophyd-async etc.). So I guess from your list "storage for workspace of application cache?" is the closest match.

storage for user scripts?
storage for visit experiment data?

I'm not sure what the D-II architecture is for storing these? At the moment I have assumed experiment data is stored in /dls/ixx/data as previously and I have sidestepped the question of user scripts. If DAQ core have a plan to also store these via PVC then I would probably have some similar requirements I could feedback but I suggest we cross the bridge when we come to it.

You are correct. I was just told by the team the detector data and nexus files are currently not accessed via PVC, it still uses Posix file system directly. I am not aware of this decision, nor for what reasons this had been decision as such.
There still no discussion on how to handle storage for user scripts so far.
So what you asked here is for deployment source codes, thanks for confirming this.

Of the criteria, the most difficult will be (in no order):

Seeing when a PVC was last edited - I think this will be tricky in the general case, but more possible if scoped to the currently mounted PVC. We already have code (not shipped) to tag PVCs with when they were last used. Hopefully this largely fulfils the use case.
Auto-deletion - Again, we have an un-shipped feature to auto-delete on pre-defined criteria. We're unable to ship this because we don't have a good solution to alerting staff prior to deletion.

I think the rest are achievable. However the human interaction with them is all quite messy:

Mounting a specific PVC / creating a new one which pulls from main will be done by changing a field in the Values.yaml file. It can also be overridden in ArgoCD (not recommended).
Remotely viewing whether a PVC has uncommitted changes can done via the /python_environment endpoint, eg. this response body having edited P48:

{
  "installed_packages": [
    {
      "name": "dls-dodal",
      "version": "https://github.com/DiamondLightSource/dodal.git @main",
      "location": "/app/.venv/lib/python3.11/site-packages &&/workspace/dodal/src &&",
      "is_dirty": false,
      "source": "scratch"
    },
    {
      "name": "htss-rig-bluesky",
      "version": "https://github.com/DiamondLightSource/htss-rig-bluesky.git @foobar",
      "location": "/app/.venv/lib/python3.11/site-packages &&/workspace/htss-rig-bluesky/src &&",
      "is_dirty": true,
      "source": "scratch"
    }
  ],
  "scratch_enabled": true
}

A PVC's last use is stamped into its Kubernetes annotations. This enabled PVC auto-deletion, but is not very human friendly. You can get the timestamp with a Kubectl command, or by looking at the beamlines' Kubernetes dashbaord or ArgoCD:

Feedback on whether this needs to be rationalised into a more singular experience would be great. Nonetheless they're technically possible.

A link to the PR with the PVC auto deletion + time stamping logic #1514

@coretl

re @coretl question. My view is that for a dev PVC for FastCS or epics-containers IOCs we should keep it very simple and have one per service being 'deved'. However, if we are to unify the approach to creating these PVCs then having all of the above requirements available, yet also make it easy to pick the simple 1-to-1 pattern would also be fine.

My view is that for a dev PVC for FastCS or epics-containers IOCs we should keep it very simple and have one per service being 'deved'

This is interesting, how will you deal with the situation where you have a hotfixed IOC and you roll forward to a release but this doesn't work and you have to roll back?

My view is that for a dev PVC for FastCS or epics-containers IOCs we should keep it very simple and have one per service being 'deved'

This is interesting, how will you deal with the situation where you have a hotfixed IOC and you roll forward to a release but this doesn't work and you have to roll back?

My expectation is that rolling forward would mean switching off debugging mode and just using the native container without any overlay PVC, thus leaving the hotfix in place. But, if I wanted to juggle two different hotfixes then maybe I do need versioning.

@DominicOram

My expectation is that rolling forward would mean switching off debugging mode and just using the native container without any overlay PVC, thus leaving the hotfix in place. But, if I wanted to juggle two different hotfixes then maybe I do need versioning.

Oh wait, I missed this point: If you need multiple hotfix versions then you can just use branches in the git repo that is sitting in your PVC (and those can be pushed to the remote too so you are not 100% relying on the PVC).

That is the normal dev cycle and only requires a single PVC. So I come down strongly on the single PVC and keep things simple side. @DominicOram let me know if you have any objections?

@dan-fernandes

My expectation is that rolling forward would mean switching off debugging mode and just using the native container without any overlay PVC, thus leaving the hotfix in place. But, if I wanted to juggle two different hotfixes then maybe I do need versioning.

Oh wait, I missed this point: If you need multiple hotfix versions then you can just use branches in the git repo that is sitting in your PVC (and those can be pushed to the remote too so you are not 100% relying on the PVC).

That is the normal dev cycle and only requires a single PVC. So I come down strongly on the single PVC and keep things simple side. @DominicOram let me know if you have any objections?

I believe (though @dan-fernandes might correct me) that in our case the PVC contains the full venv. It feels like switching branches like you're suggesting has the potential to lead to issues with dependencies if your two hotfixes have different dependencies?

I believe (though @dan-fernandes might correct me) that in our case the PVC contains the full venv. It feels like switching branches like you're suggesting has the potential to lead to issues with dependencies if your two hotfixes have different dependencies?

I'm pretty sure that just needs the instructions to read:

git checkout my-hotfix1
uv sync

I believe (though @dan-fernandes might correct me) that in our case the PVC contains the full venv. It feels like switching branches like you're suggesting has the potential to lead to issues with dependencies if your two hotfixes have different dependencies?

I'm pretty sure that just needs the instructions to read:
git checkout my-hotfix1
uv sync

What if you've made a hotfix in a dependency? I think we may just have to have different workflows here. From the DAQ side, at least atm, it's common that a PVC will contain fixes across xx-bluesky/dodal/ophyd-async

For FastCS projects my workflow would be:

Switch to debug mode which populates the PVC
Locate the bug with print debugging or similar live in the venv
Clone the repo (dependency or transitive dependency) into the PVC, then uv add --editable /path/to/dep, this will make the change in pyproject.toml that sets [tool.uv.sources] to the path
Fix the bug in the local clone and push the branch to github somewhere

Now the hotfix is deployed, but you are still in debug mode. To go back to production mode you would have to:

Switch project to get the source from upstream with uv add "dep @ git+https://github.com/org/dep" --branch fix/the-bug
Push and release a container with these changes in
Point to that container

I would say that in run fixes leave the service in debug mode, and in the shutdown you go back to production mode.

If you need a second conflicting hotfix then you could clone into /path/to/dep-for-2nd-hotfix and install from that instead. However that seems unlikely to me, wouldn't you just pile hotfix on top of hotfix until everything works?

@coretl

Discussed in person with @coretl. In theory I am ok with only ever having one PVC, my main concern is that people might accidentally overwrite a PVC with hotfixed changes and lose them. I think if we did the following:

Made pushing from the PVC easy - Allow pushing to GH from PVC (or document how to/make easier) #1660
Do not allow PVCs to be overwritten if they have changes in that have not been pushed to origin

Then I would be happy

I'm developing https://epics-containers.github.io/podbench/main/index.html

Which natively forwards your sshagent and git config to the vscode session just like a normal devcontainer.
has semantics for initializing a PVC but then never touches it, its up to you and git to make changes.

Navigation Menu

Repository navigation

Use cases for BlueAPI Deployment PVC #1570

Acceptance Criteria

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Use cases for BlueAPI Deployment PVC #1570

Description

Acceptance Criteria

Activity

fajinyuan commented on Jun 26, 2026

coretl commented on Jun 26, 2026

DominicOram commented on Jun 26, 2026

fajinyuan commented on Jun 26, 2026

dan-fernandes commented on Jun 26, 2026

Alexj9837 commented on Jun 26, 2026

gilesknap commented on Jul 3, 2026

DominicOram commented on Jul 6, 2026

gilesknap commented on Jul 20, 2026

gilesknap commented on Jul 29, 2026

DominicOram commented on Jul 29, 2026

gilesknap commented on Jul 31, 2026

DominicOram commented on Jul 31, 2026

coretl commented on Aug 4, 2026

DominicOram commented on Sep 8, 2026

gilesknap commented on Sep 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions