ClusterStorageContainer environment variable with `valueFrom` triggers "value and valueFrom both set" admission error

[mvlassis]

Bug Description

Creating an InferenceService seems to fail with the latest/edge version of kserve. This occurs only when the https_proxy, http_proxy, and no_proxy config options are set on the kserve-controller charm.

The error should be related to kserve/kserve#5516.

Explanation

KServe runs the following to build the storage-initializer container:

jsonResult := StrategicMergePatch(baseJSON, overrideJSON)   // (a) merge 2 JSONs
json.Unmarshal(jsonResult, targetContainer)                 // (b) decode back into the existing struct

An error seems to occurs while doing the merging, and some environment variables are populated with both a value and valueFrom field, resulting in an incorrect spec.

To Reproduce

• Deploy solution from feat: add multi-tenancy support for KFP charmed-kubeflow-solutions#218, run UATs from feat: add multi-tenancy tests for KFP charmed-kubeflow-uats#285

Environment

• latest/edge of kserve-controller, revision 1338

Relevant Log Output

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    modelFormat: sklearn
    serving.kserve.io/deploymentMode: Standard
    sidecar.istio.io/inject: "false"
  creationTimestamp: "2026-07-28T14:39:39Z"
  finalizers:
  - inferenceservice.finalizers
  generation: 1
  name: wine-regressor3
  namespace: admin
  resourceVersion: "442946"
  uid: 94577106-077f-4798-bbdc-c854f9fd6ef3
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      storageUri: s3://kfp/0/456bbd1459cd4888b5b6cb683f357cfe/artifacts/model
    serviceAccountName: kserve-controller-s3
status:
  clusterServingRuntimeName: kserve-sklearnserver
  components:
    predictor: {}
  conditions:
  - lastTransitionTime: "2026-07-28T15:19:32Z"
    message: Predictor HTTPRoute not created
    reason: Predictor Deployment NotReady
    status: "False"
    type: IngressReady
  - lastTransitionTime: "2026-07-28T15:19:32Z"
    message: 'Pod "wine-regressor3-predictor-546ccb8b49-5h5wp" is invalid: [spec.initContainers[0].env[0].valueFrom:
      Invalid value: "": may not be specified when `value` is not empty, spec.initContainers[0].env[1].valueFrom:
      Invalid value: "": may not be specified when `value` is not empty, spec.initContainers[0].env[3].valueFrom:
      Invalid value: "": may not be specified when `value` is not empty, spec.initContainers[0].env[4].valueFrom:
      Invalid value: "": may not be specified when `value` is not empty]'
    reason: FailedCreate
    status: "False"
    type: PredictorReady
  - lastTransitionTime: "2026-07-28T15:19:32Z"
    message: Predictor HTTPRoute not created
    reason: Predictor Deployment NotReady
    status: "False"
    type: Ready
  - lastTransitionTime: "2026-07-28T14:39:40Z"
    severity: Info
    status: "False"
    type: Stopped
  deploymentMode: Standard
  modelStatus:
    copies:
      failedCopies: 0
    states:
      activeModelState: ""
      targetModelState: Pending
    transitionStatus: InProgress
  observedGeneration: 1

Additional Context

• Bumping the kserve image to 0.19.0 and above will fix the issue.
• Another workaround for the tests where this is relevant is to reset the https_proxy, http_proxy, and no_proxy config options for the kserve-controller charm.

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-8889.

This message was autogenerated