Running a Job

Building a Job YAML Spec

Similar to our introduction to pods, we will use a sample YAML spec that is already present in the repo.

apiVersion: batch/v1

kind: Job

metadata:
  name: pi

spec:
  template:
    spec:
      containers:
        - name: pi
          image: perl:5.34
          command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
          resources:
            limits:
              memory: 12Gi
              cpu: 2
            requests:
              memory: 10Gi
              cpu: 2
      restartPolicy: Never
  backoffLimit: 0

Let's note a couple of key differences from our pod definition:

apiVersion: batch/v1

We need specify that we are using a batch API, not just V1

kind: Job

Specify that we are running a job

spec:
  template:
    spec:
      containers:

For the batch/v1 API, we have to put the spec of our containers into a nested spec --> template --> spec.

    restartPolicy: Never

We do not want our jobs to auto-restart, because generally their failures are deterministic.

  backoffLimit: 0

Should this container go into backoff due to an error, how many times should it retry to start the container. We specify 0, because again, failures are normally deterministic.

The rest of the spec is identical to our pod spec.

Starting the Job

Once you have a YAML spec created for your job, you can queue your job using the apply command:

kubectl apply -f MYFILE.yaml

Once you have done this, you can look at jobs:

kubectl get jobs

And then you can see the pods created for the job to run:

kubectl get pods

And finally you can follow the logs of the pod to see the stdout and stderr of your process with:

kubectl logs -f MYPOD

Deleting your Job

After your job has either finished or errored, you can delete all the pods and job spec using the delete command:

kubectl delete job MYJOB

Reminder that even successful jobs will not automatically be deleted. But you can delete all successful jobs in a namespace (be cautious that there are not other users' jobs in your namespace) with this command:

kubectl delete jobs --field-selector status.successful=1

Note: Again this is a destructive process so be sure that you have all needed data copied off of the local storage of the pod.