Automating Jobs

Step 0: Prerequisites

1. You have a container built and pushed to a Container Registry. For Nautilus GitLab, there are instructions here
2. You have the data staged to Persistent Storage
3. The PVC you want to use has the mode of ReadWriteMany. This is necessary to allow multiple pods to read and write to your PVC at once.
4. You have installed envsubst and kubectl

Step 1: Creating the Template Job Spec

To begin, we will need to create a template YAML spec file. For this example, we will create a job that will save a text file named "helloworld.txt" in 4 directories, but these steps can be replicated for any parallelizable task.

apiVersion: batch/v1
kind: Job

metadata:
  name: helloworld-$DIRPATH

spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: helloworld-container
          image: ubuntu:20.04
          workingDir: /output/$DIRPATH
          imagePullPolicy: IfNotPresent
          command: ["touch", "helloworld.txt"]
          resources:
            limits:
              memory: 4Gi
              cpu: 2
            requests:
              memory: 4Gi
              cpu: 2
          volumeMounts:
            - mountPath: /output
              name: $PVC
      volumes:
        - name: $PVC
          persistentVolumeClaim:
            claimName: $PVC

Notice how we've used 2 environment variables in our template. One of them is what we want to iterate over, the directory name in the PVC. The second is the name of the PVC we'd like to use.

Step 2: Creating the Bash Script

Next, we want to create a bash script that will kick off our jobs for us:

Dirs="mydir1 mydir2 mydir3 mydir4"
Pvc="myPVCName"

for dirpath in $Dirs; do
    PVC=$Pvc DIRPATH=$Dirpath envsubst < spec.yml | kubectl apply -f -
done

This bash file will iterate through the directories in $Dirs and substitute each directory into our Spec YML file everywhere we see $DIRPATH. This now fully prepared KubeCTL YML Spec is then passed to kubectl apply and the job is created.

Note: /output/mydir1 and /output/mydir2, etc. will need to already exist on the PVC or this will error when the jobs start.

Step 3: Running the Bash Script

With the bash script and Template Job Spec in place, we need to run our bash script to create the jobs:

chmod +x script.sh
./script.sh

The output of the script, if run correctly, will be:

$ ./script.sh
job.batch/helloworld-mydir1 created
job.batch/helloworld-mydir2 created
job.batch/helloworld-mydir3 created
job.batch/helloworld-mydir4 created

Once the jobs have finished, we can see that the jobs have successfully created the 4 text files:

$ ls /output/mydir*
mydir1:
helloworld.txt

mydir2:
helloworld.txt

mydir3:
helloworld.txt

mydir4:
helloworld.txt