Updated on December 26th, 2023 in #linux

Automate Suspending Kubernetes Cron Jobs with find and sed

Treat yourself by automating mundane tasks like setting a bunch of YAML properties from true to false and vice versa.

Quick Jump: Demo Video

This example is specific to Kubernetes cron jobs but it can be applied to anything!

With Kubernetes you can set suspend: true or suspend: false on a cron job. This is handy because a suspended cron job won’t get triggered, this is often more user friendly than temporarily deleting the entire cron job and adding it back later.

This is especially true if you subscribe to the GitOps philosophy and use tools like Argo CD to monitor the state of a git repo and sync it to be the true state of your cluster.

In other words, when you want to make changes to your cluster you commit and push config file changes, not run kubectl commands directly against your cluster.

You may choose to temporarily suspend cron jobs if you’re putting your site into maintenance mode such as when performing major database upgrades. You might block access to your public site at the firewall level but internally your Kubernetes cron jobs would still be firing unless you suspend them.

Let’s say you have 3 different apps and each of them have anywhere between 2 and 15 cron jobs. Let’s call it 20 cron jobs total. It’s mildly annoying to open your code editor and do a project wide find / replace to set suspend: true or suspend: false so you can commit these changes afterwards.

Wouldn’t it be nice if you had a run script shortcut to where you can run ./run cronjobs:disable and ./run cronjobs:enable?

The run script functions could look like this:

function cronjobs:disable {
  find "kubernetes/apps/overlays/"*"/cron-job.yaml" -type f \
    -exec perl -pi -e "s/^  suspend: false$/  suspend: true/g" {} \;
}

function cronjobs:enable {
  find "kubernetes/apps/overlays/"*"/cron-job.yaml" -type f \
    -exec perl -pi -e "s/^  suspend: true$/  suspend: false/g" {} \;
}

In the above case, kubernetes/apps/overlays/"*"/cron-job.yaml would be a path to wherever you keep your Kubernetes configs. I like using Kustomize but that’s not important here. The * in this case would expand to all of your apps, such as app a, b and c. This way all of your app’s cron-job.yaml files get modified.

We’re using perl instead of sed because I’ve found it to be more portable on both Linux and macOS when it comes to performing in-place edits. The version of sed on macOS does not work the same as the GNU version of sed but perl does.

But if you want to use sed instead you can. The same logic applies to both. We’re looking for very specific lines in the file and then globally replacing them.

Here’s what the Kubernetes cron job config might look like. In this case it’s a cron job that runs every 5 minutes and when it runs it curls a specific API endpoint. Check out the suspend property on line 7, that’s what gets string replaced:

---
apiVersion: "batch/v1"
kind: "CronJob"
metadata:
  name: "app-do-something"
spec:
  suspend: false
  schedule: "*/5 * * * *"
  concurrencyPolicy: "Forbid"
  jobTemplate:
    spec:
      backoffLimit: 3
      template:
        spec:
          restartPolicy: "OnFailure"
          containers:
          - name: "app"
            imagePullPolicy: "Always"
            image: "curlimages/curl:8.5.0"
            command:
            - "ash"
            - "-c"
            - |-
              curl -s \
                -H "Authorization:Bearer ${APP_API_BEARER_TOKEN}" \
                -X POST \
                http://a-app/api/v1/do_something

The video below shows disabling and enabling cron jobs on a few Kubernetes configs.

Demo Video

Timestamps

• 0:18 – Looking at a Kubernetes cron job config
• 1:05 – This could be useful for temporary maintenance
• 2:07 – Demoing the disable and enable run script functions
• 2:52 – Going over the disable and enable functions
• 3:21 – Using Kustomize and Argo CD to stick with the GitOps philosophy
• 3:50 – Back to the disable and enable functions
• 5:55 – Was it worth it? Yes, now the intent is clear that we do this

Do you think you’ll add this to your project? Let us know below!