Kubernetes CronJob: Delete Completed Pods

Kubernetes CronJobs are a powerful feature that allow you to schedule tasks to run at specific intervals, similar to the traditional Unix cron utility. These scheduled tasks create Pods to execute the defined jobs. Over time, as CronJobs run repeatedly, a large number of completed Pods can accumulate in the cluster. These completed Pods can consume storage and other resources, and clutter the namespace, making it difficult to manage and monitor the active workloads. Therefore, understanding how to delete completed Pods from CronJobs is crucial for maintaining a healthy and efficient Kubernetes cluster.

Table of Contents

  1. Core Concepts
  2. Typical Usage Example
  3. Common Practices
  4. Best Practices
  5. Conclusion
  6. References

Core Concepts

CronJobs

CronJobs in Kubernetes are objects that create Jobs on a repeating schedule. A CronJob uses a cron - like schedule expression to define when a Job should be created. For example, a CronJob can be set to run a Job every hour, every day at a specific time, etc. Each time a CronJob runs, it creates a new Job object, which in turn creates one or more Pods to perform the actual task.

Completed Pods

A Pod is considered completed when its containers have finished executing, either successfully or with an error. Completed Pods can be in one of two states: Succeeded or Failed. Succeeded means all containers in the Pod have terminated with a zero exit code, while Failed indicates that at least one container has terminated with a non - zero exit code.

Pod Deletion

Deleting completed Pods is a way to free up resources in the cluster. When a Pod is deleted, its associated resources such as storage, network, and CPU/memory allocations are released. This helps in optimizing the use of cluster resources and reducing the administrative overhead of managing a large number of completed Pods.

Typical Usage Example

Step 1: Create a CronJob

First, let’s create a simple CronJob that runs a Pod to print a message every minute.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: hello-cronjob
spec:
  schedule: "* * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - echo "Hello, Kubernetes!"
          restartPolicy: OnFailure

Save the above YAML in a file named hello-cronjob.yaml and apply it to your Kubernetes cluster using the following command:

kubectl apply -f hello-cronjob.yaml

Step 2: List Completed Pods

After a few minutes, you can list the completed Pods created by the CronJob using the following command:

kubectl get pods --field-selector=status.phase==Succeeded

Step 3: Delete Completed Pods

To delete the completed Pods, you can use the following command:

kubectl delete pods --field-selector=status.phase==Succeeded

Common Practices

Manual Deletion

The most straightforward way to delete completed Pods is to use the kubectl command as shown in the example above. This method is suitable for small clusters or when you want to perform a one - time cleanup.

Automated Scripts

You can write scripts to automate the deletion process. For example, a shell script can be scheduled using a system - level cron job to run the kubectl commands at regular intervals. Here is a simple example of a shell script:

#!/bin/bash
kubectl delete pods --field-selector=status.phase==Succeeded

Save the script as delete-completed-pods.sh, make it executable (chmod +x delete-completed-pods.sh), and then add it to your system’s cron table to run at a desired interval.

Best Practices

Retention Policy

Before deleting completed Pods, it’s a good practice to define a retention policy. For example, you may want to keep completed Pods for a certain period of time (e.g., 24 hours) for auditing or debugging purposes. You can use tools like kube-cleanup-helper to implement a retention policy.

Label - Based Deletion

Instead of deleting all completed Pods, you can use labels to selectively delete Pods. For example, if your CronJobs are labeled with a specific key - value pair, you can use the label selector in the kubectl command to delete only the relevant completed Pods.

kubectl delete pods --field-selector=status.phase==Succeeded -l app=my - cronjob - app

Monitoring and Logging

Keep track of the Pod deletion process by monitoring the events and logs. You can use tools like Prometheus and Grafana to monitor the number of completed Pods and the frequency of deletions. This helps in detecting any issues with the deletion process and ensures that the cluster remains healthy.

Conclusion

Deleting completed Pods from Kubernetes CronJobs is an important task for maintaining a clean and efficient cluster. By understanding the core concepts, following typical usage examples, and implementing common and best practices, you can effectively manage the lifecycle of completed Pods. Whether you choose manual deletion, automated scripts, or more advanced methods, it’s essential to have a strategy in place to ensure that your cluster resources are used optimally.

References