Kubernetes Custom Resource Definition (CRD) Operators

Kubernetes has emerged as the de facto standard for container orchestration, enabling seamless deployment, scaling, and management of containerized applications. While Kubernetes comes with a rich set of built - in resources, there are often scenarios where developers need to manage custom resources specific to their applications. This is where Custom Resource Definitions (CRDs) and Operators come into play. CRDs allow users to define new resource types in Kubernetes, extending the Kubernetes API. Operators, on the other hand, are a way to package, deploy, and manage these custom resources, automating the operational tasks associated with them. In this blog post, we will explore the core concepts of Kubernetes CRD Operators, provide a typical usage example, discuss common practices, and share some best practices.

Table of Contents

  1. Core Concepts
    • Custom Resource Definitions (CRDs)
    • Operators
  2. Typical Usage Example
    • Defining a CRD
    • Creating an Operator
    • Using the CRD and Operator
  3. Common Practices
    • CRD Design
    • Operator Development
  4. Best Practices
    • Error Handling
    • Testing
    • Monitoring
  5. Conclusion
  6. References

Core Concepts

Custom Resource Definitions (CRDs)

A Custom Resource Definition is a way to teach Kubernetes about a new kind of resource. It defines the schema for a custom resource, similar to how built - in resources like Pods or Services have predefined schemas.

When you create a CRD, you specify the following:

  • API Group: A logical grouping of related resources. For example, example.com could be an API group for custom resources related to a particular application.
  • Version: The version of the API, such as v1.
  • Kind: The name of the custom resource, like MyApp.
  • Spec and Status: The spec field is used to define the desired state of the resource, while the status field represents the current state.

Operators

An Operator is a Kubernetes controller that watches for changes in custom resources and takes actions to ensure that the actual state of the system matches the desired state defined in the custom resource’s spec.

Operators typically follow a control loop pattern:

  1. Watch: Continuously monitor the custom resources for changes.
  2. Reconcile: When a change is detected, compare the desired state (spec) with the actual state and take corrective actions.
  3. Update: Update the status field of the custom resource to reflect the current state.

Typical Usage Example

Defining a CRD

Let’s assume we want to create a custom resource for managing a simple database. Here is an example of a CRD definition in YAML:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                size:
                  type: integer
                version:
                  type: string
            status:
              type: object
              properties:
                phase:
                  type: string
  scope: Namespaced
  names:
    plural: databases
    singular: database
    kind: Database
    shortNames:
      - db

Creating an Operator

We can use the Kubebuilder framework to create an operator for our Database custom resource. Here is a high - level overview of the steps:

  1. Install Kubebuilder: Follow the official Kubebuilder installation guide.
  2. Create a new project:
kubebuilder init --domain example.com
kubebuilder create api --group example.com --version v1 --kind Database
  1. Implement the reconciliation logic in the Database controller:
package controllers

import (
        "context"

        "k8s.io/apimachinery/pkg/runtime"
        ctrl "sigs.k8s.io/controller-runtime"
        "sigs.k8s.io/controller-runtime/pkg/client"

        examplecomv1 "github.com/example/my - operator/api/v1"
)

// DatabaseReconciler reconciles a Database object
type DatabaseReconciler struct {
        client.Client
        Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=example.com,resources=databases,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=example.com,resources=databases/status,verbs=get;update;patch

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        var database examplecomv1.Database
        if err := r.Get(ctx, req.NamespacedName, &database); err != nil {
                return ctrl.Result{}, client.IgnoreNotFound(err)
        }

        // Here we can implement the logic to create, update or delete the database based on the spec
        // and update the status

        return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
        return ctrl.NewControllerManagedBy(mgr).
                For(&examplecomv1.Database{}).
                Complete(r)
}

Using the CRD and Operator

Once the CRD is created and the operator is deployed, we can create a Database custom resource:

apiVersion: example.com/v1
kind: Database
metadata:
  name: my - database
spec:
  size: 10
  version: "1.0"

The operator will then watch for changes in this resource and take appropriate actions.

Common Practices

CRD Design

  • Keep it Simple: Design your CRD schemas to be as simple as possible. Avoid over - complicating the spec and status fields.
  • Use Versioning: Just like with any API, use versioning to manage changes to your custom resource schema.
  • Follow Kubernetes Conventions: Use the same naming and structure conventions as built - in Kubernetes resources.

Operator Development

  • Use Existing Frameworks: Tools like Kubebuilder and Operator SDK can significantly simplify the development process.
  • Separate Concerns: Keep the reconciliation logic modular and separate from other parts of the code.
  • Use Kubernetes APIs Correctly: Follow the best practices for using Kubernetes APIs, such as handling errors and retries.

Best Practices

Error Handling

  • Graceful Degradation: When an error occurs during reconciliation, the operator should handle it gracefully and not crash. It should log the error and retry the operation if possible.
  • Error Reporting: Provide detailed error messages in the status field of the custom resource to help with debugging.

Testing

  • Unit Testing: Write unit tests for the reconciliation logic to ensure that it works as expected.
  • Integration Testing: Test the operator in a Kubernetes cluster to verify its behavior in a real - world environment.

Monitoring

  • Metrics: Expose relevant metrics about the operator’s performance, such as the number of reconciliations and the time taken for each reconciliation.
  • Logging: Use structured logging to make it easier to analyze the operator’s behavior.

Conclusion

Kubernetes CRD Operators are a powerful tool for extending the functionality of Kubernetes and automating the management of custom resources. By understanding the core concepts, following common practices, and implementing best practices, developers can create robust and reliable operators. Whether you are managing a simple database or a complex microservices architecture, CRD Operators can help you streamline the operational tasks and ensure the stability of your applications.

References