Kubernetes CRD Versioning: A Comprehensive Guide

Kubernetes Custom Resource Definitions (CRDs) allow users to extend the Kubernetes API with their own custom resources. As the development of these custom resources progresses, there comes a need to manage different versions of these resources. Versioning in Kubernetes CRDs is crucial for maintaining compatibility, enabling smooth upgrades, and evolving the API over time. This blog post aims to provide an in - depth understanding of Kubernetes CRD versioning, covering core concepts, typical usage examples, common practices, and best practices.

Table of Contents

  1. Core Concepts of Kubernetes CRD Versioning
  2. Typical Usage Example
  3. Common Practices
  4. Best Practices
  5. Conclusion
  6. References

1. Core Concepts of Kubernetes CRD Versioning

Multiple Versions in a CRD

A single CRD can have multiple versions defined. Each version represents a different API schema for the custom resource. For example, you might have a v1 and v2 version of a custom resource. The versions are specified in the spec.versions field of the CRD definition.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mycustomresources.example.com
spec:
  group: example.com
  names:
    kind: MyCustomResource
    plural: mycustomresources
    singular: mycustomresource
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                field1:
                  type: string
    - name: v2
      served: true
      storage: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                field1:
                  type: string
                field2:
                  type: integer

Served and Storage Versions

  • Served Version: A served version is an API version that the Kubernetes API server will accept requests for. In the above example, both v1 and v2 are served versions, meaning clients can send requests using either version.
  • Storage Version: The storage version is the version in which the custom resource is actually stored in etcd. Only one version can be the storage version at a time. In the example, v1 is the storage version.

Conversion Webhooks

When there are multiple versions of a CRD, there may be a need to convert resources between different versions. Conversion webhooks are used to perform these conversions. They are HTTP endpoints that the Kubernetes API server calls to convert a resource from one version to another.

2. Typical Usage Example

Step 1: Define a CRD with Multiple Versions

Let’s assume we are creating a custom resource for managing databases. We start with a v1 version and later introduce a v2 version with additional fields.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  names:
    kind: Database
    plural: databases
    singular: database
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                databaseName:
                  type: string
                username:
                  type: string
    - name: v2
      served: true
      storage: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                databaseName:
                  type: string
                username:
                  type: string
                password:
                  type: string

Step 2: Create a Conversion Webhook

We need to create a conversion webhook to convert between v1 and v2 versions. Here is a simple example using a Go server:

package main

import (
        "encoding/json"
        "fmt"
        "net/http"

        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
        "k8s.io/apimachinery/pkg/runtime"
        "k8s.io/apimachinery/pkg/runtime/schema"
        "k8s.io/apimachinery/pkg/runtime/serializer"
        "k8s.io/apimachinery/pkg/apis/meta/v1beta1"
)

func convertHandler(w http.ResponseWriter, r *http.Request) {
        var conversionReview v1beta1.ConversionReview
        err := json.NewDecoder(r.Body).Decode(&conversionReview)
        if err != nil {
                http.Error(w, err.Error(), http.StatusBadRequest)
                return
        }

        // Implement conversion logic here
        // For simplicity, we just return the same object for now
        conversionReview.Response = &v1beta1.ConversionResponse{
                Result: metav1.Status{
                        Status: metav1.StatusSuccess,
                },
                ConvertedObjects: []runtime.RawExtension{
                        conversionReview.Request.Objects[0],
                },
        }

        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(conversionReview)
}

func main() {
        http.HandleFunc("/convert", convertHandler)
        fmt.Println("Starting conversion webhook server on port 8080")
        http.ListenAndServe(":8080", nil)
}

Step 3: Update the CRD to Use the Conversion Webhook

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  names:
    kind: Database
    plural: databases
    singular: database
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                databaseName:
                  type: string
                username:
                  type: string
    - name: v2
      served: true
      storage: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                databaseName:
                  type: string
                username:
                  type: string
                password:
                  type: string
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        url: "http://localhost:8080/convert"
      conversionReviewVersions: ["v1beta1"]

3. Common Practices

Gradual Version Rollout

When introducing a new version of a CRD, it is a good practice to gradually roll it out. Start by making the new version served but not the storage version. This allows users to start using the new version without affecting the existing stored data.

Testing Version Compatibility

Before making a new version the storage version, thoroughly test the compatibility between different versions. This includes testing the conversion webhooks to ensure that resources can be converted correctly between versions.

Documentation

Maintain clear documentation about the changes in each version of the CRD. This helps users understand what has changed and how to migrate their resources to the new version.

4. Best Practices

Use OpenAPI Schema

Define the OpenAPI schema for each version of the CRD. This helps in validating the input data and provides clear documentation about the structure of the custom resource.

Keep Conversion Logic Simple

The conversion logic in the conversion webhooks should be as simple as possible. Complex conversion logic can lead to bugs and make it difficult to maintain.

Versioning Strategy

Adopt a clear versioning strategy, such as Semantic Versioning. This makes it easier for users to understand the significance of the changes in each version.

Conclusion

Kubernetes CRD versioning is an essential aspect of managing custom resources in a Kubernetes cluster. By understanding the core concepts, using typical usage examples, following common practices, and implementing best practices, software engineers can effectively manage different versions of their custom resources. This ensures compatibility, smooth upgrades, and the ability to evolve the API over time.

References