Building Operators: The Tech Stack

It's harder than it looks

Writing a robust controller from scratch that correctly handles retries, exponential backoff, caching (to avoid hammering the API server), efficient watches, and leader election (so only one controller replica acts at a time) is serious distributed systems engineering. Most teams should not attempt this from scratch.

Fortunately, the Kubernetes community has standardized on tooling that handles the hard parts for you.

controller-runtime

controller-runtime is a set of Go libraries that abstracts away the low-level mechanics of talking to the Kubernetes API. It provides:

• A Manager that handles shared caches, health checks, and leader election.
• A Reconciler interface — you implement one function: Reconcile(ctx context.Context, req Request) (Result, error).
• Efficient watches that trigger your reconciler only when relevant objects change.
• Automatic retry with backoff when your reconciler returns an error.

Kubebuilder and Operator SDK

Kubebuilder (maintained by the Kubernetes SIG API Machinery team) and Operator SDK (maintained by Red Hat, built on top of Kubebuilder) are scaffolding frameworks. They generate the boilerplate project structure, CRD manifests from your Go type definitions, Dockerfiles, and deployment YAMLs. You fill in the business logic.

A minimal reconciler in Go looks like this:

func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // 1. Fetch the desired state
    var db mycompanyv1.Database
    if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 2. Check if the StatefulSet already exists
    var sts appsv1.StatefulSet
    err := r.Get(ctx, req.NamespacedName, &sts)
    if errors.IsNotFound(err) {
        // 3. It doesn't exist — create it
        newSts := r.buildStatefulSet(&db)
        return ctrl.Result{}, r.Create(ctx, newSts)
    }

    // 4. It exists — ensure it matches desired state
    r.updateStatefulSet(&sts, &db)
    return ctrl.Result{}, r.Update(ctx, &sts)
}

The framework calls Reconcile() whenever a Database object changes, and also on a periodic re-sync interval to catch any drift. Your job is to make this function idempotent and correct.