Crate staircase 

staircase - Kubernetes Step-Based Operator

During the eventual consistency of Kubernetes, controllers need to be idempotent. It is easy to end up in an uncovered state that needs manual cleanup, which should be avoided in production environments.

The pattern this crate supports is a step-based controller. It helps you build step-based reconcilers that integrate with kube and k8s_openapi.

Step-Based Controller

A controller compares the desired state with the current state. If they differ, it applies changes until they match again. Those changes may happen through the Kubernetes API, such as creating or updating Deployments and Jobs, or through external services, such as REST APIs or gRPC. Any of those changes can fail, be reverted, or be observed late, so the controller must be able to resume from every intermediate state.

A good idea to reduce complexity is do only ever do one change at a time. Within your controll-loop, when seeing that the states DO NOT match, you decide whats the most important change, and do that. After this change, you requeue your resource and continue. In the next iteration you are hopefully left with n-1 differences.

We call an iteration of your control-loop: Run. Each Run is made of ordered Steps. The Plan decides which Steps are executed and in which order. Most steps run sequentially. Parallel steps are allowed only when they are Immutable, meaning they gather data or evaluate conditions without modifying Kubernetes or external systems.

Installation

Add the following dependency to your Cargo.toml:

[dependencies]
staircase = "0.0.7"
## staircase uses kube 4.0.0, which depends on k8s-openapi 0.28. Choose a Kubernetes version as a k8s-openapi feature.
kube = { version = "4.0.0", features = ["runtime", "derive"] }
k8s-openapi = { version = "0.28", features = ["v1_36"] }
serde_json = { version = "1" }

Usage

See examples/simple.rs for a detailed example.

// Derive a custom resource as usual.
#[derive(CustomResource, Deserialize, Serialize, Clone, Debug, JsonSchema)]
#[kube(group = "example", version = "v1", kind = "Foo", status = "FooStatus", namespaced)]
pub struct FooSpec { /* ... */ }

// Mutable steps are explicit and may return Modified.
use std::collections::BTreeMap;

use k8s_openapi::{
    ByteString,
    api::core::v1::Secret,
    apimachinery::pkg::apis::meta::v1::ObjectMeta,
};
use kube::{Api, ResourceExt, api::PostParams};
use staircase::controller::{Mutable, MutableStepOutcome, MutableStepResult, RunContext, Step};

pub struct CreateSecret {}
pub struct CreateDeployment {}

impl Step for CreateSecret {
    type Error = kube::Error;
    type InData = ();
    type Mode = Mutable;
    type OutData = Secret;
    type Resource = Foo;

    async fn run(
        &self,
        context: &RunContext<Self::Resource>,
        data: Self::InData,
    ) -> MutableStepResult<Self::Resource, Self::OutData, Self::Error> {
        // Do work here, for example create random passwords.
        let name = context.resource.name_any();
        let namespace = context.resource.namespace().expect("Foo is namespaced");
        let secrets: Api<Secret> = Api::namespaced(context.client.clone(), &namespace);

        if let Some(secret) = secrets.get_opt(&name).await? {
            return Ok(MutableStepOutcome::NoModification { data: secret });
        }

        let secret = Secret {
            metadata: ObjectMeta {
                name: Some(name.clone()),
                namespace: Some(namespace),
                ..Default::default()
            },
            data: Some(BTreeMap::from([(
                "pass".to_string(),
                ByteString(b"foo".to_vec()),
            )])),
            ..Default::default()
        };

        secrets.create(&PostParams::default(), &secret).await?;
        Ok(MutableStepOutcome::Modified { status: None })
    }
}

impl Step for CreateDeployment {
    type Error = kube::Error;
    type InData = Secret;
    type Mode = Mutable;
    type OutData = ();
    type Resource = Foo;

    async fn run(
        &self,
        _context: &RunContext<Self::Resource>,
        data: Self::InData,
    ) -> MutableStepResult<Self::Resource, Self::OutData, Self::Error> {
        let _secret = data;
        // The password exists now. Create a Deployment here and wire it to the Secret.
        Ok(MutableStepOutcome::Modified { status: None })
    }
}

// Specify step order within a reconciler.
use std::sync::Arc;

use staircase::controller::{BoxPlan, Plan, Reconciler};

struct CustomReconciler {
    client: kube::Client,
}

impl Reconciler for CustomReconciler {
    type InitialData = ();
    type Resource = Foo;
    type EvaluationResource = Arc<Foo>;
    type EvaluationOutcome = ();
    type Error = kube::Error;
    type Plan = BoxPlan<Self::Resource, Self::Error, Self::InitialData, ()>;

    async fn evaluate_plan(&self, resource: Self::EvaluationResource) -> (
        (),
        (),
        Arc<Self::Resource>,
        Self::Plan,
    ) {
        ((), (), resource, Plan::from(CreateSecret {}).then(CreateDeployment {}).boxed())
    }

    async fn client(&self) -> Result<kube::Client, kube::Error> { Ok(self.client.clone()) }
}

For small steps, you can also use closure-backed helpers instead of defining a new struct:

use staircase::controller::{MutableStepOutcome, RunContext, mutable_step};

let step = mutable_step("create-secret", |context: RunContext<Foo>, data: ()| async move {
    let _ = data;
    let name = context.resource.name_any();

    // Check and apply one fix here.
    Ok::<_, kube::Error>(MutableStepOutcome::Modified { status: None })
});

Run and Step Outcomes

A Step should check one fix. If the fix is missing, the step applies it and returns MutableStepOutcome::Modified. The current Run stops immediately, and the returned RunOutcome::Modified tells the caller to requeue the resource.

If a step does not need to change anything, it returns NoModification { data }. The data value is passed to the next step, which avoids fetching or recomputing the same state repeatedly within one Run.

If a step cannot make a useful decision yet, it returns Stop. This stops the current Run without reporting a modification. Use this when the controller is waiting for external state to become observable.

Rules of a Step-Based Approach

• Make at most one meaningful change per Run.
• After a Step modifies Kubernetes or an external system, stop the current Run and requeue.
• Each Step verifies one fix and applies it only when necessary.
• Immutable steps must not modify Kubernetes or external systems. They may gather data, check readiness, and run in parallel.
• Status updates returned together with Modified are best-effort. They are allowed to fail, and reconcile correctness must not depend on them.
• If later logic depends on a status value, write that status in its own Step and observe it in a later Run.
• When any operation fails, or when the operator restarts, the next Run must continue as if the failed run never completed.

When Two Things Seem Necessary

Sometimes a workflow appears to require two changes at once, for example placing an order in an external API and then annotating a Custom Resource with the order id.

Prefer splitting that workflow into observable steps. Ideally, the external service has an order_exists(id) endpoint. Then the controller can use two steps:

1. If !order_exists(cr.id), place the order.
2. If order_exists(cr.id), update the custom resource.

Features

• controller_metrics - measures runtimes and results via opentelemetry.
• controller_trace - adds tracing spans and run ids via tracing.
• util - provides utility functions that make integrating staircase with kube easier.
• derive - re-exports derive macros from staircase-derive.
• _k8s_openapi_latest - technical feature used to enable the latest feature of k8s_openapi. This is especially useful when compiling in CI or on docs.rs. For direct use, prefer pinning a specific Kubernetes version.

Contributing

Pull Requests welcome! Start by checking open issues or feature requests in our GitLab repo.

License

MIT License or APACHE-2.0 License

Modules

controller

Step-based controller building blocks.

resources

Helpers for describing Kubernetes resources owned by a custom resource.

util

Various utils, that are not necessarily required for a step-based operator, but handy in some situations.

Derive Macros

ComponentizedResource

Derive macro available if staircase is built with features = ["derive"].

LabeledResource

Derive macro available if staircase is built with features = ["derive"].

NamedResource

Derive macro available if staircase is built with features = ["derive"].