clickhouse-cloud-api

Typed Rust client for the ClickHouse Cloud API.

Development

Structure

Field optionality

The OpenAPI spec uses two conventions for marking fields required vs optional:

• Schemas with a required array (newer/beta endpoints) use standard OpenAPI semantics.
• Schemas without required (GA/legacy endpoints) treat fields whose description starts with "Optional" as optional. Everything else is implicitly required.

Additional rules:

• PATCH request schemas (name contains Patch and ends with Request) are always all-optional.
• Nullable fields (type: ["string", "null"] or oneOf with null) are always Option<T>, even if required.

In models.rs, required non-nullable fields use bare types (T) and optional/nullable fields use Option<T>. All fields keep #[serde(default)] so deserialization is tolerant of partial data.

Scripts

# Show a JSON manifest of required/optional fields per schema
python3 scripts/resolve-field-requirements.py

# Update models.rs field types to match the spec
python3 scripts/update-models-optionality.py

# Check for drift between the live spec and the library (dry run)
python3 scripts/check-openapi-drift.py --dry-run

When the spec adds proper required arrays everywhere

The spec team plans to add standard required metadata to all schemas in a future version. When that happens:

# 1. Download the updated spec
curl -s https://api.clickhouse.cloud/v1 -o clickhouse_cloud_openapi.json

# 2. Update models.rs
python3 scripts/update-models-optionality.py

# 3. Fix any test assertions for fields that changed
cargo test -p clickhouse-cloud-api

The resolution logic automatically prefers required arrays over the description heuristic, so no structural changes are needed.

Testing

cargo test -p clickhouse-cloud-api          # all tests
cargo test --test spec_coverage_test        # spec coverage + field optionality only
cargo test --test client_test               # wiremock-based client tests
cargo test --test models_test               # serde round-trip tests

Live-API lifecycle suites are #[ignore]d by default (they provision real resources):

cargo test --test integration_test -- --ignored --nocapture           # ClickHouse service CRUD
cargo test --test integration_postgres_test -- --ignored --nocapture  # Postgres service CRUD

ClickPipes E2E binaries live under tests/clickpipes/ and are declared as named [[test]] entries in Cargo.toml. Each per-source binary provisions its own ClickHouse Cloud service and exercises one source; clickpipe_e2e_test runs every stage in parallel against a single shared service:

cargo test --test clickpipe_e2e_test -- --ignored --nocapture            # all sources, one CHC service
cargo test --test clickpipe_s3_test -- --ignored --nocapture             # per-source: S3
cargo test --test clickpipe_kafka_test -- --ignored --nocapture          # per-source: Kafka (Redpanda)
cargo test --test clickpipe_kinesis_test -- --ignored --nocapture        # per-source: Kinesis
cargo test --test clickpipe_mongo_test -- --ignored --nocapture          # per-source: MongoDB
cargo test --test clickpipe_mysql_test -- --ignored --nocapture          # per-source: MySQL
cargo test --test clickpipe_postgres_ec2_test -- --ignored --nocapture   # per-source: Postgres-on-EC2
cargo test --test clickpipe_postgres_cdc_test -- --ignored --nocapture   # CHC-managed Postgres CDC
cargo test --test clickpipe_smoke_test -- --ignored --nocapture          # create-only smoke against a shared service

All require CLICKHOUSE_CLOUD_API_KEY, CLICKHOUSE_CLOUD_API_SECRET, CLICKHOUSE_CLOUD_TEST_ORG_ID, CLICKHOUSE_CLOUD_TEST_PROVIDER, and CLICKHOUSE_CLOUD_TEST_REGION in the environment, and are wired into the scheduled Cloud Integration GitHub Actions workflow. The ClickPipes E2E suites additionally need AWS credentials and an eu-west-1 region quota; clickpipe_smoke_test reads a pre-provisioned service ID from CLICKHOUSE_CLOUD_TEST_CLICKPIPE_SERVICE_ID.

The spec_coverage_test suite checks three things against the checked-in spec:

1. Every OpenAPI operation has a matching pub async fn in client.rs
2. Every OpenAPI schema has a matching pub struct/pub enum in models.rs
3. Every field's Option<T> vs T matches the spec's required/optional semantics

There are also #[ignore]d variants that run the same checks against the live spec.

Optionality exemptions

Occasionally the spec marks a field as required but the API actually treats it as optional (e.g. sending an empty string triggers a 400 BAD_REQUEST). In these cases we override the field to Option<T> in models.rs and add the field to the OPTIONALITY_EXEMPTIONS list in spec_coverage_test.rs.

The test prints a NOTE: line for every exemption that fires, and fails if an exemption becomes stale (i.e. the spec was corrected upstream and the override is no longer needed). To add a new exemption, add a ("RustStructName", "specFieldName") entry to the constant with a comment explaining why.