clickhouse-cloud-api

Typed Rust client for the ClickHouse Cloud API.

Development

Structure

Field optionality

The OpenAPI spec uses two conventions for marking fields required vs optional:

• Schemas with a required array (newer/beta endpoints) use standard OpenAPI semantics.
• Schemas without required (GA/legacy endpoints) treat fields whose description starts with "Optional" as optional. Everything else is implicitly required.
• Known partial required arrays use the union of that array and the description heuristic, as configured by the analyzer.

Additional rules:

• PATCH request schemas (name contains Patch and ends with Request) are always all-optional.
• Nullable fields (type: ["string", "null"] or oneOf with null) are always Option<T>, even if required.

In models.rs, required non-nullable fields use bare types (T) and optional/nullable fields use Option<T>. All fields keep #[serde(default)] so deserialization is tolerant of partial data.

Deprecated fields

The OpenAPI spec marks some response fields as deprecated (e.g. Service.tier, ApiKey.roles, Member.role). In almost all cases, these are not needed. The Cloud API library disables them by default, gated by a Cargo feature flag deprecated-fields. Enable this feature if you need to consume deprecated fields.

Scripts

# Show a JSON manifest of required/optional fields per schema
python3 scripts/resolve-field-requirements.py

# Regenerate the DEPRECATED_FIELDS constant from the snapshot
python3 scripts/regenerate-deprecated-fields.py

# Regenerate the BETA_OPERATIONS constant from the snapshot
python3 scripts/regenerate-beta-lists.py

# Check for drift between the live spec and the library (dry run)
python3 scripts/check-openapi-drift.py --dry-run

Field optionality is maintained by hand — edit models.rs directly when the drift check flags a mismatch.

Testing

cargo test -p clickhouse-cloud-api          # all tests
cargo test -p clickhouse-openapi-analyzer   # analyzer fixtures + executable parity
cargo test -p clickhouse-cloud-api --test spec_coverage_test # shared snapshot report
cargo test -p clickhouse-cloud-api --test client_test        # wiremock client tests
cargo test -p clickhouse-cloud-api --test models_test        # serde round trips

Live-API lifecycle suites are #[ignore]d by default (they provision real resources):

cargo test --test integration_test -- --ignored --nocapture           # ClickHouse service CRUD
cargo test --test integration_postgres_test -- --ignored --nocapture  # Postgres service CRUD

ClickPipes E2E binaries live under tests/clickpipes/ and are declared as named [[test]] entries in Cargo.toml. Each per-source binary provisions its own ClickHouse Cloud service and exercises one source; clickpipe_e2e_test runs every stage in parallel against a single shared service:

cargo test --test clickpipe_e2e_test -- --ignored --nocapture            # all sources, one CHC service
cargo test --test clickpipe_s3_test -- --ignored --nocapture             # per-source: S3
cargo test --test clickpipe_kafka_test -- --ignored --nocapture          # per-source: Kafka (Redpanda)
cargo test --test clickpipe_kinesis_test -- --ignored --nocapture        # per-source: Kinesis
cargo test --test clickpipe_mongo_test -- --ignored --nocapture          # per-source: MongoDB
cargo test --test clickpipe_mysql_test -- --ignored --nocapture          # per-source: MySQL
cargo test --test clickpipe_postgres_ec2_test -- --ignored --nocapture   # per-source: Postgres-on-EC2
cargo test --test clickpipe_postgres_cdc_test -- --ignored --nocapture   # CHC-managed Postgres CDC
cargo test --test clickpipe_smoke_test -- --ignored --nocapture          # create-only smoke against a shared service

All require CLICKHOUSE_CLOUD_API_KEY, CLICKHOUSE_CLOUD_API_SECRET, CLICKHOUSE_CLOUD_TEST_ORG_ID, CLICKHOUSE_CLOUD_TEST_PROVIDER, and CLICKHOUSE_CLOUD_TEST_REGION in the environment, and are wired into the scheduled Cloud Integration GitHub Actions workflow. The ClickPipes E2E suites additionally need AWS credentials and an eu-west-1 region quota; clickpipe_smoke_test reads a pre-provisioned service ID from CLICKHOUSE_CLOUD_TEST_CLICKPIPE_SERVICE_ID.

spec_coverage_test sends the checked-in sources and snapshot through the private clickhouse-openapi-analyzer crate. That same analyzer powers the scheduled live-spec issue, so operation, model, field, optionality, beta, deprecation, enum, snapshot, and stale-exemption findings share one implementation. The single ignored test runs the same report against the live spec.

Optionality exemptions

Occasionally the spec cannot be followed literally because verified API behavior differs. All such policy lives in crates/clickhouse-openapi-analyzer/src/config.rs, including optionality, extra-field, deprecated-field, extra-enum-value, non-OpenAPI-method, partial required-schema, and unsupported-enum configuration.

Add an exemption only for a deliberate runtime behavior and document why the spec cannot be followed. New unsupported-enum acknowledgements also require a tracking issue. The analyzer reports stale field/enum exemptions and vanished unsupported locations so obsolete entries are removed during normal drift remediation. See the repository AGENTS.md for exact key formats and the full remediation and verification procedure.