hemmer-provider-generator 0.3.2

Automatically generate Hemmer infrastructure providers from cloud SDK specifications
hemmer-provider-generator-0.3.2 is not a library.

Hemmer Provider Generator

CI License

Automatically generate Hemmer infrastructure providers from cloud SDK specifications.

Transform any cloud provider's official SDK specification into a complete, working Hemmer provider package—no manual coding required.

✨ Features

  • Universal Spec Support: Parse Smithy, OpenAPI, Discovery, and Protobuf specifications
  • Multi-Cloud: Support for AWS, GCP, Azure, Kubernetes, and gRPC services
  • Unified Providers: Generate single providers with multiple services (Phase 6 complete)
  • Auto-Detection: Automatically detects spec format from file extension and content
  • Complete Generation: Generates provider.k manifest, Rust code, tests, and documentation
  • Production Ready: Fully tested (57 tests), clippy-clean, formatted code
  • Zero Manual Coding: End-to-end automation from spec to provider package

🚀 Quick Start

Installation

Option 1: Install from crates.io (Recommended)

All Platforms:

cargo install hemmer-provider-generator

Option 2: Quick Install Script

Linux/macOS:

curl -fsSL https://raw.githubusercontent.com/hemmer-io/hemmer-provider-generator/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/hemmer-io/hemmer-provider-generator/main/install.ps1 | iex

Option 3: Install from GitHub

All Platforms:

cargo install --git https://github.com/hemmer-io/hemmer-provider-generator.git

Option 4: Build from source

Linux/macOS:

# Clone the repository
git clone https://github.com/hemmer-io/hemmer-provider-generator.git
cd hemmer-provider-generator

# Build and install
cargo install --path crates/cli

# Or just build
cargo build --release
./target/release/hemmer-provider-generator --help

Windows (PowerShell):

# Clone the repository
git clone https://github.com/hemmer-io/hemmer-provider-generator.git
cd hemmer-provider-generator

# Build and install
cargo install --path crates/cli

# Or just build
cargo build --release
.\target\release\hemmer-provider-generator.exe --help

Usage

1. Parse a Spec File (Inspect Without Generating)

# Auto-detect format
hemmer-provider-generator parse --spec storage-v1.json -v

# Explicit format
hemmer-provider-generator parse \
  --spec service.pb \
  --format protobuf \
  --service myservice

Output: Service definition summary with resource count and CRUD operations

2. Generate a Single Service Provider

# Generate from GCP Discovery document
hemmer-provider-generator generate \
  --spec storage-v1.json \
  --service storage \
  --output ./providers/gcp-storage

# Generate from AWS Smithy spec
hemmer-provider-generator generate \
  --spec s3-model.json \
  --format smithy \
  --service s3 \
  --output ./providers/aws-s3

# Generate from Kubernetes OpenAPI spec
hemmer-provider-generator generate \
  --spec kubernetes-api.json \
  --service kubernetes \
  --output ./providers/k8s

Output: Complete provider package with provider.k, Cargo.toml, and Rust code

3. Generate Unified Multi-Service Provider

Generate a single provider package with multiple cloud services:

Option A: Explicit Spec List

hemmer-provider-generator generate-unified \
  --provider aws \
  --specs s3-model.json,dynamodb-model.json,lambda-model.json \
  --service-names s3,dynamodb,lambda \
  --output ./provider-aws

Option B: Directory Scanning (Recommended)

# Recursively scans directory for all spec files
hemmer-provider-generator generate-unified \
  --provider aws \
  --spec-dir /path/to/aws-sdk-models/models/ \
  --filter s3,dynamodb,lambda \
  --output ./provider-aws \
  -v

Key Features:

  • 🔍 Recursive Discovery: Automatically finds all spec files in directory tree
  • 🎯 Smart Filtering: --filter flag to select specific services by name
  • 📦 Unified Output: Single provider with multiple services
  • 🚀 Large-Scale: Tested with 400+ AWS services, 18 GCP services, K8s specs

Real-World Examples:

# Generate complete AWS provider (all services)
hemmer-provider-generator generate-unified \
  --provider aws \
  --spec-dir ~/aws-sdk-models/models/ \
  --output ./provider-aws

# Generate filtered GCP provider (compute + storage + bigquery)
hemmer-provider-generator generate-unified \
  --provider gcp \
  --spec-dir ~/google-api-go-client/ \
  --filter compute,storage,bigquery \
  --output ./provider-gcp

# Generate Kubernetes provider (apps + core APIs)
hemmer-provider-generator generate-unified \
  --provider kubernetes \
  --spec-dir ~/kubernetes/api/openapi-spec/v3/ \
  --filter apps,core \
  --output ./provider-k8s

📋 Supported Spec Formats

Format Cloud Provider(s) Source Repositories Status
Smithy AWS aws/api-models-aws (406 services) ✅ Tested
OpenAPI 3.0 Kubernetes, Azure kubernetes/kubernetes ✅ Tested
Discovery Google Cloud googleapis/google-api-go-client (436 resources) ✅ Tested
Protobuf gRPC Services Compiled .proto files (FileDescriptorSet) ✅ Supported

Getting Spec Files

AWS (Smithy)

git clone https://github.com/aws/api-models-aws.git
cd api-models-aws/models
# Contains 406 service specs (S3, DynamoDB, Lambda, etc.)

Google Cloud (Discovery)

git clone https://github.com/googleapis/google-api-go-client.git
cd google-api-go-client
# Contains 436 resources across all GCP services

Kubernetes (OpenAPI)

git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes/api/openapi-spec/v3
# Contains OpenAPI specs for all K8s APIs

Protobuf (gRPC)

# Compile .proto files to FileDescriptorSet
protoc --descriptor_set_out=service.pb \
  --include_imports \
  service.proto

🏗️ Architecture

Spec File → Auto-Detect Format → Parse → ServiceDefinition IR → Generate → Provider Package

Cloud-Agnostic Design: All parsers output the same intermediate representation (ServiceDefinition), making the generator completely cloud-agnostic.

📦 Generated Provider Structure

provider-{service}/
├── Cargo.toml                    # Package manifest with SDK dependencies
├── README.md                     # Auto-generated documentation
├── provider.k                    # KCL manifest with resource schemas
└── src/
    ├── lib.rs                   # Provider struct and resource accessors
    └── resources/
        ├── mod.rs               # Resource exports
        └── {resource}.rs        # Individual resource implementations

Example Generated Output

provider.k:

schema StorageProvider:
    schema Bucket:
        name: str
        location: str?
        storage_class: str?

src/resources/bucket.rs:

pub struct Bucket<'a> {
    provider: &'a crate::StorageProvider,
}

impl<'a> Bucket<'a> {
    pub async fn create(&self, name: String, ...) -> Result<String> { }
    pub async fn read(&self, id: &str) -> Result<()> { }
    pub async fn update(&self, id: &str, ...) -> Result<()> { }
    pub async fn delete(&self, id: &str) -> Result<()> { }
}

🎯 Real-World Examples

Example 1: Complete AWS Provider (406 Services)

# Clone AWS SDK specs
git clone --depth 1 https://github.com/aws/api-models-aws.git /tmp/aws-sdk

# Generate unified AWS provider with specific services
hemmer-provider-generator generate-unified \
  --provider aws \
  --spec-dir /tmp/aws-sdk/models/ \
  --filter s3,dynamodb,lambda,ec2,rds,sqs,sns \
  --output ./provider-aws \
  -v

# Result: Single provider-aws package with 7 services

Example 2: Google Cloud Provider (Storage, Compute, BigQuery)

# Clone Google API specs
git clone --depth 1 https://github.com/googleapis/google-api-go-client.git /tmp/gcp-api

# Generate unified GCP provider
hemmer-provider-generator generate-unified \
  --provider gcp \
  --spec-dir /tmp/gcp-api/ \
  --filter storage,compute,bigquery \
  --output ./provider-gcp \
  -v

# Result: 18 services matched, 436 resources parsed

Example 3: Kubernetes Provider (Core + Apps APIs)

# Clone Kubernetes specs
git clone --depth 1 https://github.com/kubernetes/kubernetes.git /tmp/k8s

# Generate unified Kubernetes provider
hemmer-provider-generator generate-unified \
  --provider kubernetes \
  --spec-dir /tmp/k8s/api/openapi-spec/v3/ \
  --filter apps,core \
  --output ./provider-k8s \
  -v

# Result: 2 services, 9 resources

Example 4: Single Service Provider (AWS S3 Only)

# Download single Smithy spec
curl -o s3.json https://raw.githubusercontent.com/aws/api-models-aws/main/models/s3.json

# Generate single-service provider
hemmer-provider-generator generate \
  --spec s3.json \
  --format smithy \
  --service s3 \
  --output ./providers/aws-s3

# Result: provider-s3 package with 38 resources

🔧 Development

Building

# Build all crates
cargo build --workspace

# Run tests
cargo test --workspace --all-features

# Run clippy
cargo clippy --workspace -- -D warnings

# Format code
cargo fmt --all

Running Tests

# All tests
cargo test --workspace

# Specific parser
cargo test --test smithy_parser_test
cargo test --test openapi_parser_test
cargo test --test discovery_parser_test
cargo test --test protobuf_parser_test

# With output
cargo test -- --nocapture

📚 Project Structure

This is a Cargo workspace with 4 crates:

  • common/ - Shared types (ServiceDefinition IR, FieldType, errors)
  • parser/ - Spec format parsers (Smithy, OpenAPI, Discovery, Protobuf)
  • generator/ - Code generation engine (Tera templates)
  • cli/ - Command-line interface

🎓 How It Works

Single Service Generation

  1. Parse: Load and parse spec file using appropriate parser
  2. Transform: Convert to cloud-agnostic ServiceDefinition IR
  3. Generate: Apply Tera templates to create provider files
  4. Output: Complete provider package ready to use

Multi-Service Generation (Unified Provider)

  1. Discover: Recursively scan directory for .json and .pb files
  2. Filter: Match service names against --filter patterns
  3. Parse: Parse all discovered specs into ServiceDefinitions
  4. Aggregate: Combine services into single ProviderDefinition
  5. Generate: Create unified provider package with all services (🚧 in progress)

Auto-Detection Logic

The CLI automatically detects spec format from:

  • File extension: .pb → Protobuf, .json → Parse content
  • Filename patterns: smithy-model.json, storage-discovery.json, *openapi*.json
  • Content markers:
    • "smithy" + "shapes" → Smithy
    • "openapi" + "paths" → OpenAPI
    • "discoveryVersion" + "resources" → Discovery

Service Name Filtering

When using --filter, service names are matched against spec filenames:

  • --filter s3 matches: s3.json, s3-2006-03-01.json, s3control.json
  • --filter storage matches: storage.json, storage-v1-api.json, storagetransfer.json
  • Multiple filters: --filter s3,dynamodb,lambda matches any of the three

🧪 Testing

  • 57 total tests across workspace
  • 4 integration tests (one per spec format)
  • 3 unified generation tests
  • Multi-platform CI (Ubuntu, macOS, Windows)
  • All tests passing ✅

🔗 Related Projects

  • Hemmer - Infrastructure-as-Code tool that uses these providers
  • KCL - Configuration language used for provider manifests

📄 License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

🤝 Contributing

See CONTRIBUTING.md for contribution guidelines.

For development context and architecture details, see CLAUDE.md.

🎯 Status

Production Ready ✅

All 6 planned phases are complete:

  • ✅ Phase 1: Foundation & Planning
  • ✅ Phase 2: AWS SDK Parser (Smithy)
  • ✅ Phase 3: Generator Core
  • ✅ Phase 4: Multi-Cloud Parsers (OpenAPI, Discovery, Protobuf)
  • ✅ Phase 5: CLI Interface & Production Readiness
  • ✅ Phase 6: Unified Multi-Service Providers (#16, #22, PR #19, PR #23)

Feature Status

Feature Status Notes
Single-service generation ✅ Complete Fully tested
Smithy parser ✅ Complete 406 AWS services
OpenAPI parser ✅ Complete Kubernetes, Azure
Discovery parser ✅ Complete 436 GCP resources
Protobuf parser ✅ Complete gRPC services
Directory scanning ✅ Complete Recursive discovery
Service filtering ✅ Complete Pattern matching
Multi-service parsing ✅ Complete Parse & aggregate multiple services
Multi-service generation ✅ Complete Full code generation for unified providers
Cross-platform install ✅ Complete Linux, macOS, Windows

📊 Test Results

Tested with real SDK repositories:

  • AWS: 406 services scanned, 91 resources parsed (S3, DynamoDB, Lambda)
  • GCP: 18 services matched, 436 resources parsed (Storage, Compute, BigQuery)
  • Kubernetes: 2 services, 9 resources parsed (Apps, Core APIs)

Version: 0.3.2 Last Updated: 2025-11-01