Expand description
BigQuery Schema Generator
A Rust library and CLI tool for generating BigQuery schemas from JSON or CSV data.
Unlike BigQuery’s built-in auto-detect which only examines the first 500 records, this tool processes all records in the input to generate a complete schema.
§Example
use bq_schema_gen::{SchemaGenerator, GeneratorConfig, SchemaMap};
use serde_json::json;
let config = GeneratorConfig::default();
let mut generator = SchemaGenerator::new(config);
let mut schema_map = SchemaMap::new();
// Process records
let record = json!({"name": "test", "count": 42});
generator.process_record(&record, &mut schema_map).unwrap();
// Get the BigQuery schema
let schema = generator.flatten_schema(&schema_map);Re-exports§
pub use error::Error;pub use error::ErrorLog;pub use error::Result;pub use input::CsvRecordIterator;pub use input::JsonRecordIterator;pub use output::schema_to_json_string;pub use output::write_schema_ddl;pub use output::write_schema_debug_map;pub use output::write_schema_json;pub use output::write_schema_json_schema;pub use output::OutputFormat;pub use schema::bq_schema_to_map;pub use schema::read_existing_schema_from_file;pub use schema::BqMode;pub use schema::BqSchemaField;pub use schema::BqType;pub use schema::EntryStatus;pub use schema::GeneratorConfig;pub use schema::InputFormat;pub use schema::SchemaEntry;pub use schema::SchemaGenerator;pub use schema::SchemaMap;pub use validate::validate_json_data;pub use validate::SchemaValidator;pub use validate::ValidationError;pub use validate::ValidationErrorType;pub use validate::ValidationOptions;pub use validate::ValidationResult;pub use watch::run_watch;pub use watch::WatchConfig;pub use watch::WatchState;
Modules§
- diff
- Schema diff module for comparing BigQuery schemas.
- error
- Error types for the BigQuery schema generator.
- inference
- Type inference for BigQuery schema generation.
- input
- Input readers for JSON and CSV formats.
- output
- Output formatting for BigQuery schema.
- schema
- Schema generation and representation for BigQuery.
- validate
- Schema validation module for validating data against BigQuery schemas.
- watch
- Watch mode for automatic schema regeneration on file changes.
Functions§
- generate_
schema_ from_ csv - High-level function to generate schema from a CSV reader.
- generate_
schema_ from_ json - High-level function to generate schema from a JSON reader.