Crate protoweld

Expand description

§Protoweld

Protoweld is a powerful command-line tool designed to automate the compilation of Protocol Buffer (.proto) files into language-specific code for multiple projects. It simplifies the process of managing and generating protobuf code across different programming languages and projects, making it ideal for microservices architectures and multi-language codebases.

§Features

🚀 Multi-language Support: Compile .proto files to Go, .NET (C#), and Rust
📁 Multi-project Management: Handle multiple projects with different configurations in a single YAML file
🔧 Automatic Dependency Checking: Verifies required tools are installed before compilation
🎯 Flexible Configuration: Customize compilation options per project
🔌 Plugin Support: Configure custom gRPC plugins (e.g., for .NET)
📦 Smart File Organization: Automatically organizes generated code into specified folders
🛠️ Rust-specific Optimizations: Special handling for Rust projects with Tonic and Prost

§Installation

§Standard Installation

Protoweld is available on crates.io and can be installed using Cargo:

cargo install protoweld

After installation, the protoweld binary will be available in your $HOME/.cargo/bin directory (make sure it’s in your PATH).

§Prerequisites

Protoweld requires the following tools to be installed on your system:

Rust and Cargo - Required for installation via cargo install. Install Rust
Protocol Buffers Compiler (protoc) - Required for compiling proto files. Installation Guide

§Building from Source

If you prefer to build from source or want to use the latest development version:

Clone the repository:

git clone https://github.com/XACMJudge/protoweld.git
cd protoweld

Build the project:

cargo build --release

The binary will be available at target/release/protoweld

§Installing Dependencies for Each Language

§Go

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

§.NET

Install the .NET SDK
Install gRPC Tools NuGet package (the plugin path will be in your NuGet packages folder)

§Rust

cargo install protoc-gen-tonic
cargo install protoc-gen-prost

§Configuration

Protoweld uses a YAML configuration file to define projects and their compilation settings. The configuration file structure is as follows:

active_projects:
  - path: <project-name>
    lang: <GoLang|DotNet|Rust>
    associated_proto_files:
      - <path-to-proto-file-1>
      - <path-to-proto-file-2>
    compiled_proto_folder: "<output-folder-path>"
    plugin_path: "<optional-plugin-path>"
    compile_options:
      "<option-key>": "<option-value>"
      "<option-key>": ""

§Configuration Fields

§Required Fields

path (string): A unique identifier for the project
lang (string): Target programming language. Must be one of: GoLang, DotNet, or Rust
associated_proto_files (array of strings): List of paths to .proto files to compile
compiled_proto_folder (string): Output directory where generated code will be placed

§Optional Fields

plugin_path (string): Path to a custom gRPC plugin (required for .NET projects)
compile_options (map): Additional compilation options passed to protoc

§Compile Options

The compile_options field allows you to pass custom flags to the Protocol Buffers compiler. Common options include:

-I or --proto_path: Specify import paths for proto files
--descriptor_set_out: Generate a descriptor set file
--include_imports: Include all imported files in the descriptor set
--experimental_allow_proto3_optional: Enable proto3 optional fields

Note: The output flags (--go_out, --csharp_out, --prost_out, etc.) are automatically handled by Protoweld and should not be specified in compile_options.

§Usage

Run Protoweld with the path to your configuration file:

protoweld -f <path-to-config.yaml>

Or using the short form:

protoweld -f config.yaml

§Example

protoweld -f input/example.yaml

§Example Configuration

Here’s a complete example configuration file (input/example.yaml):

active_projects:
  - path: database-server
    associated_proto_files:
      - ./entities/protos/database-server/operations.proto
    compiled_proto_folder: "./database-server/"
    lang: GoLang
    compile_options:
      "-I": entities

  - path: security
    lang: DotNet
    associated_proto_files:
      - ./entities/protos/security/users.proto
      - ./entities/protos/security/auth.proto
      - ./entities/protos/schemas/security.proto
      - ./entities/protos/database-server/operations.proto
    compiled_proto_folder: "./security/Protos"
    plugin_path: /home/user/.nuget/packages/grpc.tools/2.72.0/tools/linux_x64/grpc_csharp_plugin
    compile_options:
      "-I": entities
      "--descriptor_set_out": ./security/descriptors.pb
      "--include_imports": ""
      "--experimental_allow_proto3_optional": ""

  - path: judge-server
    lang: Rust
    associated_proto_files:
      - ./entities/protos/database-server/operations.proto
    compiled_proto_folder: "./judge-server/protos"
    compile_options:
      "-I": entities

§Supported Languages

§Go (GoLang)

Protoweld compiles .proto files to Go code using:

protoc-gen-go for message types
protoc-gen-go-grpc for gRPC service definitions

Generated Output: Go source files in the specified compiled_proto_folder

§.NET (DotNet)

Protoweld compiles .proto files to C# code using:

protoc with --csharp_out for message types
grpc_csharp_plugin for gRPC service definitions

Requirements:

Must specify plugin_path pointing to the grpc_csharp_plugin executable
Typically located in: ~/.nuget/packages/grpc.tools/<version>/tools/<platform>/grpc_csharp_plugin

Generated Output: C# source files in the specified compiled_proto_folder

§Rust

Protoweld compiles .proto files to Rust code using:

protoc-gen-prost for message types (Prost)
protoc-gen-tonic for gRPC service definitions (Tonic)

Special Features:

Automatically organizes generated files into proper Rust module structure
Creates mod.rs files for each package
Handles file renaming and module imports automatically

Generated Output:

Rust source files organized by package in the specified compiled_proto_folder
Each package gets its own module with proper mod.rs structure

§How It Works

Parsing: Protoweld reads and parses the YAML configuration file
Validation: For each project, it validates that:
- Required dependencies are installed
- Proto files exist and contain valid package declarations
- Output directories can be created
Compilation: For each project, it:
- Assembles the appropriate protoc command with language-specific flags
- Executes the compilation
- Handles language-specific post-processing (especially for Rust)
Output: Generated code is placed in the specified compiled_proto_folder for each project

§Why reorganize the Rust generated files?

Protoweld applies extensive transformations to the Rust-generated files mainly to ensure a smooth developer experience with tools like rust-analyzer and to make the generated code ready for ongoing development, not just for one-off code generation.

By default, when the gRPC plugins (protoc-gen-prost for message types and protoc-gen-tonic for services) generate Rust code from .proto files, they output raw .rs files that closely mirror the original proto package and file structure. However, these raw files often lack a module structure compatible with how Rust and its tooling (such as rust-analyzer and cargo) expect code to be organized. For example, there may be missing mod.rs files, incorrect or missing module imports, and file names that don’t align with idiomatic Rust conventions.

Why does Protoweld reorganize these files?

Rust Tooling Compatibility: Tools like rust-analyzer, and even cargo itself, expect Rust code to be organized into modules using either mod.rs files or the new directory-as-module conventions. The flat file structure output by the plugins does not provide this, making IDE navigation, auto-completion, and refactoring much less effective.
Correct Imports and Modules: The generated files may refer to each other using flat paths, or miss Rust-specific mod declarations entirely, which can break module imports or make it difficult to include generated code in your own crate.
Development-Ready Code: Instead of just dumping generated code, Protoweld restructures it so that you can immediately use the generated modules in your own Rust project without manual adjustments. This includes creating mod.rs files, renaming files to valid Rust identifiers, and setting up imports and visibility declarations.

Summary: Protoweld’s file transformations are primarily driven by the needs of real-world Rust development, ensuring that generated Protobuf code is both compiler- and IDE-friendly, resulting in a much smoother workflow for developers.

§Project Structure

protoweld/
├── Cargo.toml              # Rust project configuration
├── README.md               # This file
├── LICENSE                 # License file
├── input/                  # Example configuration files
│   └── example.yaml       # Example YAML configuration
└── src/                    # Source code
    ├── main.rs            # Entry point
    ├── lib.rs             # Library root
    ├── types/             # Type definitions
    │   ├── mod.rs
    │   └── cli.rs         # CLI argument parsing
    ├── parser/            # YAML configuration parser
    │   ├── mod.rs
    │   ├── protoweld_parser.rs
    │   └── types.rs       # Parser types and structures
    ├── executor/          # Code generation executor
    │   ├── mod.rs
    │   └── protoweld_executor.rs
    ├── compilers/         # Language-specific compilers
    │   ├── mod.rs
    │   ├── protobuf_compiler.rs  # Base compiler trait
    │   ├── shared.rs      # Compiler factory
    │   └── langs_compilers/
    │       ├── mod.rs
    │       ├── compiler_types.rs
    │       ├── go_compiler.rs
    │       ├── dotnet_compiler.rs
    │       └── rust_compiler.rs
    └── os/                # OS abstraction layer
        ├── mod.rs
        ├── types.rs       # OS manager trait
        ├── shared.rs      # OS manager factory
        └── unix_manager.rs

§Error Handling

Protoweld provides clear error messages for common issues:

Missing dependencies: Lists which required tools are not installed
Invalid proto files: Reports proto files missing package declarations
Configuration errors: Validates YAML structure and required fields
Compilation failures: Passes through protoc error messages

§Troubleshooting

§“Failed to check installation of dependencies”

Ensure all required tools for your target language are installed and available in your PATH:

For Go: protoc-gen-go and protoc-gen-go-grpc
For .NET: dotnet and protoc
For Rust: protoc-gen-tonic and protoc-gen-prost

§“Package keyword missing in [proto-file]”

Your .proto file must include a package declaration. Example:

syntax = "proto3";
package mypackage;

message MyMessage {
  // ...
}

This is specially required for Rust projects.

§“The plugin [plugin-name] must have a path in plugin_path option”

For .NET projects, you must specify the plugin_path to the grpc_csharp_plugin executable. Find it in your NuGet packages folder or install gRPC Tools.

§Path Issues

Use relative paths from the configuration file’s location
Ensure proto file paths are correct and files exist
Output folders will be created if they don’t exist

§Future Plans

Protoweld is actively being developed with the following features planned:

§High Priority

Wildcard Support in File Lists: Support glob patterns in associated_proto_files to automatically discover and include proto files (e.g., ./protos/**/*.proto or ./entities/**/*.proto)
Windows Platform Support: Investigate whether Windows-specific code is needed or if Rust’s high-level cross-platform APIs are sufficient. Currently, Protoweld uses Unix-specific implementations, but Rust’s standard library may provide adequate cross-platform abstractions
Additional Language Support: Expand support to more programming languages, including:
- Python (using grpcio-tools)
- Java (using protoc-gen-java and protoc-gen-grpc-java)
- TypeScript/JavaScript (using protoc-gen-ts and @grpc/proto-loader)
- C++ (using protoc-gen-cpp)
- PHP (using protoc-gen-php)
- Ruby (using grpc-tools)
- Swift (using protoc-gen-swift)
Automatic Plugin Discovery: Automatically search for and locate gRPC plugins in common installation locations (e.g., $HOME/.cargo/bin, $HOME/.local/bin, NuGet packages, npm global packages) to reduce manual configuration

§Additional Proposals

Incremental Compilation: Only recompile proto files that have changed since the last run, improving performance for large projects
Watch Mode: Automatically recompile proto files when source files change, similar to cargo watch or file system watchers
Enhanced Error Messages: Provide actionable error messages with suggestions for fixing common issues (e.g., “Did you mean to install protoc-gen-go?”)
Configuration Validation: Validate configuration files before compilation starts, catching errors early
Parallel Compilation: Compile multiple projects in parallel to reduce total build time
Dry-Run Mode: Preview what would be compiled without actually executing the compilation
Verbose/Debug Output: Add detailed logging modes to help debug compilation issues
Configuration Templates: Generate example configuration files for common project structures
Proto Dependency Graph: Analyze and visualize dependencies between proto files
Version Pinning: Support pinning specific versions of protoc and language plugins
CI/CD Integration: Add features specifically for continuous integration environments (e.g., caching, artifact management)
Configuration Inheritance: Allow projects to inherit common settings from a base configuration
Multi-version Support: Support compiling the same proto files to multiple language versions (e.g., Go 1.18 and 1.19)

§Contributing

Contributions are welcome! If you’d like to help implement any of these features or have other ideas, please feel free to submit a Pull Request or open an issue to discuss.

§License

See the LICENSE file for details.

§Acknowledgments

Protoweld is built to simplify Protocol Buffer compilation workflows in multi-language projects, particularly useful for microservices architectures and distributed systems.

Modules§

compilers: Compiler module for language-specific Protocol Buffer code generation.
executor: Executor module for orchestrating proto file compilation across projects.
os: Operating system abstraction layer for cross-platform operations.
parser: Parser module for reading and deserializing Protoweld configuration files.
types: Types module containing CLI argument definitions and related types.

Crate protoweld

Crate protoweld Copy item path

§Protoweld

§Features

§Installation

§Standard Installation

§Prerequisites

§Building from Source

§Installing Dependencies for Each Language

§Go

§.NET

§Rust

§Configuration

§Configuration Fields

§Required Fields

§Optional Fields

§Compile Options

§Usage

§Example

§Example Configuration

§Supported Languages

§Go (GoLang)

§.NET (DotNet)

§Rust

§How It Works

§Why reorganize the Rust generated files?

§Project Structure

§Error Handling

§Troubleshooting

§“Failed to check installation of dependencies”

§“Package keyword missing in [proto-file]”

§“The plugin [plugin-name] must have a path in plugin_path option”

§Path Issues

§Future Plans

§High Priority

§Additional Proposals

§Contributing

§License

§Acknowledgments

Modules§

Crate protoweld