Crate protify

Expand description

Protify is a Rust-first framework for protobuf that generates packages from rust code, with validation included.

It aims to make working with protobuf feel (almost) as easy as using serde. It flips the logic of the typical proto workflow around, so that all the elements of a package can be defined in rust with a rich set of macros and attributes, and the resulting contracts can be generated from the rust code, rather than the other way around.

It also offers a rich validation framework that can be used to programmatically create highly customizable validators that can also be transposed into protobuf options to provide portability to other systems.

ℹ️ NOTE: This readme is generated from the rust documentation, so most of the links will not show up in Github. Read this in the docs.rs page to ensure that links work correctly.

You can visit the package setup section of the guide to learn more about how to set up protify.

§From Rust To Proto

Each element of a protobuf file can be defined with a proc or attribute macro. Options for each element can be composed and assigned programmatically.

Unlike in prost, enums representing protobuf oneofs can be reused for multiple messages (with limitations explained in the reusing oneofs section).

use protify::*;

// Creates a new package
proto_package!(MY_PKG, name = "my_pkg");
// Creates a new file
define_proto_file!(
	MY_FILE,
	name = "my_file.proto",
	package = MY_PKG,
	extensions = [MyExt]
);

fn create_option(value: i32) -> ProtoOption {
	proto_option!("(my_custom_opt)" => value)
}

#[proto_extension(target = MessageOptions)]
pub struct MyExt {
	#[proto(tag = 5000)]
	cool_opt: String,
}

#[proto_service]
enum MyService {
	Service1 { request: MyMsg, response: MyMsg },
}

// We can directly plug this into a tonic handler!
#[proto_message]
#[proto(reserved_numbers(22, 23..30))]
#[proto(reserved_names("name1", "name2"))]
pub struct MyMsg {
	// Programmatically creating options
	#[proto(options = [ create_option(25) ])]
	pub id: i32,
	#[proto(oneof(tags(1, 2)))]
	pub oneof: Option<MyOneof>,
}

#[proto_oneof]
pub enum MyOneof {
	#[proto(tag = 1)]
	A(i32),
	#[proto(tag = 2)]
	B(u32),
}

#[proto_message]
pub struct MyMsg2 {
	pub id: i32,
	// Reusing the same oneof
	#[proto(oneof(tags(1, 2)))]
	pub oneof: Option<MyOneof>,
}

// Tags are assigned automatically
// and take the reserved numbers in consideration
#[proto_enum]
#[proto(reserved_numbers(20, 25..30))]
pub enum MyEnum {
	Unspecified,
	A,
	B,
}

For a full guide on how to set up a package, visit the package setup section.

§Proxies As A Translation Layer

Messages and oneofs can be proxied. Doing so will generate a new struct/enum with the same name, followed by a Proto suffix (i.e. MyMsg -> MyMsgProto).

Proxied messages/oneofs unlock the following features:

A field/variant can be missing from the proto struct, but present in the proxy (akin to the skip attribute with serde)
Enums can be represented with their actual rust enum type, rather than being pure integers
Oneofs don’t need to be wrapped in Option
Messages don’t need to be wrapped in Option
We can use types that are not supported by prost and map them with custom conversions

By default, the macro will generate a conversion from proxy to proto and vice versa that just calls .into() for each field/variant. So if the field’s prost type implements From with the proxy field and vice versa, no additional attributes are required.

To provide custom conversions, you can use the from_proto and into_proto attributes on the container (to replace the automatically generated impl as a whole) or on individual fields/variants.

The proxied structs/enums will also implement ProxiedMessage or ProxiedOneof, whereas the proxies will implement MessageProxy and OneofProxy.

use protify::*;
use std::sync::Arc;

proto_package!(MY_PKG, name = "my_pkg");
define_proto_file!(MY_FILE, name = "my_file.proto", package = MY_PKG);

// Generates a MsgProto struct that is prost-compatible
#[proto_message(proxied)]
pub struct Msg {
	// Requires setting the type manually as the type
	// is not prost-compatible
	#[proto(string)]
	// Must provide a custom `into_proto` impl because `Arc<str>` does not support `Into<String>`
	#[proto(into_proto = |v| v.as_ref().to_string())]
	pub name: Arc<str>,
	// Ignored field. Conversion from proto will use `Default::default()` unless a custom
	// conversion is specified
	#[proto(ignore)]
	pub rust_only: i32,
	// In proxied messages, we can use `default` for oneofs
	// so that using `Option` is not required.
	// The default conversion will call `ProxiedOneofProto::default().into()`
	// if the field is `None` in the proto struct.
	#[proto(oneof(proxied, default, tags(1, 2)))]
	pub oneof: ProxiedOneof,
	// We can do the same for messages too
	#[proto(message(default))]
	pub message_with_default: Msg2,
	// We can use the enum directly as the type
	#[proto(enum_)]
	pub enum_: TestEnum,
}

#[proto_enum]
pub enum TestEnum {
	Unspecified,
	A,
	B,
}

// Direct implementation. The prost attributes will be directly
// injected in it
#[proto_message]
pub struct Msg2 {
	pub id: i32,
	// In direct impls, enums are just integers
	#[proto(enum_(TestEnum))]
	pub enum_: i32,
}

// Generates the `ProxiedOneofProto` enum
#[proto_oneof(proxied)]
pub enum ProxiedOneof {
	#[proto(string, tag = 1, into_proto = |v| v.as_ref().to_string())]
	A(Arc<str>),
	#[proto(tag = 2)]
	B(u32),
}

impl Default for ProxiedOneofProto {
	fn default() -> Self {
		Self::B(1)
	}
}

fn main() {
	let msg = MsgProto::default();
	// Using the `ProxiedMessage` trait
	let proxy = msg.into_proxy();
	// Using the `MessageProxy` trait
	let msg_again = proxy.into_message();
}

§Interacting With Databases

An important benefit that comes from having a “rust-first” approach when defining our models is that they can easily be used for operations such as db queries, without needing to create separate structs to map to the generated protos, or injecting the attributes as plain text with the prost-build helper, which can be unergonomic and brittle.

And with proxies, the interactions with a database becomes even easier, because we can have the proto-facing struct with a certain shape, while the proxy can represent the state of a message after its data has been mapped, for example, to an item queried from the database.

You can take a look at the test-server crate in the repo for an example of database interaction in a tonic handler.

use diesel::prelude::*;
use protify::proto_types::Timestamp;
use protify::*;

proto_package!(DB_TEST, name = "db_test", no_cel_test);
define_proto_file!(DB_TEST_FILE, name = "db_test.proto", package = DB_TEST);

mod schema {
	diesel::table! {
	  users {
		id -> Integer,
		name -> Text,
		created_at -> Timestamp
	  }
	}
}

// If we want to use the message as is for the db model
#[proto_message]
#[derive(Queryable, Selectable, Insertable)]
#[diesel(table_name = schema::users)]
#[diesel(check_for_backend(diesel::sqlite::Sqlite))]
pub struct User {
	#[diesel(skip_insertion)]
	pub id: i32,
	pub name: String,
	#[diesel(skip_insertion)]
	// We need this to keep `Option` for this field
	// which is necessary for protobuf
	#[diesel(select_expression = schema::users::columns::created_at.nullable())]
	#[proto(timestamp)]
	pub created_at: Option<Timestamp>,
}

// If we want to use the proxy as the db model, for example
// to avoid having `created_at` as `Option`
#[proto_message(proxied)]
#[derive(Queryable, Selectable, Insertable)]
#[diesel(table_name = schema::users)]
#[diesel(check_for_backend(diesel::sqlite::Sqlite))]
pub struct ProxiedUser {
	#[diesel(skip_insertion)]
	pub id: i32,
	pub name: String,
	#[diesel(skip_insertion)]
	#[proto(timestamp, from_proto = |v| v.unwrap_or_default())]
	pub created_at: Timestamp,
}

fn main() {
	use schema::users::dsl::*;

	let conn = &mut SqliteConnection::establish(":memory:").unwrap();

	let table_query = r"
    CREATE TABLE users (
      id INTEGER PRIMARY KEY AUTOINCREMENT,
      name TEXT NOT NULL,
      created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
      );
    ";

	diesel::sql_query(table_query)
		.execute(conn)
		.expect("Failed to create the table");

	let insert_user = User {
		id: 0,
		name: "Gandalf".to_string(),
		created_at: None,
	};

	diesel::insert_into(users)
		.values(&insert_user)
		.execute(conn)
		.expect("Failed to insert user");

	let queried_user = users
		.filter(id.eq(1))
		.select(User::as_select())
		.get_result(conn)
		.expect("Failed to query user");

	assert_eq!(queried_user.id, 1);
	assert_eq!(queried_user.name, "Gandalf");
	// The timestamp will be populated by the database upon insertion
	assert_ne!(queried_user.created_at.unwrap(), Timestamp::default());

	let proxied_user = ProxiedUser {
		id: 0,
		name: "Aragorn".to_string(),
		created_at: Default::default(),
	};

	diesel::insert_into(users)
		.values(&proxied_user)
		.execute(conn)
		.expect("Failed to insert user");

	let queried_proxied_user = users
		.filter(id.eq(2))
		.select(ProxiedUser::as_select())
		.get_result(conn)
		.expect("Failed to query user");

	assert_eq!(queried_proxied_user.id, 2);
	assert_eq!(queried_proxied_user.name, "Aragorn");

	// Now we have the message, with the `created_at` field populated
	let msg = queried_proxied_user.into_message();

	assert_ne!(msg.created_at.unwrap(), Timestamp::default());
}

§Validating Content

Whenever models or API contracts are defined and used, the need for validation follows close behind.

Because of this, protify ships with a validation framework that integrates validation logic with schema definitions.

The implementors of Validator hold two roles at the same time: on the one hand, they handle the validation logic on the rust side, and on the other hand, they can also produce a schema representation, so that their settings can be represented as options in a protobuf file, so that they can be ported between different applications.

All the provided validators map their options to the protovalidate options, but you can also create customized validators that map to other custom options.

Because every validator is type-safe and comes with an ergonomic builder methods to be built with, defining validators becomes a vastly superior experience than manual composition in protobuf files, where the process is repetitive, rigid, and lacking essential features of modern programming such as type safety and LSP integration, as well as programmatic composition.

Validators can be assigned to oneofs/messages as a whole or to individual fields/variants to be incorporated in thier validation logic.

use protify::*;
use std::collections::HashMap;

proto_package!(MY_PKG, name = "my_pkg");
define_proto_file!(MY_FILE, name = "my_file.proto", package = MY_PKG);

// We can define logic to programmatically compose validators
fn prefix_validator(prefix: &'static str) -> StringValidator {
	StringValidator::builder().prefix(prefix).build()
}

#[proto_message]
// Top level validation using a CEL program
#[proto(validate = |v| v.cel(cel_program!(id = "my_rule", msg = "oopsie", expr = "this.id == 50")))]
pub struct MyMsg {
	// Field validator
	// Type-safe and lsp-friendly!
	// The argument of the closure is the IntValidator builder,
	// so we are going to get autocomplete suggestions
	// for its specific methods.
	#[proto(validate = |v| v.gt(0))]
	pub id: i32,

	// Repeated validator
	#[proto(validate = |v| v.items(|i| i.gt(0)))]
	pub repeated_nums: Vec<i32>,

	// Map validator
	#[proto(validate = |m| m.keys(|k| k.gt(0)).values(|v| v.min_len(5)))]
	pub map_field: HashMap<i32, String>,

	#[proto(oneof(tags(1, 2)))]
	#[proto(validate = |v| v.required())]
	pub oneof: Option<MyOneof>,
}

#[proto_oneof]
pub enum MyOneof {
	#[proto(tag = 1)]
	// Same thing for oneof variants
	#[proto(validate = |v| v.gt(0))]
	A(i32),
	// Multiple validators, including a programmatically built one!
	#[proto(tag = 2, validate = [ |v| v.min_len(5), prefix_validator("abc") ])]
	B(String),
}

ℹ️ NOTE: Validators that are provided via closures will be cached in a Lazy struct (normally a LazyLock or a wrapper for OnceBox in a no_std environment), so they are only initialized once (except for the OneofValidator which is small enough).

§Custom Validators

The Validator trait allows for the construction of custom validators.

A validator can be a struct (stateful) or just a function, wrapped with the from_fn helper.

Each validator only needs to implement a single method, execute_validation, which receives a ValidationCtx and an Option of the target type. All the other methods are automatically derived.

The schema and check_consistency methods can be optionally implemented, as described later in this section and in the correctness section.

use protify::*;

proto_package!(MY_PKG, name = "my_pkg");
define_proto_file!(MY_FILE, name = "my_file.proto", package = MY_PKG);

// Function validator
fn validate_number(ctx: &mut ValidationCtx, val: Option<&i32>) -> ValidationResult {
	let mut is_valid = IsValid::Yes;

	if val.is_none() {
		// IsValid is a boolean-like enum, so it can be used with
		// bit operators
		is_valid &= ctx.add_required_violation(
			// Optionally, we can provide a custom error message
			Some("number must be set".to_string()),
		)?; // Validators use `Result`s to handle `fail-fast` scenarios
		// and trigger early exit
	}

	Ok(is_valid)
}

pub struct CustomValidator;

// If a validator contains some heavy or complex logic,
// it can be initialized once and then reused.
// This is done by default for validators defined with closures in attributes.
static CACHED_VALIDATOR: Lazy<CustomValidator> = Lazy::new(|| CustomValidator);

impl Validator<MyMsg> for CustomValidator {
	type Target = MyMsg;

	fn execute_validation(
		&self,
		ctx: &mut ValidationCtx,
		val: Option<&Self::Target>,
	) -> ValidationResult {
		validate_number(ctx, val.map(|v| &v.id))
	}
}

#[proto_message]
// Using the custom validators at the top level
#[proto(validate = [ CustomValidator, *CACHED_VALIDATOR ])]
pub struct MyMsg {
	// Using a function validator
	#[proto(validate = from_fn(validate_number))]
	pub id: i32,

	#[proto(oneof(tags(1, 2)))]
	// Using a variety of closure and custom validators
	#[proto(validate = [ |v| v.required(), CustomValidator, *CACHED_VALIDATOR ])]
	pub oneof: Option<MyOneof>,
}

impl Validator<MyOneof> for CustomValidator {
	type Target = MyOneof;

	fn execute_validation(
		&self,
		ctx: &mut ValidationCtx,
		val: Option<&Self::Target>,
	) -> ValidationResult {
		if let Some(oneof) = val
			&& let MyOneof::A(int) = oneof
		{
			validate_number(ctx, Some(&int))
		} else {
			Ok(IsValid::Yes)
		}
	}
}

#[proto_oneof]
// Same thing for oneofs
#[proto(validate = [ CustomValidator, *CACHED_VALIDATOR ])]
pub enum MyOneof {
	#[proto(tag = 1, validate = from_fn(validate_number))]
	A(i32),
	#[proto(tag = 2)]
	B(u32),
}

You can have a look at the testing crates in the repo for more complex examples of custom validator usage.

§Customizing Error Messages

In order to facilitate things like i18n, every provided validator allows for customization of the error messages, without requiring a whole custom validator to be designed purely for this purpose.

Each builder features a method called with_error_messages, which accepts a BTreeMap that maps the violations that it may produce to error messages.

use protify::*;
use protify::proto_types::{
  protovalidate::violations_data::*,
};
use std::collections::BTreeMap;
use maplit::btreemap;

let validator = StringValidator::builder().min_len(3).with_error_messages({
    let locale = std::env::var("LOCALE").unwrap_or("en/GB".to_string());
    let error_message = if locale == "it/IT" {
        "il nickname deve contenere almeno 3 caratteri"
    } else {
        "the nickname must be at least three characters long"
    };

    btreemap!{
        StringViolation::MinLen => error_message
    }
}).build();

let violations = validator.validate("a").unwrap_err().into_violations();

assert_eq!(violations[0].message(), "the nickname must be at least three characters long");

§Schema Representation

In order to make validation settings portable, each validator can optionally implement the schema method, which outputs a ProtoOption that will be added to the receiving message/oneof in the proto file.

All default validators implement this method and output the options in the protovalidate format.

use indoc::indoc;
use protify::*;

proto_package!(MY_PKG, name = "my_pkg");
define_proto_file!(MY_FILE, name = "my_file.proto", package = MY_PKG);

#[proto_message]
pub struct MyMsg {
	#[proto(validate = |v| v.min_len(3).max_len(32))]
	pub name: String,
}

fn main() {
	let schema = MyMsg::proto_schema();

	assert_eq!(
		MyMsg::proto_schema().render_schema().unwrap(),
		indoc! {"
            message MyMsg {
              string name = 1 [
                (buf.validate.field) = {
                  string: {
                    min_len: 3,
                    max_len: 32
                  }
                }
              ];
            }"
		}
	)
}

§Protovalidate Support

The best way to use protify is to define a schema from scratch using rust code. However, it is also possible to generate validators that are defined in pre-built proto files using the protovalidate syntax. You can learn more about this in the reflection usage section.

§Feature Flags

tonic — Enables direct conversion from validation errors to tonic::Status.
serde — Enables serde for all schema representations.
inventory (enabled by default) — Enables automatic collection for elements of a package. (Required std)
chrono (enabled by default) — Enables timestamp now features in CEL and timestamp validation.
chrono-wasm — Enables usage of wasmbind for chrono’s now methods.
std (enabled by default) — Enables the std library features.
common-types — Enables schema features for google.type types.
rpc-types — Enables schema features for google.rpc types.
reflection — Enables usage with reflection, to generate validation logic from pre-built protos with protovalidate annotations.
cel (enabled by default) — Enables CEL validation.
regex (enabled by default) — Enables regex-based validators.

Re-exports§

pub use prost;
pub use ::cel;cel
pub use proto_types;
pub use macros::*;
pub use validators::*;

Modules§

guide: Guide
macros
validators

Macros§

cel_program: Defines a new CelProgram.
inherit_proto_file: Brings a pre-defined proto file handle in scope so that it can be picked up by the proto items defined in the module where it’s called.
option_list: This macro can be used to create an OptionValue::List from an iterator of items that implement Into OptionValue.
option_message: This macro can be used to create an object-like protobuf option. It follows the syntax of crates like maplit, for creating key-value pairs.
proto_option: This macro can be used to generate a ProtoOption with a concise syntax.
use_proto_file: Brings a pre-defined proto file handle in scope so that it can be picked up by the proto items defined in the module where it’s called.

Structs§

Bytes: A cheaply cloneable and sliceable chunk of contiguous memory.
EnumSchema: A struct representing a protobuf enum.
EnumSchemaBuilder: Use builder syntax to set the inputs and finish with build().
EnumVariant: A struct representing a protobuf enum variant.
EnumVariantBuilder: Use builder syntax to set the inputs and finish with build().
Extension: A struct representing a protobuf Extension.
ExtensionBuilder: Use builder syntax to set the inputs and finish with build().
Field: Struct that represents a protobuf field.
FieldBuilder: Use builder syntax to set the inputs and finish with build().
FieldError: Represents an error that occurred in a validator defined at the field level.
FieldPathElement: FieldPathElement provides enough information to nest through a single protobuf field.
FileImports: HashSet wrapper for a file’s imports. Skips insertion if the file is equal to the origin file.
FileReference
Lazystd: A value which is initialized on the first access.
MessageSchema: A struct that represents a protobuf message.
MessageSchemaBuilder: Use builder syntax to set the inputs and finish with build().
Oneof: Schema representation for a protobuf oneof.
OneofBuilder: Use builder syntax to set the inputs and finish with build().
OptionList: An list-like protobuf option.
OptionMessage: An object-like protobuf option, with keys and values.
OptionMessageBuilder: A builder for building an OptionMessage with a map-like syntax.
OverlappingListsError: An error that occurs when items are present in lists of allowed and forbidden values at the same time.
Package: A struct representing a protobuf package.
ProtoFile: Struct that represents a protobuf file and its contents.
ProtoOption: A struct representing a protobuf option.
ProtoPath: A struct that represents a path to an item (message or enum) in a protobuf package.
Service: A struct representing a protobuf service.
ServiceBuilder: Use builder syntax to set the inputs and finish with build().
ServiceHandler: A struct that represents a protobuf service handler.
ServiceHandlerBuilder: Use builder syntax to set the inputs and finish with build().
TestError: The context of an error that occurred while running a correctness check during testing.
ValidatorSchema: A schema representation for a validator.
Violations: Violations is a collection of Violation messages. This message type is returned by Protovalidate when a proto message fails to meet the requirements set by the Rule validation rules. Each individual violation is represented by a Violation message.

Enums§

ConsistencyError: An error that can occur while checking the correctness of the inputs for validators.
Edition: The protobuf edition for a file.
ExtensionTarget: Valid extension targets for a Proto3 file.
FieldType: The types in which a protobuf field can be represented.
MessageEntry: An entry in a message, which can be a field or a oneof containing several fields.
OptionValue: An enum representing values for protobuf options.
ProtoMapKey: An enum representing the types that are supported as map keys in protobuf.
ProtoScalar: An enum representing the scalar types in protobuf.
ProtoType: An enum representing a protobuf type.

Traits§

AsProtoField: A trait that specifies the protobuf field type to which a rust type should be mapped (i.e. single, optional, repeated, map).
AsProtoMapKey: A sealed trait that defines the kind of protobuf map key that a rust type maps to.
AsProtoType: A trait that specifies the protobuf type to which a rust type should be mapped.
FileSchema: Trait that can generate a ProtoFile.
IntoBytes: Helper trait to convert a type to Bytes.
IntoBytesRegexregex: Utility trait to create a Regex.
IntoRegexregex: Utility trait to create a Regex.
MessagePath: A trait that returns the protobuf path of a given message.
MessageProxy: Implemented for message proxies by the proto_message macro.
OneofProxy: Implemented for oneof proxies by the proto_oneof macro.
PackageSchema: The trait that generates a Package instance.
ProtoEnum: Trait for enums with a protobuf representation.
ProtoEnumSchema: Trait for enums with a protobuf representation.
ProtoExtension: Implemented by the proto_extension macro.
ProtoMessage: A trait that is responsible for generating a protobuf schema representation for a rust struct.
ProtoOneof: Trait responsible for generating the schema representation of a oneof.
ProtoService: A trait that enables the generation of a protobuf service schema.
ProxiedMessage: Implemented for proxied messages by the proto_message macro.
ProxiedOneof: Implemented for proxied oneofs by the proto_oneof macro.
ValidatedOneof: Trait responsible for executing validators that have been assigned to a oneof.

Crate protify

Crate protify Copy item path

§From Rust To Proto

§Proxies As A Translation Layer

§Interacting With Databases

§Validating Content

§Custom Validators

§Customizing Error Messages

§Schema Representation

§Protovalidate Support

§Feature Flags

Re-exports§

Modules§

Macros§

Structs§

Enums§

Traits§

Crate protify