Protocheck

Protocheck is a crate that allows you to leverage protovalidate annotations to automatically generate validation logic for the structs generated from your protobuf messages.

This allows you to define your validation schemas only once, directly in your protobuf files, and then use libraries such as this one or others like protovalidate-es in the Typescript ecosystem to execute the validation logic.

Getting started

To get started, you need to use protocheck-build as a build dependency in your crate, which will use protocheck-proc-macro to add all the validation logic to your structs. The setup will look more or less like this (this is taken directly from the tests crate)

// In your build.rs file
use prost_build::Config;
use protocheck_build::compile_protos_with_validators;

fn main() -> Result<(), Box<dyn std::error::Error>> {
  println!("cargo:rerun-if-changed=proto/");
  println!("cargo:rerun-if-changed=proto_deps/");

  let out_dir = std::path::PathBuf::from(std::env::var("OUT_DIR").expect("Could not find OUT_DIR"));
  let final_descriptor_path = out_dir.join("file_descriptor_set.bin");

  let proto_include_paths = &["proto", "proto_deps"];

  let proto_files = &["proto/myapp/v1/tests.proto"];

  let mut config = Config::new();
  config
    .file_descriptor_set_path(final_descriptor_path.clone())
    // Enable the use of bytes::Bytes for `bytes` fields
    .bytes(["."])
    .out_dir(out_dir.clone());

  // Call the build helper
  compile_protos_with_validators(&mut config, proto_files, proto_include_paths, &["myapp.v1"])?;

  // Compile protos
  config.compile_protos(proto_files, proto_include_paths)?;

  // Set the env for the file descriptor location
  println!(
    "cargo:rustc-env=PROTO_DESCRIPTOR_SET={}",
    final_descriptor_path.display()
  );

  Ok(())
}

compile_protos_with_validators takes these arguments:

1. The config struct.
2. The proto files and include paths being used by the config.
3. The list of packages to apply validators to. If a given message contains Cel validation or is validated as a field by another message, its package name must be included in this list.

The function will then create an intermediary descriptor, iterate its messages, and use the information extracted from them to add the derives and attributes to the actual config that are needed by protocheck-proc-macro to add the validation logic.

Noteworthy features

1. It does not require reflection, except at build time.

This is a major benefit for two reasons.

First, it removes the need to include a reflection library in the consuming app's binary.

And, most importantly, it avoids the overhead that is introduced by using reflection to determine the structure of a message when validating it.

Rather than using reflection, this crate leverages the TryIntoCelValue derive macro to generate a method called try_into_cel_value which will directly convert any given struct (or field) into the appropriate cel::Value (only failing in case of a Duration or Timestamp field being out of the allowed range for chrono types).

2. It uses native rust code for validation except for custom Cel rules.

Unlike other similar libraries, all of the standard validators are implemented in rust code. This means that the cel interpreter (provided by the cel crate) is used only for custom rules explicitely defined in Cel, and can be disabled altogether if custom rules are not used.

3. Extra safety checks for rules definitions

Because of human error, some of these situations may arise:

• A list of allowed values and a list of forbidden values have some items in common
• A string field is trying to use BytesRules
• An "enum.in" rule (list of allowed values for an enum field) includes values that are not part of that enum
• An "lt" rule (meaning 'less than') specifies a value that is smaller than the "gt" (greater than) rule
• ... and other corner cases

This crate handles these situations by emitting a compilation error, which will report the specific field (and the specific values) involved in the error.

Example:

message Oopsie {
  string mystring = 1 [(buf.validate.field).string = {
    min_len: 10
    max_len: 2
  }];
}

Error message:

error: Error for field myapp.v1.Oopsie.mystring: min_len cannot be larger than max_len

4. Strenghtened compile-time safety for Cel programs

When the protobuf_validate proc macro is being processed, it will attempt to create a test case for any given Cel expression being used, generating some default values for the given message or field and trying to execute a Cel program with those defaults. This ensures that if a Cel expression is fundamentally invalid (for example for a type mismatch), the error will be caught at compile time and not at runtime. (With some caveats explained below)

5. Lazy initialization

All Cel programs are generated using LazyLock so they are only initialized once. The same thing goes for other static elements being used in the validators, such as regexes or allowed/forbidden list of values (more on that below).

6. Smart list validation

When a list rule is defined ("in" or "not_in"), the crate will generate two kinds of validators.

If the list is up to 15 elements long, it will generate a validator that uses a slice to check for presence.

If the list is longer than that, it will instead use a HashSet (also initialized in a LazyLock) containing the list of values, to ensure faster lookup times.

How to validate messages

After the validate method has been added to a struct, validating it is as simple as calling my_struct.validate().

The validate method returns a Result<(), Violations>, where the Violations struct contains a vector of individual Violation elements, which contain the context behind a given validation error, such as the parent messages (if the field was part of a nested message) of the invalid field, along with the error message and the rule id for that given rule.

Both Violations and the invidivual Violation structs come with several utility methods, such as violation_by_rule_id, which allows you to select a particular violation from the list, or field_path_str, which conveniently takes a list of FieldPathElement and turns it into a single string path such as person.friends.0.address.street_name.

The protocheck-proc-macro crate also adds a generic trait [ProtoValidator] that calls the validate method, which can be used to develop things like a tower::Layer that accepts any struct with a validator, calls the method and returns the result.

Example:

message JediFight {
  Anakin anakin = 1;
  ObiWan obi_wan = 2;
}

message ObiWan {
  bool has_high_ground = 1 [(buf.validate.field).bool.const = true];
}

message Anakin {
  bool has_high_ground = 1 [(buf.validate.field).bool.const = false];
}

#[test]
fn example() {
  let obi_wan = ObiWan {
    has_high_ground: false,
  };
  let anakin = Anakin {
    has_high_ground: true,
  };
  let jedi_fight = JediFight {
    anakin: Some(anakin),
    obi_wan: Some(obi_wan),
  };

  let Violations { violations } = jedi_fight.validate().unwrap_err();

  violations.iter().for_each(|v| {
    println!(
      "Field path: {}, Error message: {}",
      v.field_path_str().unwrap(),
      v.message()
    )
  });
}

Output:

Field path: anakin.has_high_ground, Error message: must be equal to false
Field path: obi_wan.has_high_ground, Error message: must be equal to true

The tests crate contains many other examples of validation schemas being implemented.

Custom validation with Cel

With the cel feature (enabled by default), you can also specify some validation rules defined with the Cel syntax, which can be applied to entire structs or to singular fields.

Example:

Let's change the above to this:

message ObiWan {
  // Message-level validation
  option (buf.validate.message).cel = {
    id: "obi-wan.high_ground"
    message: "obi-wan must have the high ground."
    expression: "this.has_high_ground == true"
  };

  // Field-level validation
  bool has_high_ground = 1 [(buf.validate.field).cel = {
    id: "obi-wan.high_ground"
    message: "obi-wan must have the high ground."
    expression: "this == true"
  }];
}

Now the output will be:

Field path: obi_wan, Error message: obi-wan must have the high ground.
Field path: obi_wan.has_high_ground, Error message: obi-wan must have the high ground.

This is particularly useful for cases when we need to apply message-wide validation which analyzes more than one field at a time.

The typical example would be:

message User {
  option (buf.validate.message).cel = {
    id: "passwords_match"
    message: "the two passwords do not match"
    expression: "this.password == this.confirm_password"
  };

  string password = 1;
  string confirm_password = 2;
}

let user = User {
  password: "abc".to_string(),
  confirm_password: "abcde".to_string(),
};

let Violations { violations } = user.validate().unwrap_err();

println!(
  "Field path: {}, Error message: {}",
  // Message-wide violations do not have a field path unless they are nested in another message  
  // But they do have a rule id
  violations[0].rule_id(),
  violations[0].message()
);

Outcome:

Field path: passwords_match, Error message: the two passwords do not match

Caveats

• While the compile-time check for the validity of a Cel expression helps to catch most if not all errors relative to the Cel program compilation and execution, it is still very encouraged to have some tests that trigger the validation logic at runtime (it's just as easy as calling .validate() once again) to be absolutely sure that the Cel program is not causing any issues.

This is because the Cel validation function can obviously not panic and crash the whole app if a Cel program failed to execute, so it will just return a generic error to the user while logging the actual error.

This means that if there is an unattended error, then it would silently keep generating these generic and unhelpful error messages for users until it would be reported or noticed in the logs.

But the good news is that the compile time check prevents the majority of these situations, and adding a very simple test on top of that can eradicate that problem entirely.

• For certain cases where the instructions are conflicting but could be intentional, such as using the "const" rule for a field while also having other validators, the other rules will simply be ignored and no error will be shown. This is to allow for cases when you want to have a temporary override for a field's validation without needing to remove the other validators.

• Validation for bytes fields only works when using bytes::Bytes as the rust type for them.

• The types for the well known protobuf messages must be imported from proto-types (re-exported in this crate in the [types] module). These are based on the prost-types implementation, with some extra helpers and methods that make validation smoother or even possible at all in some cases.

compile_protos_with_validators automatically takes care of calling compile_well_known_types and assigning all of the .google.protobuf types to the ones defined in proto-types. The same thing goes for the types belonging to the protovalidate specification.