Crate serde_avro_fast

Getting started

let schema: serde_avro_fast::Schema = r#"
{
	"namespace": "test",
	"type": "record",
	"name": "Test",
	"fields": [
		{
			"type": {
				"type": "string"
			},
			"name": "field"
		}
	]
}
"#
.parse()
.expect("Failed to parse schema");

#[derive(serde_derive::Serialize, serde_derive::Deserialize, Debug, PartialEq)]
struct Test<'a> {
	field: &'a str,
}

let rust_value = Test { field: "foo" };
let avro_datum = &[6, 102, 111, 111];

// Avro datum deserialization
assert_eq!(
	serde_avro_fast::from_datum_slice::<Test>(avro_datum, &schema)
		.expect("Failed to deserialize"),
	rust_value
);

// Avro datum serialization
assert_eq!(
	serde_avro_fast::to_datum(
		&rust_value,
		Vec::new(),
		&mut serde_avro_fast::ser::SerializerConfig::new(&schema)
	)
	.expect("Failed to serialize"),
	avro_datum
);

Object container file encoding

Otherwise called “avro files”, avro object container files contain a header that holds the schema, followed by an arbitrary number of avro objects.

For this use-case, please see the object_container_file_encoding module documentation.

Deriving schema from Rust structs

If the Rust program is the source of truth for the schema definition, it is useful to define the schema as a derive on the relevant Rust structs. This can be achieved using the serde_avro_derive crate:

use serde_avro_derive::BuildSchema;

#[derive(BuildSchema)]
struct Foo {
	primitives: Bar,
}

#[derive(BuildSchema)]
struct Bar {
	a: i32,
	b: String,
}

let schema: serde_avro_fast::Schema = Foo::schema()?;

// This will generate the following schema:
let _schema_str = r#"{
  "type": "record",
  "name": "crate_name.path.to.Foo",
  "fields": [{
    "name": "primitives",
    "type": {
      "type": "record",
      "name": "Bar",
      "fields": [
        { "name": "a", "type": "int" },
        { "name": "b", "type": "string" }
      ]
    }
  }]
}"#;

See the serde_avro_derive documentation for more details.

An idiomatic (re)implementation of serde/avro (de)serialization

At the time of writing, the other existing libraries for Avro (de)serialization do tons of unnecessary allocations, HashMap lookups, etc… for every record they encounter.

This version is a more idiomatic implementation, both with regards to Rust and to serde.

It is consequently >10x more performant (cf benchmarks):

apache_avro/small       time:   [386.57 ns 387.04 ns 387.52 ns]
serde_avro_fast/small   time:   [19.367 ns 19.388 ns 19.413 ns] <- x20 improvement

apache_avro/big         time:   [1.8618 µs 1.8652 µs 1.8701 µs]
serde_avro_fast/big     time:   [165.87 ns 166.92 ns 168.09 ns] <- x11 improvement

Re-exports

pub use schema::Schema;

Modules

de

Defines everything necessary for avro deserialization

object_container_file_encoding

Support for object container files

schema

Navigate, modify and initialize the Schema

ser

Defines everything necessary for avro serialization

Functions

from_datum_reader

Deserialize from an avro “datum” (raw data, no headers…) impl BufRead

from_datum_slice

Deserialize from an avro “datum” (raw data, no headers…) slice

from_single_object_reader

Deserialize from an avro single object encoding impl BufRead

from_single_object_slice

Deserialize from an avro single object encoding slice

to_datum

Serialize an avro “datum” (raw data, no headers…)

to_datum_vec

Serialize an avro “datum” (raw data, no headers…)

to_single_object

Serialize to an avro single object encoding

to_single_object_vec

Serialize to an avro single object encoding