Module serializers 

Flatbuffer serialization. Flatbuffer serialization for Icechunk metadata.

How serializers work:

• Main goal is to make sure newer version of Icechunk can read metadata files created using older versions. In this way, a repository can evolve during its life. As users upgrade their Icechunk versions they don’t need to migrate their data.

• Of course we may choose to limit backwards compatibility after certain number of versions or a time limit.

• Performance is critical, so we cannot copy much data around during the process of serialization/deserialization

• For serialization:

  • We define a new XSerializer for each metadata file type X. Example: SnapshotSerializer.
  • This type implement serde::Serialize
  • This type holds only references to the same fields as X
  • This type implements From<&X> (notice by reference) Example:

    impl<'a> From<&'a Snapshot> for SnapshotSerializer<'a> {
    ...
    }

  • Because the serializer only holds references it’s essentially free to call snapshot.into() to get one.
  • Then this object is serialized using serde.

• For deserialization:

  • We define a new XDeserializer for each metadata file type X. Example: SnapshotDeserializer.
  • This type implement serde::Deserialize
  • This type holds the same fields as X by value
  • X implements From<XDeserializer> (notice by value). Example:

     impl From<SnapshotDeserializer> for Snapshot {
     ...
     }

  • Because the deserializer can be destructed and X implements From, it’s essentially free to call obtain the original type X
  • Then this new type XDeserializer is deserialized using serde and converted with into.

• serializers.current.rs holds all the serializers and deserializers for the current version of the spec

• serializers.version_foo.rs holds all the serializers and deserializers for version foo of the spec

• The serializers module root has functions serialize_X and deserialize_X that take a spec version number and use the right (de)-serializer to do the job.

Constants

MAX_APPARENT_SIZE_INFLATION

Upper bound on how much larger the verifier’s apparent-size counter runs than the buffer it walks, over every element type we serialize.

MIN_BYTES_PER_TABLE

Lower bound on the buffer bytes one flatbuffers table costs: a 4 byte soffset plus its 4 byte slot in the enclosing vector. Vtables are deduplicated by the builder, so they don’t count towards the floor.

VERIFY_THRESHOLD_BYTES

Files at or above this size should be verified before being written.

Functions

deserialize_manifest

deserialize_repo_info

deserialize_snapshot

deserialize_transaction_log

serialize_manifest

serialize_repo_info

serialize_snapshot

serialize_transaction_log

verify_buffer

Check that buffer can be read back as file_type.