Crate swh_mosaic 

swh-mosaic: MOdular Storage of Archived and Indexed Contents from Software Heritage

MOSAIC is a file format designed to efficiently store and randomly read contents archived by Software Heritage. Target content is source code and therefore small objects (median size: 3kb), indexed over one (or more) of the possible objstorage keys.

The motivations and design of this file format are thoroughly explained in SWH Enhancement Proposal #5. Format evolutions are described in CHANGELOG.md in the package’s sources. The format is described in EBML but this reference implementation is a bit more constrained than what EBML can describe.

Installing with cargo install swh-mosaic provides a binary to read, write or check MOSAIC files, use swh-mosaic --help to see its detailed help. The crate also provides libraries for lower-level access.

Example use

fn main() -> anyhow::Result<()> {
    use swh_mosaic::creator::MosaicCreator;
    use swh_mosaic::reader::MmapMosaicReader;
    use swh_mosaic::IdxDescription;
    use tempfile::TempDir;

    let temp_dir = TempDir::new()?;
    let mosaic_path = temp_dir.path().join("demo.mosaic");
    let comment = String::from("This is a MOSAIC demo");
    let object = "print('Hello World!')".as_bytes().to_vec();
    let object_sha1 = vec![1u8; 20];
    let object_sha256 = vec![1u8; 32];

    let compression_level = if false {
        // copy objects directly to the Mosaic file
        None
    } else {
        // or, have MosaicCreator compress each object at this zstd level:
        Some(3)
    };

    // When creating a file, we must provide
    // - the target Tile size (MOSAIC groups and checksums contents by Tile)
    // - comments (may be empty) are short or long text string to describe the file
    // - the indexes this file will use
    // - the compression mode, if any.
    let mut creator = MosaicCreator::new(
        &mosaic_path,
        32000000,
        vec![comment.clone()],
        vec![IdxDescription::Sha1Fmphgo, IdxDescription::Sha256Fmphgo],
        compression_level
    )?;

    // We must provide each object's keys, in the same order as specified to `MosaicCreator`
    creator.add(vec![object_sha1.clone(), object_sha256.clone()], object.clone())?;

    // Indexes are written on closing.
    creator.close()?;

    // This shortcut file opener requires we choose which index will be loaded for
    // next lookups:
    let mut reader = MmapMosaicReader::new(&mosaic_path, IdxDescription::Sha1Fmphgo)?;
    assert_eq!(reader.comments, vec![comment]);
    assert_eq!(reader.objects_counter, 1);

    let object_from_sha1 = reader.lookup(&object_sha1)?.unwrap();
    assert_eq!(object, object_from_sha1);

    // Although unusual, it is possible to switch the active index later:
    let indexes: Vec<_> = reader.list_indexes()?.iter().map(|x| x.0).collect();
    assert_eq!(indexes, [IdxDescription::Sha1Fmphgo, IdxDescription::Sha256Fmphgo]);

    reader.load_index(IdxDescription::Sha256Fmphgo)?;
    let object_from_sha256 = reader.lookup(&object_sha256)?.unwrap();
    assert_eq!(object, object_from_sha256);

    Ok(())
}

Modules

backends

This module contains implementations of MosaicBackend

commands

implementations of swh-mosaic subcommands

creator

Tools to create MOSAIC files

ebml

our EBML toolbox: read/write tags and VINTs.

reader

Tools to read MOSAIC files

updater

Tools to delete object(s) in a MOSAIC file

writer

Tools to write MOSAIC files

Structs

Position

New type wrapping usize and meant to represent a position or an offset in a MOSAIC.

Size

New type wrapping u64 and meant to represent the size of various elements in a MOSAIC.

Enums

CompressionMethod

Enumeration of objects’ compression methods, as can be declared in CompressionMethod elements

IdxDescription

Values allowed in the IdxDescription element, describing the key and key map used by each index.