Crate ahda 

ahda is a library and a command-line client for:

• Compressing plain text pseudoalignment data.
• Decompressing binary pseudoalignment data in any supported format.
• Converting between plain text pseudoalignment formats output by different tools.
• Concatenating non-overlapping binary alignment data.
• Performing set operations on compressed pseudoalignment data.

The following plain text formats are supported:

• Bifrost
• Fulgor
• Metagraph (--query-mode labels only)
• SAM (input only)
• Themisto

An additional custom plain text format meant to display all data contained in the records is also provided:

• Ahda .tsv

The default format for plain text outputs is Ahda .tsv.

Internally, ahda uses roaring bitmaps to store the pseudoalignments.

Usage

Command line

The ahda CLI supports the following subcommands:

• ahda encode compress pseudoalignment data from a supported format.
• ahda decode decompress pseudoalignment data to a supported format.
• ahda convert convert between supported plain text formats.
• ahda cat concatenate binary data that doesn’t contain duplicated queries.
• ahda set perform set operations on compressed pseudoalignment data.

Note that encode may need access to the .fastq input file and the names of the pseudoalignment targets. These are required to create an encoded record that can be converted to any of the supported plain text formats, because the plain text formats contain varying levels of information about the input data.

Encoding Themisto and Metagraph data can be performed without the .fastq file by renaming the queries because the index of each query (in the .fastq file) is provided in the plain text format.

Encoding SAM and Bifrost data can be performed without the names of the pseudoalignment targets because they are provided in the plain text format.

Rust API

The API provides several functions for operating on structs that implement Read and/or Write. These are meant for use cases where an entire stream should be processed.

For use cases requiring access to a single record at a time, the following structs are provided:

• Decoder: takes a Read containing the encoded bytes and decodes them into PseudoAln.
• BitmapDecoder: takes an iterator over the indexes of set bits and decodes them into PseudoAln.
• Encoder: takes an iterator over PseudoAln records and encodes them into a Vec<u8>.
• Parser: takes a Read containing plain text pseudoalignment bytes and converts them into PseudoAln.
• Printer: takes an iterator over PseudoAln records and formats them into plain text data.

These structs can additionally be chained together to eg. read encoded data and print it in a plain text format, or to parse plain text data and encode it.

See documentation for the appropriate functions or structs for usage examples.

C++ API

ahda provides a C++ API for encoding and decoding pseudoalignment data into memory. The API is available with the feature cxxbridge.

Encoding requires converting the pseudoalignment to a flattened form and obtaining the indexes of the set bits (positive alignments) that should be included in a bitmap representation.

The encoding API supports writing either a complete .ahda record or one block at a time.

Decoding can be performed by reading the bytes of a full .ahda record into memory, or by reading in one block at a time. The output from the decoding API are the indexes of the set bits in the (flattened) bitmap representation.

The decoding API also supports reading in the target sequence names, the query sequence names, and the positions of the query sequences in the original query input.

File format specification

The binary .ahda file format has the following structure:

File header and flags

• A FileHeader consisting of exactly 32 bytes. The header must start with the bytes 97, 104, 100, 97 indicating that the binary file is an .ahda record, and must be followed by two bytes indicating the ahda library version. The library version is followed by a single byte indicating the ahda file format version. The remaining bytes in the header contain information about the binary data contained in the file.

• A FileFlags consisting of a variable number of bytes. The number of bytes must be given in the flags_len field of FileHeader. The FileFlags block may be compressed, in which case flags_len must provide the number of compressed bytes. The compression method used must be provided in the metadata_compression field of FileHeader.

The FileFlags block contains the name of the sample stored within the file and the names of the alignment target sequences. The alignment target sequences must be provided in the order that they appear in the alignment.

Each .ahda file must only have one FileHeader and FileFlags.

Blocks

Alignment data is stored in a number of blocks that follow the file header and flags. Each block has the following structure:

• A BlockHeader consisting of exactly 32 bytes. The header contains information about the records stored in this block.

• A BlockFlags consisting of a variable number of bytes. The number of bytes must be given in the flags_len field of BlockHeader. The BlockFlags block may be compressed, in which case flags_len must provide the number of compressed bytes. The compression method used must be provided in the metadata_compression field of BlockHeader.

• A block containing the (compressed) bitmap corresponding to the alignment data. The type of the bitmap must match the bitmap_type field of BlockHeader. The number of bytes containing the bitmap must be provided in the block_len field of BlockHeader.

The BlockFlags block must contain the fields promised by the fields_present field of FileHeader. The BlockFlags may contain additional fields that are promised by the fields_present field of BlockHeader but not by the FileHeader.

Each .ahda block must consist of exactly one BlockHeader and BlockFlags. An .ahda file may have any number of blocks.

A valid .ahda file ends with a block and must not include a footer.

Modules

compression

Wrappers for supported bitmap and metadata compression schemes.

decoder

Decoder implementations from Read and bitmap iterators.

encoder

Encoder implementation for an iterator over PseudoAln records.

errors

Error types from the ahda library.

headers

File and block headers used in the encoded format.

parser

Parser for reading plain text data into memory from Read.

printer

Printer for outputting PseudoAln records as plain text in any supported Format.

Structs

EncodeOpts

Options to functions that encode input data.

PseudoAln

A decompressed pseudoalignment record.

Enums

AhdaFormatVersion

Ahda file format version

AhdaVersion

Ahda library version

Format

Supported plain text formats.

MergeOp

Supported set operations for decode_from_read_into_roaring.

Functions

concatenate_from_read_to_write

Merge compressed data by concatenating all blocks.

convert_from_read_to_write

Convert plain text data from Read to plain text data to Write.

decode_from_read

Decode all pseudoalignments from Read to memory.

decode_from_read_into_roaring

Merge bitmap from Read to an existing bitmap with Union

decode_from_read_to_roaring

Reads the full bitmap and combined block flags from a file

decode_from_read_to_write

Decode all pseudoalignments from Read and format to Write.

decode_to_write

Decode from memory and format to Write.

encode_from_read

Parse all plain-text pseudoalignments from Read and encode to memory.

encode_from_read_to_write

Parse all plain-text pseudoalignments from Read and encode to Write.

encode_to_write

Encode from memory to something that implements Write.

merge_from_read_to_write

Merge bitmaps from Read(s) and write encoded result to Write.