encode_jsonl_to_xben

ben::encode

Function encode_jsonl_to_xben 

Source 
pub fn encode_jsonl_to_xben<R: BufRead, W: Write>(
    reader: R,
    writer: W,
    variant: BenVariant,
    n_threads: Option<u32>,
    compression_level: Option<u32>,
) -> Result<()>
Expand description

This function takes a JSONL file and compresses it to the XBEN format.

The JSONL file is assumed to be formatted in the standard

{"assignment": [...], "sample": #}

format. While the BEN format is a simple bit-packed (streamable!) run-length encoded assignment vector, the XBEN format uses LZMA2 dictionary compression on the byte level to achieve better compression ratios. In order to use XBEN files, the decode_xben_to_ben function must be used to decode the file back into a BEN format.

§Arguments

  • reader - A buffered reader for the input file
  • writer - A writer for the output file
  • variant - The BEN variant to use (Standard or MkvChain)
  • n_threads - The number of threads to use for compression (optional)
  • compression_level - The compression level to use (0-9, optional)

§Returns

A Result type that contains the result of the operation