Struct ParMultiGzipReader 

pub struct ParMultiGzipReader<R>
where
    R: Read,
{ /* private fields */ }

This reader facilitates parallel decompression of BCF data compressed in the BGZF format—a specialized version of the multi-member gzip file format. It utilizes internal buffers to sequentially ingest compressed data from various gzip blocks, leveraging the rayon crate to achieve concurrent decompression. This design addresses the potential bottleneck in data processing speed that occurs when decompression is not executed in parallel, ensuring more efficient handling of compressed data streams. Example:

use bcf_reader::*;
use std::fs::File;
use std::io::BufReader;
use std::io::Write;
// read data generated by bcftools
// bcftools query -f '[\t%GT]\n' test.bcf | bgzip -c > test_gt.gz
let mut gt_str = String::new();
smart_reader("testdata/test_gt.gz")
    .unwrap()
    .read_to_string(&mut gt_str)
    .unwrap();
// read data via bcf-reader
let max_gzip_block_in_buffer = 10;
let reader = File::open("testdata/test.bcf").map(BufReader::new).unwrap();
let mut f =
    ParMultiGzipReader::from_reader(reader, max_gzip_block_in_buffer, None, None).unwrap();
let s = read_header(&mut f).unwrap();
let header = Header::from_string(&s).unwrap();
let mut record = Record::default();
let mut gt_str2 = Vec::<u8>::new();
while let Ok(_) = record.read(&mut f) {
    for (i, bn) in record.fmt_gt(&header).enumerate() {
        let bn = bn.unwrap();
        let (noploidy, dot, phased, allele) = bn.gt_val();
        assert_eq!(noploidy, false); // missing ploidy
        let mut sep = '\t';
        if i % 2 == 1 {
            if phased {
                sep = '|';
            } else {
                sep = '/';
            }
        }
        if dot {
            write!(gt_str2, "{sep}.").unwrap();
        } else {
            write!(gt_str2, "{sep}{allele}").unwrap();
        }
    }
    write!(gt_str2, "\n").unwrap();
}
let gt_str2 = String::from_utf8(gt_str2).unwrap();
// compare bcftools results and bcf-reader results
for (a, b) in gt_str
    .split(|c| (c == '\n') || (c == '\t'))
    .zip(gt_str2.split(|c| (c == '\n') || (c == '\t')))
{
    assert_eq!(a, b);
}

See ParMultiGzipReader::from_reader for an example to jump to a target genome interval.

Implementations

impl<R> ParMultiGzipReader<R>

where R: Read,

pub fn from_reader( reader: R, ngzip_max: usize, coffset: Option<u64>, uoffset: Option<u64>, ) -> Result<Self>

Constructs a new ParMultiGzipReader by specifying the ngzip_max parameter, which defines the maximum number of gzip blocks that the internal buffers can handle simultaneously. This parameter should ideally be set to the number of CPU cores available to optimize parallel decompression performance, thereby leveraging the hardware’s concurrency capabilities.

The coffset parameter indicates the offset to the first byte of a target gzip block of the input reader. The input reader should point to the start byte of a gzip block as indicated by coffset before passing the reader to ParMultiGzipReader::from_reader; otherwise, Seek::seek should be used on the input reader to adjust the position accordingly. Note that ParMultiGzipReader does not call Seek::seek on the reader.

The uoffset parameter specifies the number of bytes to skip within the first decompressed gzip data. Since skipping within uncompressed data requires decompression, this offset is applied within the ParMultiGzipReader::from_reader method.

Examples

use bcf_reader::*;
use std::{
    fs::File,
    io::{BufReader, Seek},
};
// index file
let csi = Csi::from_path("testdata/test3.bcf.csi").unwrap();
// reader
let mut reader = File::open("testdata/test3.bcf")
    .map(BufReader::new)
    .unwrap();

// calculate first offsets of the target postion
let start = 1495403 - 1;
let end = 1495746 - 1;
let chrom_id = 0;
let bin_id = csi.get_bin_id(start, start + 1 as i64);
let bin_details = csi.get_bin_details(chrom_id, bin_id).unwrap();
let (coffset, uoffset) = bin_details.chunks()[0].chunk_beg.get_coffset_uoffset();

// seek to the target bgzip block
reader.seek(std::io::SeekFrom::Start(coffset)).unwrap();
// create the parallelizable reader by wraping around the existing reader
// and specifing offsets
let mut reader =
    ParMultiGzipReader::from_reader(reader, 1, Some(coffset), Some(uoffset)).unwrap();

let mut record = Record::default();
let mut pos_found = vec![];
while let Ok(_) = record.read(&mut reader) {
    let pos = record.pos() as i64;
    // the bin containing the start position of target interval may have records
    // before the start position, so skip them.
    if pos < start {
        continue;
    }
    // read the record until out of the target interval
    else if pos >= end {
        break;
    }
    pos_found.push(pos);
}
assert_eq!(pos_found, vec![start]);

Parameters

• reader: The input reader from which gzip blocks will be read.
• ngzip_max: The maximum number of gzip blocks that can be processed in parallel.
• coffset: An optional offset to the start of a gzip block in the input reader.
• uoffset: An optional offset within the first block of decompressed data.

Returns

Returns a new instance of ParMultiGzipReader.

pub fn get_coffset_uoffset(&self) -> (u64, u64)

Trait Implementations

impl<R> Read for ParMultiGzipReader<R>

where R: Read,

fn read(&mut self, buf: &mut [u8]) -> Result<usize>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut<'_>]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

fn is_read_vectored(&self) -> bool

🔬This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Reads all bytes until EOF in this source, placing them into buf.

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Reads all bytes until EOF in this source, appending them to buf.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Reads the exact number of bytes required to fill buf.

fn read_buf(&mut self, buf: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Pull some bytes from this source into the specified buffer.

fn read_buf_exact(&mut self, cursor: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Reads the exact number of bytes required to fill cursor.

fn by_ref(&mut self) -> &mut Self

where Self: Sized,

Creates a “by reference” adapter for this instance of Read.

fn bytes(self) -> Bytes<Self>

where Self: Sized,

Transforms this Read instance to an Iterator over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R>

where R: Read, Self: Sized,

Creates an adapter which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

where Self: Sized,

Creates an adapter which will read at most limit bytes from it.

fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error>

where Self: Sized,

🔬This is a nightly-only experimental API. (read_array)

Read and return a fixed array of bytes from this source.