Kira Bio Tools CD-Hit-compatible FASTQ reader

Streaming FASTQ reader with CD-HIT–compatible input handling (plain and .gz), a safe, idiomatic Rust API, and optional async support.

Single-line FASTQ by default (sequence and quality occupy exactly one line each) to match common CD-HIT expectations.
Multi-line FASTQ supported via an option for broader compatibility.
Gzip auto-detection by extension or magic bytes.
Error policy: skip malformed records (CD-HIT-like, default) or fail fast.
Streaming: process records one-by-one without loading the whole file.
Optional mmap for faster plain-file reads.
Async API behind the async feature (Tokio + async-compression).
MSRV pinned to Rust 1.85; edition = 2024.

Status
Installation
Features
Design goals
Quick start (sync)
Quick start (async)
Single-line vs multi-line FASTQ
Error policy
Resynchronization behavior
API overview
Performance notes
Testing & benches
Versioning & MSRV
License

Status

Production-ready streaming reader for FASTQ (plain and gzip).
Cross-platform CI for Linux, macOS, Windows.
Intended for use in pipelines where CD-HIT input behavior must be mirrored.

Installation

[dependencies]

kira_cdh_compat_fastq_reader = "*"

Optional features:

[dependencies]

kira_cdh_compat_fastq_reader = { version = "0.1", features = ["async", "mmap", "zlib"] }

gzip — enabled by default (gzip via flate2 with miniz_oxide backend).
zlib — switch flate2 to system zlib backend (closer to CD-HIT’s zlib path).
mmap — enable memmap2 for plain files (reduces syscalls).
async — enable async API (Tokio + async-compression).

MSRV: 1.85.0 or newer (pinned).

Features

CD-HIT–compatible defaults: single-line mode and a resilient “skip-bad-and-continue” policy.
Auto gzip detection: by .gz extension or magic bytes (1F 8B).
Streaming iterator: reads record-by-record; constant memory overhead regardless of file size.
Clear error reporting: format errors include line/byte context.
Minimal dependencies: core functionality keeps dependency surface small; performance extras are opt-in.

Design goals

Safety first: no unsafe parsing; owned record buffers; clear error types.
Predictable behavior: strong defaults that mirror CD-HIT expectations.
Composability: easy to integrate in larger pipelines (sync or async).
KISS/DRY: keep the public API small and focused.

Quick start (sync)

use kira_cdh_compat_fastq_reader::{FastqReader, ReaderOptions, ErrorPolicy, LineMode};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // CD-HIT–compatible defaults:
    let opts = ReaderOptions {
        error_policy: ErrorPolicy::Skip, // keep going on malformed records
        fastq_only: true,                // reject FASTA '>' headers
        line_mode: LineMode::Single,     // single-line seq/qual
    };

    let mut rdr = FastqReader::from_path("reads.fastq.gz", opts)?;

    for rec in &mut rdr {
        let rec = match rec {
            Ok(r) => r,
            Err(e) => { eprintln!("skipped: {e}"); continue; }
        };
        println!("id={} len={}", rec.id, rec.len());
    }
    Ok(())
}

From stdin:

use std::io::{self, BufReader};
use kira_cdh_compat_fastq_reader::{FastqReader, ReaderOptions, ErrorPolicy, LineMode};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let opts = ReaderOptions { error_policy: ErrorPolicy::Return, fastq_only: true, line_mode: LineMode::Single };
    let stdin = io::stdin();
    let rdr = BufReader::new(stdin.lock());
    let mut fq = FastqReader::from_bufread(rdr, opts);
    for rec in &mut fq {
        let r = rec?;
        println!("{}", r.id);
    }
    Ok(())
}

Quick start (async)

Enable the async feature: kira_cdh_compat_fastq_reader = { version = "0.1", features = ["async"] }

use kira_cdh_compat_fastq_reader::{AsyncFastqReader, ReaderOptions, ErrorPolicy, LineMode};

#[tokio::main(flavor = "multi_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let opts = ReaderOptions {
        error_policy: ErrorPolicy::Skip,
        fastq_only: true,
        line_mode: LineMode::Single,
    };

    let mut rdr = AsyncFastqReader::from_path("reads.fastq.gz", opts).await?;

    while let Some(item) = rdr.next_record().await {
        let rec = match item {
            Ok(r) => r,
            Err(e) => { eprintln!("skipped: {e}"); continue; }
        };
        println!("id={} len={}", rec.id, rec.len());
    }
    Ok(())
}

You can also wrap any AsyncBufRead via:

// AsyncFastqReader::from_async_bufread(reader, opts)

Single-line vs multi-line FASTQ

Single-line (default): after the @header, sequence is exactly one line, + is one line, quality is exactly one line. This matches typical Illumina output and how CD-HIT often sees inputs.

Multi-line: sequence and/or quality may span multiple lines. Enable via:

ReaderOptions { line_mode: LineMode::Multi, ..Default::default() }

Note: Single-line mode is both stricter and faster. If your datasets are multi-line, switch to LineMode::Multi.

Error policy

enum ErrorPolicy {
    Skip,   // default: skip malformed records and continue (CD-HIT-like)
    Return, // fail fast on first malformed record
}

Typical format errors include:

Missing header @ (or encountering FASTA > in FASTQ-only mode).
Missing + line.
Unexpected EOF inside a record.
Length mismatch between sequence and quality.
Empty sequence.

All errors carry an I/O context (byte offset and line number).

Resynchronization behavior

With ErrorPolicy::Skip, the parser attempts to resynchronize at the next line starting with @. This mirrors the robust “keep going” behavior often expected in CD-HIT pipelines when inputs contain occasional malformed records.

API overview

Types

FastqReader — synchronous streaming reader (plain or .gz).
AsyncFastqReader — asynchronous streaming reader (feature async).
FastqRecord — { id, desc: Option<String>, seq: Vec<u8>, qual: Vec<u8> }.
ReaderOptions — { error_policy, fastq_only, line_mode }.
ErrorPolicy — Skip or Return.
LineMode — Single or Multi.
FastqError / FormatError — detailed error types with context.

Construction

// sync
let mut r = FastqReader::from_path("reads.fastq.gz", opts)?;
// or
let mut r = FastqReader::from_bufread(my_buf_reader, opts);

// async
let mut ar = AsyncFastqReader::from_path("reads.fastq.gz", opts).await?;
// or
let mut ar = AsyncFastqReader::from_async_bufread(my_async_bufread, opts);

Iteration

// sync
for item in &mut r {
    let rec = item?; // or handle Skip policy
    // ...
}

// async
while let Some(item) = ar.next_record().await {
    let rec = item?; // or handle Skip policy
    // ...
}

Performance notes

Plain FASTQ + mmap (--features mmap): can reduce syscalls and improve throughput on fast storage (commonly +5–30% vs buffered reads).
Gzip:
- Default flate2 backend (miniz_oxide) provides solid performance.
- --features zlib switches to system zlib for closer parity with CD-HIT’s zlib path.
I/O-bound workloads benefit most from larger buffers and sequential access patterns; CPU-bound cases (e.g., heavy downstream processing) usually dwarf parse costs.

Use cargo bench to evaluate on your hardware and datasets.

Testing & benches

# default features (gzip enabled)

cargo test


# all features

cargo test --all-features


# benches

cargo bench

Tests cover:

Basic parsing (single-line).
Gzip files.
Skip vs Return policies.
Async parsing (behind async feature).

Versioning & MSRV

MSRV: >=1.85.
SemVer: public API follows semantic versioning. Breaking changes trigger a major version bump.

License

Licensed under GPLv2 like a CD-Hit.

kira_cdh_compat_fastq_reader 0.1.3