# dino_seq
Dino Seq is a small Rust parser crate for streaming FASTQ and FASTA inputs. It
focuses on low-allocation record access for downstream tools rather than
preprocessing workflows.
The default crate is the raw parser core. Compression is a transport layer:
enable `gzip`, `bgzf`, or `transport` only when file-magic convenience matters.
## Which API Should I Use?
| Raw FASTQ/FASTA hot path from a file | `FastqReader::new(File)` / `FastaReader::new(File)` | concrete reader type, no boxed transport |
| Borrowed record iteration | `next_batch` | reusable slab plus record side tables |
| Count/stat only | `count_fastq_read`, `count_fastq_bytes`, `FastaReader::stats`, `count_fasta_*` | avoids building record views where possible |
| Single-pass processing | `visit_records`, `visit_fastq_bytes`, `visit_fasta_bytes` | avoids retaining batch side tables |
| Pipeline-owned output buffers | `FastqReader::next_chunk_with_sink` | streams bounded chunks into caller-owned storage |
| Strict two-line FASTA | `visit_two_line_fasta_*`, `count_two_line_fasta_*` | skips ordinary multiline FASTA folding |
| Indexed reference ranges | `FastaIndex`, `IndexedFastaReader`, `reference_chunks_into` | copies only requested sequence runs into caller buffers |
| Convenience path with file-magic transport | `open_fastq`, `open_fasta` | boxed raw/gzip/BGZF detection behind explicit features |
## Scope
Dino Seq provides:
- borrowed FASTQ batches over any `Read`
- one-pass FASTQ visitors for callers that do not need batch side tables
- FASTQ count/stat fast paths that do not build record views
- bounded FASTQ chunk sinks for callers that fill their own output buffers
- streaming multiline FASTA batches and visitors
- ordered paired-end validation for separate R1/R2 or adjacent interleaved reads
- optional BGZF, gzip, mmap, libdeflate, SIMD, and pack-path features
Dino Seq does not trim adapters, filter reads, align reads, produce QC reports,
or synchronize reordered mates.
Hard contracts:
- FASTQ input is ordinary four-line FASTQ; multiline FASTQ is unsupported.
- Paired FASTQ validation is ordered only. Dino Seq validates separate R1/R2
streams or adjacent interleaved records, but it does not synchronize
reordered mates.
- Ordinary multiline FASTA is supported, but APIs that return a contiguous
`seq()` slice fold sequence lines into reusable parser storage. Use strict
two-line APIs when the file shape allows avoiding that work.
## FASTQ Entry Points
- `FastqReader::next_batch`: use when the caller needs `RecordRef` side tables.
- `FastqReader::new(std::fs::File::open(path)?)`: raw-file hot path with a
concrete reader type.
- `FastqReader::count_records`, `count_fastq_read`, and `count_fastq_bytes`:
use for count/stat workloads that do not need record views.
- `FastqReader::visit_records`: use for whole-stream one-pass consumers.
- `FastqReader::next_chunk_with_sink`: use for aligners or pipelines that need
bounded chunks in caller-owned structs.
- `visit_fastq_bytes`: use when the complete FASTQ input is already resident.
- `open_fastq` and `open_fastq_with_config`: convenience boxed file openers.
They open raw files by default and detect gzip/BGZF when those features are
enabled.
```rust
use dino_seq::FastqReader;
let data = b"@r1\nACGT\n+\nIIII\n";
let mut reader = FastqReader::new(&data[..]);
while let Some(batch) = reader.next_batch()? {
for record in batch.records() {
assert_eq!(record.seq(), b"ACGT");
}
}
# Ok::<(), dino_seq::FastqError>(())
```
Use `FastqReader::next_chunk_with_sink` when downstream code wants to fill
owned buffers without building a batch side table.
## FASTA Entry Points
- `FastaReader::next_batch`: stream multiline FASTA batches.
- `FastaReader::new(std::fs::File::open(path)?)`: raw-file hot path with a
concrete reader type.
- `FastaReader::visit_records`: visit records without retaining a batch.
- `visit_fasta_bytes`: parse resident multiline FASTA bytes.
- `visit_two_line_fasta_bytes` and `visit_two_line_fasta_read`: strict fast
paths for canonical `>header` / `sequence` FASTA.
- `open_fasta` and `open_fasta_with_config`: convenience boxed file openers.
They open raw files by default and detect gzip/BGZF when those features are
enabled.
```rust
use dino_seq::FastaReader;
let data = b">chr1\nAC\nGT\n";
let mut reader = FastaReader::new(&data[..]);
let batch = reader.next_batch()?.unwrap();
let record = batch.records().next().unwrap();
assert_eq!(record.name_without_gt(), b"chr1");
assert_eq!(record.seq(), b"ACGT");
# Ok::<(), dino_seq::FastqError>(())
```
## Features
- default: raw FASTQ/FASTA parser core only
- `gzip`: ordinary gzip input by gzip magic
- `bgzf`: BGZF reader, writer, detection, indexing, and parallel block helpers
- `transport`: gzip + BGZF convenience transports
- `libdeflate`: optional libdeflate BGZF backends
- `gzip-libdeflate`: `gzip` + `libdeflate` for explicit buffered gzip openers
- `mmap`: resident file visitors and counters backed by read-only memory maps
- `simd`: stable `std::arch` acceleration where supported
## Unsafe And Optional Features
The default parser core is dependency-light: stable Rust plus `memchr`, with no
compression or memory-map dependency. Optional features add explicit systems
surfaces:
- `mmap` maps files through the operating system for resident-file visitors and
counters. It is useful for local files that should be scanned as byte slices,
not for stdin, pipes, or compressed streams.
- `simd` enables stable `std::arch` kernels for pack paths where the target
supports them.
- `gzip`, `bgzf`, `transport`, `libdeflate`, and `gzip-libdeflate` are transport
or backend choices, not required parser semantics.
Unsafe code is allowed only for these low-level integration points, such as
memory mapping and architecture intrinsics. The raw FASTQ/FASTA reader APIs do
not require callers to use unsafe code.
## Peer Fit
This is a workload map, not a speed ranking.
| General low-copy FASTA/FASTQ parsing | Good when caller wants explicit batch/visitor/count/chunk choices | `seq_io`, `needletail` |
| FASTQ-only no-copy callbacks or record sets | Good when FASTA, pairing, or chunk sinks are also needed | `fastq` |
| Parser-owned parallel FASTX processing | Prefer only when caller wants Dino Seq surfaces and owns scheduling elsewhere | `paraseq` |
| Broad bioinformatics algorithms and ecosystem types | Parser-core dependency, not a replacement | `bio` |
| Sequence manipulation, normalization, kmers | Out of scope except pack side channels | `needletail`, `bio` |
| Aligner/pipeline-owned buffers | Strong fit through `next_chunk_with_sink` | compare against direct integration cost in existing parser |
| BGZF/indexed FASTA reference access | Good when explicit BGZF/index knobs are needed | `noodles`, `rust-htslib`-style ecosystems depending on format scope |
## Adoption Docs
- [API surface](docs/API_SURFACE.md): intended public surface before `1.0`.
- [Benchmarking](BENCHMARKING.md): in-tree smoke benchmark policy and external
artifact boundary.
- [Security policy](SECURITY.md): supported input-security scope and hard
unsupported cases.
- [Changelog](CHANGELOG.md): release-facing behavior and feature changes.
Publication benchmark artifacts live outside this crate so generated datasets,
peer harnesses, and large result tables do not become package weight. Keep
small explanatory figures in this package when they document the public API.
## CLI
```bash
cargo run --release --bin dino-seq -- stats reads.fastq
cargo run --release --bin dino-seq -- stats --format fasta reference.fa
cargo run --release --bin dino-seq -- checksum --format fastq reads.fastq
cargo run --release --bin dino-seq -- checksum --format fasta reference.fa
cargo run --release --bin dino-seq -- fasta-index reference.fa
cargo run --release --bin dino-seq -- fasta-fetch reference.fa --fai reference.fa.fai --name chr1 --start 0 --end 100
```
## Checks
```bash
cargo fmt --all --check
cargo test -p dino-seq --no-default-features
cargo test -p dino-seq --all-features
cargo clippy -p dino-seq --all-targets --all-features -- -D warnings
cargo bench --all-features
cargo package --allow-dirty --list
```