taxutils 1.1.7

Rust utilities for NCBI taxonomy data and FASTA files
Documentation

taxutils-rs

Rust port of the Python taxutils package, optimized for parallelism and incorporation into Rust repositories with its crate. The port was performed entirely by codex with GPT-5.6-sol. Available for download from bioconda.

The Rust package provides both:

  • the tu command-line program
  • the taxutils Rust library crate, [crates.io]

Citation

@article{obrien2026taxutils,
  title   = {taxutils: Tools for managing and operating on NCBI taxonomy and accession mappings},
  author  = {O'Brien, William and Lee, Seungmo and Agarwal, Vivek and Eskin, Eleazar},
  journal = {bioRxiv},
  year    = {2026},
  doi     = {10.64898/2026.09.18.752821}
}

DOI: 10.64898/2026.09.18.752821

Installation

Install via conda or mamba from the bioconda forge:

conda install bioconda::taxutils-rs

Or install the command-line program from crates.io:

cargo install taxutils
tu --help

The Cargo package and library are named taxutils; the installed executable is named tu. To use the library in another Rust project:

cargo add taxutils

Data directory

Like the Python package, taxutils-rs stores downloaded NCBI taxonomy and accession resources in ./taxutils/ by default. Set TAXUTILS_GLOBALS to use a persistent location instead:

export TAXUTILS_GLOBALS=/path/to/taxutils/cache
tu filter -i input.fasta -o filtered.fasta --keep-taxids 2697049

The Rust library reads TAXUTILS_GLOBALS when TaxutilsOptions::default() or TaxutilsBuilder::new() is constructed. Set the variable before constructing the builder. Applications that manage configuration directly can set the same path without an environment variable:

let tu = taxutils::TaxutilsBuilder::new()
    .save_folder("/path/to/taxutils/cache")
    .low_memory(false)
    .keep_accession_downloads(false)
    .build()?;
# Ok::<(), anyhow::Error>(())

Managed resources include:

  • names.dmp and nodes.dmp from the NCBI taxdump
  • targets.json
  • nucl_gb.accession2taxid.gz
  • optional nucl_wgs.accession2taxid.gz
  • nucl.accession2taxid.db when using indexed SQLite lookup

Library

use taxutils::{TaxonomicUtils, TaxutilsBuilder, TopologyStat};

let mut tu = TaxutilsBuilder::new()
    .low_memory(true)
    .wgs(false)
    .build()?;

let accession = tu.parse_accession(">NC_045512.2 SARS-CoV-2", true);
tu.load_a2t(&[accession], None, false, None)?;

let branch = tu.get_branch(2697049);
let subtree = tu.get_subtree(694009);
let lca = tu.get_lca(2697049, 694009);
let distance = tu.get_distance(2697049, 694009);
let profile = tu.topology(2697049, Some("F"))?;
let scale = tu.topology_stat(
    2697049, Some("F"), TopologyStat::TopologyScale)?;
# Ok::<(), anyhow::Error>(())

TaxutilsOptions::default() reads TAXUTILS_GLOBALS at construction time and otherwise uses ./taxutils/, matching the Python package.

Low-memory accession lookups stream gzip data through a decoder while scanning borrowed row buffers on another thread. GB and WGS are scanned concurrently when WGS is enabled. Each source uses one decoder worker and one scanning thread; wider speculative gzip pools were slower and used much more memory on NCBI mapping data. Scans keep bounded buffers plus the requested and matching accessions, without building an index or unpacking files to disk. Concatenated gzip members are supported. Empty queries return immediately, and forward lookups stop once all requested accessions have been found in a source. Reverse lookups and queries with missing accessions still require a full scan. Use indexed mode for repeated lookups against the same cache.

Source builds require Rust 1.88 or newer. Build with cargo build --release for lookup throughput; unoptimized development builds are substantially slower.

The library API is centered on TaxonomicUtils and provides the Python object capabilities:

  • parse_accession, load_a2t, and get_t2a
  • get_branch, get_subtree, get_ancestor, is_leaf, is_child, and is_descendent
  • get_lca, get_distance, sort_taxa, and format_tree
  • topology and topology_stat
  • get_rank_order and higher_than_rank
  • extract_accessions, clean_fasta_headers, grep_fasta, and filter_fasta for embedding the same bounded, ordered FASTA workflows used by tu
  • lookup_accession_taxids and lookup_taxid_accessions for bulk mapping without constructing the full taxonomy object
  • ensure_accession_database for an atomic, tuned SQLite bulk build shared by Rust applications and the Python extension

SQLite databases are populated and indexed in a sibling temporary file, then atomically installed only after the build succeeds. Both NCBI dumps are already sorted by accession, so they are decompressed in parallel and merged into one ascending stream that fills the table in key order. a2t is keyed on the accession itself (WITHOUT ROWID), which removes a whole second copy of every accession and makes the taxid index covering for reverse lookups.

SQLite scratch files (including refresh staging tables and index-sort spills) are confined to a private .taxutils-sqlite-* directory inside the resolved save folder. This uses TAXUTILS_GLOBALS by default, with the usual explicit save-folder override; no additional environment variable or option is needed. There is no fallback to system temporary directories. If scratch cannot be created or written, the operation reports an error. Scratch directories are removed when connections close, including after errors and cancellation. A hard process kill can leave an operation's directory behind; these are not automatically swept because another process may still be using them.

TaxutilsBuilder::refresh(true) updates existing sources in place instead of rebuilding them. Adding WGS to a GB-only database is the deliberate exception: the builder performs one ordered atomic rebuild, avoiding a massive temporary delta table and hundreds of millions of indexed row mutations. An already-downloaded WGS gzip is reused unless refresh was explicitly requested. Because the incoming dumps and the stored table share an ordering, one lockstep pass classifies every row as an insert, an update, or a deletion, and accessions withdrawn upstream are removed. Sources whose validators still match what was recorded are not downloaded at all, and a server that cannot be reached is never read as a change.

ensure_accession_database_with_cancel, lookup_accession_taxids_with_cancel and lookup_taxid_accessions_with_cancel accept a CancellationToken. They stop promptly - including inside CREATE INDEX, via SQLite's interrupt handle - leaving no temporary file and no change to an installed database.

Set TaxutilsBuilder::keep_accession_downloads(false) when using indexed mode to discard the compressed NCBI inputs after the database is ready. They are downloaded again only if a later compressed-file lookup or rebuild needs them.

Python uses one method name for both scalar and pandas/NumPy inputs. Rust keeps the same scalar names and provides explicit, parallel batch forms:

Python capability Rust scalar Rust batch
parse_accession parse_accession parse_accessions
get_ancestor get_ancestor get_ancestors
is_leaf is_leaf are_leaves
is_child is_child are_children
is_descendent is_descendent are_descendents
topology topology topologies
topology(..., stat=...) topology_stat topology_stats

The object exposes names, nodes, parent, target_taxa, and a2t, using typed Rust collections and records (Vec, HashMap, HashSet, TaxonNode, and TopologyProfile) where Python returns pandas or NumPy containers.

The FASTA functions return structured counts and are shared by the tu binary and language bindings, so both interfaces use the same implementation.

Command line

tu extract input.fasta -o accessions.txt
tu clean -i input.fasta -o clean.fasta
tu deduplicate -i input.fasta -o unique.fasta
tu deduplicate -i input.fasta  # atomic in-place rewrite
tu grep -i input.fasta -a NC_045512.2,NC_001422.1 -o hits.fasta
tu filter -i input.fasta -o filtered.fasta --keep-taxids 2697049
tu filter -i input.fasta --keep-taxids 2697049  # atomic in-place filtering
tu filter -i input.fasta -o filtered.fasta --remove-taxids 9606

All five commands accept plain, gzip, and zstd inputs, detected by magic bytes rather than filename. Accession and taxid query files support the same formats. Explicit output paths select compression by their final suffix (case-insensitive): .gz uses gzip, .zst or .zstd uses zstd, and other suffixes produce plain text. extract applies this rule to its accession-text output too. When clean, deduplicate, or filter omits --output, the rewrite preserves the input's detected compression, even if its filename has no compression suffix. An explicit output path selects compression by suffix, including when it names the input.

tu grep -i genomes.fna.zst -a NC_045512.2 -o hits.fna
tu clean -i genomes.fna.gz -o cleaned.fna.zst
tu deduplicate -i genomes.fna.zst
tu extract genomes.fna.zst -o accessions.txt.zst
tu filter -i genomes.fna.zst --keep-taxids 2697049

Compression streams in process without external tools or unpacking the whole input to disk. Concatenated gzip members and zstd frames, including skippable frames, are supported. Zstd output uses level 3; gzip uses the library default. All outputs are written to unique temporary files beside the destination and installed atomically only after decoding, encoding, and flushing succeed. Errors or cancellation preserve existing destinations; replacement retains existing file permissions.

deduplicate keeps the first record for each parsed accession, including its version, and removes later records with that accession even if their sequences differ. Identical sequences with different accessions remain. Retained headers, sequence bytes, and record order are preserved. An unparseable header fails without replacing the destination. Output is written atomically, including when --output is omitted or names the input. Memory use includes one set entry per unique accession.

The CLI uses all available logical CPUs for accession parsing and filtering. Use tu --threads N <command> ... (or place --threads N after the command) to cap worker threads. FASTA records are written in bounded, ordered batches, so parallel execution does not reorder records. filter parses accessions and performs indexed SQLite lookups for each batch while scanning the FASTA once. Compression is sequential; record parsing and filtering retain their existing parallel execution.

Taxid and accession arguments may also name text files. grep --no-version matches accessions without versions. filter uses the indexed SQLite mode just like the Python CLI and is the command that uses the resources under TAXUTILS_GLOBALS. extract, clean, deduplicate, and grep operate directly on FASTA data and do not require the NCBI resource cache.

Performance model

  • FASTA records are streamed in bounded batches instead of retaining sequence data in memory. filter retains lookup results for the current batch.
  • Header parsing and filter decisions run in parallel; large buffered reads and writes preserve input order without issuing I/O for every FASTA line.
  • filter reuses one SQLite connection and looks up accessions in bounded batches, avoiding a full-file accession map or a second decompression pass.
  • clean is normally limited by sequential input/output throughput. Additional threads accelerate header parsing but do not divide the input file into shards.
  • --batch-size controls records per batch for extract and filter, and approximate bytes per batch for grep.
  • GB and WGS gzip mapping files are scanned concurrently when WGS mode is enabled.
  • SQLite lookups use indexed temporary-table joins instead of one query per key.
  • NCBI dump parsing avoids allocating a temporary field vector for every row.
  • Batch accession parsing, ancestor lookup, rank checks, target expansion, node materialization, and topologies use the shared Rayon worker pool.

Alternative viral assignments

TaxutilsBuilder::new().canonical(false) enables influenza A strain matches, then H/N genotype matches, with NCBI fallback. The default is canonical(true). The policy applies to constructor accessions, load_a2t, and get_t2a; strain keys remain private and do not change TaxonNode.

For standalone lookups use AccessionMappingOptions with lookup_accession_taxids_with_options or lookup_taxid_accessions_with_options. The existing lookup functions retain canonical behavior. The options-based functions accept a CancellationToken and honor threads for batch matching.

Alternative mode downloads and trims AllNuclMetadata to viral.metadata.csv.gz. Indexed mode adds a sparse a2t_overrides table and reverse index to the shared accession database. Low-memory mode scans metadata without creating a database; reverse queries use two passes to limit memory to candidate accessions. prepare_alternative_mappings supports explicit preparation and refresh. Cached matching dictionaries and stored overrides are invalidated by resource changes.