rsomics-vcf 0.5.0

VCF and BCF inspection, transformation, filtering, indexing, and statistics
Documentation

rsomics-vcf

rsomics-vcf is the rsomics product for VCF and BCF workflows.

The current implementation provides:

  • annotate: edit headers and records or transfer typed fields from sorted VCF, BCF, BED, and tab-delimited annotation streams.
  • head: stream ordered VCF headers and an optional record prefix from plain VCF, BGZF-compressed VCF, BCF, or standard input.
  • filter: evaluate typed site and sample expressions, annotate failures, and apply genomic masks and gap filters.
  • index: create and inspect CSI indexes for BGZF VCF or BCF and TBI indexes for BGZF VCF.
  • norm: normalize against an indexed reference, split, join, atomize, and deduplicate variants while remapping typed INFO, FORMAT, and genotype fields.
  • query: project typed site, INFO, FORMAT, and genotype fields from the same input formats with sample selection and transactional named output.
  • reheader: replace headers, synchronize contigs from a FASTA index, or rename samples without changing the variant payload.
  • validate: validate VCF 4.1–4.5 or BCF 2.2 structure, schema, typed values, cardinality, genotypes, and cross-field invariants.
  • view: convert among VCF, BGZF VCF, raw BCF, and BGZF BCF while selecting records, samples, indexed regions, or streaming targets.

annotate applies checked header removals and renames, site-format ID edits, typed fixed/INFO/FORMAT/GT transfer, explicit sample mapping, allele-aware Number=A, R, and G remapping, target expressions, site marking, and indexed regions. The annotation join advances over two coordinate-sorted streams and retains only records that can still overlap the current target. Named output is transactional; all four VCF/BCF encodings and bounded BGZF compression workers use the same writer layer as the other commands.

rsomics-vcf annotate calls.vcf.gz -a db.vcf.gz -c ID,INFO/AF
rsomics-vcf annotate calls.bcf -a depths.vcf.gz -c FORMAT/DP -s tumor,normal
rsomics-vcf annotate calls.vcf.gz -x INFO/OLD --rename-annotations names.tsv

Whole FORMAT transfer excludes GT unless FORMAT/GT is requested explicitly. Missing definitions, incompatible schemas, invalid cardinalities, unavailable samples, coordinate regressions, and impossible allele maps fail nonzero. Experimental merge logic, dynamic source-column expressions, forced recovery, automatic indexing, and provenance stamping are not accepted as placeholder options.

The annotation release benchmark verifies complete normalized output against bcftools 1.24 before timing. On an Apple M2 with three warmups and ten alternating measured pairs, the 2,000,000-record interval join used 4.8 MB median peak RSS versus 124.4 MB for bcftools while taking 3.895 versus 3.735 seconds. The 300,000-record, eight-sample typed transfer used 5.1 versus 8.3 MB RSS but took 8.495 versus 1.560 seconds. This is a bounded-memory advantage, not a general throughput claim; the complete distributions and fingerprints are in PERFORMANCE.md.

reheader composes complete header replacement, FAI contig synchronization, and positional or old-to-new sample renaming in that order. It preserves the input encoding across plain VCF, BGZF VCF, raw BCF, and BGZF BCF. Named output is atomic, and every requested edit is validated before commit.

rsomics-vcf reheader -H header.vcfh -o renamed.vcf calls.vcf
rsomics-vcf reheader -f reference.fa.fai -N samples.tsv -o renamed.bcf calls.bcf
rsomics-vcf reheader -n tumor,normal calls.vcf.gz > renamed.vcf.gz

Plain VCF record bytes are copied unchanged. BGZF VCF rewrites only header frames, preserves the remaining compressed record frames byte for byte, and requires one canonical EOF block. BCF records are streamed through stable edited dictionaries so removed definitions still referenced by a record fail nonzero. Sample count mismatches, unknown mapping sources, duplicate final names, malformed FAI rows, ordinary gzip input, truncated data, and unsafe output aliases also fail instead of producing a partial artifact. Nonzero --threads is accepted only for BGZF BCF.

The 0.5 release benchmark compares complete output with bcftools 1.24 before timing. On a 2,000,000-record Apple M2 workload, plain VCF reheadering used 0.515 versus 3.075 seconds median wall time and 3.92 versus 6.39 MB median peak RSS. BGZF reheadering used 4.64 versus 7.01 MB median peak RSS but took 0.040 versus 0.020 seconds. The complete distributions, equality hashes, commands, and artifact fingerprints are in PERFORMANCE.md.

head writes VCF text, preserves header order, normalizes the standard PASS definition, removes BCF-internal IDX fields, and renders typed records compatibly with bcftools 1.24. Unlike bcftools, malformed records and invalid numeric arguments fail with a nonzero exit code.

rsomics-vcf head variants.bcf
rsomics-vcf head -n 10 variants.vcf.gz
rsomics-vcf head -s 3 < variants.bcf

-H limits metadata header lines while retaining -h for the unified rsomics help experience. -s starts output at #CHROM and then emits the requested number of records.

index creates CSI by default, supports custom CSI minimum shifts and BGZF decompression workers, and writes the completed index transactionally. It rejects ordinary gzip, missing BGZF terminators, unsorted coordinates, noncontiguous contig blocks, malformed records, and implicit replacement. --stats and --nrecords read count metadata from a variant path or directly from an existing index.

rsomics-vcf index variants.vcf.gz
rsomics-vcf index --tbi variants.vcf.gz
rsomics-vcf index --min-shift 18 variants.bcf
rsomics-vcf index --stats variants.vcf.gz
rsomics-vcf index --nrecords variants.bcf.csi

query supports fixed columns, %POS0, %END, %END0, %FIRST_ALT, %TYPE, %INFO, %INFO/TAG, %FORMAT, %LINE, array subscripts, and per-sample loops with %SAMPLE, %GT, %TGT, %IUPACGT, and declared FORMAT tags. -H and -HH produce indexed and plain projection headers. If a format contains no newline, one is appended per record unless -N is used.

rsomics-vcf query variants.bcf -f '%CHROM\t%POS\t%INFO/DP\n'
rsomics-vcf query variants.vcf.gz -s S1,S2 -f '%POS[\t%SAMPLE=%GT]\n'
rsomics-vcf query variants.bcf -f '%CHROM\t%POS0\t%END\n' -o variants.tsv

The current query contract is a single-input field projection. Region and target selection, expression filtering and functions, multi-input %MASK, %PBINOM, %N_PASS, %TBCSQ, %VKX, and undefined-tag fallback are outside this contract.

view preserves typed records across all four encodings. It supports ordered sample inclusion and exclusion, optional genotype removal, AC/AN recalculation, FILTER, ID, allele-count, and current bcftools variant-type selection. Targets stream through any supported input; regions require a CSI or TBI and use position, record, or variant overlap semantics.

rsomics-vcf view variants.bcf -O z -o variants.vcf.gz
rsomics-vcf view variants.vcf.gz -s tumor,normal -r chr1:1-100000
rsomics-vcf view variants.bcf -v snps,indels -f PASS -G

Named output is transactional, and JSON summaries require it so variant data never shares standard output with the envelope. Expression filtering, allele trimming and remapping, frequency/genotype predicates, output indexing, and compression workers are outside this first stable view contract; they are not accepted as partially implemented flags.

norm accepts every VCF and BCF input encoding supported by the product. It left-aligns and trims alleles against indexed plain or BGZF FASTA, validates or repairs REF mismatches, and can use GFF3 transcript orientation for HGVS 3-prime right alignment. Multiallelic splitting and joining preserve declared Number=A, R, and G INFO and FORMAT values, genotypes, phasing, and mixed ploidy. Complex variants can be atomized with explicit overlap and provenance policies. Duplicate removal, expression-controlled transformation, indexed regions, streaming targets, position or lexical local sorting, transactional output, JSON summaries, and bounded BGZF compression workers compose with the same pipeline.

rsomics-vcf norm -f reference.fa calls.vcf.gz -O z -o normalized.vcf.gz
rsomics-vcf norm -m --keep-sum AD --split-overlaps missing calls.bcf
rsomics-vcf norm --join-multiallelic any --strict-filter calls.vcf
rsomics-vcf norm -f reference.fa -g transcripts.gff3.gz calls.vcf

Malformed allele-indexed fields fail instead of being silently discarded. Automatic output indexing is intentionally excluded until the output and index can be committed as one transaction.

The normalization release benchmark compares exact record bodies with bcftools 1.24 before timing. It ran on an Apple M2 Mac mini with 8 GB RAM and macOS 26.6.1, using three warmups and ten measured runs:

Workload Records rsomics-vcf bcftools Throughput result
Reference-guided indel alignment 500,000 0.920 ± 0.008 s 1.170 ± 0.017 s 1.27× faster
Typed eight-sample multiallelic split 200,000 input, 400,000 output 5.457 ± 0.061 s 7.456 ± 0.047 s 1.37× faster

A single /usr/bin/time -lp run reported 6,799,360 versus 7,012,352 bytes RSS for reference alignment, and 10,518,528 versus 8,437,760 bytes for typed splitting. The committed benchmarks/norm-vs-bcftools.sh harness generates the fixtures, verifies body equality, records hashes and machine provenance, and captures the complete timing distributions.

filter evaluates the product's typed expression language over fixed, INFO, FORMAT, genotype, calculated, arithmetic, logical, regex, file-set, and statistical function values. It supports hard include or exclude selection, soft FILTER annotation modes, failed-sample genotype replacement, masks, SnpGap and IndelGap, indexed regions, streaming targets, every VCF/BCF output encoding, and bounded BGZF compression workers. Named outputs use the shared rsomics-common transaction and JSON summaries remain separate from variant data.

rsomics-vcf filter variants.vcf.gz -i 'QUAL >= 20 && INFO/DP >= 10'
rsomics-vcf filter variants.bcf -e 'FMT/DP < 10' -s LowDepth -S .
rsomics-vcf filter variants.vcf.gz -g 3:indel,mnp -G 5 -O z --threads 4 -o filtered.vcf.gz

The release benchmark uses bcftools 1.24 as the oracle on an Apple M2 Mac mini with 8 GB RAM and macOS 26.6.1. On a 250,000-record, 7.5 MiB plain VCF (f3352403c8a071f09e71da3b38901bca091a73fdd6f7b0b2a49281095964512a), filtering INFO/DP >= 20 && QUAL >= 30 to /dev/null took 57.16–59.24 ms with a 58.24 ms estimate, versus 111.94–118.35 ms and a 114.91 ms estimate for bcftools. Three warm /usr/bin/time -l runs used at most 3,702,784 bytes RSS for rsomics and 6,782,976 bytes for bcftools. The committed Criterion benchmark generates the input and records both implementations under the same conditions.

validate accepts plain or gzip/BGZF-compressed VCF, raw or BGZF-compressed BCF, and standard input. Diagnostics identify the record line and field, and invalid input exits with status 1. By default it retains at most 100 diagnostics while still reporting the complete error and warning counts.

rsomics-vcf validate variants.vcf.gz
rsomics-vcf validate variants.bcf --max-diagnostics 20
rsomics-vcf validate variants.vcf --require-evidence --json

--require-evidence additionally requires every record to carry GT, INFO/AF, or both INFO/AC and INFO/AN. This is an explicit policy rather than part of the VCF format validity contract.

License

Licensed under either the Apache License, Version 2.0 or the MIT License.