# BBNorm Parity Ledger
## Upstream Reference
The upstream BBTools snapshot is vendored at `vendor/BBTools-master`.
Key files for BBNorm behavior:
- `vendor/BBTools-master/bbnorm.sh`: wrapper, help text, and JVM launch defaults.
- `vendor/BBTools-master/current/jgi/KmerNormalize.java`: CLI parsing, k-mer table construction, normalization, error detection/correction, histograms, and multipass orchestration.
- `vendor/BBTools-master/current/bloom`, `current/kmer`, `current/stream`, `current/shared`: supporting counting, long-kmer, sequence I/O, and utility classes.
## Implemented In Rust Now
- BBNorm-style `key=value` CLI parsing for the core normalization options and many upstream aliases, including BBTools-style `config=<file[,file...]>` expansion from one-argument-per-line config files, shared BBTools file aliases (`input`/`input1`/`input2` and `output`/`output1`/`output2`) mapped to the supported read streams, BBTools' default `passes=2` state as real Rust temp-file multipass orchestration for covered outputs, and accepting BBTools `threads`/`t` controls as Rayon worker-count controls for the parallelized Rust paths.
- Plain and `.gz` FASTA/FASTQ input and output, including built-in zlib-rs gzip, parallel gzip output for `.gz` streams, and BBTools-style pigz/unpigz input hooks when `threads`, `zipthreads`, `pigz`, or `unpigz` controls request more than one worker, output-format inference from BBTools-style output extensions (`.fa`/`.fasta` writes FASTA, `.fq`/`.fastq` and unknown extensions write FASTQ), `append=t` read-output behavior where sequence outputs append but histogram-style outputs are recreated like vendored BBNorm, BBTools-style case-insensitive `null` output sinks (`out=null`, paired/bin null streams, and histogram null streams), fake FASTQ qualities for FASTA input (`fakequality`/`qfake`, `fakefastaquality`/`ffq`, `qout=33`/`qout=64` conversion), `fastawrap`/`wrap` FASTA output line wrapping including `0` as no-wrap, explicit `qin=33`/`qin=64` conversion, BBTools-style input quality normalization for called bases and no-call bases (`Q0`), configurable `changequality`/`ignorebadquality`/`ibq`/`mincalledquality`/`maxcalledquality` controls, and BBNorm quality-offset aliases such as `ascii`, `qual`, `asciiin`, `asciiout`, and `qauto`, including mixed q33/q64 single-end comma-list, paired two-file, explicit interleaved, and auto-interleaved `qin=auto`/`qauto=t` golden coverage. Generic ordinary gzip still cannot be split into independent deflate work without block/index support, but Rust now uses the faster zlib-rs backend and will stream `.gz` input through pigz/unpigz when available and requested.
- BBTools-style input base cleanup for the covered read-validation flags: `utot`, `touppercase`/`tuc`, `lowercaseton`/`lctn`, `iupacton`/`undefinedton`/`itn`, `dotdashxton`, `fixjunk`, `ignorejunk`, `flagjunk`, `tossjunk`, `crashjunk`/`failjunk`, `junk=crash`/`junk=fail`, `junk=flag`/`junk=discard`, and `junk=iupacton`, including the Java ordering where quality offset normalization happens before base cleanup and `changequality=t` controls post-cleanup quality re-capping. In the covered `KmerNormalize` path, junk flag/discard modes mark reads in Java but do not physically remove them, so Rust preserves the same output records.
- Single-end input, comma-separated input lists (`in=a.fq,b.fq`, plus paired `in=r1a.fq,r1b.fq in2=r2a.fq,r2b.fq`) for missing literal list paths with BBTools-style per-file `reads` limits, table-building list-tail semantics, comma-separated output-list fanout for input-list runs including representative keep, toss, low-bin, mid-bin, and high-bin streams, Java's first-output-driven handling where an `out2` comma-list is not fanned out unless `out` is also a list, and Java's lazy failure shape for too-short fanout lists after earlier outputs are written including paired `out2` short-list mismatches, two-file paired input (`in=<r1> in2=<r2>`) including Java's behavior where `interleaved=t` is accepted but `in2` keeps the run on the two-file paired path, `extra=<file[,file...]>` table-building inputs that contribute k-mers without being emitted including BBTools' literal existing-file rule for comma-containing filenames, Java's unlimited-extra behavior under `tablereads`, and Java-compatible rejection of missing literal `#` patterns, BBTools-style paired input expansion from missing first-stream `#` patterns (`in=reads#.fq` -> `reads1.fq`/`reads2.fq`), explicit interleaved paired input/output (`in=<reads> interleaved=t`), BBTools-style paired output splitting from first-stream `#` patterns, implicit paired-to-interleaved output when `out2`/`outt2`/bin `*2` streams are omitted, and BBTools-style auto-interleaved detection from first-record pair names or `_interleaved.` filenames.
- Exact Rust-map counting with packed short k-mer keys for `1 <= k <= 31` and BBTools-style long-kmer hash keys for `k > 31`; local S. cerevisiae and E. coli biological stress now verifies `k=40` paired keep/toss/bin/hist/rhist behavior and byte-identical `threads=1/2/auto` outputs.
- `minq`, `minprob`, and `rdk`/`removeduplicatekmers` during table construction; `canonical=f` follows BBNorm's effective short-kmer behavior where `KmerNormalize` still counts/analyzes short k-mers canonically through `ReadCounter(k, true, ...)`, while `k > 31` follows BBTools long-kmer hash behavior.
- Deterministic one-pass keep/toss normalization using the upstream depth-percentile logic and Java-compatible `FastRandomXoshiro(0)` read-random sequence.
- `deterministic=f`/`dr=f`/`det=f` requests keep ordered Rust output but use a per-run nondeterministic seed for Java-shaped read-selection coin tosses; default deterministic mode remains reproducible.
- `target`, `max`/`maxdepth` with BBTools' `max >= target` clamp, `min`, `minkmers` with BBTools' minimum-1 clamp, `minlen`/`ml` including BBTools-style KMG suffix parsing, `reads`/`maxreads`, `tablereads`/`buildreads` including BBTools-style KMG suffix parsing, `percentile`, `hdp`, `ldp`, `edr`, `ht`, `lt`, `keepall`, `rename`, `tossbadreads`, final-stage/first-stage toss aliases, `requirebothbad`/`removeifeitherbad`, `saverarereads`, `discardbadonly`, and `uselowerdepth` core behavior for the covered single-pass real-dataset cases.
- `fixspikes`/`fs` coverage-profile correction using BBTools' precise left/right adjacent k-mer lookup logic before read-depth sorting and histogram accumulation, including the long-kmer hashed-key behavior for `k > 31`; the biological long-kmer stress runs the paired `k=40 fixspikes=t` path on local S. cerevisiae and E. coli across `threads=1/2/auto`.
- Common quality trimming parity for `qtrim`, `qtrim=window`/`qtrim=w,<len>`, `trimq`/`trimquality`, `trimleft`/`trimright`, `optitrim`, and `trimgoodinterval`: the Rust engine counts the untrimmed table first, then trims reads with BBTools-style optimal/simple/window trimming before read coverage, histograms, keep/toss decisions, renaming, and output. Position-specific comma-separated trim qualities such as `trimq=10,20` now emit a note and use the first threshold for ASAP working behavior.
- BBNorm-inactive trim parser controls (`trimclip`, `trimpoly*`, `filterpoly*`, `maxnonpoly`, and `forcetrim*` aliases) are accepted as no-ops because `KmerNormalize` parses but does not use those fields in the covered normalization path.
- BBNorm-inactive read-filter parser controls (`maxlen`, `minlenfraction`, `maxns`, `mingc`, `maxgc`, `usepairgc`, `minconsecutivebases`, `maq`, `maqb`, `mbq`, `chastityfilter`, and `trimbadsequence`) are accepted as no-ops for the same reason; `minlen` remains active and golden-tested separately.
- Byte-identical `rename=t` output headers for covered single-end, paired two-file, and explicit interleaved single-pass paths using BBTools-style numeric read IDs, including Java-shaped ECC-active `e1=0` and `e1=0,e2=0` header fields.
- `hist` and `histout` k-mer depth histograms with upstream-style raw and estimated-unique columns.
- `rhist` and `rhistout` read/base depth histograms.
- Shared BBTools side-output stats histogram controls (`qhist`, `bqhist`, `qchist`, `aqhist`, `bhist`, `lhist`, `gchist`, `enthist`, `idhist`, `mhist`, `ihist`, `qahist`, `indelhist`, `ehist`, and related sizing/toggle aliases) are accepted for ASAP usability. Rust now emits covered primary quality, base-quality, quality-count, average-quality, overall-base-quality, read-length, GC-bin, base-content, entropy, sequence-input identity-shaped, no-alignment match-shaped, and no-alignment error-shaped histograms for these controls, using one shared trimmed primary-input scan for read-local artifacts. Since this Rust BBNorm path does not align reads, `idhist` records input reads at 100 sequence identity, `mhist` treats called bases as match-shaped observations and no-calls as `N`, `qahist` records quality observations as no-observed-error matches, `ehist` records zero observed alignment errors per read, and `ihist`/`indelhist` emit BBTools-shaped no-observed-alignment fallback artifacts rather than claiming real insert/indel measurements. It applies `maxhistlen` to emitted quality/length/base side histograms, applies `gcbins` to emitted GC histograms, applies `entropybins`/`entropyns`/`entropyk`/`entropywindow` to emitted entropy histograms, and applies `idbins`/`idhistbins` to emitted identity histograms. Vendored `KmerNormalize` rejects representative side-output stats controls in this path, so the phiX guard verifies Rust output against the local Java baseline without those controls while checking the Rust-only side-output artifacts; `scripts/parity_side_output_stats_biological_stress.sh` additionally validates the emitted artifacts on local paired biological reads and checks byte-identical output across `threads=1/2/auto`.
- `peaks` and `peaksout` peak-calling output from BBNorm k-mer histograms, including Java-style raw-to-unique conversion, progressive smoothing, peak statistics, BBNorm's long peak-tuning option names (`minheight`, `minvolume`, `minwidth`, `minpeak`, `maxpeak`, `ploidy`, `maxpeakcount`/`maxpc`/`maxpeaks`), and Rust-friendly CallPeaks short aliases (`h`, `v`, `w`, `minp`, `maxp`) mapped to the same filters.
- Depth-bin outputs (`outlow`, `outmid`, `outhigh`, plus paired `*2` streams or `#` split patterns) with `lowbindepth` and `highbindepth`; local S. cerevisiae and E. coli biological stress now verifies paired bin totals and byte-identical `threads=1/2/auto` outputs.
- `outuncorrected`/`outu` output paths for no-error-correction runs; these are opened and remain empty like vendored BBNorm when `ecc` is off. The first table-based ECC path now writes corrected reads to normal keep/toss outputs and Java-golden-covers forced uncorrectable high-quality suspected errors routed to `outuncorrected`, including both single-end and two-file paired output streams; real-derived biological stress also verifies paired `outuncorrected` routing and paired `markuncorrectableerrors` quality marking across `threads=1/2/auto`.
- BBNorm-compatible read-analysis coverage semantics: all possible read k-mer windows participate in depth/error percentiles, with valid but uncounted windows scored as `0` and invalid/N windows scored as `-1`; depth histograms still skip negative invalid windows like Java.
- Real-dataset Java parity harnesses for the bundled paired phiX sample (`resources/sample1.fq.gz` and `resources/sample2.fq.gz`) in single-pass keep-all histogram/binning mode, read-limit modes (`reads=10`, `tablereads=10`, `reads=0.01k`, and `tablereads=0.01k`), input/output-pattern modes (`in=...#...` paired expansion, representative single-end and paired comma-list input modes including output fanout, shared `input`/`output` file-alias fallback coverage, `out=...#...` splitting, paired implicit interleaved output without `out2`, paired `in2` plus `interleaved=t`, `append=t`, comma-separated `config=<file>` expansion, default omitted-`passes` single-pass fallback coverage, wrapper-advertised sampling-option fallback coverage, and representative `extra=...` table-building input mode), no-ECC `outuncorrected` empty output mode, table-mode toggles (`canonical=f`, `rdk=f`, and combined `canonical=f rdk=f`), threshold clamps (`target=100 max=50` and `minkmers=0`), `fixspikes=t`, histogram option modes (`histcol=1`, `histcol=2`, `zerobin=t`, `printzerocoverage=t`, and `histout`/`rhistout` after deterministic error-toss output), deterministic paired error-toss keep/toss normalization mode including `threads=2` output parity, renamed keep-all mode, threshold/stage edge cases (`lowthresh=0`, `highthresh=1`, `errordetectratio=2`, combined `lowthresh=1 highthresh=1 errordetectratio=2`, `requirebothbad=t`, `removeifeitherbad=t`, `saverarereads=t` with both `requirebothbad=t` and `removeifeitherbad=t`, final-stage toss aliases `tossbadreadsf=t`/`tossbadreads2=t`/`ter2=t`/`tbr2=t`, first-stage-only `tossbadreads1=t`, `discardbadonly=t` on a duplicated high-depth real-derived fixture, `minlen=101` and `minlen=0.101k` tossing on a duplicated real-derived fixture, inactive trim parser options such as `trimclip=t`/`trimpolyg=10`/`ftr=10`, inactive read-filter parser options such as `maxlen=50`/`maxns=0`/`maq=40,20`, `qtrim=r trimq=10` and `qtrim=w,4 trimq=10` on real-derived low-quality-tail fixtures, representative `qin=64` input conversion, `qin=auto`, quality-offset aliases, `changequality`/called-quality caps, output-format/fake-quality/wrap modes, base-cleanup/read-validation flags, and `qin=33 qout=64` output encoding with low/high/no-call quality clamping, `uselowerdepth=t/f` with `saverarereads=t` on a mixed-depth paired real-derived fixture, non-default `percentile`/`highdepthpercentile`/`lowdepthpercentile` with `saverarereads=t` on the same mixed-depth fixture, `k=40` long-kmer histograms on a duplicated high-depth real-derived fixture, `k=40 fixspikes=t` on a representative long-kmer spike fixture, and `peaks`/`peaksout` on compact representative single-peak and multi-peak fixtures), explicit interleaved target=1 keep/toss normalization mode, and auto-interleaved target=1 keep/toss normalization mode.
- `stepsize`/`buildstepsize` are accepted as Java-parity no-ops in the covered no-ECC single-pass path; vendored `KmerNormalize` stores them as trusted-kmer detection stride controls, but the paired phiX keep/bin/hist harness is byte-identical across the covered values and Rust outputs.
- Sketch/table-sizing controls (`bits`, `hashes`, `cells`, `matrixbits`, `buildpasses`, `prefilter`, and prefilter sizing/hash knobs) are accepted in the covered single-pass path with typed validation for malformed numeric/KMG/fraction values. Constrained `cells`/`matrixbits` requests, Rust's explicit `sketchmemory`/`countminmemory` byte budget, and large-input automatic sizing now build direct fixed-memory count-min sketches for primary input counts and kept-output side counts using the parsed `bits`/`hashes` settings; explicit `cells`/`matrixbits` are treated as BBTools-style total-cell budgets and kept in one shared KCountArray-style cell universe rather than multiplied by hash count. Default `bits=32` sketches use a deterministic shared atomic table with BBTools-style conservative updates that raise hash cells to `min+count`, while smaller cell widths use the packed fixed-memory fallback with the same conservative raise-to-min update shape. Count/sketch hot paths use bounded chunk maps, incremental KCountArray bucket filling, per-kmer mask-table reuse, and duplicate-removal buffers instead of allocating a global exact table; `deterministic=f` also enables faster schedule-dependent parallel replay for bounded approximate atomic sketches. Long-kmer hashing follows BBNorm's scalar `Kmer.xor()` fingerprint path and now uses rolling BBTools-style Java word state for `k>31`, avoiding both per-window vector allocation and full-window rescans on common layouts, and `countup=t` also uses bounded kept-count sketches when those settings are requested. `prefiltercells`/`precells`, `prehashes`/`prefilterhashes`, prefilter memory aliases, and `prefiltersize`/`prefilterfraction` with a configured table memory budget now build a two-stage prefilter-plus-main bounded input sketch using parsed prefilter `bits`/`hashes` where present and an explicit KCountArray-style prefilter-limit gate; configured prefilter tables use KCountArray-style locked/conservative updates by default and honor `lockedincrement=f`/`symmetricwrite=f` as independent row increments, fraction-based prefiltering partitions the main table budget rather than adding an unbounded side allocation, `prefilter=t` remains exact for small non-sketch inputs but now uses the BBTools default 35% prefilter/main partition whenever bounded count-min counting is selected, bounded bucket masks now use `KCountArray7MTA` FastRandomXoshiro mask generation with per-table Java-style seed stepping, so two-stage prefilter/main sketches no longer reuse the same mask table, and large rows use KCountArray-style internal array placement before the prime cell modulus, with configured or active Rayon worker counts rounded up to BBTools-style minimum internal array counts. Exact-count fallback collision estimates now replay through the same bounded sketch objects rather than materializing per-kmer hash-bucket maps. Bounded sketch unique-kmer reports now include thresholded mindepth estimates for the prefilter/main high-depth split used by vendored KCountArray.estimateUniqueKmers(hashes, mindepth). Additional shared kmer-table runtime controls (`initialsize`, `ways`, table buffer length aliases, `tabletype`, `rcomp`, `maskmiddle`, stats/reporting toggles, preallocation controls, minprob routing toggles, `prefilterpasses`/`prepasses`, and `onepass`) are accepted as Rust working-path fallbacks when copied from related BBTools commands. `initialsize` and `prealloc`/`preallocate` now also pre-reserve Rust exact-count map capacity when practical while preserving exact-count output. A paired phiX smoke verifies default prefilter parity, verifies that Rust `buildpasses=2` histograms now change under trusted-kmer filtering, verifies that Rust `prehashes=1`/`prefilterhashes=1`, `prefiltercells=1k`/`precells=1k`, prefilter memory aliases, and guarded prefilter fraction runs now change under real prefilter collision behavior, and verifies that constrained Rust `cells=1k`/`matrixbits=10` plus `sketchmemory=64` histograms now change under direct fixed-memory count-min sketch behavior; a separate fallback smoke verifies the pure kmer-table runtime controls against a local Java baseline because vendored `KmerNormalize` rejects them directly, and a Rust-only local S. cerevisiae stability check verifies the reserve knobs stay byte-identical across `threads=1/2/auto` on 10000 paired biological reads.
- Runtime/output controls that do not change covered output (`ordered=f`, `verbose=t`, default-assertion `printcoverage=t`, BBTools temporary-directory controls `tmpdir`/`usetmpdir`/`usetempdir`, and FASTA `fastareadlen`/`fastareadlength`) are Java-golden-tested. `auto=f` now disables Rust large-input automatic count-min selection; `exact=t` forces exact maps even when explicit sketch sizing is present. Covered `passes=2` runs now create managed Rust temp files for intermediate keep streams and clean them automatically; when `tmpdir` is enabled, those managed temp directories are created under the requested parent.
- Shared FASTA parser controls that do not change covered `KmerNormalize` FASTA output (`fastaminread`/`fastaminlen`/`fastaminlength`, `forcesectionname`, and disabled `fastadump=f`) are Java-golden-tested as no-ops on a representative FASTA fixture, while malformed numeric minimum-read values are rejected like vendored `KmerNormalize`.
- Quality recalibration matrix controls, including pass-suffixed `_p1`/`_p2` variants (`loadq102_p1`, `observationcutoff_p1`, `qmatrixmode_p2`, and related aliases), are accepted as output-preserving no-ops for covered normalization paths.
- Shared BBTools I/O/runtime parser controls that do not change covered output (`usejni`/`jni`, `bytefile*`/`bf*`, `readbufferlength`/`readbufferlen`, `readbufferdata`, `readbuffers`, `skipvalidation`, `validate`, and `validateinconstructor`/`vic`) are Java-golden-tested as no-ops on paired phiX, including Java's lenient boolean parsing for these controls, while malformed numeric values are rejected like vendored `KmerNormalize`. BBTools preparser controls (`json`, `silent`, `printexecuting`, proxy settings, `metadatafile`, `bufferbf`, and `bufferbf1`) are accepted as Rust no-ops for output compatibility. Shared BBTools SAM/readgroup/streamer runtime controls (`sam`/`samv`, `samtools`, `nativebam`, SAM tag toggles, streamer/writer thread knobs, `xs`/`xstag`, and read-group metadata aliases) are accepted as Rust no-ops for covered FASTA/FASTQ output; vendored `KmerNormalize` rejects representative SAM controls in this path, so the guard verifies Rust output against the local Java baseline without those controls. Enabled BBTools MPI controls (`usempi`, `mpi`, `crismpi`, and `mpikeepall`) now emit local-run notes and continue through the supported local Rust path for ASAP usability. Enabled global pairing runtime controls (`pairreads` and `flipr2`) emit notes and keep Rust on explicit `in2=`, `interleaved=`, and `#` routing.
- Shared BBTools file-extension and I/O worker hints (`extin`, `extout`, `workers`, `workerthreads`, `wt`, `threadsin`, `tin`, `threadsout`, and `tout`) are accepted as Rust no-ops/fallback hints for covered FASTA/FASTQ paths; vendored `KmerNormalize` rejects representative values in this path, so the guard verifies Rust output against the local Java baseline without those controls.
- Stdin input is supported for practical Rust pipeline use: `in=stdin`, `in=-`, and `in=stdin.fq.gz` are materialized into a managed temp file before the Rust count/normalize/histogram passes, honoring enabled `tmpdir` settings and allowing safe rereads while matching the local Java file-input baseline on bundled phiX. If both `in` and `in2` point at stdin, Rust rejects the ambiguous request and tells callers to use interleaved stdin instead.
- Shared BBTools genome-build context controls (`build` and `genome`) are accepted as Rust no-ops for covered FASTA/FASTQ normalization because no reference-index metadata is used; mapping-aware filters such as `idfilter` remain rejected instead of being silently ignored.
- Shared BBTools diagnostic sizing control `testsize` is accepted as a Rust no-op for covered output. Disabled read-breaking controls (`breaklen=0`/non-positive `breaklength`) and disabled quality-recalibration aliases (`recalibrate=f`, `recalibratequality=f`, and `recal=f`) are also accepted as explicit no-ops, while output-affecting positive `breaklen` and enabled recalibration remain rejected instead of being silently ignored.
- Shared BBTools environment/performance parser controls that do not change covered output (`validatebranchless`, `fairqueues`, `fixextensions`, `parallelsort`, `gcbeforemem`, `warnifnosequence`, `warnfirsttimeonly`, `kmg`/`outputkmg`, `forceJavaParseDouble`, `simd`/SIMD sparse toggles, `aws`/`nersc`, `lowmem`, `sidechannelstats`, `entropyk`, and `entropywindow`) are Java-golden-tested as no-ops on paired phiX, while malformed entropy numeric values are rejected like vendored `KmerNormalize`.
- Multipass/countup controls (`target1`, `targetbadpercentilelow`/`tbpl`, `targetbadpercentilehigh`/`tbph`, `abrc`, `discardbadonly1`, and first-pass toss aliases) are parsed into Rust execution settings. Covered `passes=2` single-end and paired phiX cases match Java output, paired phiX keep/hist outputs now match Java for `passes=2/3/4`, intermediate passes apply Java-shaped bad-read target tightening from `targetbadpercentilelow/high`, covered `passes=2` correction staging honors Java's `ecc1`/`eccf` split on simple/noisy mixed-mutant fixtures and real-derived paired phiX, S. cerevisiae, and E. coli stress fixtures with byte-identical `threads=1/2/auto` outputs, multipass keep/toss routing follows Java final-stage toss output semantics, multipass `outuncorrected` is accepted for covered runs but follows Java's final-stage routing rather than preserving intermediate ECC fragments, single-pass ECC normalization/toss routing matches Java with and without `tossbadreads`, generalized real-derived ECC stress verifies Rust correction on controlled paired mutations from bundled phiX plus local S. cerevisiae and E. coli biological datasets, and `countup=t` now applies a Java-shaped relaxed prepass before exact count-up selection: prepass-tossed reads are skipped by default, carried forward with `addbadreadscountup=t`, sorted by low-kmer error burden, read length, expected errors, numeric ID, and read ID, and then normalized with incremental exact kept-kmer table updates. Count-up also honors `minlen`, runs table-based ECC plus overlap-only `ecco=t` mate repair and mark/trim-after-marking on kept reads, and creates renamed keep/toss/depth-bin plus correction-driven `outuncorrected` side streams; local S. cerevisiae and E. coli count-up biological stress now verifies balanced paired keep/toss/depth-bin outputs, ECC keepall behavior, and byte-identical `threads=1/2/auto` output stability, while broader side-routing combinations still need expansion.
- Table-based short-kmer error correction is now implemented for the covered `ecc`, `ecc1`, `ecc2`/`eccf`, `aec`, `cec`, `markerrors`/`meo`, `markuncorrectableerrors`/`mue`, `trimaftermarking`, and tuning-control path (`ecclimit`, `eccmaxqual`, `errorcorrectratio`/`ecr`, `echighthresh`/`echthresh`/`echt`, `eclowthresh`/`eclthresh`/`eclt`, `suflen`/`suffixlen`/`sl`, `prelen`/`prefixlen`/`pl`, `cfl`, and `cfr`). The smoke harness verifies no old fallback note, paired phiX no-error output preservation against the local Java baseline, correction of a representative one-substitution fixture, Rust correction of controlled substitutions in generalized real-derived paired stress fixtures spanning bundled phiX and two local biological datasets, Rust routing of noisier forced-uncorrectable multi-mutation paired reads to keep plus paired `outuncorrected` in the same generalized real-derived harness, Rust correction of staged `passes=2` real-derived paired stress fixtures across `threads=1/2/auto`, staged multipass forced-uncorrectable routing plus marked-quality preservation for `ecc1=f eccf=t`, `ecc1=t eccf=f`, and `ecc=t` on bundled phiX plus local S. cerevisiae and E. coli fixtures, Java-matching `outuncorrected` routing for forced single-end and paired uncorrectable high-quality suspected errors, Java-matching multipass `outuncorrected` routing where `ecc1`-only keeps marked reads but does not emit an uncorrected side stream until the final stage, Java-matching `markuncorrectableerrors=t` quality marking on the forced single-end uncorrectable fixture, Java-matching multi-site `markerrors=t ecclimit=1 cfl=t cfr=f` semantics where Java caps correction attempts but still marks every detected site in the enabled direction, Java-matching `markerrors=t trimaftermarking=t` output on a representative tail-error fixture, Java-matching `passes=2` ECC staging for `ecc1=f eccf=t`, `ecc1=t eccf=f`, and `ecc=t`, Java-matching ECC-enabled keep/toss routing for `target=2 max=2` with both `tossbadreads=f` and `tossbadreads=t`, and Java-shaped strict overlap handling for `ecco=t` plus explicit `ecco=auto` support using a bounded 1%-sample overlap probe before table-based ECC, including overlap-only biological stress that now exercises both a real-derived accepted repair fixture and a compact competing-overlap ambiguity rejection fixture in single-pass and `countup=t` modes across `threads=1/2/auto`. Rust now matches Java's strict overlap short-read gate below 35 bp, matches compact accepted-overlap high-entropy repairs plus compact repetitive/high-confidence/competing-overlap rejections, and the ECC fallback smoke now includes byte-for-byte Java guards for the compact auto-overlap and competing short-overlap ambiguity fixtures. Full Java overlap heuristics are still not collision/overlap equivalent.
- Repeatable real-dataset benchmark harness (`scripts/benchmark_real_dataset.sh`) that runs vendored Java BBNorm and release-mode Rust on the same bundled paired phiX fixture, compares every output byte-for-byte, and records elapsed seconds plus max RSS per iteration, plus Rust thread-scaling benchmarks (`scripts/benchmark_thread_scaling.sh` for bundled phiX, `scripts/benchmark_biological_dataset.sh` for one larger paired dataset under `~/Projects/biological data`, and `scripts/benchmark_biological_matrix.sh` for bounded multi-dataset preset sweeps) that default to `1 2 auto` and check byte-identical outputs across configured Rayon worker counts. `threads=max`/`threads=all` now explicitly choose all available Rayon workers; on shared desktops, validation should prefer explicit low worker counts before max-thread stress. The biological benchmark harness now accepts mode-specific extra args so quick smoke coverage can exercise default exact-count, `countup=t`, correction-heavy `countup=t ecc=t markuncorrectableerrors=t`, plain `k=40`, `k=40 fixspikes=t`, bounded `passes=2`, and bounded `passes=2 ecc=t markuncorrectableerrors=t` through one shared script, while the matrix harness now forwards `MATRIX_EXTRA_ARGS` to run one bounded Rust mode across multiple biological presets. The truncated biological Java-parity harness covers the default working mode, and the working-mode probe now passes for both plain `k=40` and `k=40 fixspikes=t`.
- Explicit errors for unsupported behavior that would otherwise change results silently; wrapper-advertised KmerCoverage sampling knobs now emit notes and keep the supported normalization path for ASAP usability. CallPeaks-only short aliases (`h`, `v`, `w`, `minp`, `maxp`) are still rejected by vendored `KmerNormalize`, but Rust now accepts them as aliases for the supported long peak filters. The vendored Java count-up path is separately guarded on the paired phiX fixture because it fails before producing a normal comparable output set there; Rust now runs an exact count-up keep/toss implementation using a sequential kept-kmer table.
## Known Parity Gaps
- Full multipass BBNorm parity is not complete yet. Covered temp-file orchestration is implemented and Java-golden-tested for compact single-end plus paired phiX `passes=2/3/4` keep/hist outputs, bounded 1k-read paired S. cerevisiae biological `passes=2/3/4` keep/bin/hist outputs, bounded 1k-read paired S. cerevisiae biological `passes=2 ecc=t` keep/bin/hist outputs with and without `markuncorrectableerrors=t`, and representative `passes=2` ECC staging, plus Rust real-derived staged ECC keep/toss stress on paired phiX, S. cerevisiae, and E. coli across `threads=1/2/auto`; staged `outuncorrected` plus marked-quality routing is now covered on bundled phiX and local biological datasets with Java-shaped final-stage-only uncorrected emission.
- Full Bloom/count-min sketch, prefilter, and cardinality/loglog parity is not implemented yet. For ASAP working behavior, Rust now builds direct fixed-memory count-min sketches for constrained main-table `cells`/`matrixbits` settings and explicit `sketchmemory`/`countminmemory` byte budgets, uses a deterministic conservative atomic table for default `bits=32` bounded sketches, supports a real two-stage prefilter-plus-main bounded input sketch for `prehashes`/`prefilterhashes`, constrained `prefiltercells`/`precells`, `filtermemory`/`prefiltermemory`/`filtermem`/`filtermemoryoverride` memory budgets, and `prefiltersize`/`prefilterfraction` memory partitioning when a table memory budget is configured, honors KCountArray-style locked/conservative writes by default plus `lockedincrement=f`/`symmetricwrite=f` independent row increments for bounded main and prefilter sketches, now exposes KCountArray7MTA-style update helpers that return the pre-increment minimum while raising cells to the new target, wraps direct atomic conservative updates in BBTools' 1999-way key-striped lock shape to keep the read-min/raise-all sequence group-atomic under Rayon, skips that lock-stripe allocation entirely for independent-row atomic sketches, and avoids redundant sort/mutex traffic for already-merged engine chunk-map replay. Bounded-sketch unique-kmer reporting now uses BBTools-style hash-adjusted used-fraction estimates, including thresholded `mindepth` estimates matching `KCountArray.estimateUniqueKmers(hashes, mindepth)`, instead of raw occupied-cell counts. Main count-min bucket selection and storage now follow the KCountArray7MTA row structure from the vendored source: row-0 double masking, 6-bit row rotation, row-indexed mask application, KCountArray7MTA FastRandomXoshiro mask generation with BBTools' low/rotated-cell rejection and per-table Java-style seed stepping, KCountArray-style internal array slot placement, and prime-adjusted row lengths under the requested total cell/memory budget; the Rust hot path now fills row buckets incrementally and reuses the selected mask table across all rows for a key. Cardinality/loglog controls, prefilter pass recursion reporting details beyond the current explicit main-table prefilter limit, remaining Java-identical sketch collision details remain explicit gaps; results can still intentionally differ from BBTools approximate-sketch behavior, and exact cardinality is intentionally unavailable in bounded count-min mode.
- Count-up normalization (`countup=t`) now has a Rust implementation using a Java-shaped relaxed prepass, byte-budgeted external sorted temp runs with bounded fan-in compaction, and incremental kept-kmer table/sketch updates, including bounded kept-count sketches when `cells`/`matrixbits`/`sketchmemory` are requested, `addbadreadscountup`/`abrc` carry-forward for prepass-tossed reads, final table updates for tossed reads, `minlen` toss filtering, table-based ECC on kept reads, overlap-only `ecco=t` mate repair, mark/trim-after-marking ECC, `markuncorrectableerrors` quality preservation on uncorrectable reads, uncorrectable-read routing, practical renamed keep/toss/depth-bin/outuncorrected side-stream creation, focused local S. cerevisiae plus E. coli biological stress including forced uncorrectable marked-quality pairs, and byte-identical `threads=1/2/auto` stability on those biological fixtures. Bounded real-data Java validation now also makes the current upstream limitation explicit: vendored Java still crashes in `normalizeInThread` on a 1k-read paired S. cerevisiae `countup=t` slice while Rust completes normally. Rust is not yet BBTools sketch/collision equivalent and still needs deeper large-dataset validation.
- Bounded real-data Java validation now also makes a second upstream limitation explicit on the local paired E. coli slice: vendored Java crashes in `PairStreamer.nextList` for both default and `passes=2`, while Rust completes and emits usable keep/bin/hist outputs on the same 1k-read slice.
- Bounded real-data Java validation now also makes a third upstream limitation explicit on the local paired S. pombe slice: vendored Java can write substantial keep/bin/hist partial outputs and then terminate in an error state after a `PairStreamer.nextList` failure for both default and `passes=2`, while Rust completes and emits usable outputs on the same 1k-read slice.
- The main remaining ECC validation gap is broader Java-equivalent overlap behavior beyond the current compact strict-overlap guards, plus broader noisy biological stress beyond the now-covered staged multipass/countup/single-pass forced-uncorrectable fixtures.
- Real biological Java parity is now covered for a small bounded default-mode slice, and bounded `k=40` plus `k=40 fixspikes=t` now both match Java on that real-data slice after aligning Rust long-kmer duplicate counting with Java's `k>31` table-build behavior and restoring Java-shaped precise spike smoothing for long-kmer fix-spike runs. Broader real-dataset Java parity across additional modes and larger read limits is still not complete.
- Obscure comma-output-list edge cases are partially covered now, but not exhaustively across every output stream and paired short-list combination.
- Two-file paired target=1 exact-output comparisons are flaky in the vendored Java reference on the local phiX fixture because upstream sometimes falls back to a time-seeded random coin; the interleaved target=1 path is golden-tested instead.
- True nondeterministic read selection is implemented for the Rust exact-count path, but distribution-level Java parity for time/thread-local random mode is not a goal yet; deterministic output remains the supported parity baseline.
- Position-specific trim-quality semantics are not fully implemented; comma-separated trim thresholds run with the first threshold and emit a note.
- Full Java ECC parity is not complete yet. The Rust table-based short-kmer correction path can correct representative single substitutions and controlled real-derived paired mutations across bundled and local biological fixtures, mark suspect qualities, trim after marking for a covered tail-error fixture, and route forced uncorrectable reads like Java. Rust now also matches Java's strict overlap short-read gate below 35 bp, applies Java-shaped entropy-driven minimum overlap gating plus expected-mismatch and probability post-filtering, rejects compact repetitive ambiguous, high-confidence mismatch, and competing short-overlap ambiguity fixtures like Java, and matches sequence plus quality output on compact accepted-overlap high-entropy fixtures. Remaining work is now broader than a single compact overlap seam: full BBMerge heuristic parity on more edge cases, complete Java rollback/staging behavior, broader marked-error trimming, and deeper noisy correction stress still need work.
- Error-correction-driven uncorrected output is Java-golden-covered for compact forced uncorrectable single-end and paired fixtures, and real-derived paired biological stress now covers `outuncorrected` plus `markuncorrectableerrors` quality marking on local S. cerevisiae and E. coli across `threads=1/2/auto`; noisier multi-mutation real-derived uncorrectable cases still need deeper stress coverage.
- Peak calling now has representative single-peak and multi-peak Java golden tests, but noisier real-dataset peak fixtures still need coverage before treating this as fully stress-tested.
- Normalization output routing remains sequential to preserve deterministic ordering, but table construction, normalization decision/rename work, kept-output counting for output histograms, sparse histogram/read-histogram analysis, and guarded long-read depth analysis now use Rayon.
## Next Parity Targets
1. Expand the Java reference fixture harness from the current deterministic paired/interleaved coverage into additional target/max/min/error-detection combinations on duplicated high-depth real fixtures.
2. Expand correction-enabled fixtures for `outuncorrected`, mark/trim modes, and multipass staging.
3. Port remaining error detection edge cases and overlap-ECC behavior.
4. Continue tightening BBTools sketch/hash/cardinality parity when output identity matters more than bounded practical behavior.
- Giant-dataset validation now has a guarded Rust-only harness (`scripts/benchmark_giant_safe.sh`) that defaults to release mode, bounded count-min, null sequence outputs, explicit thread/memory settings, and a repo-owned `/proc` monitor for elapsed time plus peak RSS so full-scale runs can be staged without filling disk or repeating the previous desktop-freeze risk. `scripts/install_benchmark_tools.sh` installs `pigz`/`unpigz` either through the system package manager when passwordless admin access is available or locally under `~/.local/bin`; benchmark scripts prepend that path automatically. `scripts/benchmark_java_rust_human.sh` runs a matched Java/Rust benchmark with desktop-safe read limits by default and records hist/rhist/output comparison status.