Module fdb 

FDB (Flanking Database) Format Module

Converts TSV flanking sequences to compressed binary FDB format. Uses external sorting with LZ4 compression for temporary files and zstd compression for final gene blocks.

File Format

[Header]
  - Magic: "FLANKDB\0" (8 bytes)
  - Version: u32 (4 bytes)
  - Gene count: u32 (4 bytes)
  - Index offset: u64 (8 bytes)
[Gene Blocks]
  - Zstd-compressed TSV data per gene
[Index]
  - Gene name → (offset, compressed_len, record_count)

Pipeline

1. Read TSV line by line → FlankingRecord
2. External sort by gene name (parallel, LZ4-compressed temp files)
3. Stream sorted records → group by gene → zstd compress → write FDB

Performance

• Memory-efficient: External sort handles files larger than RAM
• Compression: ~10x reduction in file size

Structs

FlankingRecord

Flanking record for sorting Keep this compact - extsort-iter buffers these in memory

Functions

build

Builds an FDB file from TSV input.

build_from_sorted

Build FDB from a pre-sorted TSV file (streaming, memory-efficient).