ycd-reader

ycd-reader is a Rust library for reading base-10 compressed digit files (.ycd) produced by y-cruncher.

Each YCD payload stores up to 19 decimal digits in an unsigned 64-bit little-endian block. The library restores those blocks to digit strings and returns them in caller-selected processing units.

In-memory index: YcdIndex

YcdIndex builds an in-memory index over a contiguous set of YCD files by reading every file's header once. Subsequent read_digits calls use binary search to locate the relevant file(s) and seek directly to the target block, leaving all other files untouched.

This is the recommended path when you have a large, mostly-static collection of YCD files and want to avoid scanning every header on each request.

Lifecycle

Build once         Fast reads           Refresh when files change
─────────────      ──────────────────   ───────────────────────────
YcdIndex::build    index.read_digits    index.rebuild

Building the index

use std::io;
use ycd_reader::YcdIndex;

fn main() -> io::Result<()> {
    let files = [
        "Pi - Dec - Chudnovsky - 0.ycd",
        "Pi - Dec - Chudnovsky - 1.ycd",
    ];

    // Build the index once: reads all headers and validates continuity.
    let index = YcdIndex::build(&files)?;
    println!("{} files indexed", index.len());

    Ok(())
}

Reading digits via the index

use std::io;
use ycd_reader::YcdIndex;

fn main() -> io::Result<()> {
    let files = [
        "Pi - Dec - Chudnovsky - 0.ycd",
        "Pi - Dec - Chudnovsky - 1.ycd",
    ];

    let index = YcdIndex::build(&files)?;

    // Fast random-access — only the file(s) containing the requested range
    // are opened.  All other files are skipped entirely.
    let digits = index.read_digits(999_995, 20)?;
    assert_eq!(digits, "45815130927562832084");

    Ok(())
}

Updating the index

When the file set changes (files added, removed, or replaced), call rebuild to discard the old index and build a fresh one. If the rebuild fails the old index is left intact.

# use std::io;
# use ycd_reader::YcdIndex;
# fn main() -> io::Result<()> {
# let files: &[&str] = &[];
let mut index = YcdIndex::build(files)?;

// Later, after files change:
let new_files = ["Pi - Dec - Chudnovsky - 0.ycd", "Pi - Dec - Chudnovsky - 1.ycd"];
index.rebuild(&new_files)?;
# Ok(())
# }

Stale-index detection

Each time a file is actually read, its current file_size and last-modified time are compared against the snapshot recorded at build time. A mismatch causes read_digits to return io::ErrorKind::InvalidData with an explicit "index is stale" message. There is no silent fallback to a full-scan path — the caller must explicitly call rebuild to refresh the index.

YcdIndex::read_digits errors

Random-access: read a specific digit range

YcdFileUtil::read_digits reads exactly length digits starting at a 1-based absolute position from an ordered list of contiguous YCD files. The entire file list is validated before any payload I/O begins, and the function seeks directly to the compressed block that contains the first requested digit — no bytes before the target block are read.

use std::io;
use ycd_reader::YcdFileUtil;

fn main() -> io::Result<()> {
    let files = [
        "Pi - Dec - Chudnovsky - 0.ycd",
        "Pi - Dec - Chudnovsky - 1.ycd",
    ];

    // Read 20 digits starting at position 999,995 (1-based, decimal digits only).
    // Position 1 is the first digit after the decimal point — "1" in "3.14159…"
    // The integer part, sign, and decimal point are never included.
    let digits = YcdFileUtil::read_digits(&files, 999_995, 20)?;
    assert_eq!(digits, "45815130927562832084");

    Ok(())
}

Signature

pub fn read_digits<P: AsRef<Path>>(
    files: &[P],
    one_based_start_position: usize,
    length: usize,
) -> io::Result<String>

• files — An ordered slice of YCD file paths. The first file does not need to have BlockID = 0; positions are derived from each file's header. The files must be contiguous: each file must start immediately after the preceding file ends. All files are validated before any I/O on the payload.
• one_based_start_position — 1-based index of the first digit to return. Position 1 is the first decimal digit (immediately after the "3." in Pi). The integer part, sign, and decimal point are never returned.
• length — Number of digits to return. The result string is exactly length ASCII bytes.

File-range rules

A file with TotalDigits == 0 is assumed to contain exactly Blocksize digits. If the actual payload is shorter than the logical range, UnexpectedEof is returned.

Errors

Memory note

read_digits allocates a String of exactly length bytes to hold the result. A large length requires the same amount of heap memory.

Single-file reading

use std::io;
use ycd_reader::YcdSeqBlockStream;

fn main() -> io::Result<()> {
    let mut stream = YcdSeqBlockStream::new("digits-0.ycd", 1_000)?;

    while stream.has_next() {
        let unit = stream.next()?;
        println!(
            "unit {} starts at digit {} and contains {} digits",
            unit.process_no,
            unit.start_digit,
            unit.value.len()
        );
    }

    Ok(())
}

The processing-unit size must be at least 19. The final unit can be shorter than the requested size.

Resuming single-file reading from an arbitrary position

YcdSeqBlockStream::new_from opens a YCD file and positions the stream so that the first digit returned is at a caller-specified 1-based absolute position. This lets an application save unit.start_digit at any point and later resume reading from exactly that position.

use std::io;
use ycd_reader::YcdSeqBlockStream;

fn main() -> io::Result<()> {
    // Resume from position 500_000 (1-based, relative to all digits of Pi).
    let mut stream = YcdSeqBlockStream::new_from("digits-0.ycd", 1_000, 500_000)?;

    while stream.has_next() {
        let unit = stream.next()?;
        println!(
            "unit {} starts at digit {} and contains {} digits",
            unit.process_no,
            unit.start_digit,
            unit.value.len()
        );
    }

    Ok(())
}

start_position must be inside the digit range covered by the file (digit_start .. digit_start + digit_length - 1, inclusive). The stream seeks directly to the compressed block that contains start_position; no bytes before that block are read.

Reading contiguous files (sequential)

YcdMultiFileStream joins an explicit ordered list of YCD files. It validates that each file starts immediately after the preceding file and allows a processing unit to cross file boundaries.

use std::io;
use ycd_reader::YcdMultiFileStream;

fn main() -> io::Result<()> {
    let files = ["digits-0.ycd", "digits-1.ycd", "digits-2.ycd"];
    let mut stream = YcdMultiFileStream::new(&files, 1_000)?;

    while stream.has_next() {
        let unit = stream.next()?;
        consume(&unit.value);
    }

    Ok(())
}

fn consume(_digits: &str) {}

Resuming multi-file reading from an arbitrary position

YcdMultiFileStream::new_from opens an ordered list of contiguous YCD files and positions the stream at a caller-specified 1-based absolute digit position. All files in the list are validated for header correctness and list continuity before any payload I/O begins. Files that end before start_position are skipped; only the file that contains start_position and the files that follow it are opened for streaming.

use std::io;
use ycd_reader::YcdMultiFileStream;

fn main() -> io::Result<()> {
    let files = ["digits-0.ycd", "digits-1.ycd", "digits-2.ycd"];

    // Resume from position 1_500_000 (inside the second file).
    let mut stream = YcdMultiFileStream::new_from(&files, 1_000, 1_500_000)?;

    while stream.has_next() {
        let unit = stream.next()?;
        consume(&unit.value);
    }

    Ok(())
}

fn consume(_digits: &str) {}

Header inspection

YcdFileUtil::get_ycd_header returns recognized header fields, and YcdFileUtil::get_header_size returns the byte offset at which compressed blocks begin. Headers can use CRLF or LF line endings and are not restricted to a fixed buffer size.

Only base-10 YCD files are supported. Missing or invalid required fields, invalid payload blocks, truncated files, and noncontiguous file lists are reported as std::io::Error.

Tests

cargo test

The integration suite uses tests/ycd/Pi - Dec - Chudnovsky_3000000.txt as the golden result. It covers:

• read_digits: position 1 with various lengths, arbitrary middle positions, 19-digit block boundaries (before/after/crossing), YCD file boundaries (before/after/crossing) in both 1M×3 and 2M+1M layouts, leading zeroes, reads spanning multiple files, the complete error-contract table, and direct-seek unit tests that verify block_index, offset_in_block, and blocks_to_read without running full I/O.
• Sequential streaming: all three million digits with multiple processing-unit sizes, file-boundary crossings, final partial units, and malformed inputs.
• new_from (sequential stream with start position): position-1 equivalence with new, mid-block seeks, block-boundary seeks, last-digit reads, non-zero BlockID files, multi-file boundary and mid-file starts, unit crossing file boundaries, and the complete error-contract table.

The published crates.io package excludes the large tests/ycd/** fixture data to stay within the upload size limit. Run the full integration suite from a Git checkout of this repository.

License

Licensed under either of the following, at your option:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)