sfbinpack 0.6.3

Library to read Stockfish Binpacks
Documentation

Stockfish Binpack

Rust port of the Stockfish binpack reader from the C++ version.

Binpacks store chess positions and their evaluations in a compact format. Instead of storing complete positions, they store the differences between moves. This makes them very space efficient - using only 2.5 bytes per position on average. See Anatomy for more details.

The library also works on wasm32 targets. In wasm, prefer the in-memory APIs such as CompressedTrainingDataEntryReader::from_bytes, CompressedTrainingDataEntryReader::from_slice, CompressedTrainingDataEntryWriter::new_in_memory, and CompressedTrainingDataEntryWriter::into_bytes.

Compile

Starting with crate version 0.6.3, fast BMI2 runtime dispatch is enabled by default. You do not need to pass a feature flag or add one in Cargo.toml.

cargo build --release

In Cargo.toml:

[dependencies]
sfbinpack = "0.6.3"

Usage

Run the following Cargo command in your project directory:

cargo add sfbinpack

use sfbinpack::CompressedTrainingDataEntryReader;

fn main() {
    let file = File::open("data.binpack").unwrap();
    let mut reader = CompressedTrainingDataEntryReader::new(file).unwrap();

    while reader.has_next() {
        let entry = reader.next();

        println!("entry:");
        println!("fen {}", entry.pos.fen().unwrap());
        println!("uci {:?}", entry.mv.as_uci());
        println!("score {}", entry.score);
        println!("ply {}", entry.ply);
        println!("result {}", entry.result);
        println!("\n");
    }
}

WASM Usage

use sfbinpack::{CompressedTrainingDataEntryReader, CompressedTrainingDataEntryWriter};

fn read_binpack(bytes: Vec<u8>) {
    let mut reader = CompressedTrainingDataEntryReader::from_bytes(bytes).unwrap();

    while reader.has_next() {
        let entry = reader.next();
        let _ = entry;
    }
}

fn write_binpack(entry: sfbinpack::TrainingDataEntry) -> Vec<u8> {
    let mut writer = CompressedTrainingDataEntryWriter::new_in_memory().unwrap();
    writer.write_entry(&entry).unwrap();
    writer.into_bytes().unwrap()
}

Browser Test Page

There is a simple browser test page in wasm-test/.

Build the wasm package for the web target:

cargo install wasm-pack
wasm-pack build --target web --out-dir wasm-test/pkg

Then serve the repository root with any static file server and open wasm-test/index.html.

The page imports the generated wasm-test/pkg/sfbinpack.js bundle and uses the exported parse_binpack_chunk(bytes, preview_limit) function to inspect a selected .binpack file.

The browser page does not load the entire binpack into memory. It reads the file chunk-by-chunk using File.slice(), parses each BINP chunk separately, and aggregates counts and preview rows client-side.

The preview page also stops reading more chunks once it has collected the requested number of preview rows, so requesting 10 rows does not scan the rest of a very large file.

The demo page also includes a built-in link and button for the repository's example file, test/ep1.binpack, so you can test the wasm page without uploading a local file.

GitHub Pages deployment is configured in .github/workflows/pages.yml. On pushes to main, GitHub Actions builds the wasm bundle into wasm-test/pkg and publishes the wasm-test/ directory as the Pages site.

After the first deployment, enable GitHub Pages in the repository settings and select GitHub Actions as the source if it is not already selected.

More examples can be found in the examples directory.
If you are doing some counting keep in mind to use a u64 type for the counter.

Examples

To run the examples in the examples directory, use the following command:

cargo run --release --example <example_name>

binpack_reader - Read a binpack file and print the contents. binpack_writer - Write a binpack file from a list of positions.

The examples use std::fs for local files, but the library itself does not require filesystem access. For wasm, use the in-memory APIs above.

Performance Comparison

Slightly faster when compiled with bmi2 because of _pdep_u64 trick which is missing in the upstream version.

Anatomy

Binpack

Compression

When compressing new data, it is advised to store the entire continuation of the actual game. This will allow for a much better compression ratio.
Failure to do so will result in a larger file size, than compared to other alternatives.

License

GNU General Public License v3.0

https://www.gnu.org/licenses/gpl-3.0.html