# Stockfish Binpack
Rust port of the Stockfish binpack reader from the [C++ version](https://github.com/official-stockfish/Stockfish/blob/tools/src/extra/nnue_data_binpack_format.h).
Binpacks store chess positions and their evaluations in a compact format.
Instead of storing complete positions, they store the differences between moves.
This makes them very space efficient - using only 2.5 bytes per position on
average. See [Anatomy](#anatomy) for more details.
The library also works on `wasm32` targets. In wasm, prefer the in-memory APIs
such as `CompressedTrainingDataEntryReader::from_bytes`,
`CompressedTrainingDataEntryReader::from_slice`,
`CompressedTrainingDataEntryWriter::new_in_memory`, and
`CompressedTrainingDataEntryWriter::into_bytes`.
## Compile
Starting with crate version `0.6.3`, fast BMI2 runtime dispatch is enabled by default.
You do not need to pass a feature flag or add one in `Cargo.toml`.
```bash
cargo build --release
```
In `Cargo.toml`:
```
[dependencies]
sfbinpack = "0.6.3"
```
## Usage
Run the following Cargo command in your project directory:
```shell
cargo add sfbinpack
```
```rust
use sfbinpack::CompressedTrainingDataEntryReader;
fn main() {
let file = File::open("data.binpack").unwrap();
let mut reader = CompressedTrainingDataEntryReader::new(file).unwrap();
while reader.has_next() {
let entry = reader.next();
println!("entry:");
println!("fen {}", entry.pos.fen().unwrap());
println!("uci {:?}", entry.mv.as_uci());
println!("score {}", entry.score);
println!("ply {}", entry.ply);
println!("result {}", entry.result);
println!("\n");
}
}
```
### WASM Usage
```rust
use sfbinpack::{CompressedTrainingDataEntryReader, CompressedTrainingDataEntryWriter};
fn read_binpack(bytes: Vec<u8>) {
let mut reader = CompressedTrainingDataEntryReader::from_bytes(bytes).unwrap();
while reader.has_next() {
let entry = reader.next();
let _ = entry;
}
}
fn write_binpack(entry: sfbinpack::TrainingDataEntry) -> Vec<u8> {
let mut writer = CompressedTrainingDataEntryWriter::new_in_memory().unwrap();
writer.write_entry(&entry).unwrap();
writer.into_bytes().unwrap()
}
```
### Browser Test Page
There is a simple browser test page in `wasm-test/`.
Build the wasm package for the web target:
```bash
cargo install wasm-pack
wasm-pack build --target web --out-dir wasm-test/pkg
```
Then serve the repository root with any static file server and open `wasm-test/index.html`.
The page imports the generated `wasm-test/pkg/sfbinpack.js` bundle and uses the exported
`parse_binpack_chunk(bytes, preview_limit)` function to inspect a selected `.binpack` file.
The browser page does not load the entire binpack into memory. It reads the file chunk-by-chunk
using `File.slice()`, parses each BINP chunk separately, and aggregates counts and preview rows
client-side.
The preview page also stops reading more chunks once it has collected the requested number of
preview rows, so requesting 10 rows does not scan the rest of a very large file.
The demo page also includes a built-in link and button for the repository's example file,
`test/ep1.binpack`, so you can test the wasm page without uploading a local file.
GitHub Pages deployment is configured in `.github/workflows/pages.yml`.
On pushes to `main`, GitHub Actions builds the wasm bundle into `wasm-test/pkg` and publishes
the `wasm-test/` directory as the Pages site.
After the first deployment, enable GitHub Pages in the repository settings and select
`GitHub Actions` as the source if it is not already selected.
_More examples can be found in the [examples](./examples) directory._
_If you are doing some counting keep in mind to use a `u64` type for the counter._
## Examples
To run the examples in the `examples` directory, use the following command:
```shell
cargo run --release --example <example_name>
```
`binpack_reader` - Read a binpack file and print the contents.
`binpack_writer` - Write a binpack file from a list of positions.
The examples use `std::fs` for local files, but the library itself does not
require filesystem access. For wasm, use the in-memory APIs above.
## Performance Comparison
Slightly faster when compiled with bmi2 because of _pdep_u64 trick which is missing in the upstream version.
## Anatomy
## Compression
When compressing new data, it is advised to store the entire continuation of the actual game.
This will allow for a much better compression ratio.
Failure to do so will result in a larger file size, than compared to other alternatives.
## License
GNU General Public License v3.0
<https://www.gnu.org/licenses/gpl-3.0.html>