[](https://git.sr.ht/~alip/jja)
Swiss army knife for chess file formats
=======================================
[](https://builds.sr.ht/~alip/jja?)
[](https://crates.io/crates/jja)
[](https://docs.rs/jja)
[](https://crates.io/crates/jja)
[](https://deps.rs/repo/sourcehut/~alip/jja)
[](https://github.com/mkenney/software-guides/blob/master/STABILITY-BADGES.md#beta)
[](https://git.sr.ht/~alip/jja/tree/main/item/COPYING)
This is ___jja___, a command line utility to interact with various chess file
formats. It is still in its early stages of development. The initial intention
of the author was to convert their opening books which were saved with ChessBase's
proprietary [CTG](https://www.chessprogramming.org/CTG) format to the free and open
[PolyGlot](http://hgm.nubati.net/book_format.html) format. Overtime they intend to
add support for other chess file formats (
[cbh](https://www.chessprogramming.org%2FChessBase_%28Database%29%23Formats),
[epd](https://www.chessprogramming.org/Extended_Position_Description),
[pgn](https://www.chessprogramming.org/Portable_Game_Notation),
[si4](https://www.chessprogramming.org/SCID#Database_Format),
[si5](https://www.chessprogramming.org/SCID#Database_Format)
and so on).
# Formats
As of version 0.3.1, ___jja___ supports reading/querying:
- [PolyGlot](http://hgm.nubati.net/book_format.html), aka `bin`
- [Arena](https://www.chessprogramming.org/ABK), aka `abk`
- [ChessBase](https://www.chessprogramming.org/CTG), aka `ctg`
- [ChessMaster](https://www.chessprogramming.org/Chessmaster), aka `obk`
(version 1 and 2, w\o,with text notes)
opening book files, whereas it supports writing/converting to:
- [PolyGlot](http://hgm.nubati.net/book_format.html), aka `bin`
- [Arena](https://www.chessprogramming.org/ABK), aka `abk`
opening book files.
During opening book conversion, ___jja___ uses the information provided in various
input opening book formats to come up with a **move weight** which accompanies the
move in the [PolyGlot](http://hgm.nubati.net/book_format.html) opening file.
___jja___ also writes some custom numbers in the **learn** field, such as
[NAGs](https://en.wikipedia.org/wiki/Numeric_Annotation_Glyphs) during `ctg`
conversion or **priority** during `abk` conversion. You may disable this custom
usage using `--no-learn` as it may confuse other software making use of this field.
Note, [Arena](https://www.chessprogramming.org/ABK), aka `abk`, opening book file
writing support is only supported from
[ChessBase](https://www.chessprogramming.org/CTG), aka `ctg` books. Use the command
line flags `--author`, `--comment`, `--probability-priority`, `--probability-games`,
`--probability-win-percent` to configure [ABK](https://www.chessprogramming.org/ABK)
header metadata. Game statistics (minimum number of games/wins, win percentages for
both sides) are managed automatically by ___jja___.
In-place editing for [Arena](https://www.chessprogramming.org/ABK) opening books is
also possible using `-i, --in-place=SUFFIX` command line option. Conversion from
[PolyGlot](http://hgm.nubati.net/book_format.html), aka `bin`, and
[ChessMaster](https://www.chessprogramming.org/Chessmaster), aka `obk` opening books
to [Arena](https://www.chessprogramming.org/ABK), aka `abk` opening book files is
planned for a future release.
# PGN Book Making
Since version 0.4.0, ___jja___ can make PolyGlot books out of PGN files.
This feature is similar to `polyglot make-book` with the following differences:
1. ___jja___ may directly read compressed PGN files .pgn.{bz2,gz,lz4,zst}
2. ___jja___ can process PGN files bigger than your system's available memory, by
persisting statistics in a temporary [RocksDB](https://rocksdb.org/) database.
3. ___jja___ scales move weights by default to prevent potential overflows with huge
PGN files, use `--no-scale` to disable.
4. ___jja___ can filter moves using *Filter Expressions*, allowing the user to
filter-out unwanted games and create specialised opening books.
## Filter Expressions
The filter expression string should contain filter conditions, which consist of a
tag name, a comparison operator, and a value. The following operators are supported:
- `>` (greater than)
- `>=` (greater than or equal to)
- `<` (less than)
- `<=` (less than or equal to)
- `=` (equal to)
- `!=` (not equal to)
- `=~` (regex match, case insensitive)
- `!~` (negated regex match, case insensitive)
Filter conditions can be combined using the following logical operators:
- `AND` (logical AND)
- `OR` (logical OR)
Example:
`--filter="Event =~ World AND White =~ Carlsen AND ( Result = 1-0 OR ECO = B33 )"`
Supported tags are Event, Site, Date, UTCDate, Round, Black, White, Result,
BlackElo, WhiteElo, BlackRatingDiff, WhiteRatingDiff, BlackTitle, WhiteTitle, ECO,
Opening, TimeControl, Termination, and ScidFlags.
In addition to these are four special variables, namely, Player, Elo, Title, and
RatingDiff. These variables may be used to match the relevant header from either
one of the sides. E.g the filter:
`--filter="Player =~ Carlsen"`
is functionally equivalent to
`--filter="( White =~ Carlsen OR Black =~ Carlsen )"`
Note: The filtering is designed to be simple and fast. The tokens, including
parantheses are split by whitespace. Quoting values is not allowed. For more
sophisticated filtering needs, use pgn-extract.
### Scid Flags
Scid uses one character flags, `DWBMENPTKQ!?U123456`, for each field where:
- `D` - Deleted
- `W` - White opening
- `B` - Black opening
- `M` - Middlegame
- `E` - Endgame
- `N` - Novelty
- `P` - Pawn structure
- `T` - Tactics
- `K` - Kingside play
- `Q` - Queenside play
- `!` - Brilliancy
- `?` - Blunder
- `U` - User-defined
- `1..6` - Custom flags
It is ill-advised to rely on the order of the characters flags.
Use a regex match if/when you can.
## Tips and Tricks about PGN Book Making
0. The defaults run best on my laptop and in my personal benchmarks on the
[SourceHut build
server](https://builds.sr.ht/~alip/jja/commits/main/benchmark.yml),
they're not universal truth.
1. ___jja___ processes input PGN files in parallel. You can use this to your
advantage by giving many split PGNs as input to increase parallelism and
performance.
2. Increasing batch size is good as long as you have constant memory usage. When
threads can't keep up, you'll get increased memory usage so that's the point you
really know your limit.
3. Try increasing max open files to the point you don't get too many open files from
your operating system. You may specify `--max-open-files=-1` to keep files always
open.
4. Try different compression algorithms for the temporary
[RocksDB](https://rocksdb.org/) database, or try disabling compression completely
if you have enough space. ___jja___ writes the temporary
[RocksDB](https://rocksdb.org/) database in the same directory as the first
pgn file argument. The default compression algorithm,
[Lz4](https://lz4.github.io/lz4/), and the default compression level, `4`, are
aimed at speedy conversion with relatively moderate space usage. If you run out
of space during conversion, try to use an algorithm like
[Zstd](https://github.com/facebook/zstd) with an "ultra" level, ie. a level
greater or equal to `20`.
5. Use filters which are processed during PGN traversal when possible. Due to the
fact that these filters are matched before writing the game data to the temporary
database, when use wisely, they may have a vast on impact space and memory costs
and therefore improve overall performance. These filters are `--filter=<expr>`,
`--max-ply=<ply>`, `--only-white`, and `--only-black`.
# Install
To compile from source, use `cargo install jja`. This requires the [Rust
Toolchain](https://rustup.rs/) to be installed. In addition you are going to need
[OpenSSL](https://www.openssl.org/) libraries on
[UNIX](https://en.wikipedia.org/wiki/Unix) systems. Moreover you need
[liburing](https://github.com/axboe/liburing) on [Linux](https://kernel.org). If
you're on a [Linux](https://kernel.org) system older than
[5.1](https://en.wikipedia.org/wiki/Io_uring#History) or you are unable to install
[liburing](https://github.com/axboe/liburing) for another reason, you may disable
the feature by building ___jja___ with `cargo install jja --no-default-features`.
As an alternative, release builds of ___jja___ are hosted on
[chesswob.org](https://chesswob.org/jja) for
[64-bit](https://en.wikipedia.org/wiki/64-bit_computing)
[Linux](https://en.wikipedia.org/wiki/Linux) and
[Windows](https://en.wikipedia.org/wiki/Microsoft_Windows). These versions are
signed by [GnuPG](https://gnupg.org/), using key
[D076A377FB27DE70](https://keybase.io/alip/pgp_keys.asc). To install, acquire the
latest version from [chesswob.org](https://chesswob.org/jja), verify the
[checksum](https://en.wikipedia.org/wiki/Checksum) and the
[GnuPG](https://gnupg.org/)
[signature](https://en.wikipedia.org/wiki/Digital_signature):
```
$> export JJA_VERSION=0.4.0
$> export JJA_FLAVOUR=glibc
$> curl https://keybase.io/alip/pgp_keys.asc | gpg --import
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13292 100 13292 0 0 13535 0 --:--:-- --:--:-- --:--:-- 26584
gpg: key D076A377FB27DE70: public key "Ali Polatel (Caissa AI) <alip@caissa.ai>" imported
gpg: Total number processed: 1
gpg: imported: 1
$> for f in jja-${JJA_VERSION}-${JJA_FLAVOUR}.bin{,.sha512sum,.sha512sum.asc}; do wget -q https://chesswob.org/jja/${f}; done
$> gpg --verify jja-${JJA_VERSION}.bin.sha512sum.asc jja-${JJA_VERSION}.bin.sha512sum
gpg: Signature made Sun Mar 19 20:52:41 2023 CET
gpg: using RSA key 5DF763560390A149AC6C14C7D076A377FB27DE70
gpg: Good signature from "Ali Polatel (Caissa AI) ...
$> sha512sum -c jja-${JJA_VERSION}.bin.sha512sum
jja: OK
$> sudo install -m755 jja-${JJA_VERSION}-${JJA_FLAVOUR}.bin /usr/local/bin
```
Finally you may download the builds of the latest git version via the
[SourceHut build server](https://builds.sr.ht/~alip/jja?). There're three flavours,
`windows`, `linux-glibc`, and `linux-musl`. Simply browse to the latest build and
download the artifact listed on the left. Note: these artifacts are kept for 90
days.
# Usage
- Use `jja info` to get brief information about the chess file.
- Use `jja find` to search for a position in a chess file.
- Use `jja edit` to edit opening book files and convert to
[PolyGlot](http://hgm.nubati.net/book_format.html) files.
- Use `jja make` to compile
[PGN](https://en.wikipedia.org/wiki/Portable_Game_Notation) files into
[PolyGlot](http://hgm.nubati.net/book_format.html) opening books.
___jja___ determines the type of the file using its file extension. Files with the
extension `.bin` are considered [PolyGlot](http://hgm.nubati.net/book_format.html)
books. Files with the extension `.ctg` are considered
[ChessBase](https://www.chessprogramming.org/CTG) books. Files with the extension
`.abk` are considered [Arena](https://www.chessprogramming.org/ABK) books. Files
with extension `.obk` are considered
[ChessMaster](https://www.chessprogramming.org/Chessmaster) books.
By default if the standard output is a
[TTY](https://en.wikipedia.org/wiki/Computer_terminal), ___jja___ will display
information using fancy tables. Use `--porcelain` command line option to get the
output in [CSV](https://en.wikipedia.org/wiki/Comma-separated_values)
(comma-separated values) format instead.
# Demo
<img src="https://chesswob.org/jja/jja-intro-20230326.gif"
alt="jja intro"
title="jja intro"
/>
# Acknowledgements
Thanks to ___Steinar H. Gunderson___, for publishing the [CTG
Specification](https://web.archive.org/web/20210129162445/https://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=2319),
and authoring the [remoteglot](http://git.sesse.net/?p=remoteglot;a=summary) tool:
The [CTG](https://www.chessprogramming.org/CTG) probing code in ___jja___ is very
directly ported from their
[C](https://en.wikipedia.org%2Fwiki%2FC_%28programming_language%29%) probing code,
and the specification has been an enormous help in clearing up various rough edges.
Thanks to ___Fabien Letouzey___, the author of the original
[PolyGlot](http://hgm.nubati.net/book_format.html) software: The
[PolyGlot](http://hgm.nubati.net/book_format.html) probing, book making and merging
code in ___jja___ is mostly ported from their respective
[C](https://en.wikipedia.org%2Fwiki%2FC_%28programming_language%29%) code. Thanks to
___Michel Van den Bergh___, the author of pg\_utils, a collection of tools to
interact with [PolyGlot](http://hgm.nubati.net/book_format.html) opening books: The
[PolyGlot](http://hgm.nubati.net/book_format.html) book editing code of ___jja___
uses many ideas and code excerpts from pg\_utils. Thanks to ___Peter Österlund___,
the author of [DroidFish](https://www.chessprogramming.org/DroidFish): The
[ABK](https://www.chessprogramming.org/ABK) opening book interface code in ___jja___
makes use of ideas and code excerpts from
[DroidFish](https://www.chessprogramming.org/DroidFish). Thanks to ___Jens
Nissen___, the author of [ChessX](https://chessx.sourceforge.io/): The
[CTG](https://www.chessprogramming.org/CTG) and
[ABK](https://www.chessprogramming.org/ABK) probing codes in ___jja___ use ideas and
code excerpts from [ChessX](https://chessx.sourceforge.io/). Thanks to
[LiChess](https://lichess.org), the best chess website on the planet. The `quote`
command of ___jja___ has a selection of quotes imported from the
[LiChess](https://lichess.org) codebase. Thanks to ___Shane Hudson___, the author of
[Scid vs. PC](https://scidvspc.sourceforge.net/): The `jja eco` command uses the
[ECO](https://www.chessprogramming.org/ECO) classification which has been done by
the [Scid](https://scidvspc.sourceforge.net/) project. In addition, the
[PolyGlot](http://hgm.nubati.net/book_format.html) editing code of ___jja___ uses
ideas and code from [Scid](https://scidvspc.sourceforge.net/). Thanks to ___Marcus
Bufett___, the author of
[chess-tactics-cli](https://github.com/marcusbuffett/chess-tactics-cli): The
chessboard displaying code in [PolyGlot](http://hgm.nubati.net/book_format.html) and
[ABK](https://www.chessprogramming.org/ABK) edit screens is borrowed from
[chess-tactics-cli](https://github.com/marcusbuffett/chess-tactics-cli).
# License
__jja__ is free software: you can redistribute it and/or modify it under the
terms of the __GNU General Public License__ as published by the Free Software
Foundation, either __version 3__ of the License, or (at your option) any later
version.
This program is distributed in the hope that it will be useful, but __WITHOUT
ANY WARRANTY__; without even the implied warranty of __MERCHANTABILITY__ or
__FITNESS FOR A PARTICULAR PURPOSE__. See the GNU General Public License for
more details.
You should have received a copy of the GNU General Public License along with
this program. If not, see <http://www.gnu.org/licenses/>.
# Bugs
```
Hey you, out there beyond the wall,
Breaking bottles in the hall,
Can you help me?
```
Report bugs to ___jja___'s bug tracker at <https://todo.sr.ht/~alip/jja/>.
Attaching poems with the bug report encourages consideration tremendously.
# Jin, Jîyan, Azadî
I've started hacking this on International Women's Day 2023, a day to honor the
achievements of women and advocate for their rights worldwide. As a person of
Kurdish heritage, I am particularly moved by the slogan "Woman, Life, Freedom,"
which has become a symbol of resistance against oppression and a call for
equality. In the spirit of free software and free speech, I strive to
contribute to the creation of a more just and inclusive society, where every
human being is granted the freedom to express themselves and pursue their
dreams. I also honor the memory of Mahsa Amini, whose tragic death reminds us
of the urgent need to fight for women's freedom and safety.
More on Wikipedia,
[WikiPedia::Woman,_Life,_Freedom](https://en.wikipedia.org%2Fwiki%2FWoman%2C_Life%2C_Freedom%)
# ChangeLog
## 0.4.0
- replace [atty](https://crates.io/crates/atty) with the better maintained
[is-terminal](https://crates.io/crates/is-terminal) crate.
- replace [default-editor](https://crates.io/crates/default-editor) with the more
advanced [dialoguer](https://crates.io/crates/dialoguer) crate.
- replace [colored](https://crates.io/crates/colored) with the more portable
[console](https://crates.io/crates/console) crate.
- match learned `-i`, `--irreversible` to prefer irreversible moves during random
playouts. Pawn moves, captures, moves that destroy castling rights, and moves that
cede en-passant are irreversible.
- replace the [progress_bar](https://crates-io/crates/progress_bar)
crate with the more portable and advanced
[indicatif](https://crates.io/creates/indicatif) crate.
- make learned `-B <games>`, `--batch-size=<games>` to determine write batch size in
number of games
- Linux builds require [liburing](https://github.com/axboe/liburing) to be installed
by default. This may be disabled using `--no-default-features` on installation.
- make learned about `ScidFlags`, a set of character flags used by the
[Scid](https://scidvspc.sourceforge.net/) software, which may be used in filter
expressions
- make learned `--debug` flag to print information on matching filter expressions
- quote learned many more chess quotes which were imported from goodreads.com
- make learned `--max-open-files=<int>` to specify the maximum number of open files
per thread opened by the temporary [RocksDB](https://rocksdb.org) database.
- match command is now multithreaded and uses as many threads as there're cpus on
the machine by default. Use `-T=<threads>`, `--threads=<threads>` or the
`JJA_NPROC` environment variable to override
- fix match to alternate colour between books during random playouts
- quote accepts an optional index argument to print the quote at the specified
index, rather than a random quote
- quote learned many more chess quotes which were imported from archive.org,
goodreads.com and nitter.net
- make filter learned special variables `Player`, `Elo`, `Title`, and `RatingDiff`
to match the relevant field from either colour
- make learned `--filter=<expression>` to filter PGN games by headers. The filtering
is designed to be simple and fast. The tokens, including parantheses are split by
whitespace. Quoting variables is not allowed. See `jja make --help` for more
information on Filter Expressions
- find learned `-t <max-ply>`, `--tree=<max-ply>` to display lines from the opening
book as a tree using the nice [termtree](https://crates.io/crates/termtree) crate
- drop the build dependency upon the [built](https://crates.io/crates/built) crate
- drop the unused dependency on the
[ħyphenation](https://crates.io/crates/hyphenation) crate
- replace [csv](https://crates.io/crates/csv) crate with the more
lightweight [quick-csv](https://crates.io/crates/quick-csv) crate
- replace [cli-table](https://crates.io/crates/cli-table) crate with the more
lightweight [prettytable-rs](https://crates.io/crates/prettytable-rs) crate
- use [pgn-reader](https://crates.io/crates/pgn-reader) crate instead of
[pgnparse](https://crates.io/crates/pgnparse) for parsing the `--pgn`
commandline option
- strip off the unneeded
[cli-table-derive](https://crates.io/crates/cli-table-derive) dependency
- fix hash subcommand option parsing causing panic
- ctg find prioritises wins & draws over performance
- make learned to preserve moves with null moves in the book with `-0`, `--null`
- make learned to avoid scaling weights using `--no-scale`
- make learned to configure compression for the rocksdb database using
`--compression={none,bzip2,lz4,lz4hc,snappy,zlib,zstd}`,`--compression-level=<level>`,
defaults to `lz4`, level 4
- use a temporary [RocksDB](https://crates.io/crates/rocksdb) database during pgn book
make so as to better make use of memory, this reduces the performance a little,
however in return makes importing huge PGN files possible.
- new make command to make polyglot books out of pgn files, runs multithreaded
with as many threads as the cpu number of the system by default, use `-T`, `--threads`
to override, works transparently with compressed PGN files (zstd, bzip2, gunzip, lz4)
## 0.3.2
- edit writes the name of the user and jja's version as comment to Arena opening
book metadata on ChessBase to Arena opening book conversion, override with
--author, --comment
- edit displays a unicode chess board in edit tempfile
- fix book traversal on Arena opening book to PolyGlot conversion
- enable ansi colors when running on windows terminal
- drop unixisms, cross-compiles for windows
- fix yet another bug with castle decoding on polyglot read/query
- edit no longer tries to spawn the default editor if standard output is not a TTY
- fix find and edit for Arena opening book reading, move selection is on par with
the Arena GUI
- new hash command to calculate the Zobrist hash of the given position
- fix infinite loop while converting some big Arena opening book files
- improve hashing performance by avoiding double hashing using a hasher builder for
Zobrist hashes
- improve hashing performance using
[shakmaty](https://docs.rs/shakmaty/latest/shakmaty/zobrist/trait.ZobristHash.html#tymethod.zobrist_hash)
crate's [Zobrist Hashing](https://www.chessprogramming.org/Zobrist_Hashing)
implementation rather than the internal one.
## 0.3.1
- edit learned to calculate & write ABK header game statistics fields
- edit learned to convert CTG book files to ABK book files
- edit learned --author and --comment to specify metadata for Arena opening books
- edit can edit Arena opening book (abk) files in-place with -i, --in-place=SUFFIX
- support for writing Arena (abk) opening books
- open learned to wrap long ECO opening lines into multiple lines
- find no longer panics on some abk books with entries having invalid uci
- edit takes move priority into account for weight on abk to bin conversion
- match learned --move-selection={best\_move,uniform\_random,weighted\_random} to
pick move selection algoritm for book moves
- fix castle decoding on polyglot read/query
- fix error return when no positions found in abk, obk and ctg find
- fix promotion handling in ctg edit
- many improvements to ctg find (move coloring & sorting, average statistics)
- new merge command to merge two PolyGlot opening books
- new match command to arrange book matches with random playouts
## 0.3.0
- refactor code to unify various opening book reader interfaces
- support obk version 1 as well as 2 (ChessMaster books with and without notes)
- support for reading obk (ChessMaster) books and converting them to polyglot books
- do not display progress bar if standard output is not a TTY
- support for reading abk (Arena) books and converting them to polyglot books
- new play command can be used to make random playouts using opening books
## 0.2.1
- weight conversion in ctg to polyglot edit can be tuned with
--nag-weight-{good,mistake,hard,blunder,interesting,dubious,forced}=<WEIGHT>
- edit learned --no-scale to avoid scaling weights globally to fit into 16 bits
- the code is now relatively well documented
- edit --in-place now properly deletes the output temp file on interrupt
- edit filters out moves with zero weights, use -0, --null to preserve them
## 0.2.0
- edit window lists position info (key, epd, legal moves) as comment
- edit no longer silently discards illegal moves
- edit can edit PolyGlot files in-place with -i, --in-place=SUFFIX
- edit can convert CTG opening books into PolyGlot opening books
- default to start position when no --fen or --pgn is given for edit and find
- info prints number of total pages in CTG books
## 0.1.1
- new positions can be added to polyglot files
- many bugs fixed with polyglot edit
- quote command added to print a random chess quote
- open command added to query ECO classification
## 0.1.0
- edit polyglot files, only editing present positions work
- read polyglot files
- read ctg files