Expand description
A very overengineered rust crate for compressing and decompressing data in the RefPack format utilized by many EA games of the early 2000s
RefPack
RefPack, also known as QFS, is a semi-standardized compression format utilized by many games published by Electronic Arts from the 90s to the late 2000s. In many cases, it was deployed with a custom header format.
Structure
RefPack shares many similarities with lz77 compression; it is a lossless compression format which relies on length-distance pairs to existing bytes within the decompression buffer. Where it differs from lz77 is that rather than a single format for “Literal” control codes and “Pointer” control codes, RefPack uses 4 distinct control codes for different sizes of pointers and literal blocks. A fifth control code is also present to indicate end of stream rather than requiring a size to be specified before decompression.
Codes
RefPack utilizes one “Literal” bytes-only control code similar to lz77, but with limited precision to multiples of 4. The remaining three control codes are varying sizes of “Pointer” control codes, for small, medium, and large back-references and lengths. The limited precision of the “Literal” control code is compensated for via “Pointer” control codes also having the ability to write up to 3 literal bytes to the stream
See Command for further details.
Decompression
Decompression simply requires reading from a stream of RefPack
data until
a stopcode is reached.
See decompression for further details
Compression
Compressing via RefPack is largely similar to lz77 compression algorithms, and involves a sliding window over the data to search for repeating blocks, and then writing to the stream as the previously specified codes.
See compression for further details
Headers
While the actual data block of RefPack has only one known implementation, multiple types of headers for the library have been identified.
Other Implementations
RefPack has been implemented in various other languages and for various games:
- RefPack.cpp (download): Original canonical implementation of RefPack by Frank Barchard for Origin Software. Utilized by some early Origin Software games.
- JDBPF: Early Simcity 4 Java Library for reading DBPF files which utilize RefPack
- JDBPFX: Later currently maintained fork of JDBPF
- DBPFSharp: Simcity 4 DBPF Library written in C#
- Sims2Tools: Sims 2 DBPF Library written in C#
This Crate
This crate is a rust implementation designed to compress and decompress refpack data with any header format. It uses generics to support arbitrary header formats to allow pure usage of this library without having to write “glue” code to parse header info.
Put simply, this means that you get the benefit of being able to use any format however you like without any performance overhead from dynamic dispatch, as well as being able to implement your own arbitrary formats that are still compatible with the same compression algorithms.
Usage
refpack-rs
exposes two functions: compress
and decompress
, along with
easy
variants with easier but less flexible of usage.
compress
and decompress
take mutable references to a buffer to read and
write from, that implements std::io::Read
and std::io::Write
,
respectively.
decompress
will read from the buffer until it encounters a stopcode (byte
within (0xFC..=0xFF)), while compress
will read in the provided length.
all compression and decompression functions accept one generic argument constrained to the Format trait. Implementations be “unconstructable” types, with the recommended type being an empty enum.
Implementations
Format | Games | Header |
---|---|---|
Reference | Various 90s Origin Software and EA games | Reference |
Maxis | The Sims, The Sims Online, Simcity 4, The Sims 2 | Maxis |
SimEA | The Sims 3, The Sims 4 | SimEA |
Example
use std::io::{Cursor, Seek};
use refpack::format::Reference;
let mut source_reader = Cursor::new(b"Hello World!".to_vec());
let mut out_buf = Cursor::new(vec![]);
refpack::compress::<Reference>(
source_reader.get_ref().len(),
&mut source_reader,
&mut out_buf,
)
.unwrap();
The easy variants are compress_easy
and decompress_easy
, which take a
&[u8]
and return a Result<Vec<u8>, RefPackError>
.
Internally they simply call compress
and decompress
with a Cursor
to
the input and output buffers, however they are more convenient to use in
many cases.
Re-exports
pub use crate::data::compression::compress;
pub use crate::data::compression::easy_compress;
pub use crate::data::decompression::decompress;
pub use crate::data::decompression::easy_decompress;
Modules
- things relating the actual compressed data block. Anything past the header info, the actual compression algorithms themselves, control codes, etc.
- Possible compression formats to utilize
- Module for things relating to the header of the data which include decompressed length, sometimes flags or a magic number, and sometimes compressed length.
Enums
- Possible errors returned by compression and decompression functions
Type Aliases
- Wrapper for Result specified to [RefPackError]