SPUD (Structured Payload of Unintelligible Data)

SPUD is a custom binary file format designed for efficient storage and transmission of structured data. It uses type tags and field name interning to achieve a compact representation. This implementation is written in Rust.

Features

• Compact Binary Representation: Uses binary encoding for data types.
• Basic Data Types: Supports null, boolean, various integer (i8-i64, u8-u64) and float (f32, f64) types, strings, and raw binary blobs.
• Field Name Interning: Assigns unique IDs to field names within a file to reduce redundancy and save space.
• Versioning: Files start with a version identifier (set at compile time) to ensure compatibility.
• Simple Structure: Consists of a version header, a field name map, the data payload, and an End-Of-File marker ([0xDE, 0xAD, 0xBE, 0xEF]).
• Serde Integration: Seamless conversion between Rust structs and the SPUD format via serde.

File Structure

A .spud file generally follows this structure:

1. Version String: The version string (bytes).
2. Field Name Map:
   • A sequence of (length_byte, field_name_bytes, id_byte).
   • Ends with the FieldNameListEnd marker (0x01).
3. Data Payload:
   • A sequence of (FieldNameId_marker, field_id_byte, type_tag_byte, value_bytes).
   • String and BinaryBlob types include their length before the data.
   • Arrays and Objects are delimited by ArrayStart, ArrayEnd, ObjectStart, and ObjectEnd type tags.
4. EOF Marker: The sequence 0xDE, 0xAD, 0xBE, 0xEF.

Usage

Encoding (Creating a SPUD file)

You can build SPUD files either manually or by serializing Rust structs with serde.

Manual Usage

use spud::spud_builder::SpudBuilder;

let mut builder = SpudBuilder::new();

builder
    .add_value("name", "Example Object")
    .add_value("version", 1u8)
    .add_value("enabled", true)
    .add_value("description")
    .add_value("value", 123.45f64)
    .add_value("raw_data", &[0x01, 0x02, 0x03, 0x04]);

builder.build_file("output_dir", "my_spud_data");

println!("SPUD file created!");

Serde Usage

use spud::spud_builder::SpudBuilder;
use serde::Serialize;

#[derive(Serialize)]
struct MyData {
    name: String,
    version: u8,
    enabled: bool,
    description: Option<String>,
    value: f64,
    raw_data: Vec<u8>,
}

let data = MyData {
    name: "Example Object".to_string(),
    version: 1,
    enabled: true,
    description: None,
    value: 123.45,
    raw_data: vec![0x01, 0x02, 0x03, 0x04],
};

let mut builder = SpudBuilder::from_serde(&data);
builder.build_file("output_dir", "my_spud_data");

Decoding (Reading a SPUD file)

You can decode SPUD files manually or deserialize them into Rust structs with serde.

Manual Usage

use spud::spud_decoder::SpudDecoder;

let mut decoder = SpudDecoder::new_from_path("output_dir/my_spud_data.spud");

println!("Decoding SPUD file:");
decoder.decode();
println!("Decoding finished.");

Serde Usage

use spud::spud_decoder::SpudDecoder;
use serde::Deserialize;

#[derive(Deserialize)]
struct MyData {
    name: String,
    version: u8,
    enabled: bool,
    description: Option<String>,
    value: f64,
    raw_data: Vec<u8>,
}

let mut decoder = SpudDecoder::new_from_path("output_dir/my_spud_data.spud");
let data: MyData = decoder.deserialize().unwrap();

Roadmap / TODO

Here are some planned features and improvements:

• [ ] More Types:
  • [x] Decimals
  • [ ] Timestamp
  • [ ] DateTime (no timezone)
  • [ ] Enums
• [ ] serde Integration: Implemented serde::Serialize for SpudBuilder and serde::Deserialize for SpudDecoder.

Known bugs

There are some bugs that are yet to be fixed, most of them are minor and are not listed here.

Contributing

Contributions are welcome! Please open an issue to discuss changes or submit a pull request.

README.md partially generated by Gemini