uxie 0.3.0

Data fetching library for Pokemon Gen 4 romhacking - map headers, C parsing, and more
Documentation

uxie

A data fetching library for Pokemon Gen 4 romhacking. Provides unified access to ROM data from DSPRE projects and decompilation sources.

Table of Contents

Background

Working with Pokemon Gen 4 ROMs involves multiple data sources: binary files extracted via ndstool or ds-rom, DSPRE projects (folders containing arm9.bin and unpacked/ filesystem), and modern JSON/C source structures from decompilation projects (pokeplatinum/pokeheartgold). Each source has different formats and conventions.

uxie provides a unified interface for reading ROM data regardless of source. Its RomHeader::open() function auto-detects the format and returns a consistent struct. The library also includes utilities for reading map headers, parsing C enums and defines, and querying relationships between game data.

Etymology

uxie takes its name from Uxie, the legendary Pokemon known as the "Being of Knowledge."

Performance

uxie is optimized for high-performance symbol resolution and project loading:

  • ~200ms loading time for full pokeplatinum decompilation projects (50,000+ symbols)
  • O(1) Resolution: Project-wide symbols are pre-resolved and cached, making lookups nearly instantaneous.
  • Lightweight Inheritance: Collecting constants for individual script files uses a parent-pointer overlay instead of cloning the global symbol table, reducing per-file overhead by 99%.
  • Parallel Source Loading: All headers, JSON text banks, event files, and build artifacts are loaded in parallel via rayon.
  • Minimized Syscall Overhead: Canonical paths are cached globally to avoid redundant filesystem lookups.

Performance benchmarks on a typical development machine:

  • Loading 50,000+ constants from pokeplatinum: ~200ms
  • Single constant resolution (cached): < 1 microsecond
  • Complex expression evaluation (e.g., RGB(r,g,b), bitwise operations): < 5 microseconds

Features

  • Smart Discovery: Automatically detects game version, internal project names, and table offsets. No manual configuration required for standard projects.
  • Unified ROM Access: Auto-detects and reads data from DSPRE projects and decompilation sources.
  • High-Level Workspace: Unified API for managing symbols, script mappings, and text banks across a project
  • Full C Expression Evaluation: Pratt parser implementation with correct operator precedence for all C operators (+, -, *, /, %, &, |, ^, <<, >>, ~, !, parentheses)
  • Parallel Loading: Multi-threaded header file loading via rayon for significantly faster project initialization
  • Standard HashMap API: Public getters return std::collections::HashMap for easy interoperability with Rust's standard library
  • Enhanced Symbol Table: Automatically resolves cross-references between constants, supports .txt files with incremental indexing, and parses event JSON files for object IDs.
  • Built-in Constants: TRUE and FALSE are pre-defined, ensuring compatibility with C-style boolean expressions.
  • Map Header Parsing: Unified access to area data, scripts, and events across all Gen 4 games.
  • Format Agnostic: Seamlessly bridge legacy binary formats and modern JSON/YAML source data.
  • Bidirectional Script Resolution: Resolve script constants from names to values AND values to names using a shortest-name heuristic.

Install

CLI (Command Line Interface)

The easiest way to install the uxie CLI is via cargo:

cargo install uxie

Verify installation:

uxie --version

Library Usage

Add uxie to your project's dependencies. Using cargo add:

cargo add uxie

Or manually add to your Cargo.toml:

[dependencies]
uxie = "0.3.0"

For full API documentation, visit docs.rs/uxie.

Build from Source

To build from the latest source code:

git clone https://github.com/KalaayPT/uxie.git
cd uxie
cargo build --release

The binary will be at target/release/uxie. Verify installation:

./target/release/uxie --version

CLI Usage

The Being of Knowledge offers several ways to query the secrets of the Sinnoh and Johto regions.

Common Flags

Most commands (map, event, encounter, resolve-script) share these common options:

  • --project, -p: Path to the project root (defaults to current directory).
  • --decomp, -d: Optional local path OR GitHub raw URL (e.g., https://raw.githubusercontent.com/pret/pokeplatinum/master/) for symbol resolution.
  • --json: Output the result as a structured JSON object.

header

Read ROM header information. Auto-detects format from a file path or project directory:

# From current directory
uxie header

# From a specific ROM file or header.bin
uxie header path/to/header.bin

# Output as JSON
uxie header --json

map

Read map header data, including internal and pretty names.

# Read map header ID 0 from current project
uxie map 0

# Read from DSPRE project but resolve symbols using a remote decomp
uxie map 3 -p /path/to/dspre/ -d https://raw.githubusercontent.com/pret/pokeplatinum/master/

Example Output:

Map Header 0 (Platinum)
Internal Name:   D21R0101
Pretty Name:     Twinleaf Town
=========================
Area Data ID:    0
Matrix ID:       0
Script File ID:  0
...

event & encounter

Load and resolve event or encounter data:

# Resolve events for map 3
uxie event 3

# Fetch encounters for map 100 in JSON format
uxie encounter 100 --json

symbols

Parse C header files or symbol list files:

# Parse and resolve complex expressions
uxie symbols constants.h

# Parse a .txt list with incremental indexing
uxie symbols variables.txt

# Output only enums as JSON
uxie symbols constants.h --only-enums --json

resolve-script (untested)

Bidirectionally resolve constants within a script file (Names -> Values AND Values -> Names):

# Resolve using a specific decomp root (optional)
uxie resolve-script game_script.s --decomp /path/to/pokeplatinum/

Integration

uxie is designed to bridge the gap between different toolchains in the Gen 4 romhacking ecosystem:

  • Universal Symbols: Bridges C headers, assembly constants, and binary offsets into a single, searchable namespace.
  • Text & Archives: Seamlessly maps text archive IDs to their symbolic names and content, linking binary data to human-readable strings.
  • Format Fluidity: Provides a unified bridge between legacy binary formats (NARC, arm9.bin) and modern source-controlled data (JSON, YAML, C).
  • Toolchain Agnostic: Handles both legacy binary projects and modern decompilation trees transparently, allowing your tools to work anywhere knowledge is stored.

Library Usage

High-Level Workspace

The Workspace struct is the primary entry point. It automatically detects the project type (DSPRE or Decompilation) and sets up the environment.

use uxie::Workspace;

// Auto-detect and load from any supported project path
let workspace = Workspace::open("path/to/project")?;

// Resolve a constant name to its value
let val = workspace.resolve_constant("VARS_START"); // Some(16384)

// Bidirectional resolution: Names -> Values AND Values -> Names
let script = "SetFlag FLAG_UNK_0x000A";
let resolved = workspace.resolve_script_symbols(script);
println!("{}", resolved); // "SetFlag 10"

let binary_script = "SetFlag 10";
let symbolic = workspace.resolve_script_symbols(binary_script);
println!("{}", symbolic); // "SetFlag FLAG_UNK_0x000A" (shortest name heuristic)

SymbolTable API

The SymbolTable provides a powerful API for parsing C headers and evaluating expressions with correct C operator precedence.

use uxie::SymbolTable;
use std::collections::HashMap;

let mut symbols = SymbolTable::new();

// Load all headers from a directory in parallel (handles .h, .hpp, .txt, .py, .json)
symbols.load_headers_from_dir("include/constants")?;

// Resolve a constant from any of the loaded files
if let Some(val) = symbols.resolve_constant("ITEM_POKE_BALL") {
    println!("ID: {}", val);
}

// Evaluate arbitrary C expressions with correct precedence
// Handles bitwise OR, shifts, arithmetic, and nested parentheses
let expr_val = symbols.evaluate_expression("(1 << 8) | (2 << 4) | 3");
println!("Expression value: {:?}", expr_val); // Some(275)

// Access all constants using standard HashMap for easy interoperability
let all_defines: HashMap<String, i64> = symbols.get_all_defines();
for (name, value) in &all_defines {
    println!("{} = {}", name, value);
}

// Access enum data using standard HashMap
let enums: HashMap<String, Vec<(String, Option<i64>)>> = symbols.get_enums_std();
for (enum_name, variants) in &enums {
    println!("enum {}:", enum_name);
    for (variant, value) in variants {
        println!("  {} = {:?}", variant, value);
    }
}

The Pratt parser implementation ensures correct operator precedence matching C standard:

  • Bitwise OR (|) has lower precedence than shifts
  • Arithmetic operators (*, /, %) have higher precedence than (+, -)
  • Parentheses and unary operators (~, !, -, +) are handled correctly

Reading ROM Headers

use uxie::RomHeader;

// Auto-detect format from path
let header = RomHeader::open("path/to/header.bin")?;
let header = RomHeader::open("path/to/dspre-project/")?;

// Access header data
println!("Game: {:?}", header.detect_game());       // Some(Platinum)
println!("Family: {:?}", header.detect_game_family()); // Some(Platinum)
println!("Region: {:?}", header.region());          // Some("USA")

Reading Map Headers

use uxie::{Arm9Provider, GameFamily};

// Low-level provider access
let provider = Arm9Provider::new(
    "path/to/arm9.bin",
    0xE601C,  // Table offset
    559,      // Count
    GameFamily::Platinum,
);

let header = provider.get_map_header(0)?;
println!("Script file: {}", header.script_file_id());

Working with DSPRE Projects

use uxie::Workspace;

// Workspace handles DSPRE projects transparently
let workspace = Workspace::open("path/to/dspre-project")?;

println!("Project Type: {:?}", workspace.project_type); // Dspre
println!("Detected Game: {:?}", workspace.game);

Parsing C Headers

For complex projects, use SymbolTable to load multiple headers and resolve cross-references:

use uxie::SymbolTable;

let mut symbols = SymbolTable::new();

// Load all headers from a directory (handles .h, .hpp, and .txt)
symbols.load_headers_from_dir("include/constants")?;

// Resolve a constant from any of the loaded files
if let Some(val) = symbols.resolve_constant("ITEM_POKE_BALL") {
    println!("ID: {}", val);
}

You can also use the low-level parsing functions for single files:

use uxie::c_parser::{parse_enum, parse_and_resolve_defines};

let content = std::fs::read_to_string("header.h")?;

// Parse enums
if let Some(e) = parse_enum(&content) {
    for variant in &e.variants {
        println!("{} = {:?}", variant.name, variant.value);
    }
}

// Parse and resolve defines (with correct C precedence)
let defines = parse_and_resolve_defines(&content);
for d in &defines {
    if let Some(resolved) = d.resolved {
        println!("#define {} {} (= {})", d.name, d.value, resolved);
    }
}

Supported Games

Game Code Family
Diamond ADAE (US), ADAJ (JP), ADAP (EU) DP
Pearl APAE (US), APAJ (JP), APAP (EU) DP
Platinum CPUE (US), CPUJ (JP), CPUP (EU) Platinum
HeartGold IPKE (US), IPKJ (JP), IPKP (EU) HGSS
SoulSilver IPGE (US), IPGJ (JP), IPGP (EU) HGSS

License

uxie is free software licensed under the MIT License. See LICENSE for details.