dream_archive
Pure-Rust Bethesda archive tooling for common PC archive layouts. It reads, lists, opens file-like readers, extracts, and builds the archive families used by Morrowind, Oblivion, Fallout, Skyrim, Fallout 4, and Starfield-era games, subject to the format rows below rather than wishful thinking.
The crate keeps archive paths as bytes. That is intentional. Old Bethesda tools and mods do not always agree on Unicode, code pages, or reality in general. ASCII paths work as ordinary Rust strings; legacy localized BSA paths should use an explicit encoding helper instead of a hopeful guess.
Capability matrix
| Format | Read | Extract | Write | Notes |
|---|---|---|---|---|
| BA2 GNRL | yes | yes | yes | General-file BA2 archives. |
| BA2 DX10 | yes | yes | yes | Extraction emits DDS files by reconstructing standard DDS headers from BA2 texture metadata. Private/vendor DDS header fields are not preserved. Writer accepts supported DDS files or explicit texture metadata plus raw payload bytes. |
| BA2 GNMF | metadata | no | no | Sony GNM (.gnf) texture archives. Explicitly out of scope for now; metadata is parsed only so callers can identify them instead of getting mystery meat. |
| TES3 BSA | yes | yes | yes | Morrowind-era archives. |
| TES4 BSA | yes | yes | yes | Oblivion/Fallout/Skyrim PC archives, including hash-only and embedded-name layouts. |
| Console/Xbox layouts | no | no | no | Out of scope. |
| XMem compression | no | no | no | Out of scope. |
“Write support” above means the writer semantics for that specific row are implemented. It does not mean “every BA2 payload type is writable.” Words mean things. Annoying, but useful.
BA2 parsing recognizes the supported PC BA2 versions used across Fallout 4/Fallout 76/Starfield-era archives, including ZIP and supported LZ4 layouts. GNMF remains metadata-only.
GNMF is not a missing general-file feature. It is Sony GNM texture data with console-style swizzle/unswizzle requirements. This crate does not currently extract or write it. If you need GNMF support, bring real fixtures and a clear compatibility target; otherwise we are not guessing our way into a PlayStation texture pipeline.
30-second usage
Open any supported archive and list recoverable paths:
use Archive;
Read a required file:
use Archive;
Open a required file as a Read implementation, for callers that want ordinary
file-like streaming instead of a Vec<u8>:
use Read as _;
use Archive;
Uncompressed archive entries read directly from archive storage. Compressed entries may be decoded into an internal buffer before the reader is returned so format errors remain format errors instead of appearing later as generic I/O failures. A reader API is not a wizard. Annoying, but quite useful.
Extract everything with paths stored in the archive:
use Archive;
Some archive layouts do not contain enough text to name every file. Hash-only
TES4 archives need the TES4-specific extract_to_with_paths API with a path
dictionary. BA2 archives without string tables still have hashes, but
Entry::path() returns None through the top-level facade because no filename
text exists to recover. No path does not mean empty filename.
use Archive;
Building archives
Build a TES3/Morrowind BSA from a directory:
use Tes3BsaBuilder;
Build a TES4-family BSA using a PC game profile:
use ;
Build a BA2 GNRL archive:
use ;
Builder filesystem inputs are deferred. add_file and add_dir validate archive
paths, reject duplicates, and record source file metadata when called, but they
read payload bytes during write_path, write_seek, or to_vec. If a deferred
source file disappears, becomes unreadable, or changes size before writing, the
write fails rather than emitting an archive whose table lies about its payloads.
For filesystem output, prefer write_path or write_seek. to_vec necessarily
buffers the final archive because it returns a Vec<u8>; useful for tests and
small archives, not magic.
When rewriting an archive, unchanged entries can be attached from the parsed
source archive instead of first extracting them into a Vec<u8>:
use Arc;
use ;
BA2 and TES4 builders expose the same add_archive_entry pattern, plus
add_archive_entry_with_compression where per-file compression policy matters.
The first implementation decodes existing entries and writes them according to
the destination builder policy. Raw compressed-byte copy is deliberately not
claimed here. That would be a different feature, not a vibe.
Build a BA2 DX10 texture archive from DDS files:
use ;
The first path is the archive path to store. The second path is the filesystem DDS source to read.
The DDS path parses supported DDS headers, maps them to BA2 texture metadata,
validates the payload size, strips the DDS header, and stores the raw texture
payload. It does not transcode formats or generate mips. If you already have BA2
texture metadata, add_texture_bytes accepts raw texture payload bytes only —
not a complete DDS file with its header still attached.
DX10/DDS support here is archive plumbing, not a texture processing library. It does not decode pixels, preserve vendor/private DDS fields, handle texture arrays/volumes, transcode formats, repair payloads, or aim for DirectXTex parity. A broader DDS crate belongs with a renderer; this crate only does the pieces BA2 writing and extraction need today.
Supported DDS input for BA2 DX10 writing is deliberately narrow: tightly packed 2D textures and cubemaps using the DXGI/FourCC/plain formats this crate can reconstruct on extraction. The payload byte count must exactly match the DDS dimensions, format, mip count, and cubemap face count. Texture arrays and 3D volume textures are rejected.
Archive paths are not filesystem paths
Archive paths are virtual filesystem names stored inside the archive. They are byte strings, not necessarily Unicode OS paths.
- Normal ASCII mod paths can be passed as
&str; this is fine only when the archive path bytes are actually UTF-8/ASCII. - Legacy localized BSA paths should be encoded explicitly.
- The crate does not guess encodings, because guessing corrupts mods and then everyone has a bad afternoon.
Example for a legacy Windows code page path:
use ;
For GNMF BA2 archives, inspect metadata rather than attempting extraction:
use ;
Feature flags
Default features enable BA2 and both BSA families.
ba2: BA2 reader/extractor plus GNRL and DX10 writers.bsa-tes3: TES3 BSA support.bsa-tes4: TES4-family BSA support.bsa: both BSA families.parallel: parallel extraction with Rayon.lua: enables themluaLuau bindings,ba2,bsa, and the re-exporteddream_path's Luau companion helpers. It does not select anmluaruntime.standalone-lua: selectsmlua's Luau backend (built from source); intended for this crate's tests, examples, and documentation builds rather than normal downstream library use.
Luau bindings
Enable it in Cargo.toml:
= { = "0.2", = ["lua"] }
As of 0.2.0 the bindings target Luau instead of LuaJIT, and
the whole Lua-facing surface follows the same conventions as DreamWeave's other
Luau APIs: functions, methods, table fields, and enum-like string values are
camelCase (openPath, readFileRequired, folderHash, "bsaTes4",
"skyrimSe"), types are PascalCase (Builder, Dx10Builder), and numeric
constants are UPPER_SNAKE (archiveTypes.MESHES, version.V8). The
conventional global is dreamArchive, next to dreamPath. Scripts written
against 0.1's snake_case names need renaming; nothing else about the behaviour
changed.
Embedding applications must choose the mlua runtime centrally. If you just want
to run this crate's examples or tests without an application's feature graph, use
standalone-lua instead. Building this crate by itself with lua but no mlua
runtime selected is intentionally incomplete. Selecting runtimes in every
lower-level crate is how you get one build graph wearing several fake moustaches.
The lua feature exposes a byte-first Luau API through dream_archive::lua for
Rust embedders. It does not install a standalone require("@dreamArchive") C
module by itself; register the mlua table in your application. Use the
dream_archive::dream_path re-export for companion path helpers instead of
adding a separate dream_path dependency just to reach the same API. Lua strings
are archive path bytes and payload bytes. Filesystem arguments are the exception:
all openPath, detectPath, writePath, extractTo*,
extractEntryToPath, addDir, and source paths such as addFile /
addDdsFile are converted as UTF-8 host paths. Archive paths passed to
readFile, addBytes, addFile's first argument, hash helpers, and
normalizePath remain raw archive path bytes. bsa.encodeFilename() and
builder addEncodedPath() are the opposite boundary: they take UTF-8 text and
produce or insert legacy-encoded archive filename bytes. If your Unix filesystem
path is not valid UTF-8, use the Rust API directly.
Annoying, but less dishonest than pretending all paths are the same kind of
string.
Luau builders follow the Rust deferred-source behavior: addFile records a host
path and reads it when writePath / toBytes runs. Format-specific archive
entry tables include both index and id fields; today they are the same
1-based number. Pass that id to builder:addArchiveEntry(...) to preserve an
entry from an already-open archive without round-tripping it through a Lua
string. BA2 and TES4 builders also expose
addArchiveEntryWithCompression(path, archive, id, policy).
The module exposes:
- top-level detection/open/read/extract facade:
openPath,openBytes,detectPath,guessFormat, archiveentries,readFile*,extractFile*,readEntry,extractEntry,extractEntryToPath, andextractTo; - BA2-specific archive inspection, hashes, GNRL builder, and DX10 builder;
- BSA encoding/decoding helpers and path normalization;
- TES3/TES4 archive inspection, hashes, builders, TES4 profiles, name modes, compression policy, and archive type bits.
Register dream_archive::dream_path next to dreamArchive when scripts need
the shared virtual path helpers. The re-exported dreamPath handles path
normalization/helpers; dreamArchive handles open/list/read/extract/build
archive mechanics; dream_archivetool should remain the layer that decides
filesystem rewrite, diff, and verification policy.
Common calls look like this:
| Luau call | Result |
|---|---|
dreamArchive.openBytes(bytes) |
generic BA2/TES3/TES4 archive |
dreamArchive.guessFormat(bytes) |
"ba2", "bsaTes3", "bsaTes4", or nil |
archive:entries() |
array-style table of copied entry metadata |
archive:readFile(path) |
payload bytes or nil |
archive:readFileRequired(path) |
payload bytes or error |
archive:extractFile(path) |
extracted payload bytes or nil |
archive:extractFileRequired(path) |
extracted payload bytes or error |
archive:extractTo(target) |
writes files for side effect; errors throw |
archive:readEntry(1) |
reads the first entry; entry indices are 1-based |
builder:addBytes(path, bytes) |
adds archive path bytes and payload bytes |
builder:toBytes() |
raw archive bytes |
entries() returns a fully materialized Lua table, copying entry metadata into
Lua values. That is convenient for scripts and not a streaming iterator.
readFile*, extractFile*, and extractEntry also materialize the complete
payload as a Lua string; for large archives that means Rust buffering plus a Lua
string copy. The Rust open_file* -> Read archive APIs are not exposed as Lua
userdata readers. Luau gets byte strings or filesystem extraction; if you need an
actual Rust Read, stay in Rust. openBytes(bytes) copies the Lua archive
string into Rust-owned storage, and builder toBytes() / toString() build a
Rust archive buffer and then copy it into Lua. For large archives, prefer
openPath() and writePath(). Likewise TES4 extractToWithPaths() copies
the supplied contiguous Lua sequence (1..n, no gaps) of candidate archive path
byte strings before it starts matching hash-only entries. It is not a key/value
dictionary; non-sequence keys are ignored.
local hashOnly = dreamArchive.bsa.tes4.openPath("HashOnly.bsa")
hashOnly:extractToWithPaths("out", {
"meshes/foo.nif",
"textures/foo.dds",
})
TES4 hash-only entries cannot expose names they do not have. Their Luau entry
tables use nil for path, folder, and name; use folderHash.hex and
fileHash.hex when you need exact identity.
Use dreamArchive.open* when you want one generic reader for BA2/TES3/TES4.
Use dreamArchive.ba2.open*, dreamArchive.bsa.tes3.open*, or
dreamArchive.bsa.tes4.open* when you need format-specific metadata or helper
APIs such as BA2 info() or TES4 extractToWithPaths().
BA2 archive compression accepts nil, "none"/"store", "zip", or "lz4".
Per-file compression overrides accept nil, "inherit", "store", or
"compress". setZlibLevel(level) accepts 0..=9 and rejects other values at
the setter call, not later during toBytes().
TES4 builders take a profile through setProfile: "oblivion", "fallout3",
"falloutNewVegas", "skyrimLe", or "skyrimSe", and a name mode through
setNameMode: "strings", "hashOnly", "embedded", or
"stringsAndEmbedded". The profile, nameMode, compression, and encoding
tables map each of those values to itself, so either spelling works:
builder:setProfile(dreamArchive.bsa.tes4.profile.skyrimSe) or
builder:setProfile("skyrimSe").
DX10 BA2 texture building needs either a DDS source (addDdsBytes /
addDdsFile) or a raw texture payload plus the BA2 texture metadata. The bytes
passed to addTextureBytes() are raw texture payload bytes, not a complete DDS
file:
local dx10 = dreamArchive.ba2.Dx10Builder.new()
dx10:addTextureBytes("textures/foo.dds", {
height = 1,
width = 1,
mipCount = 1,
format = 61,
flags = 0,
tileMode = 0,
}, rawTexturePayload)
TES4 archive type bits are numeric flags. Combine them with Luau's bit32
library (or addition, when each flag appears once):
builder:setArchiveTypes(bit32.bor(
dreamArchive.bsa.tes4.archiveTypes.MESHES,
dreamArchive.bsa.tes4.archiveTypes.TEXTURES
))
Legacy BSA filenames must still be encoded explicitly from Luau:
local path = dreamArchive.bsa.encodeFilename(
"textures/zażółć.dds",
dreamArchive.bsa.encoding.windows1250
)
local bytes = archive:readFileRequired(path)
Optional read/extract APIs return nil when the archive does not contain the
path. detectPath and guessFormat return nil for unknown or unsupported
headers. The *Required variants raise an error instead:
local maybe = archive:readFile("meshes/foo.nif")
if maybe == nil then
-- absent from the archive
end
local ok, err = pcall(function()
archive:extractFileRequired("missing.nif")
end)
if not ok then
print(err)
end
normalizePath returns normalized archive path bytes, not a Unicode-cleaned OS
path. It normalizes to forward slashes; stored entry names and hash helper
normalized fields may display backslashes depending on the archive family.
TES3/TES4 64-bit hashes are exposed as exact component fields plus a fixed width
hexadecimal string; they are not exposed as Lua numbers because Luau numbers are
doubles, which are not a safe exact carrier for arbitrary u64 values.
local ba2 = dreamArchive.ba2.hashFile("Meshes/Foo.NIF")
-- ba2.directory, ba2.file, ba2.extension, ba2.normalized
local tes3 = dreamArchive.bsa.tes3.hashFile("Meshes/Foo.NIF")
-- tes3.lo, tes3.hi, tes3.hex, tes3.normalized
local tes4 = dreamArchive.bsa.tes4.hashFile("Meshes/Foo.NIF")
-- tes4.last, tes4.last2, tes4.length, tes4.first, tes4.crc, tes4.hex
-- no numeric u64 field; use hex for exact identity
To support Luau's require, register the module yourself. Luau require looks
registered modules up by an @-prefixed name:
Compatibility policy
The target is real PC archive compatibility. Console archive layouts and XMem compression are not implemented. If a future PC archive proves a currently unsupported path is real, useful, and not just format archaeology with a hat, it can be added with tests.
Support
Has dream_archive been useful to you?
If so, please consider amplifying the signal through my ko-fi.
Thank you for using dream_archive.