Expand description
BundleFile (files/BundleFile.py), read lazily.
BundleFile::open reads the header, blocks-info and directory and
decodes no data block, so a registry over thousands of bundles is cheap
and BundleFile::uncompressed_size (the declared total) is known before
any decode (for a caller that plans with it, plan section 4.1).
One layout (plan section 3.6). UnityPy always reads entries from the
b"".join of every block’s actual output (files/BundleFile.py:173-186), so
an entry’s offset depends on the actual length of every block before it.
Before a lazy read returns bytes ending at joined offset X, every block
starting below X has a known length (lengths): a stored block’s is
known at open when slot 0 holds the built-in stored codec
(crate::codec::Codecs::stores_as_is) and its bytes are within the
block cap (crate::limits::Limits::max_block_output, which a stored
block meets on every path, as it does in load_all); an unmeasured
block wholly before the read is decoded with
its output discarded (a measuring decode: charged while it runs, returned
after). An entry is the Python slice data[offset : offset + size] of the
joined data (File.py:78-81; crate::stream::py_read). A lazy read
that succeeds returns exactly the slice of BundleFile::load_all’s
joined data, and a block that fails before or inside the read fails it
as it fails load_all; a block that fails only after the read fails
load_all and not the read (plan section 3.13, difference (c)).
Block outcomes. A block’s measurement is kept once known, failures
included: lengths holds each block’s length or the error its decode
gave, so a block that fails is decoded once and every later read that
needs it fails at once, as UnityPy fails once. A refusal that depends on
what else is held (the meter, the budget) is not kept
(crate::once::get_or_publish_result).
Runs. The blocks one read overlaps are decoded into one buffer, a run,
cached by its block span (a span of stored blocks that are their own
bytes is one slice of the input, cached as a run too). A read whose span
a cached run covers is a zero-copy slice of it; any other read decodes
its own whole span into a new run, so a block that two spans share is
decoded once per span. load_all decodes every block into one run.
BundleFile::entry_range is a read like any other: it measures, slices
a covering cached run when there is one, and otherwise decodes its span
into a run it caches, so N ranges in one block decode it once.
What a read costs (index). The bundle keeps where each block of the
known prefix ends (the prefix sums of the known lengths, in a table
charged at open), and its cached runs in an index by first block (a
B-tree charged a share per run), in which no run contains another. A read
finds its first block and its span’s end by binary search of the prefix
and a covering run by one search of the index, extends the prefix only
past its end (so each block is learned once per open), and decodes only
the blocks of a span no run covers: O(log B + log R) steps besides the
blocks it learns or decodes, for B blocks and R runs. Every step counts
as the open’s work, 8 bytes a step (a row a search reads, a block a read
learns, decodes again or checks for a stored slice, a run a publish
drops), so K reads over B blocks count O(K log B) steps, where each read
walked the blocks from block 0 and scanned every run, uncounted
(tests/bundles/runs.rs).
Memory and work. Every decode (first, measuring, repeated) writes
into a crate::budget::DecodeBuf on the input’s meter, one segment per
block (each capped at max_block_output), so whatever the runs hold is
charged, and a hostile directory whose spans overlap fails with
LimitKind::InputMemory instead of growing; every byte a decode writes
also counts as its open’s work (LimitKind::DecodeWork, on the meter,
which is this open of the input: a reopen after eviction is a new meter
on the same memory, crate::limits::Meter::reopen), which is not
returned while the open lives (a refused growth counts nothing, and
writes on several threads never pass the bound together). The header,
blocks, directory, files and the per-row
cells are charged at open; each parsed entry charges its own file, and
a kept error charges its heap. An entry’s cell keeps its parse’s
failure, so the row is not parsed again, but every directory row is its
own cell, and rows over one span repeat one parse: what the meter granted
a parse that ends in an error the cell keeps, on its own thread, counts
as the open’s work too, so K such rows stop at the open’s DecodeWork
limit instead of costing K parses no limit sees (round 4;
tests/bundles/work.rs). A parse that succeeds keeps its tables (each
row pays for them), and what its AssetBundle read builds and drops is
counted by the read (files::serialized, “Work”; round 5). A failure the
cell does not keep (the meter’s or the budget’s refusal) is
crate::once’s to count. A block’s stored bytes count too, before a
codec reads them, as credit for what it writes (decode, “The input is
work”; round 6: LZ4 scans a whole block before it writes, so a nested
bundle whose blocks-info or block fails at its end cost each row over it
a scan no count saw).
Twins. Rows with the same offset and size read the same bytes, and
when their names agree on the one rule that changes what a parse builds
from given bytes (a raw suffix, crate::detect::parsed_as) they parse
alike, but for the name each file and each wrapped error carries. Such
rows are twins, and they answer a failure as one cell does: the first
failure the input decides kept for any of them (the first twin’s cell
holds it) is every twin’s answer, under each one’s own name, so K rows
over one failing span parse it once (round 6). A twin that parses
successfully parses for itself, as its file carries its name. The table
of first twins is a sort of the directory’s keys at open, charged with
the other per-row tables.
Decodes and user codecs run with no
lock held: a run is decoded first and then published into runs, and an
entry is parsed first and then published into its cell
(crate::once::get_or_publish_result). Publishing a run may grow the
run index under its mutex, which charges the meter and so calls the
budget; the budget’s contract forbids it to block or call the core
(crate::budget::MemoryBudget), and a run replaced or dropped under
the lock is released after it. The known prefix takes no lock: a reader
loads how many blocks it holds and searches them while another thread
extends it.
UnityPy rules ported exactly (plan section 3.6):
- the extra byte after the data flags only for non-UnityFS signatures
(
files/BundleFile.py:104-106; rabex reads it for UnityFS,files/bundlefile/mod.rs:616-618); - the flag set by engine version (
files/BundleFile.py:113-121); the decryptor read before the 16-byte alignment (:123-124, then:129-131; rabex reads it after,files/bundlefile/mod.rs:638-645); the alignment at format 7 and up or a 2019 engine at 2019.4.15 or later; - blocks-info at the end read at
len - compressed(files/BundleFile.py:134-137); padding at the start only underArchiveFlagswith 0x200 (:170-171); - short reads: a truncated block is read short and decoded as it is; its
bytes_readis what the dump writes ascompressed; - LZMA’s output can outgrow its declared size: the block cap
(
crate::limits::Limits::max_block_output) and the input’s meter follow the actual bytes, never the declared ones.
UnityWeb/UnityRaw: format 6 takes the UnityFS layout. Other formats take
read_web_raw, with the legacy payload decoded once at open. UnityArchive is
crate::Unsupported::UnityArchive, as UnityPy raises (files/BundleFile.py:41-42).
Owner: package C.
Structs§
- Block
Info - One block (or the blocks-info section).
- Bundle
Counts - What a bundle’s block reads have done since it was opened (plan section 8, R2 and R3). Its own blocks only: a nested bundle counts its own.
- Bundle
File - A bundle: header and directory parsed, data decoded on demand.
- Bundle
Header - The header, as read.
- Bundle
Save Options - Bundle save layout and compression. Children use the same layout and
their default packer, as UnityPy calls their
save()with no arguments. - DirEntry
- One directory row.
- File
Slot - One key of UnityPy’s
filesdict: a path at its first position, holding the last directory entry of that path.
Enums§
- Bundle
Packer - Bundle packer. Tuple flags follow the code’s order, not its docstring.
- Layout
Kind - Which of UnityPy’s two readers a bundle takes.