Skip to main content

Module bundle

Module bundle 

Source
Expand description

BundleFile (files/BundleFile.py), read lazily.

BundleFile::open reads the header, blocks-info and directory and decodes no data block, so a registry over thousands of bundles is cheap and BundleFile::uncompressed_size (the declared total) is known before any decode (for a caller that plans with it, plan section 4.1).

One layout (plan section 3.6). UnityPy always reads entries from the b"".join of every block’s actual output (files/BundleFile.py:173-186), so an entry’s offset depends on the actual length of every block before it. Before a lazy read returns bytes ending at joined offset X, every block starting below X has a known length (lengths): a stored block’s is known at open when slot 0 holds the built-in stored codec (crate::codec::Codecs::stores_as_is) and its bytes are within the block cap (crate::limits::Limits::max_block_output, which a stored block meets on every path, as it does in load_all); an unmeasured block wholly before the read is decoded with its output discarded (a measuring decode: charged while it runs, returned after). An entry is the Python slice data[offset : offset + size] of the joined data (File.py:78-81; crate::stream::py_read). A lazy read that succeeds returns exactly the slice of BundleFile::load_all’s joined data, and a block that fails before or inside the read fails it as it fails load_all; a block that fails only after the read fails load_all and not the read (plan section 3.13, difference (c)).

Block outcomes. A block’s measurement is kept once known, failures included: lengths holds each block’s length or the error its decode gave, so a block that fails is decoded once and every later read that needs it fails at once, as UnityPy fails once. A refusal that depends on what else is held (the meter, the budget) is not kept (crate::once::get_or_publish_result).

Runs. The blocks one read overlaps are decoded into one buffer, a run, cached by its block span (a span of stored blocks that are their own bytes is one slice of the input, cached as a run too). A read whose span a cached run covers is a zero-copy slice of it; any other read decodes its own whole span into a new run, so a block that two spans share is decoded once per span. load_all decodes every block into one run. BundleFile::entry_range is a read like any other: it measures, slices a covering cached run when there is one, and otherwise decodes its span into a run it caches, so N ranges in one block decode it once.

What a read costs (index). The bundle keeps where each block of the known prefix ends (the prefix sums of the known lengths, in a table charged at open), and its cached runs in an index by first block (a B-tree charged a share per run), in which no run contains another. A read finds its first block and its span’s end by binary search of the prefix and a covering run by one search of the index, extends the prefix only past its end (so each block is learned once per open), and decodes only the blocks of a span no run covers: O(log B + log R) steps besides the blocks it learns or decodes, for B blocks and R runs. Every step counts as the open’s work, 8 bytes a step (a row a search reads, a block a read learns, decodes again or checks for a stored slice, a run a publish drops), so K reads over B blocks count O(K log B) steps, where each read walked the blocks from block 0 and scanned every run, uncounted (tests/bundles/runs.rs).

Memory and work. Every decode (first, measuring, repeated) writes into a crate::budget::DecodeBuf on the input’s meter, one segment per block (each capped at max_block_output), so whatever the runs hold is charged, and a hostile directory whose spans overlap fails with LimitKind::InputMemory instead of growing; every byte a decode writes also counts as its open’s work (LimitKind::DecodeWork, on the meter, which is this open of the input: a reopen after eviction is a new meter on the same memory, crate::limits::Meter::reopen), which is not returned while the open lives (a refused growth counts nothing, and writes on several threads never pass the bound together). The header, blocks, directory, files and the per-row cells are charged at open; each parsed entry charges its own file, and a kept error charges its heap. An entry’s cell keeps its parse’s failure, so the row is not parsed again, but every directory row is its own cell, and rows over one span repeat one parse: what the meter granted a parse that ends in an error the cell keeps, on its own thread, counts as the open’s work too, so K such rows stop at the open’s DecodeWork limit instead of costing K parses no limit sees (round 4; tests/bundles/work.rs). A parse that succeeds keeps its tables (each row pays for them), and what its AssetBundle read builds and drops is counted by the read (files::serialized, “Work”; round 5). A failure the cell does not keep (the meter’s or the budget’s refusal) is crate::once’s to count. A block’s stored bytes count too, before a codec reads them, as credit for what it writes (decode, “The input is work”; round 6: LZ4 scans a whole block before it writes, so a nested bundle whose blocks-info or block fails at its end cost each row over it a scan no count saw).

Twins. Rows with the same offset and size read the same bytes, and when their names agree on the one rule that changes what a parse builds from given bytes (a raw suffix, crate::detect::parsed_as) they parse alike, but for the name each file and each wrapped error carries. Such rows are twins, and they answer a failure as one cell does: the first failure the input decides kept for any of them (the first twin’s cell holds it) is every twin’s answer, under each one’s own name, so K rows over one failing span parse it once (round 6). A twin that parses successfully parses for itself, as its file carries its name. The table of first twins is a sort of the directory’s keys at open, charged with the other per-row tables.

Decodes and user codecs run with no lock held: a run is decoded first and then published into runs, and an entry is parsed first and then published into its cell (crate::once::get_or_publish_result). Publishing a run may grow the run index under its mutex, which charges the meter and so calls the budget; the budget’s contract forbids it to block or call the core (crate::budget::MemoryBudget), and a run replaced or dropped under the lock is released after it. The known prefix takes no lock: a reader loads how many blocks it holds and searches them while another thread extends it.

UnityPy rules ported exactly (plan section 3.6):

  • the extra byte after the data flags only for non-UnityFS signatures (files/BundleFile.py:104-106; rabex reads it for UnityFS, files/bundlefile/mod.rs:616-618);
  • the flag set by engine version (files/BundleFile.py:113-121); the decryptor read before the 16-byte alignment (:123-124, then :129-131; rabex reads it after, files/bundlefile/mod.rs:638-645); the alignment at format 7 and up or a 2019 engine at 2019.4.15 or later;
  • blocks-info at the end read at len - compressed (files/BundleFile.py:134-137); padding at the start only under ArchiveFlags with 0x200 (:170-171);
  • short reads: a truncated block is read short and decoded as it is; its bytes_read is what the dump writes as compressed;
  • LZMA’s output can outgrow its declared size: the block cap (crate::limits::Limits::max_block_output) and the input’s meter follow the actual bytes, never the declared ones.

UnityWeb/UnityRaw: format 6 takes the UnityFS layout. Other formats take read_web_raw, with the legacy payload decoded once at open. UnityArchive is crate::Unsupported::UnityArchive, as UnityPy raises (files/BundleFile.py:41-42).

Owner: package C.

Structs§

BlockInfo
One block (or the blocks-info section).
BundleCounts
What a bundle’s block reads have done since it was opened (plan section 8, R2 and R3). Its own blocks only: a nested bundle counts its own.
BundleFile
A bundle: header and directory parsed, data decoded on demand.
BundleHeader
The header, as read.
BundleSaveOptions
Bundle save layout and compression. Children use the same layout and their default packer, as UnityPy calls their save() with no arguments.
DirEntry
One directory row.
FileSlot
One key of UnityPy’s files dict: a path at its first position, holding the last directory entry of that path.

Enums§

BundlePacker
Bundle packer. Tuple flags follow the code’s order, not its docstring.
LayoutKind
Which of UnityPy’s two readers a bundle takes.