Skip to main content

Module walk

Module walk 

Source
Expand description

Reading and writing values through a typetree.

TypeTreeHelper.read_typetree (helpers/TypeTreeHelper.py:123-158) picks a reader by the object’s size: with byte_size > 0 UnityPyBoost reads the object’s bytes and nothing else (boost); with byte_size == 0 the pure-Python reader starts at byte_start in the whole file and may read on past the object (python). Then check_read compares the bytes read with byte_size, in either direction. UnityPyBoost has no writer, so write ports UnityPy’s pure-Python write_typetree (:161-179, :407-458), which does not mirror the reader (plan section 3.11).

Memory. Every allocation a read makes for the value (a list, a map, a tuple, a primitive run) is charged to ReadCtx::meter, the input’s, before it is made, at crate::limits::heap_cost of its capacity, and the charges ride in the Parsed (grow through crate::limits::MeteredVec, then hand its charge to the Parsed). A refused charge is ReadDetail::Charge (kind other). The writer’s output grows the same way. What a read reserves before the bytes it holds are read is bounded apart from the tree’s width and the read’s depth: a dict at most DICT_FIRST entries, growing as its children are set; the lists a read holds open at once, max_prealloc elements in all (Work::first_room), each growing geometrically as it is filled (review round 7 and its sweep: a dict reserved one entry a child, and each list open max_prealloc of its own, so a ReferencedObject re-entering a wide ref type, or vectors nested in vectors, held that many times over for a few bytes of data; tests/walk_reentry.rs).

Work. Every read and every write counts what it does toward its open’s work (crate::limits::Limits::max_decode_work, the bound every other kind of the open’s work counts toward), whatever its outcome: VALUE_WORK for each value it builds, writes or skips, and for each element of a primitive run a read holds as a run, or each primitive a write writes, its bytes (Work); each byte of a string a write copies, or passes over to find its characters; STEP_WORK for each step that builds no value: a slot a dict’s table of positions grows by, an entry a lookup by name passes, a name a writer’s table indexes or probes (and the bytes of a name it compares or hashes); and its ref-type lookups, a ref type a lookup compares or indexes, or a probe, VALUE_WORK and the name bytes it reads (dict::RefIndex). Objects may share their bytes, and a read refused by a bound of its own (max_zero_size_elements, max_values_per_object) keeps nothing, so K objects over the same bytes repeated it K times with no bound on the input (fix pass 4’s sweep: every object of a small file over one vector of 2^22 empty elements, each refused after 2^20); every object searches its file’s one list of ref types again, so K objects over R ref types took K x R steps (fix pass 5’s sweep); and a read failing at a class’s first child had passed over all of its children to size its dict (review round 6: K objects over a class of k children, K x k steps; review round 7: the pass that stayed for a class whose children repeat a name refused such reads with DecodeWork where UnityPy read each to its own failure). No read or write passes over a class’s children before it reads or writes them.

Each step is counted before it is taken, on the read’s or the write’s OpenWork, which adds the steps to the open’s meter a chunk (WORK_CHUNK) at a time, in one atomic check-and-add: a chunk that would take the open past its allowance is refused, and the read or the write fails with LimitKind::DecodeWork (kind other for a read); one whose own work, counted when it ends, took the open past its allowance fails too (its own failure first, if it has one). So the reads and writes that succeed never do more work together than the allowance, on any number of threads, and each one in progress does at most a chunk and one step past it before it stops (review round 6: a read added its work only when it ended, so two reads on two threads both passed the start’s check and ran to their end past the allowance). Once the open’s work has passed its allowance, a read or a write is refused before it starts (tests/walk_work.rs, tests/walk_prepass.rs).

Owner: the scaffold for the types in this file (frozen); package B owns read_typetree’s body and the three submodules.

Modules§

boost
UnityPyBoost’s reader (UnityPyBoost/TypeTreeHelper.cpp:67-1056), for objects with byte_size > 0. The rules, each pinned by a test (plan section 3.10, R1-R16): bounds checks pos + n > end, a length’s before the bytes it counts; align4 from the object’s start with no bounds check; strings always aligned; TypelessData aligned only by its flag; the switch on the node’s own kind first, and only a kind it does not handle (Unknown, PPtr, Array, ReferencedObjectData, ManagedReferencesRegistry) is an array when its first child is Array (R9); primitive and pair elements read as one run and align once, a pair run checking its arity before the first element, so even when empty; other elements read and align one by one; the ManagedReferencesRegistry scope; ReferencedObject data resolved through the file’s ref types in UnityPyBoost’s order (R14: no file, ref_types absent or not a list, no type read yet, a missing class/ns/asm, all ref_type; then an empty class or a matched ref type with no tree skips data; no match is ref_type; a class that is not a Str skips data when its Python object’s size field is 0, else is ref_type, plan R14); PyDict_SetItem for repeated names; float32 widened (a signalling NaN quieted under super::FloatMode::UnityPy).
emit
A direct read for a binding: UnityPyBoost’s reader (super::boost’s rules, R2-R15) over the trees and objects that need none of its bookkeeping, each value handed to a Sink as it is read, with no crate::value::Value in between (UnityPyBoost builds its Python objects as it reads, TypeTreeHelper.cpp:702-883).
python
UnityPy’s pure-Python reader (helpers/TypeTreeHelper.py:182-361), for objects with byte_size == 0. It starts at byte_start in the whole SerializedFile and may read on past the object, with Python’s reader rules at any position (crate::stream’s helpers; the position may go negative). Alignment is relative to the file, not the object: align_stream rounds the absolute position. Its differences from UnityPyBoost are docs/CANONICAL_DUMP.md section 7’s list: aligned primitive arrays become tuples, with 16-bit signedness swapped; an aligned SInt8 array raises at every length (read_byte_array(reader, size) before any loop), and an array of aligned TypelessData raises only when its length is above 0 (read_bytes() inside the loop; an empty one reads as []); a negative or out-of-range string length reads as "", and a string of length 0 or read as "" that way is not aligned after (read_aligned_string returns before align_stream: the base reader’s, streams/EndianBinaryReader.py:146-153, which FUNCTION_READ_MAP holds for a string node and a pair run’s child, and for an array’s elements read from bytes the memoryview reader’s override, :266-273, the same body); in an array whose aligned element is a vector, a negative inner length reads as [], unless the inner element is an aligned primitive, whose struct format rejects it (struct.error, other); arrays of aligned classes align once and skip the registry check; TypelessData length -1 reads to the end of the file and other negative lengths follow Python slice semantics (so bytes_read can be negative), on the memory reader: a file loaded from a path is read by the stream reader’s rules (read_with_reader), where such a length raises and no read passes the end; a pair with three or more children reads its first two, and an Array with three or more reads its elements through m_Children[1] (:216, :309), where UnityPyBoost raises both; an aligned pair array reads a child whose type is a key of FUNCTION_READ_MAP (every primitive, string, TypelessData) through that function, without the child’s own align flag (a non-empty string still aligns itself, as read_aligned_string does), and any other child (a class, a vector, a pair) through read_value, which keeps its flag (TypeTreeHelper.py:285-293); a ReferencedObject reads type before anything else (a missing key is KeyError, other; a type with no class, ns or asm is KeyError and a non-dict type AttributeError, :89-97, both other where UnityPyBoost reports ref_type) and raises SerializedFile has no ref_types for an empty or absent list before the empty-class skip (no ref_types attribute is an AttributeError whose message the dump maps to ref_type); an aligned pair element with fewer than two children raises IndexError (other) at every length, since m_Children[0] and [1] are read before the loop (:282-283), while an empty array of non-aligned malformed pairs, or of three-child pairs, reads as [] (the probes: CANONICAL_DUMP.md section 7); likewise an aligned vector element whose Array has fewer than two children raises IndexError at every length, the empty one included, since m_Children[0].m_Children[1] is read before the loop (:309), where UnityPyBoost reads an empty one as [] (read_value_array reads an aligned element’s pair children and its vector’s element before its loop, and indexes everywhere else inside it); outside an array, a vector reads its length, and refuses a negative one, before it indexes its Array’s element (:213-216), and a pair reads its first child before it indexes its second (:194-195), so a malformed one fails with whatever those reads raise first (a vector whose Array has fewer than two children is negative_length at length -1 and IndexError, other, at 0), where UnityPyBoost checks each arity first and reports other: this reader never fails a compiled BadVector or BadPair on entry; and an aligned ReferencedObject array element whose class is empty or whose ref type has no tree raises (super::ReadDetail::RefTypeNoTree). Each difference is an oracle fixture of package B (plan section 7.3).
write
UnityPy’s pure-Python writer (helpers/TypeTreeHelper.py:161-179, :381-458). Dispatch is on the type name first (FUNCTION_WRITE_MAP), then pair, ReferencedObject, an Array first child, and a class. Under super::WriteRules::UnityPy each array element aligns after itself when its node is flagged, then the vector or Array flag aligns once; a class writes value[child.m_Name] for every child, a missing key failing and an extra key ignored; ReferencedObject data needs a non-empty ref_types (:99-100) and a ref type with a tree: an empty class, or a matched ref type with no tree, fails (KeyError: 'data' when the reader left data out, else an AttributeError on the None node); integers are range-checked as struct.pack checks them.

Structs§

FieldProgress
Progress of a binding’s ordered object construction.
ReadCtx
What the reader needs besides the bytes and the tree.
ReadError
A reader failure, with its kind decided structurally (never parsed from a message).
ReadOptions
The walker’s options.
WriteCtx
What the writer needs besides the value and the tree.

Enums§

FloatMode
How a float32 signalling NaN reaches the value.
L2Kind
The canonical dump’s L2 error kinds (CANONICAL_DUMP.md section 4.3).
ParseError
Why reading an object failed: the L2 outcome other than success.
ReadDetail
What a reader failure was.
ReadErrorKind
A reader failure’s L2 kind.
RefTypes
assetsfile.ref_types as the reader sees it (UnityPyBoost TypeTreeHelper.cpp:585-681; the pure-Python writer, TypeTreeHelper.py:88-109).
WriteError
Why writing a value failed (the dump’s L3 "error"). Each variant stands for the exceptions Self::exceptions names, the classes UnityPy 1.25.2’s write_typetree raises there; a variant UnityPy has no counterpart for names none.
WriteRules
The writer’s alignment rules.

Constants§

DICT_FIRST
The most entries a dict reserves before its first child is read: one for each child up to this many. A dict grows past it geometrically as its children are set, so a read holds no more than this many entries a dict that it has not filled, beyond as many again as it has filled (review round 7: every ReferencedObject level re-entering a ref type of k children held k entries reserved while the next level was read, gigabytes for a few kilobytes of data).
STEP_WORK
The open’s work one step costs a read or a write that builds no value, in bytes: a slot a dict’s table of positions grows by, an entry a lookup by name passes, and an entry a name table indexes. 8, as a step of a bundle read (plan 3.6, “What a read costs”). A name a step compares or hashes counts its bytes besides.
VALUE_WORK
The open’s work one value costs a read or a write, in bytes (crate::limits::Limits::max_decode_work): the size of the Value that holds it, whether the value is kept or not, as a decode counts every byte it writes whether its output is kept or not. A child a read or a write skips costs as much. An element of a primitive run that a read holds as a run costs its width instead (Work::run, Work::elements), and so does a primitive a write writes.
WORK_CHUNK
The most work a read or a write counts on its own before it adds it to its open’s meter, in bytes (OpenWork): so the most it can be past its open’s allowance before it learns of it, and the most each read or write in progress on other threads can add past it.

Functions§

read_typetree
TypeTreeHelper.read_typetree over bytes (UnityPy’s memory reader): boost::read over the Python slice file[byte_start : byte_start + byte_size] (crate::stream::py_read: short at the end of a truncated file, from the end for a negative start) when byte_size > 0, else python::read from byte_start; then, with check_read, bytes_read != byte_size is ParseError::CheckRead carrying the value.
read_typetree_with_reader
read_typetree from the file’s reader of kind (crate::files::SerializedFile::reader_kind): reset() sets the position to byte_start, which the stream reader refuses when negative or at or above 2^63 (super::ReadDetail::Stream, kind other), as UnityPy’s does on every machine; a byte_start past the largest file the file system takes is UnityPy’s OSError on that machine, and taken here (crate::stream::ReaderKind, PARITY.md section 19.3). The Boost path is given read_bytes(byte_size) from there (on both readers the bytes that exist), and the pure-Python reader reads by the reader’s rules (python::read_with_reader).