Expand description
Reading and writing values through a typetree.
TypeTreeHelper.read_typetree (helpers/TypeTreeHelper.py:123-158) picks
a reader by the object’s size: with byte_size > 0 UnityPyBoost reads the
object’s bytes and nothing else (boost); with byte_size == 0 the
pure-Python reader starts at byte_start in the whole file and may read on
past the object (python). Then check_read compares the bytes read
with byte_size, in either direction. UnityPyBoost has no writer, so
write ports UnityPy’s pure-Python write_typetree (:161-179,
:407-458), which does not mirror the reader (plan section 3.11).
Memory. Every allocation a read makes for the value (a list, a map, a
tuple, a primitive run) is charged to ReadCtx::meter, the input’s,
before it is made, at crate::limits::heap_cost of its capacity, and
the charges ride in the Parsed (grow through
crate::limits::MeteredVec, then hand its charge to the Parsed). A
refused charge is ReadDetail::Charge (kind other). The writer’s
output grows the same way. What a read reserves before the bytes it holds
are read is bounded apart from the tree’s width and the read’s depth: a
dict at most DICT_FIRST entries, growing as its children are set;
the lists a read holds open at once, max_prealloc elements in all
(Work::first_room), each growing geometrically as it is filled (review
round 7 and its sweep: a dict reserved one entry a child, and each list
open max_prealloc of its own, so a ReferencedObject re-entering a
wide ref type, or vectors nested in vectors, held that many times over
for a few bytes of data; tests/walk_reentry.rs).
Work. Every read and every write counts what it does toward its
open’s work (crate::limits::Limits::max_decode_work, the bound every
other kind of the open’s work counts toward), whatever its outcome:
VALUE_WORK for each value it builds, writes or skips, and for each
element of a primitive run a read holds as a run, or each primitive a
write writes, its bytes (Work); each byte of a string a write copies,
or passes over to find its characters; STEP_WORK for each step that
builds no value: a slot a dict’s table of positions grows by, an entry a
lookup by name passes, a name a writer’s table indexes or probes (and
the bytes of a name it compares or hashes); and its ref-type lookups, a
ref type a lookup compares or indexes, or a probe, VALUE_WORK and the
name bytes it reads (dict::RefIndex). Objects may share their bytes,
and a read refused by a bound of its own (max_zero_size_elements,
max_values_per_object) keeps nothing, so K objects over the same bytes
repeated it K times with no bound on the input (fix pass 4’s sweep: every
object of a small file over one vector of 2^22 empty elements, each
refused after 2^20); every object searches its file’s one list of ref
types again, so K objects over R ref types took K x R steps (fix pass 5’s
sweep); and a read failing at a class’s first child had passed over all
of its children to size its dict (review round 6: K objects over a class
of k children, K x k steps; review round 7: the pass that stayed for
a class whose children repeat a name refused such reads with
DecodeWork where UnityPy read each to its own failure). No read or
write passes over a class’s children before it reads or writes them.
Each step is counted before it is taken, on the read’s or the write’s
OpenWork, which adds the steps to the open’s meter a chunk
(WORK_CHUNK) at a time, in one atomic check-and-add: a chunk that
would take the open past its allowance is refused, and the read or the
write fails with LimitKind::DecodeWork (kind other for a read); one
whose own work, counted when it ends, took the open past its allowance
fails too (its own failure first, if it has one). So the reads and writes
that succeed never do more work together than the allowance, on any
number of threads, and each one in progress does at most a chunk and one
step past it before it stops (review round 6: a read added its work only
when it ended, so two reads on two threads both passed the start’s check
and ran to their end past the allowance). Once the open’s work has passed
its allowance, a read or a write is refused before it starts
(tests/walk_work.rs, tests/walk_prepass.rs).
Owner: the scaffold for the types in this file (frozen); package B owns
read_typetree’s body and the three submodules.
Modules§
- boost
- UnityPyBoost’s reader (
UnityPyBoost/TypeTreeHelper.cpp:67-1056), for objects withbyte_size > 0. The rules, each pinned by a test (plan section 3.10, R1-R16): bounds checkspos + n > end, a length’s before the bytes it counts;align4from the object’s start with no bounds check; strings always aligned;TypelessDataaligned only by its flag; the switch on the node’s own kind first, and only a kind it does not handle (Unknown,PPtr,Array,ReferencedObjectData,ManagedReferencesRegistry) is an array when its first child isArray(R9); primitive andpairelements read as one run and align once, apairrun checking its arity before the first element, so even when empty; other elements read and align one by one; theManagedReferencesRegistryscope;ReferencedObjectdata resolved through the file’s ref types in UnityPyBoost’s order (R14: no file,ref_typesabsent or not a list, notyperead yet, a missingclass/ns/asm, allref_type; then an empty class or a matched ref type with no tree skipsdata; no match isref_type; aclassthat is not aStrskipsdatawhen its Python object’s size field is 0, else isref_type, plan R14);PyDict_SetItemfor repeated names; float32 widened (a signalling NaN quieted undersuper::FloatMode::UnityPy). - emit
- A direct read for a binding: UnityPyBoost’s reader (
super::boost’s rules, R2-R15) over the trees and objects that need none of its bookkeeping, each value handed to aSinkas it is read, with nocrate::value::Valuein between (UnityPyBoost builds its Python objects as it reads,TypeTreeHelper.cpp:702-883). - python
- UnityPy’s pure-Python reader (
helpers/TypeTreeHelper.py:182-361), for objects withbyte_size == 0. It starts atbyte_startin the wholeSerializedFileand may read on past the object, with Python’s reader rules at any position (crate::stream’s helpers; the position may go negative). Alignment is relative to the file, not the object:align_streamrounds the absolute position. Its differences from UnityPyBoost aredocs/CANONICAL_DUMP.mdsection 7’s list: aligned primitive arrays become tuples, with 16-bit signedness swapped; an alignedSInt8array raises at every length (read_byte_array(reader, size)before any loop), and an array of alignedTypelessDataraises only when its length is above 0 (read_bytes()inside the loop; an empty one reads as[]); a negative or out-of-range string length reads as"", and a string of length 0 or read as""that way is not aligned after (read_aligned_stringreturns beforealign_stream: the base reader’s,streams/EndianBinaryReader.py:146-153, whichFUNCTION_READ_MAPholds for astringnode and apairrun’s child, and for an array’s elements read from bytes the memoryview reader’s override,:266-273, the same body); in an array whose aligned element is a vector, a negative inner length reads as[], unless the inner element is an aligned primitive, whosestructformat rejects it (struct.error,other); arrays of aligned classes align once and skip the registry check;TypelessDatalength -1 reads to the end of the file and other negative lengths follow Python slice semantics (sobytes_readcan be negative), on the memory reader: a file loaded from a path is read by the stream reader’s rules (read_with_reader), where such a length raises and no read passes the end; apairwith three or more children reads its first two, and anArraywith three or more reads its elements throughm_Children[1](:216,:309), where UnityPyBoost raises both; an alignedpairarray reads a child whose type is a key ofFUNCTION_READ_MAP(every primitive,string,TypelessData) through that function, without the child’s own align flag (a non-emptystringstill aligns itself, asread_aligned_stringdoes), and any other child (a class, a vector, apair) throughread_value, which keeps its flag (TypeTreeHelper.py:285-293); aReferencedObjectreadstypebefore anything else (a missing key isKeyError,other; atypewith noclass,nsorasmisKeyErrorand a non-dicttypeAttributeError,:89-97, bothotherwhere UnityPyBoost reportsref_type) and raisesSerializedFile has no ref_typesfor an empty or absent list before the empty-class skip (noref_typesattribute is anAttributeErrorwhose message the dump maps toref_type); an alignedpairelement with fewer than two children raisesIndexError(other) at every length, sincem_Children[0]and[1]are read before the loop (:282-283), while an empty array of non-aligned malformedpairs, or of three-childpairs, reads as[](the probes:CANONICAL_DUMP.mdsection 7); likewise an aligned vector element whoseArrayhas fewer than two children raisesIndexErrorat every length, the empty one included, sincem_Children[0].m_Children[1]is read before the loop (:309), where UnityPyBoost reads an empty one as[](read_value_arrayreads an aligned element’spairchildren and its vector’s element before its loop, and indexes everywhere else inside it); outside an array, a vector reads its length, and refuses a negative one, before it indexes itsArray’s element (:213-216), and apairreads its first child before it indexes its second (:194-195), so a malformed one fails with whatever those reads raise first (a vector whoseArrayhas fewer than two children isnegative_lengthat length -1 andIndexError,other, at 0), where UnityPyBoost checks each arity first and reportsother: this reader never fails a compiledBadVectororBadPairon entry; and an alignedReferencedObjectarray element whose class is empty or whose ref type has no tree raises (super::ReadDetail::RefTypeNoTree). Each difference is an oracle fixture of package B (plan section 7.3). - write
- UnityPy’s pure-Python writer (
helpers/TypeTreeHelper.py:161-179,:381-458). Dispatch is on the type name first (FUNCTION_WRITE_MAP), thenpair,ReferencedObject, anArrayfirst child, and a class. Undersuper::WriteRules::UnityPyeach array element aligns after itself when its node is flagged, then the vector orArrayflag aligns once; a class writesvalue[child.m_Name]for every child, a missing key failing and an extra key ignored;ReferencedObjectdata needs a non-emptyref_types(:99-100) and a ref type with a tree: an empty class, or a matched ref type with no tree, fails (KeyError: 'data'when the reader leftdataout, else anAttributeErroron theNonenode); integers are range-checked asstruct.packchecks them.
Structs§
- Field
Progress - Progress of a binding’s ordered object construction.
- ReadCtx
- What the reader needs besides the bytes and the tree.
- Read
Error - A reader failure, with its kind decided structurally (never parsed from a message).
- Read
Options - The walker’s options.
- Write
Ctx - What the writer needs besides the value and the tree.
Enums§
- Float
Mode - How a float32 signalling NaN reaches the value.
- L2Kind
- The canonical dump’s L2 error kinds (
CANONICAL_DUMP.mdsection 4.3). - Parse
Error - Why reading an object failed: the L2 outcome other than success.
- Read
Detail - What a reader failure was.
- Read
Error Kind - A reader failure’s L2 kind.
- RefTypes
assetsfile.ref_typesas the reader sees it (UnityPyBoostTypeTreeHelper.cpp:585-681; the pure-Python writer,TypeTreeHelper.py:88-109).- Write
Error - Why writing a value failed (the dump’s L3
"error"). Each variant stands for the exceptionsSelf::exceptionsnames, the classes UnityPy 1.25.2’swrite_typetreeraises there; a variant UnityPy has no counterpart for names none. - Write
Rules - The writer’s alignment rules.
Constants§
- DICT_
FIRST - The most entries a dict reserves before its first child is read: one
for each child up to this many. A dict grows past it geometrically as its
children are set, so a read holds no more than this many entries a dict
that it has not filled, beyond as many again as it has filled (review
round 7: every
ReferencedObjectlevel re-entering a ref type ofkchildren heldkentries reserved while the next level was read, gigabytes for a few kilobytes of data). - STEP_
WORK - The open’s work one step costs a read or a write that builds no value, in bytes: a slot a dict’s table of positions grows by, an entry a lookup by name passes, and an entry a name table indexes. 8, as a step of a bundle read (plan 3.6, “What a read costs”). A name a step compares or hashes counts its bytes besides.
- VALUE_
WORK - The open’s work one value costs a read or a write, in bytes
(
crate::limits::Limits::max_decode_work): the size of theValuethat holds it, whether the value is kept or not, as a decode counts every byte it writes whether its output is kept or not. A child a read or a write skips costs as much. An element of a primitive run that a read holds as a run costs its width instead (Work::run,Work::elements), and so does a primitive a write writes. - WORK_
CHUNK - The most work a read or a write counts on its own before it adds it to
its open’s meter, in bytes (
OpenWork): so the most it can be past its open’s allowance before it learns of it, and the most each read or write in progress on other threads can add past it.
Functions§
- read_
typetree TypeTreeHelper.read_typetreeover bytes (UnityPy’s memory reader):boost::readover the Python slicefile[byte_start : byte_start + byte_size](crate::stream::py_read: short at the end of a truncated file, from the end for a negative start) whenbyte_size > 0, elsepython::readfrombyte_start; then, withcheck_read,bytes_read != byte_sizeisParseError::CheckReadcarrying the value.- read_
typetree_ with_ reader read_typetreefrom the file’s reader ofkind(crate::files::SerializedFile::reader_kind):reset()sets the position tobyte_start, which the stream reader refuses when negative or at or above 2^63 (super::ReadDetail::Stream, kindother), as UnityPy’s does on every machine; abyte_startpast the largest file the file system takes is UnityPy’sOSErroron that machine, and taken here (crate::stream::ReaderKind,PARITY.mdsection 19.3). The Boost path is givenread_bytes(byte_size)from there (on both readers the bytes that exist), and the pure-Python reader reads by the reader’s rules (python::read_with_reader).