Expand description
UnityPy’s Environment (environment.py:28-363): loaded files, the CAB
registry, dependency loading, pointer dereference, resource reads and the
container.
Two load modes (plan section 3.13):
LoadMode::Eageris UnityPy: a bundle decodes every block, then parses every entry in directory order and registers eachSerializedFileand raw entry as a CAB bycrate::pystr::simplify_nameas it parses it (files/File.py:76-90). A failing entry fails the load, andload_fileraises before the bundle entersfiles(environment.py:154-157), but the CABs of the entries parsed before it stay registered and resolvable (partial registration: probed on UnityPy 1.25.2 with a bundle[good CAB, its .resS, bad CAB], after whichenv.filesis empty andenv.cabsholds the first two).LoadMode::Lazyis the batch engine’s: a bundle registers its entries’ names at open and parses an entry the first time it is asked for (computed with no lock held and published once: two threads that ask at once may both parse, and the first to publish wins).
On a scene whose blocks all decode, whose entries all parse and none of
which is a container, the two modes give identical results, the
container pass’s RuntimeError included. Elsewhere they differ in
exactly four ways (plan section 3.13, PARITY.md section 19.3, each
pinned by a test in tests/env/modes.rs): (a) a failing entry fails the load
under Eager (with the partial registration above) but only the get
that parses it under Lazy (an entry shadowed by a later one of its path
is then never parsed; crate::files::BundleFile::load_all reproduces
Eager’s error); (b) a nested container’s own name is a key under
Lazy, whose slot answers None, where Eager registers no such key
(and keeps an earlier CAB of that path under it); its entries are
registered when it is reached, before any lookup that resolves its
slot, starts from a file inside it, names it as the asking file’s
sibling or resource candidate, or walks the container, and before a
registration drops its slot (an evicted dependency’s before the next
container pass), with the position and the value an eager load gives
them, so only a lookup by name alone of one of its entries can miss
before then (registry’s module docs state the invariant);
(c) a block that fails to decode, or trips a limit,
after the last byte any read needs fails the load under Eager (UnityPy
decodes every block before any entry, files/BundleFile.py:173-186) and
nothing under Lazy; (d) a directory whose spans nest or overlap makes
Lazy decode and keep the shared blocks once per span, so it can stop
at the input’s meter (crate::LimitKind::InputMemory) or decode-work
bound (crate::LimitKind::DecodeWork) where Eager’s one run fits.
Threads (plan section 3.16, item 6). Each key’s registration, each
files entry and each path_dict entry is published whole, so no call
sees one half made. A load’s or a reach’s set of keys is not published at
once: another thread may find one key of a load, or of a nested
container being reached, and miss a later one (review round 6: these
docs said concurrent calls act as if made one after another, which that
does not). A call that reaches a nested container returns only after its
whole subtree is registered, and a load only after all its keys and its
files entry are. A dependency’s load under Lazy (load_dependency,
or a resolver’s load made as a dependency, its files entry under
keep_resolver_loads included) publishes its first key only once every
one is charged, so one that fails publishes none (registry’s module
docs, “A failed dependency load”); a caller’s load, and any load under
Eager, publishes key by key, and one that fails keeps those it
published. A call’s result depends on what was done before it,
by its thread or another, as a Python dict’s does: a load, a resolver
load, and under Lazy the first call that reaches a nested container.
A container pass’s preload walk that a load on another thread meets
gives, for each key the load changes, what UnityPy’s walk gives with
that change made at the point of the walk at which it was made, between
two of its steps (watch‘s module docs, W1; review round 14); a key
the load adds does not fail the pass, where UnityPy’s raises.
What a set of calls leaves does not depend on the order of the ones that
are not loads: the registry orders entries by rank (registry),
files by the loads’ numbers the same way (Environment::files),
and each path_dict entry keeps the least stamp (crate::container).
So once racing loads have returned, the registry and files hold what
the same loads make one after another in the order of their numbers
(review round 16: files kept the order the threads listed in, so a
load that took the lower number but listed last won files and lost
the registry, for good).
Memory. Each load makes its input’s meter
(crate::limits::Meter::for_input): crate::budget::Reservation::Input
for what a caller loads, crate::budget::Reservation::Dependency for a
dependency. Every Source is bounded by
crate::limits::Limits::max_input_bytes before its bytes are detected
(LimitKind::InputBytes): a Source::Path is read by read_path,
refused from its length before anything is read and charged to that
meter before its bytes exist; a Source::Fetch is refused from the
size its fetcher states (Fetch::size_hint) before it is fetched, and
from the length of what it returns; Source::Bytes from their length.
Each registry slot, its key and its share of the
registry’s tables are charged to the meter of the file that registered
it (registry), and each files entry, its share of the list and its
lowered key to the loaded file’s (Environment::files); the LRU’s
evictions allocate nothing (registry). Listings (Environment::objects,
Environment::container, Registry::keys) are iterators: they hold
a handle per nesting level and allocate nothing per record.
Dependencies (files loaded to resolve a pointer or a resource, or by
Environment::load_dependency) are kept in an LRU bounded by
EnvOptions::dep_cache_bytes (the sum of their meters). The registry
holds no strong handle on a dependency, only what reopens it
(registry), so evicting one drops the LRU’s handle and with it the
dependency’s runs and parsed entries, and the bytes of a source that can
be read again (Source::Path, Source::Fetch) with their charge,
once nothing else holds them; the next read opens it again from its
source on crate::limits::Meter::reopen of its meter: the same memory
account, and the new open’s own work allowance, so eviction churn never
exhausts a valid dependency’s allowance, and a handle of the old open
that someone still holds (a nested bundle a resource read kept) goes on
counting against the old open’s, which nothing resets. A
Source::Bytes dependency keeps its bytes, which are the caller’s.
An ordinary CAB lookup after reopening a dependency can see changed bytes,
and it follows the indices registered at the first open, so a source whose
bundle directory was reordered can resolve a CAB name to a different entry.
Retained-object reuse via Environment::deref_held_reusing assumes
stable bytes across reopens. If the source changes after eviction, reuse
may still return the previously retained object.
The files a caller loaded are never evicted. Every call that reads or
registers through a dependency’s meter (a load, a lookup, a dereference,
a resource read, an object’s tree with the script hook, the container
pass), when the pool refuses it, evicts the others and retries once, by
one rule however the calls nest (registry’s with_room;
tests/env/calls.rs lists every public call and how it meets the
pool); the refusal is never cached
(crate::once::get_or_publish_result), so a later read with room in
the pool succeeds. A read the caller makes itself on a dependency’s
handle (an object’s value) is no call of the Environment’s, and its
refusal is the caller’s.
Resolver loads in files (probed on UnityPy 1.25.2, plan section
3.13). UnityPy keeps every file it loads, those it loads to resolve a
pointer or a resource included, in env.files, for good:
- a resource read’s fallback (
SerializedFile.load_dependencies: the file’s externals and the four candidates) callsload_file(name, True), whose second positional parameter isparent, so those files are not dependencies: keyed by the name asked, their objects are inenv.objectsand their CABs’ rows inenv.container, and a second fallback loads the file again and replaces the entry in place; find_fileloads a file as a dependency (itsis_dependencydefaults to true, and a dereference takes the default), keyed by the path it resolved (env.pathjoined with the name:load_filekeys a path by itself);env.objectsandenv.containerskip it, and sinceenv.containerwalksenv.cabs, where a dependency’s CAB replaces a CAB of the same key, the replaced file’s rows leave the container while its objects stay inenv.objects, which walksenv.files. Withis_dependency=Falsethe file is a caller’s load under that key. A raw reader carries no flag (parse_filereturns the reader itself), so it is no dependency togetattr(f, "is_dependency", False).
EnvOptions::keep_resolver_loads (true in EnvOptions::unitypy) does
exactly that: Environment::files lists each resolver load as a
FileEntry, is_dependency as UnityPy sets it, and none is evicted.
With it false (EnvOptions::batch) resolver loads are LRU
dependencies, evictable, and in neither files, objects nor
container, whatever find_file’s is_dependency: PARITY.md
section 19.3’s row “resolver loads under keep_resolver_loads false”
(find_file’s and a dereference’s; review round 12: no row listed
them) and section 19.2’s SerializedFile.load_dependencies row (a
resource fallback’s). An evicted dependency is opened again from its
source by the next read that needs it (section 19.3’s row “a dependency
the LRU evicts”). Environment::load_dependency, which UnityPy has
no call for, never adds to files.
Locks. The registry, the files list, the LRU and a file’s
path_dict are locked only to look up or publish; a file is loaded, an
entry parsed and a resolver, fetcher or provider called with no lock held
(crate::once; tests/env/locks.rs runs loads on another thread while a
resolver or a fetch waits for them). The one
caller code that may run under them is a crate::budget::MemoryBudget
or crate::budget::PermitHome (a growth charged while a table is
published, a replaced value’s charge returned), whose contract forbids
blocking and calling the core (plan section 3.16).
Where UnityPy reaches the filesystem (find_file’s walk of env.path,
load_file’s relative paths), the core asks a CabResolver: the core
itself never opens a path it was not given.
Owner: package D. Option constructors are implemented by the scaffold.
Re-exports§
pub use pptr::DerefError;pub use pptr::PPtr;pub use registry::Cab;pub use registry::HeldCab;pub use registry::Keys;pub use registry::RawCab;pub use registry::Registry;pub use resource::Candidate;pub use resource::SUFFIXES;pub use resource::resource_candidates;
Modules§
- pptr
PPtrand its dereference (classes/PPtr.py:15-107).- registry
- The CAB registry: UnityPy’s
env.cabsdict (environment.py:252-280). - resource
- Resource lookup (
helpers/ResourceReader.py:10-38).
Structs§
- Container
Item - A container row with the object it points at, as
env.containeryields. - Container
Items - The iterator
Environment::containerreturns, CABs in registry order. - Dependency
File - Reopens a registered dependency without pinning its files in the cache.
- EnvOptions
- Environment options.
- EnvStats
- Counters, for the batch report and tests.
- Environment
- UnityPy’s
Environment. A cheap handle: clones share one environment. - File
Entries - The iterator
Environment::filesreturns: a cursor at the position of the last entry it yielded, which entries listed meanwhile do not disturb. - File
Entry - One entry of
Environment::files: UnityPy’senv.files[name], with the file’sis_dependencyflag. - File
Occurrence - A serialized file’s occurrence, comparable before its data is loaded. Dependency occurrences are stable across cache reopens.
- Objects
- The iterator
Environment::objectsreturns: a handle per nesting level (the position of the lastfilesentry it stepped into, the bundle, the entry, the object index).
Enums§
- Load
Mode - When a container’s entries are parsed.
- Mono
Behaviour Error - Failure of the explicit MonoBehaviour generator operation. Only
ValueErrorfailures are swallowed by automatic fallback; other failures remain visible. - Source
- Something to load. Every kind is bounded by
Limits::max_input_bytes: bytes longer than it, however they came, are refused (LimitKind::InputBytes) before anything detects, parses or keeps them, so a refused fetch’s bytes drop at once, with any charge they carry.
Traits§
- CabResolver
- Finds a file the registry does not hold, where UnityPy looks in
env.path:find_file’s fallback afterget_cab(environment.py:310-354), and eachload_file(name, True)of a resource read’s fallback (files/SerializedFile.py:330-349). - Fetch
- Bytes the caller fetches on demand (a batch engine’s mmap or object store).
Functions§
- read_
path - Reads
pathwhole forSource::Path. A file whose length is abovelimits.max_input_bytesis refused before anything is read (LimitKind::InputBytes). The buffer is allocated once at the stat’d length, charged tometerfirst, and filled; the end of the file is then checked with a read into a fixed stack buffer, so a file that ends where its length says costs exactly its buffer (heap_cost(len)and the owner record,crate::budget::OWNER_COST), never a growth past it. Only a file that outlasts its stat’d length (one that grows while it is read, a pipe, a device) grows the buffer, charged as it grows, and the read stops at the bound. The returned bytes carry the charge.