Skip to main content

Module pystr

Module pystr 

Source
Expand description

Python string semantics on raw bytes, as UnityPy applies them to names.

UnityPy compares CAB and resource names after ntpath.basename and str.lower() (environment.py:357-363, classes/PPtr.py:66-77, helpers/ResourceReader.py:11-18). Python lower-cases a surrogate-escaped string: valid UTF-8 runs with Unicode full case mapping (Python 3.11 has Unicode 14.0), invalid bytes passed through. Rust’s tables are newer, so a generated exception table (crate::tables::lower_exception) pins the code points whose mapping changed after 14.0 (tools/gen/, package A; plan risk R7). The ntpath functions behave as Python 3.11’s do, which tools/fixtures/env.py records (tests/fixtures/d/pystr.json).

Which bytes are characters. A surrogate-escaped string is UTF-8 decoded with surrogateescape: every valid UTF-8 sequence is one character, and every byte of an invalid sequence is a lone surrogate (U+DC80-U+DCFF), which lower() keeps and the encoder turns back into the byte. Python’s decoder and std::str::Utf8Chunks split a byte string into the same valid runs (both reject surrogates, overlong forms and code points past U+10FFFF), so a run of std::str::Utf8Chunks is lowered as a str and its invalid bytes are copied.

Final sigma. str.lower() maps U+03A3 to U+03C2 in the Final_Sigma context (handle_capital_sigma in Python’s unicodeobject.c): a cased character before it and none after it, skipping case-ignorable ones. The two classes come from Python 3.11’s own str.lower() (crate::tables::is_sigma_cased, crate::tables::is_case_ignorable, Unicode 14.0), not from Rust’s newer tables, which differ for characters Unicode assigned after 14.0. A lone surrogate (an invalid byte) is neither cased nor case-ignorable to Python, so the context never crosses a run’s edge and a run lowered on its own gets Python’s answer.

Memory. Each function that returns a string computes it from an input’s name (a directory path, an external path, a resource path, up to the input’s size) and charges it to that input’s meter before it is allocated (RawStr::charged’s cost): a refusal allocates nothing. A result equal to its input (a name already lower-case and without a directory) is returned as the input’s own bytes, uncopied, and a result that is a part of its input (a basename already lower-case) as a slice of them. Comparisons that only need to know whether two names are equal (lower_eq) compare without building the lowered string at all, and nothing else is allocated: a capital sigma’s context is read in place.

Owner: package D.

Functions§

clean_external_path
PPtr.deref’s external path cleaning: strip a leading archive:/, then a leading assets/, keep what follows the last /, lower-case it; charged to meter.
is_split_path
Whether UnityPy’s load_file reads a path as the parts of a split file: reSplit.match(path) (environment.py:25, :111-116), with reSplit (.*?([^\/\\]+?))\.split\d+, a match at the start that need not reach the end. So .split and a digit must follow at least one character that is no / or \, and no \n (which . does not match) may come before the last separator ahead of it. Python’s \d on a str pattern is Py_UNICODE_ISDECIMAL (sre’s SRE_UNI_IS_DIGIT): a character with a decimal digit value, which Unicode gives exactly the category Nd, read from Python 3.11’s own table (crate::tables::nd_digit, Unicode 14.0), so a name such as x.split² (No) or x.splitⅠ (Nl) is one file, as UnityPy reads it (tests/fixtures/d/pystr.json records \d over every code point). Linear in the path.
lower_eq
Whether a.lower() == lowered, compared piece by piece without building a.lower() (classes/PPtr.py:75-76 compares each files key so); it allocates nothing.
ntpath_basename
ntpath.basename: after splitdrive, the part after the last / or \.
ntpath_splitext
ntpath.splitext: (root, ext), leading dots not counting as an extension.
py_lower
str.lower() on surrogate-escaped bytes, charged to meter.
simplify_name
simplify_name: ntpath.basename(name).lower(), the CAB registry’s key, charged to meter.
simplify_name_vec
A transient CAB lookup key. Keeps computed bytes in a fallibly allocated, charged vector instead of constructing a Bytes shared owner (CR10).