Module python_packed_resources::specifications

Specifications

Embedded Resources Data

The embedded Python interpreter can reference a data blob defining resources. These resources can be consumed by the custom meta path importer provided by this crate. This meta path importer parses a serialized data structure at run time, converts it into a Rust data structure, and uses the discovered resources to influence run-time behavior.

From a super high level, the embedded resources data structure defines an iterable of resources. A resource is an entity with a name, metadata, and blob fields. The most common resource is a Python module/package. But other resource types (such as shared libraries) are defined.

The format of this serialized data structure is as follows.

The first 8 bytes is a magic header identifying the content as our data type and the version of it. The first 7 bytes are pyembed and the following 1 byte denotes a version. The following sections denote the different magic headers/versions and their semantics.

pyembed\x01

Version 1 of the embedded resources data.

From a high-level, the serialized format consists of:

• A global header describing the overall payload.
• An index describing the blob sections present in the payload.
• An index describing each resource and its content.
• A series of blob sections holding the data referenced by the resources index.

A resource is composed of various fields that describe it. Examples of fields include the resource name, source code, and bytecode. The resources index describes which fields are present and where to find them in the payload.

The actual content of fields (e.g. the raw bytes containing source code) is stored in field-specific sections after the index. Each field has its own section and data for all resources is stored next to each other. e.g. you will have all the data for resource names followed by all data for module sourcecode.

The low-level data format is described below. All integers are little-endian.

The first 13 bytes after the magic header denote a global header. The global header consists of:

• A u8 denoting the number of blob sections, blob_sections_count.
• A u32 denoting the length of the blob index, blob_index_length.
• A u32 denoting the total number of resources in this data, resources_count.
• A u32 denoting the length of the resources index, resources_index_length.

Following the global header is the blob index. The blob index describes the various blob sections present in the payload following the resources index.

Each entry in the blob index logically consists of a set of fields defining metadata about each blob section. This is encoded by a start of entry u8 marker followed by N u8 field type values and their corresponding metadata, followed by an end of entry u8 marker. The blob index is terminated by an index of index u8 marker. The total number of bytes in the blob index including the end of index marker should be blob_index_length.

Following the blob index is the resources index. Each entry in this index defines a sparse set of metadata describing a single resource. Entries are composed of a series of u8 identifying pieces of metadata, followed by field-specific supplementary descriptions. For example, a value of 0x02 denotes the length of the resources's name and is immediately followed by a u16 holding said length. See the section below for each field tracked by this index.

Following the resources index is blob data. Blob data is logically consisted of different sections holding data for different fields for different resources. But there is no internal structure or separators: all the individual blobs are just laid out next to each other.

Blob Field Types

The Blob Index allows attributing a sparse set of metadata with every blob section entry. The type of metadata being conveyed is defined by a u8. Some field types have additional metadata following that field.

The various field types and their semantics follow.

0x00 - End of index. This field indicates that there are no more blob index entries and we've reached the end of the blob index.

0x01 - Start of blob section entry. Encountering this value signals the beginning of a new blob section. From a specification standpoint, this isn't strictly required. But it helps ensure parser state.

0xff - End of blob section entry. Encountering this value signals the end of the current blob section definition. The next encountered u8 in the index should be 0x01 to denote a new entry or 0x00 to denote end of index.

0x02 - Resource field type. This field defines which resource field this blob section is holding data for. A u8 following this one will contain the resource field type value (see section below).

0x03 - Raw payload length. This field defines the raw length in bytes of the blob section in the payload. The u64 containing that length will immediately follow this u8.

0x04 - Interior padding mechanism. This field defines interior padding between elements in the blob section. Following this u8 is another u8 denoting the padding mechanism. 0x01 indicates no padding. 0x02 indicates NULL padding (a 0x00 between elements). If not present, no padding is assumed. If the payload data logically consists of discrete resources (e.g. Python package resource files), then padding applies to these sub-elements as well.

Resource Field Types

The Resources Index allows attributing a sparse set of metadata with every resource. A u8 indicates what metadata is being conveyed. Some field types have additional metadata following this [u8] further defining the field. The values of each defined metadata type follow.

0x00 - End of index. Special type to denote the end of an index.

0x01 - Start of resource entry. Signals the beginning of a new resource. From a specification standpoint this isn't strictly required. But it helps ensure parser state.

0x02 - Resource flavor. Declares the type of resource this entry represents. A u8 defining the resource flavor immediately follows this byte. See the section below for valid resource flavors.

0xff - End of resource entry. The next encountered u8 in the index should be an end of index or start of resource marker.

0x03 - Resource name. A u16 denoting the length in bytes of the resource name immediately follows this byte. The resource name must be valid UTF-8.

0x04 - Package flag. If encountered, the resource is identified as a Python package.

0x05 - Namespace package flag. If encountered, the resource is identified as a Python namespace package.

0x06 - In-memory Python module source code. A u32 denoting the length in bytes of the module's source code immediately follows this byte.

0x07 - In-memory Python module bytecode. A u32 denoting the length in bytes of the module's bytecode immediately follows this byte.

0x08 - In-memory Python module optimized level 1 bytecode. A u32 denoting the length in bytes of the module's optimization level 1 bytecode immediately follows this byte.

0x09 - In-memory Python module optimized level 2 bytecode. Same as previous, except for bytecode optimization level 2.

0x0a - In-memory Python extension module shared library. A u32 denoting the length in bytes of the extension module's machine code immediately follows this byte.

0x0b - In-memory Python resources data. If encountered, the module/package contains non-module resources files and the number of resources is contained in a u32 that immediately follows. Following this u32 is an array of (u16, u64) denoting the resource name and payload size for each resource in this package.

0x0c - In-memory Python distribution resource. Defines resources accessed from importlib.metadata APIs. If encountered, the module/package contains distribution metadata describing the package. The number of files being described is contained in a u32 that immediately follows this byte. Following this u32 is an array of (u16, u64) denoting the distribution file name and payload size for each virtual file in this distribution.

0x0d - In-memory shared library. If set, this resource is a shared library and not a Python module. The resource name field is the name of this shared library, with file extension (as it would appear in a dynamic binary's loader metadata to indicate a library dependency). A u64 denoting the length in bytes of the shared library data follows. This shared library should be loaded from memory.

0x0e - Shared library dependency names. This field indicates the names of shared libraries that this entity depends on. The number of library names is contained in a u16 that immediately follows this byte. Following this u16 is an array of u16 denoting the length of the library name for each shared library dependency. Each described shared library dependency may or may not be described by other entries in this data structure.

0x0f - Relative filesystem path to Python module source code. A u32 holding the length in bytes of a filesystem path encoded in the platform-native file path encoding follows. The source code for a Python module will be read from a file at this path.

0x10 - Relative filesystem path to Python module bytecode. Similar to the previous except the filesystem path holds Python module bytecode.

0x11 - Relative filesystem path to Python module bytecode at optimization level 1. Similar to the previous except for what is being pointed to.

0x12 - Relative filesystem path to Python module bytecode at optimization level 2. Similar to the previous except for what is being pointed to.

0x13 - Relative filesystem path to Python extension module shared library. Similar to the previous except the file holds a Python extension module loadable as a shared library.

0x14 - Relative filesystem path to Python package resources. The number of resources is contained in a u32 that immediately follows. Following this u32 is an array of (u16, u32) denoting the resource name and filesystem path to each resource in this package.

0x15 - Relative filesystem path to Python distribution resources. Defines resources accessed from importlib.metadata APIs. If encountered, the module/package contains distribution metadata describing the package. The number of files being described is contained in a u32 that immediately follows this byte. Following this u32 is an array of (u16, u32) denoting the distribution file name and filesystem path to that distribution file.

Resource Flavors

The data format allows defining different types/flavors of resources. This flavor of a resource is identified by a u8. The declared flavors are:

0x00 - No flavor. Should not be encountered.

0x01 - Python module/package. This type represents a normal Python module.

0x02 - Builtin Python extension module. This type represents a Python extension module that is built in (compiled into) the interpreter itself or is otherwise made available to the interpreter via PyImport_Inittab such that it should be imported with the builtin importer.

0x03 - Frozen Python module. This type represents a Python module whose bytecode is frozen and made available to the Python interpreter via the PyImport_FrozenModules array and should be imported with the frozen importer.

0x04 - Python extension. This type represents a compiled Python extension. Extensions have specific requirements around how they are to be loaded and are differentiated from regular Python modules.

0x05 - Shared library. This type represents a shared library that can be loaded into a process.

Design Considerations

The design of the embedded resources data format was influenced by a handful of considerations.

Performance is a significant consideration. We want everything to be as fast as possible. Possible dimensions influencing performance include parse time, payload size, and I/O access patterns.

The payload is designed such that the index data is at the beginning so a reader only has to read a contiguous slice of data to fully understand the data within. This is in opposition to jumping around the entire data structure to extract metadata of the data within. This means that we only need to page in a fraction of the total backing data structure in order to initialize our custom importer. In addition, the index data is read sequentially. Sequential I/O should always be faster than random access I/O.

x86 is little endian, so we use little endian integers so we don't need to waste cycles on endian transformation.

We store all data for the same field next to each other in the data structure. This is in opposition to say packing all of resource A's data then resource B's, etc. We do this to help maximize locality for similar data. This can help with performance because often the same field for multiple resources is accessed together. e.g. an importer will access a bunch of module bytecode entries at the same time. This locality helps minimize the number of pages that must be read. Locality can also help yield higher compression ratios.

Everything is designed to facilitate a reader leveraging 0-copy. If a reader has the data structure in memory, we don't want to require it to copy memory in order to reference entries. In Rust speak, we should be able to hold &[u8] references everywhere.

There is no checksumming of the data because we don't want to incur I/O overhead to read the entire blob. It could be added as an optional feature.

A potential area for optimization is use of general compression. Various fields should compress well - either in streaming mode or by utilizing compression dictionaries. Compression would undermine 0-copy, of course. But in environments where we want to optimize for size, it could be desirable.