cpex-hosts-python

Runs existing Python CPEX plugins out-of-process from the Rust PluginManager. Each plugin gets its own cached virtualenv and a long-lived worker.py subprocess, spoken to over newline-delimited JSON on stdio — the same protocol the Python CLI already uses, so a plugin that works there works here unchanged.

Registers under the plugin kind isolated_venv.

Why out-of-process

The runtime never links libpython, so a plugin crash, a C-extension segfault, or a dependency conflict cannot take the gateway down. Each plugin's dependencies live in its own venv, so two plugins can pin incompatible versions of the same package. This supersedes issue #20's original python:// in-process (PyO3) framing.

Usage

Register the factory before loading config:

use cpex_core::factory::PluginFactoryRegistry;
use cpex_hosts_python::{IsolatedVenvFactory, KIND};

let mut factories = PluginFactoryRegistry::new();
factories.register(KIND, Box::new(IsolatedVenvFactory));

let manager = PluginManager::from_config(config, &factories)?;

Config schema

Host-specific settings live in the plugin entry's opaque config: map, because PluginConfig has no fields for them:

plugins:
  - name: pii-filter
    kind: isolated_venv
    hooks: [tool_pre_invoke]
    config:
      # Required — the fully-qualified Python class.
      class_name: my_pkg.filters.PiiFilter

      # Optional. Installed into the venv on a cache miss.
      requirements_file: requirements.txt

      # Optional, with defaults matching the Python reference implementation.
      script_path: cpex/framework/isolated/worker.py
      max_content_size: 10000000   # bytes, per outbound task
      timeout_secs: 30             # per invocation

Plugin directories are not configurable

The host always uses plugins/ at the project root — the process working directory — as the worker's sys.path entry and as the directory the venv is built under. There is no config key for it:

• a plugin_dirs: key inside a plugin's config: block is ignored (the factory logs a warning naming the plugin), and
• the top-level plugin_dirs: YAML key is ignored too — cpex-core parses it, warns, and discards it.

Both are ignored rather than rejected, so a config carrying either still loads.

This is deliberate. cpex plugin install writes the plugin into <project root>/plugins/ and builds its venv there, so both sides already agree on the location; a config key only creates a way for them to disagree, and the failure mode is an ImportError inside the worker at invoke time — far from the config that caused it. The worker independently restricts plugin dirs to its own ALLOWED_PLUGIN_DIRS, which includes its working directory (inherited from this host), so the resolved directory is importable by construction.

Hooks and payloads

Both hook families work. The cmf. prefix selects the CMF message payload; each legacy name maps to its own typed payload; anything unrecognized falls back to a generic JSON payload.

A plugin error is returned to the executor, which applies the plugin's configured on_error policy (fail / ignore / disable). This host does not interpret that policy itself.

Extensions

The capability-filtered Extensions the executor produced for a plugin is serialized onto the task under an extensions field, so a 3-arg (payload, context, extensions) hook sees out-of-process what its in-process equivalent would. Slot visibility comes from the filtered view — this host does not re-derive capability filtering.

Two rules apply in both directions:

• raw_credentials never rides this channel. Raw tokens travel the capability-gated credential DTO instead (see below).
• Authorization, Cookie, and X-API-Key are stripped from http.request_headers and http.response_headers (spec §3.5), matched case-insensitively.

A plugin's returned extensions come back on the response's modified_extensions field — the field the Python PluginResult model already carries, so returning extensions works the same way in and out of process. The host rebuilds an OwnedExtensions and hands it to the executor's existing copy-on-write merge, which enforces the mutability tiers. The host adds no tier logic of its own; what it does add is two structural guarantees the merge relies on:

Reusing the inbound Arcs is load-bearing: the executor validates the immutable tier with Arc::ptr_eq, and a JSON round trip allocates a fresh Arc for every slot. Deserializing the whole object from the wire would fail that check on every immutable slot and get the entire return rejected, even for a plugin that only touched custom.

Write authority comes from the WriteToken the executor mints per plugin from its declared capabilities and carries on the inbound view. WriteToken::new() is pub(crate) to cpex-core, so this host cannot mint one — a gated write with no token behind it is dropped rather than honored.

Credentials

The token fields on the framework's credential types are #[serde(skip)], so no generic serialization of an Extensions carries token bytes — which would leave identity resolvers and token delegators unable to work out-of-process. This host adds a narrow, capability-gated exception on a separate channel: raw credential material does reach the worker process, deliberately. cpex-core's RawCredentialsExtension module docs state the same boundary from the other side.

A plugin declares what it needs:

    capabilities: [read_inbound_credentials]   # or read_delegated_tokens

Only identity_resolve and token_delegate are eligible — they are the only hooks whose Python payload models a raw token (IdentityPayload.raw_token, DelegationPayload.bearer_token). For a declaring plugin on one of those hooks, the host attaches a dedicated credential object to the task JSON, built by reading the in-memory token directly. Production credential types keep their serde guard and are never serialized; the FFI, Python-bindings, and audit paths are untouched.

Fail closed. A plugin that declared nothing gets no credential field and no token material. A plugin that declared a capability the request cannot honor gets an error — not a silent no-token dispatch, which a resolver could read as "no authentication required".

Worker requirement. The credential field is only consumed by a worker.py that implements reconstruct_credential_payload. An older worker silently drops it. Assert a minimum cpex version for credential-bearing plugins.

Residual exposure

The capability gate controls which plugin receives a token. It does not constrain what happens next: once the plaintext is resident in the worker process, every transitively installed dependency in that venv can read it. That is a materially larger and less audited trust boundary than the in-process host, and neither the gate nor the transport closes it. Sending raw credentials to an out-of-process plugin means accepting that venv's whole dependency tree into the credential trust boundary.

The transport itself is sound by construction: the credential object rides on the worker's stdin, a private pipe inherited only by the child. There is no listening socket, and no other local process can read it.

Venv cache

Reuse is keyed on a SHA256 of the requirements file plus the persisted manifest's version and content hash. Layout mirrors the Python CLI's: .venv and .cpex/venv_cache under <plugin_dir>/<class_root>.

Two deliberate behaviors:

• A missing manifest signal means "no signal", not "mismatch." Metadata written by an older CLI has no manifest_version / manifest_hash key. Treating an absent key as a mismatch would wipe and rebuild every existing venv on the first run after an upgrade.
• Both the manifest and the metadata filename are keyed on the full class name. Plugins sharing a package share one venv directory by design. The Python CLI keys only the manifest per class and derives the metadata filename from the venv directory name, so those plugins share one metadata file and each install invalidates its neighbour's cache — a rebuild loop. Keying both closes it, at the cost of one extra rebuild the first time this host meets a venv the Python CLI created.

Testing

Unit tests run against a stub worker and need only python3:

cargo test -p cpex-hosts-python

The end-to-end tests need a cpex Python source tree, and skip with a printed reason when one is absent rather than failing:

CPEX_PYTHON_SOURCE=/path/to/cpex-python \
  cargo test -p cpex-hosts-python --features testing --test isolated_venv_e2e --test credential_e2e

Two environmental caveats, both upstream of this crate:

• Older Python framework checkouts import McpError while declaring mcp>=1.26, which renamed that symbol to MCPError. The e2e tests downgrade mcp in their pre-built venv only for those older checkouts. Current checkouts use the declared version, and the host reuses the cached venv in both cases.
• The Python hook registry registers no cmf.* hooks, so a CMF round-trip cannot be proven end to end yet. The host's CMF routing is covered by unit tests, and the e2e suite asserts that a CMF hook is rejected cleanly rather than silently allowed.