# CPU profiling
`scripts/profile_cpu.py` runs ignored Rust profiling workloads with temporary synthetic settings and local fixtures. It removes credential-looking environment variables from child processes and does not refresh live model catalogs.
## Requirements
- All platforms: Rust/Cargo, `python3`.
- macOS: built-in `sample` and `spindump`; optional Instruments.
- Linux: `perf`; optional FlameGraph and `hyperfine`.
## Default credential-free run
```sh
python3 scripts/profile_cpu.py --repo . --all --min-seconds 3
```
Automatic wrapper artifacts:
```text
target/profiling/issue-48/profile-results.json
target/profiling/issue-48/profile-summary.md
target/profiling/issue-48/<scenario>.log
target/profiling/issue-48/tui-render-pipeline.json
```
The wrapper does not collect `.sample.txt` files or idle PTY samples. Collect those separately using the manual [macOS sampling](#macos-sampling) workflow.
Run one scenario:
```sh
python3 scripts/profile_cpu.py --repo . --scenario provider_sse_parser --min-seconds 10
cargo test --release profile_cpu_sse_parser -- --ignored --nocapture
```
Self-test:
```sh
python3 scripts/profile_cpu.py --self-test
```
## Mission Control render pipeline
Default render profiling is credential-free and uses Ratatui `TestBackend`; it does not enter raw mode, alternate screen, provider transport, network, or real stdout ANSI output. Historical baseline results live in [`docs/archive/cpu-benchmark-history.md`](../archive/cpu-benchmark-history.md#render-pipeline-history).
Run full harness through Python wrapper:
```sh
python3 scripts/profile_cpu.py --repo . --scenario tui_render_pipeline --min-seconds 5
```
Run all render-pipeline scenarios once through the exact aggregate test (a partial filter also selects six individual tests):
```sh
cargo test --release --lib profiling::profile_harness::render::profile_tui_render_pipeline -- --exact --ignored --nocapture
```
Run one scenario directly:
```sh
cargo test --release profile_tui_render_pipeline_idle -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_streaming -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_heavy_transcript -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_scroll -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_resize -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_modal -- --ignored --nocapture
```
Direct Cargo runs print results to stdout; they do not write a JSON artifact. The Python wrapper writes:
```text
target/profiling/issue-48/tui-render-pipeline.json
```
Backend modes are selected by `PROFILE_TUI_RENDER_BACKEND`:
```sh
PROFILE_TUI_RENDER_BACKEND=test cargo test --release --lib profiling::profile_harness::render::profile_tui_render_pipeline -- --exact --ignored --nocapture
PROFILE_TUI_RENDER_BACKEND=crossterm-sink cargo test --release --lib profiling::profile_harness::render::profile_tui_render_pipeline -- --exact --ignored --nocapture
PROFILE_TUI_RENDER_BACKEND=crossterm-real cargo test --release --lib profiling::profile_harness::render::profile_tui_render_pipeline -- --exact --ignored --nocapture
```
| backend mode | purpose | caveat |
| --- | --- | --- |
| `test` | default Ratatui `TestBackend`; measures process-side render and diff cost | records 0 for stdout/write/flush; no terminal emulator paint |
| `crossterm-sink` | measures generated ANSI bytes and crossterm write/flush work against an in-memory sink | still does not measure terminal emulator paint |
| `crossterm-real` | writes to real stdout for manual terminal-path checks | env-gated; use only in disposable terminal; manual real-TTY validation has not been recorded in baseline |
Metrics:
| metric | meaning | interpretation |
| --- | --- | --- |
| `terminal_draw_ms` | end-to-end `Terminal::draw` wall time in default `TestBackend` mode | closest process-side total draw number for baseline tables |
| `controlled_draw_ms` | phase-instrumented manual frame path wall time | use when comparing `render_draw_ms`, `diff_ms`, backend draw, write, and flush phases |
| `render_draw_ms` | Mission Control `render::draw` CPU time | historical baseline bottleneck: 71-82% of p50 frame time; rerun to assess current cost |
| `diff_ms` | Ratatui buffer diff duration | not the historical baseline bottleneck: 0.049-0.111 ms |
| `diff_cells` | changed-cell count emitted by Ratatui diff | high values can raise terminal output, but baseline resize still diffed 7995 cells in 0.111 ms p50 |
| `changed_cell_ratio` | changed cells divided by terminal area | resize/modal first frames should be high; steady idle should be near zero |
| `backend_draw_ms` | backend `draw` call duration inside controlled path | zero/near-zero for `TestBackend`; useful for crossterm modes |
| `stdout_bytes` | ANSI/output bytes generated by crossterm backend | zero for `TestBackend`; terminal I/O ceiling needs crossterm modes |
| `write_calls` / `write_ms` | writer call count and write duration | zero for `TestBackend`; sink/real backends isolate output overhead |
| `flush_calls` / `flush_ms` | flush count and flush duration | process flush time is not terminal emulator paint telemetry |
| `frames_per_sec` | scripted harness throughput | not display refresh rate; compare same machine/profile/backend/area only |
Interpretation rules:
1. Compare same backend, terminal area, Rust profile, scenario mix, and machine.
2. If `render_draw_ms` dominates p50 frame time, optimize Mission Control render/state projection first.
3. If `diff_ms` stays sub-millisecond while `diff_cells` is high, Ratatui diff is not blocking frame throughput.
4. If `stdout_bytes`, `write_ms`, or `flush_ms` rise in crossterm modes, terminal I/O may cap real-world FPS even when `TestBackend` is fast.
5. Treat `TestBackend` results as process-side render ceiling only. Real-world ceiling depends on terminal emulator write/flush/paint behavior.
## Typing while output is queued
```sh
cargo test --release --locked --lib profile_typing_under_streaming_load \
-- --ignored --nocapture --test-threads=1
```
This offline diagnostic uses existing controller drain, input-classification, input-reduction, and draw components with `TestBackend` at 120 × 36. It compares idle typing, a 1,024-event streaming queue, and the same queue with 120 prior synthetic turns. Each scenario excludes 10 warmup samples and emits 100 raw rows as `typing_under_load` JSON. Setup and assertions stay outside timing; assertions verify the edited draft appears in the drawn prompt while streaming output remains queued.
Columns: `drain_ms`, `edit_to_draw_ms`, `drain_then_edit_to_draw_ms`, `events_drained`, `backlog_after_draw`. Edit timing starts immediately after the drain, before applying its scheduling state and classifying input. The draft resets between samples; the streaming response grows throughout the run. These are correlated component-workflow observations, not independent whole-session trials.
The diagnostic scripts component order, not the complete event loop. It excludes input capture/queue wait, timer pacing, concurrent producer contention, terminal writes and physical display. No latency threshold gates ordinary tests. Repeat separate invocations when comparing changes; retain raw rows and machine/build context.
Initial measurements and interpretation limits: [interaction baseline](../archive/interaction-baseline-2026-09-27.md).
## Real application PTY workflows
```sh
cargo build --release --locked --bin magi-code
python3 scripts/profile_interaction.py --cycles 60 --idle-seconds 10 \
--out-dir target/profiling/pty-interaction/new-run
```
`profile_interaction.py` launches a private snapshot of the actual CLI at 160 × 48 cells with disposable working directory, `HOME`, and `MC_HOME`. The report hashes the executed copy rather than a shared build path. An allowlisted environment and nonforwarding loopback proxy keep configured provider traffic local without supplying credentials. The scripted provider exercises text replies, shell-tool continuation, streaming and process-tree cancellation, and recovery prompts. This is not an OS network sandbox.
Reports include draft-write-to-output timing, submission/request/output boundaries, cancellation feedback and observed cleanup, startup model-label observation, application RSS, idle CPU/output, and shutdown checks. Application CPU and RSS exclude the observer and tool children. Separate observer-thread CPU counters cover turns and idle windows; they are not disjoint wall-time stages. Plain content with delayed completion is separate from streaming content prefixed by `</think>`, matching the current Chat Completions reasoning parser.
Limits: Unix PTYs and `ps`, verified on macOS only; no new dependencies. `--cycles` accepts 1-200 (default 20), `--idle-seconds` accepts 1-300 (default 10). A unique output directory is generated when omitted; explicitly named directories must not exist. Each cancellation wait and process fixture is bounded. Fixture cleanup never signals reported callback PIDs.
Require command exit 0 and `valid: true` in `results.json`. Missing workflow evidence, unexpected process exit, fixture failures, surviving reported helpers, forced shutdown, or failed terminal restoration invalidate the run. Raw failure output is capped at 2 MiB. The fixed-grid ANSI observer waits for complete synchronized updates but approximates Unicode widths: these are reconstructed-output observations, not physical pixels or terminal-compatibility results. Small draft/cancellation sample counts and bounded turn loops do not establish tail latency or leak freedom.
Optional diagnostics on macOS:
```sh
python3 scripts/profile_interaction.py --cycles 60 --idle-seconds 10 --diagnostics \
--out-dir target/profiling/pty-interaction/new-profiled
```
`--diagnostics` adds `application.sample.txt` (10 ms sampling interval, 120-second maximum), `observer.prof` (main Python thread only), and bounded raw/screen captures for both idle windows. Captures record truncation; native sample files can contain local paths. A failed or missing native sample invalidates the run. Sampling starts after initial startup and may cover only a prefix of long workloads. Profiles perturb timing: keep diagnostic runs separate from ordinary baselines, and do not sum nested cumulative profile times. Open the observer profile with `python3 -m pstats <run>/observer.prof`.
Add `--reduced-motion` to disable animation in the disposable fixture only. Compare both orders when attributing output to animation; this does not change user settings or measure physical-terminal cost.
For observer-side delay attribution, add `--trace-timing` to ordinary or diagnostic runs. It records monotonic select/read/processing timestamps, processing-thread CPU, byte counts, input-write completion and first literal marker bytes. Coverage includes initial scheduled cycles plus lifecycle turns, `/new`, compaction and session switching. Each trace ends when reconstructed output confirms its marker; pacing, idle, cancellation and resource counters stay outside. Differential ANSI output can split a marker with escapes, so absent literal bytes do not mean absent feedback. Select waits include scheduling; processing wall time can include descheduling or cursor-query writes. These traces alone cannot distinguish application waiting, kernel delivery and host contention.
Trace storage is capped at 20,000 pump rows across the run; truncation or missing operation traces invalidate the run. Keep traced and untraced repetitions separate and compare both orders.
On macOS, add `--stall-threshold-ms 250 --trace-timing` for concurrent application and observer native samples when a traced operation crosses the threshold. Accepted threshold: 25-5000 ms. A watchdog checks every 10 ms and starts at most four capture pairs per run, each requesting one second at 1 ms intervals. Both samplers start before either is awaited; samples and logs use `stall-N-{application,observer}` filenames. Reports retain operation, trigger, launch and reap timestamps, target PIDs, commands, exit codes and file sizes. Sampler failures invalidate the run; child waits share a 15-second deadline, followed by kill/reap cleanup. This mode cannot combine with continuous `--diagnostics` sampling.
Native `sample` output is an aggregate, not timestamped stacks. Launch time precedes attachment/collection; sampling may begin after a short stall ends and extend into later work. The watchdog shares Python's process and can itself be delayed by scheduling or the interpreter lock. Captures pause no application thread intentionally, but attachment and sampling perturb timing. Keep low-threshold capture-validation runs separate from natural-stall evidence. A run without captures does not establish absence of stalls beyond traced operations or the four-pair limit.
Optional paced session lifecycle/resource measurement on macOS:
```sh
python3 scripts/profile_interaction.py --cycles 20 --idle-seconds 10 \
--lifecycle-rounds 6 --lifecycle-pause 20 \
--out-dir target/profiling/pty-interaction/new-lifecycle
```
`--lifecycle-rounds` accepts 0-12 (default 0). After the ordinary workflows, it creates a second session and repeats 20 turns, `/compact`, a continuation prompt, session-picker switching, and another prompt. It verifies durable inputs/replies in the selected session, the compaction checkpoint, and summary use in subsequent requests. Summaries are synthetic: correctness checks do not assess semantic quality or archival recovery.
`--lifecycle-pause` accepts 1-60 seconds (default 30) after each compaction and switch. Lifecycle mode also waits one second before provider submissions, outside latency timing; it does not test rapid resubmission. macOS `ps -M` and `lsof` provide thread and numeric file-descriptor counts at workflow boundaries and approximately every five seconds during these pauses. Each counter command has a five-second timeout. Counts are sequential, application-only samples and can miss brief resources; mapped-file and working-directory entries are not numeric descriptors. Active JSONL byte counts exclude archives. Synthetic session inspection is capped at 32 MiB per active file.
Measured results and limitations: [PTY interaction baseline](../archive/pty-interaction-baseline-2026-09-27.md), [paired profiling and 200-turn follow-up](../archive/pty-profiling-followup-2026-09-27.md), [eight-minute session lifecycle runs](../archive/session-lifecycle-resources-2026-09-27.md), [first-display/resize, observer tracing and 19.5-minute investigation](../archive/performance-investigation-2026-09-27.md).
## Large-response and text-modal comparisons
These ignored, credential-free workflows compare current render paths with local reference algorithms, not historical binaries or terminal-emulator paint:
```sh
cargo test --release --locked --lib streaming_viewport_large_response_workload -- --ignored --nocapture
cargo test --release --locked --lib text_modal_large_document_workload -- --ignored --nocapture
```
Streaming uses independent states with identical synthetic events and alternates measurement order. It reports initial indexing separately, then medians across 30 appends. `rebuild` disables only completed-prefix reuse; `full_reference` materializes and measures every line. Text-modal results include `Terminal::draw` with `TestBackend`, across 21 top/middle/end draws.
For repeated first-display and resize observations:
```sh
cargo test --release --locked --lib profile_large_response_first_display_and_resize \
-- --ignored --nocapture --test-threads=1
```
The diagnostic rotates three fixture types across ten fresh display states each. For every state it measures widths `80, 80, 120, 120, 40, 80` at 24 rows: first projection, cached projection, two newly encountered widths, and return to retained width. Ingestion, assertions and result cleanup are outside timing. It verifies visible content survives resize and returns unchanged at width 80. These are projection costs, not complete frame/input latency; no performance threshold gates tests. Repeat fresh process invocations and report their medians separately from single samples.
Earlier viewport measurements, cold-index costs, and limitations are preserved in the [CPU observations](../archive/cpu-hotspot-observations.md#large-response-and-text-modal-comparisons). Compare cold and warm paths separately; faster repeated projection does not imply faster first display.
## Scenarios
| scenario | harness | exercises |
| --- | --- | --- |
| startup_discovery | `profile_cpu_startup_discovery` | `src/config`, `src/instructions`, `src/skills`, `src/sessions`, `src/model_catalog` local cache reads |
| provider_sse_parser | `profile_cpu_sse_parser` | `src/providers/stream.rs::StreamParser` using Responses and Chat Completions SSE fixtures |
| rendering_heavy_transcript | `profile_cpu_rendering_heavy_transcript` | `src/tui/state`, `src/tui/render/transcript.rs`, `src/rendering/markup.rs`, `src/rendering/highlight.rs` |
| tui_render_pipeline | `profile_tui_render_pipeline` | full Mission Control `Terminal::draw`, controlled `render::draw`, Ratatui diff, TestBackend/crossterm byte and flush accounting |
| tui_streaming_simulation | `profile_cpu_tui_streaming_simulation` | assistant deltas, activity events, transcript/activity caches |
| tool_timeout_cleanup | `profile_cpu_tool_timeout_cleanup` | `src/tools/process.rs::terminate_child_tree_and_wait` process-group cleanup |
| idle Mission Control TUI | manual real terminal or non-interactive PTY | static `--no-session` wakeups/CPU while no prompt is submitted |
## Skill discovery benchmark
Run `PROFILE_CPU_ITERATIONS=25 cargo test --release profile_cpu_discover_skills -- --ignored --nocapture` to measure synthetic skill discovery. [Earlier results](../archive/cpu-hotspot-observations.md#issue-381-skill-discovery-benchmark) include fixture details and measurement limits.
## macOS sampling
Harness sample:
```sh
PROFILE_CPU_MIN_SECONDS=20 PROFILE_CPU_ITERATIONS=1 \
cargo test --release profile_cpu_rendering_heavy_transcript -- --ignored --nocapture &
cargo_pid=$!
sleep 1
pgrep -P "$cargo_pid" -fl magi_code
test_pid=<pid-from-pgrep>
sample "$test_pid" 10 -mayDie -file target/profiling/issue-48/rendering-heavy-transcript.sample.txt
wait "$cargo_pid"
```
Idle Mission Control TUI real-terminal command:
```sh
MC_HOME="$(mktemp -d)" cargo run --bin magi-code -- --no-session
# In another terminal:
pgrep -fl "magi-code.*--no-session"
sample <pid> 30 -file target/profiling/issue-48/idle-tui.sample.txt
spindump <pid> 10 -file target/profiling/issue-48/idle-tui.spindump.txt
```
Idle rules: do not submit a prompt, do not run `/login`, do not configure provider credentials, do not refresh model catalogs, exit with `/quit` after sampling. A non-interactive `/usr/bin/script` PTY baseline is acceptable for issue #48 if it launches `magi-code --no-session`, keeps stdin open, uses isolated `MC_HOME`, captures `sample`, and is labeled as PTY rather than physical terminal evidence. Recorded baselines below retain the launch flags used at the time.
## Linux perf parity
The Linux commands provide an equivalent workflow; they have not been run unless a Linux host is named in the baseline notes:
```sh
PROFILE_CPU_MIN_SECONDS=20 PROFILE_CPU_ITERATIONS=1 \
perf record -F 99 -g -- cargo test --release profile_cpu_tui_streaming_simulation -- --ignored --nocapture
perf report --stdio > target/profiling/issue-48/tui-streaming.perf-report.txt
perf script > target/profiling/issue-48/tui-streaming.perf-script.txt
```
Optional FlameGraph:
```sh
perf script | stackcollapse-perf.pl > target/profiling/issue-48/tui-streaming.folded
flamegraph.pl target/profiling/issue-48/tui-streaming.folded > target/profiling/issue-48/tui-streaming.svg
```
## Earlier measurements
The [CPU observations](../archive/cpu-hotspot-observations.md#issue-48-historical-baseline) preserve the original baseline, hotspot report, and priorities. Those findings describe the recorded run, not current bottlenecks.
## Before/after comparison method
1. Use the same machine, branch base, and Rust profile, and the same terminal size for TUI checks.
2. Run:
```sh
python3 scripts/profile_cpu.py --repo . --all --min-seconds 10
cargo test --release profile_cpu -- --ignored --nocapture
```
3. Profile the changed area with a platform profiler:
```sh
sample <pid> 10 -file target/profiling/issue-48/<scenario>.sample.txt
# or Linux:
perf record -F 99 -g -- cargo test --release <scenario> -- --ignored --nocapture
```
4. Compare baseline table fields: `scenario`, `date/time`, `machine/OS/arch`, `Rust version/profile`, `command`, `duration/iterations`, `primary metric`, `result`, `artifact path`, `notes`.
5. Compare hotspot table fields: `scenario`, `source file`, `function/symbol`, `metric type`, `metric value`, `profiler artifact`, `impact`, `confidence`, `follow-up priority`.
6. Report delta as `(after - before) / before`, plus whether profiler top symbols moved away from target function.
## Optional live-provider profiling
Live provider profiling is not part of the default workflow. Skip it unless credentials and network are intentionally available. Do not store provider captures, bearer-style headers, provider secret strings, account identifiers, authorization codes, or secret payloads in fixtures or artifacts.