# CPU Hotspot Profiling Workflow
Maintainer workflow for issue #48 / PRD-0056. Default path is credential-free and offline: `scripts/profile_cpu.py` runs ignored Rust harness tests with a temporary synthetic `MC_HOME`, no `auth.json`, credential-looking environment variables removed from child processes, local SSE fixtures, local session/cache/skill data, and no live model-catalog refresh.
## Requirements
- All platforms: Rust/Cargo, `python3`.
- macOS: built-in `sample` and `spindump`; optional Instruments.
- Linux: `perf`; optional FlameGraph and `hyperfine`.
## Default credential-free run
```sh
python3 scripts/profile_cpu.py --repo . --all --min-seconds 3
```
Artifacts:
```text
target/profiling/issue-48/profile-results.json
target/profiling/issue-48/profile-summary.md
target/profiling/issue-48/<scenario>.log
target/profiling/issue-48/<scenario>.sample.txt
target/profiling/issue-48/idle-tui-pty.sample.txt
```
Run one scenario:
```sh
python3 scripts/profile_cpu.py --repo . --scenario provider_sse_parser --min-seconds 10
cargo test --release profile_cpu_sse_parser -- --ignored --nocapture
```
Self-test:
```sh
python3 scripts/profile_cpu.py --self-test
```
## Mission Control render pipeline
Default render profiling is credential-free and uses Ratatui `TestBackend`; it does not enter raw mode, alternate screen, provider transport, network, or real stdout ANSI output. Baseline results live in [`docs/features/cpu-benchmark-history.md`](cpu-benchmark-history.md#render-pipeline-history).
Run full harness through Python wrapper:
```sh
python3 scripts/profile_cpu.py --repo . --scenario tui_render_pipeline --min-seconds 5
```
Run all render-pipeline scenarios directly:
```sh
cargo test --release profile_tui_render_pipeline -- --ignored --nocapture
```
Run one scenario directly:
```sh
cargo test --release profile_tui_render_pipeline_idle -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_streaming -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_heavy_transcript -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_scroll -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_resize -- --ignored --nocapture
cargo test --release profile_tui_render_pipeline_modal -- --ignored --nocapture
```
Artifact:
```text
target/profiling/issue-48/tui-render-pipeline.json
```
Backend modes are selected by `PROFILE_TUI_RENDER_BACKEND`:
```sh
PROFILE_TUI_RENDER_BACKEND=test cargo test --release profile_tui_render_pipeline -- --ignored --nocapture
PROFILE_TUI_RENDER_BACKEND=crossterm-sink cargo test --release profile_tui_render_pipeline -- --ignored --nocapture
PROFILE_TUI_RENDER_BACKEND=crossterm-real cargo test --release profile_tui_render_pipeline -- --ignored --nocapture
```
| backend mode | purpose | caveat |
| --- | --- | --- |
| `test` | default Ratatui `TestBackend`; measures process-side render and diff cost | records 0 for stdout/write/flush; no terminal emulator paint |
| `crossterm-sink` | measures generated ANSI bytes and crossterm write/flush work against an in-memory sink | still does not measure terminal emulator paint |
| `crossterm-real` | writes to real stdout for manual terminal-path checks | env-gated; use only in disposable terminal; manual real-TTY validation has not been recorded in baseline |
Metrics:
| metric | meaning | interpretation |
| --- | --- | --- |
| `terminal_draw_ms` | end-to-end `Terminal::draw` wall time in default `TestBackend` mode | closest process-side total draw number for baseline tables |
| `controlled_draw_ms` | phase-instrumented manual frame path wall time | use when comparing `render_draw_ms`, `diff_ms`, backend draw, write, and flush phases |
| `render_draw_ms` | Mission Control `render::draw` CPU time | current bottleneck; baseline shows 71-82% of p50 frame time |
| `diff_ms` | Ratatui buffer diff duration | not current bottleneck; baseline stays 0.049-0.111 ms |
| `diff_cells` | changed-cell count emitted by Ratatui diff | high values can raise terminal output, but baseline resize still diffed 7995 cells in 0.111 ms p50 |
| `changed_cell_ratio` | changed cells divided by terminal area | resize/modal first frames should be high; steady idle should be near zero |
| `backend_draw_ms` | backend `draw` call duration inside controlled path | zero/near-zero for `TestBackend`; useful for crossterm modes |
| `stdout_bytes` | ANSI/output bytes generated by crossterm backend | zero for `TestBackend`; terminal I/O ceiling needs crossterm modes |
| `write_calls` / `write_ms` | writer call count and write duration | zero for `TestBackend`; sink/real backends isolate output overhead |
| `flush_calls` / `flush_ms` | flush count and flush duration | process flush time is not terminal emulator paint telemetry |
| `frames_per_sec` | scripted harness throughput | not display refresh rate; compare same machine/profile/backend/area only |
Interpretation rules:
1. Compare same backend, terminal area, Rust profile, scenario mix, and machine.
2. If `render_draw_ms` dominates p50 frame time, optimize Mission Control render/state projection first.
3. If `diff_ms` stays sub-millisecond while `diff_cells` is high, Ratatui diff is not blocking frame throughput.
4. If `stdout_bytes`, `write_ms`, or `flush_ms` rise in crossterm modes, terminal I/O may cap real-world FPS even when `TestBackend` is fast.
5. Treat `TestBackend` results as process-side render ceiling only. Real-world ceiling depends on terminal emulator write/flush/paint behavior.
## Scenarios
| scenario | harness | exercises |
| --- | --- | --- |
| startup_discovery | `profile_cpu_startup_discovery` | `src/config`, `src/instructions`, `src/skills`, `src/sessions`, `src/model_catalog` local cache reads |
| provider_sse_parser | `profile_cpu_sse_parser` | `src/providers/stream.rs::StreamParser` using Responses and Chat Completions SSE fixtures |
| rendering_heavy_transcript | `profile_cpu_rendering_heavy_transcript` | `src/tui/state`, `src/tui/render/transcript.rs`, `src/rendering/markup.rs`, `src/rendering/highlight.rs` |
| tui_render_pipeline | `profile_tui_render_pipeline` | full Mission Control `Terminal::draw`, controlled `render::draw`, Ratatui diff, TestBackend/crossterm byte and flush accounting |
| tui_streaming_simulation | `profile_cpu_tui_streaming_simulation` | assistant deltas, activity events, transcript/activity caches |
| tool_timeout_cleanup | `profile_cpu_tool_timeout_cleanup` | `src/tools/process.rs::terminate_child_tree_and_wait` process-group cleanup |
| idle Mission Control TUI | manual real terminal or non-interactive PTY | static `--tui --no-session` wakeups/CPU while no prompt is submitted |
## macOS sampling
Harness sample:
```sh
PROFILE_CPU_MIN_SECONDS=20 PROFILE_CPU_ITERATIONS=1 \
cargo test --release profile_cpu_rendering_heavy_transcript -- --ignored --nocapture &
cargo_pid=$!
sleep 1
pgrep -P "$cargo_pid" -fl magi_code
test_pid=<pid-from-pgrep>
sample "$test_pid" 10 -mayDie -file target/profiling/issue-48/rendering-heavy-transcript.sample.txt
wait "$cargo_pid"
```
Idle Mission Control TUI real-terminal command:
```sh
MC_HOME="$(mktemp -d)" cargo run --bin magi-code -- --tui --no-session
# In another terminal:
pgrep -fl "magi-code.*--tui"
sample <pid> 30 -file target/profiling/issue-48/idle-tui.sample.txt
spindump <pid> 10 -file target/profiling/issue-48/idle-tui.spindump.txt
```
Idle rules: do not submit a prompt, do not run `/login`, do not configure provider credentials, do not refresh model catalogs, exit with `/quit` after sampling. A non-interactive `/usr/bin/script` PTY baseline is acceptable for issue #48 if it launches `magi-code --tui --no-session`, keeps stdin open, uses isolated `MC_HOME`, captures `sample`, and is labeled as PTY rather than physical terminal evidence.
## Linux perf parity
Linux commands are documented parity unless a Linux host is named in baseline notes:
```sh
PROFILE_CPU_MIN_SECONDS=20 PROFILE_CPU_ITERATIONS=1 \
perf record -F 99 -g -- cargo test --release profile_cpu_tui_streaming_simulation -- --ignored --nocapture
perf report --stdio > target/profiling/issue-48/tui-streaming.perf-report.txt
perf script > target/profiling/issue-48/tui-streaming.perf-script.txt
```
Optional FlameGraph:
```sh
perf script | stackcollapse-perf.pl > target/profiling/issue-48/tui-streaming.folded
flamegraph.pl target/profiling/issue-48/tui-streaming.folded > target/profiling/issue-48/tui-streaming.svg
```
## Issue #48 baseline
Machine baseline from `target/profiling/issue-48/profile-results.json` generated on 2026-05-31T07:32:23Z. macOS stack samples were collected with `/usr/bin/sample` for all synthetic harness scenarios plus idle TUI under a non-interactive `/usr/bin/script` PTY. Linux profiler execution was not run on this Darwin host.
| scenario | date/time | machine/OS/arch | Rust version/profile | command | duration/iterations | primary metric | result | artifact path | notes |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| startup_discovery | 2026-05-31T07:32:23Z | Darwin 25.5.0 arm64 | rustc 1.95.0 release | `cargo test --release profile_cpu_startup_discovery -- --ignored --nocapture` | 3000 ms / 4401 | discovered_items | 563618466 | `target/profiling/issue-48/startup_discovery.log`; `target/profiling/issue-48/startup_discovery.sample.txt` | synthetic settings, sessions, skills, model cache |
| provider_sse_parser | 2026-05-31T07:32:23Z | Darwin 25.5.0 arm64 | rustc 1.95.0 release | `cargo test --release profile_cpu_sse_parser -- --ignored --nocapture` | 3000 ms / 102104 | events_parsed | 1735768 | `target/profiling/issue-48/provider_sse_parser.log`; `target/profiling/issue-48/provider_sse_parser.sample.txt` | local Responses + Chat Completions fixtures |
| rendering_heavy_transcript | 2026-05-31T07:32:23Z | Darwin 25.5.0 arm64 | rustc 1.95.0 release | `cargo test --release profile_cpu_rendering_heavy_transcript -- --ignored --nocapture` | 3001 ms / 263 | rendered_units | 956268 | `target/profiling/issue-48/rendering_heavy_transcript.log`; `target/profiling/issue-48/rendering_heavy_transcript.sample.txt` | markdown/code/diff transcript fixture |
| tui_streaming_simulation | 2026-05-31T07:32:23Z | Darwin 25.5.0 arm64 | rustc 1.95.0 release | `cargo test --release profile_cpu_tui_streaming_simulation -- --ignored --nocapture` | 3000 ms / 1059 | state_units | 277458 | `target/profiling/issue-48/tui_streaming_simulation.log`; `target/profiling/issue-48/tui_streaming_simulation.sample.txt` | synthetic assistant deltas and activity deltas |
| tool_timeout_cleanup | 2026-05-31T07:32:23Z | Darwin 25.5.0 arm64 | rustc 1.95.0 release | `cargo test --release profile_cpu_tool_timeout_cleanup -- --ignored --nocapture` | 3020 ms / 127 | cleanup_units | 254 | `target/profiling/issue-48/tool_timeout_cleanup.log`; `target/profiling/issue-48/tool_timeout_cleanup.sample.txt` | Unix process-group termination path |
| idle Mission Control TUI | 2026-05-31T07:24:36Z | Darwin 25.5.0 arm64 | release binary | `/usr/bin/script -q target/profiling/issue-48/idle-tui-pty.script.txt target/release/magi-code --tui --no-session` + `/usr/bin/sample <pid> 8 -mayDie` | 8 s sample / PTY idle | ps CPU + stack sample | 0.4% CPU before sample, 0.1% CPU after sample; 6832/6853 samples in `kevent` | `target/profiling/issue-48/idle-tui-pty-summary.json`; `target/profiling/issue-48/idle-tui-pty.sample.txt` | isolated temp `MC_HOME`; no prompt submitted; no provider request; no `auth.json` created; PTY baseline, not physical terminal |
## Hotspot report
| scenario | source file | function/symbol | metric type | metric value | profiler artifact | impact | confidence | follow-up priority |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| idle Mission Control TUI | `src/tui/mod.rs` + crossterm event path | `MissionControlApp::run` / `crossterm::event::poll` / `kevent` | macOS `sample` stack + `ps` CPU | 6832/6853 samples blocked in `kevent`; 0.4% -> 0.1% CPU around 8 s sample | `target/profiling/issue-48/idle-tui-pty.sample.txt`; `target/profiling/issue-48/idle-tui-pty-summary.json` | idle loop appears mostly blocked on terminal events under PTY; low idle CPU on this machine | High for PTY evidence; physical-terminal wakeups still optional confirmation | P2 |
| rendering_heavy_transcript | `src/tui/render/transcript.rs` | `cached_transcript_lines` / `append_transcript_entry_lines` | macOS `sample` stack | 3225 samples in `MissionControlState::cached_transcript_lines`; 3191 in `cached_transcript_lines`; 3163 in `append_transcript_entry_lines` | `target/profiling/issue-48/rendering_heavy_transcript.sample.txt` | heavy transcript redraw cost; markdown projection repeated across many assistant turns | High; stack sample plus harness throughput | P1 |
| rendering_heavy_transcript | `src/rendering/markup.rs` + `src/rendering/highlight.rs` | `render_markdown` / `highlight_code` | macOS `sample` stack | 3055 samples in `render_markdown`; 2391 samples in `highlight_code` | `target/profiling/issue-48/rendering_heavy_transcript.sample.txt` | code fence highlighting and Markdown parsing dominate long transcript rendering | High | P1 |
| tui_streaming_simulation | `src/tui/state/mod.rs` | `apply_activity_event` / `cached_visible_nodes` / `collect_visible` | macOS `sample` stack | 1164 samples in `apply_activity_event`; 1137 in `cached_visible_nodes`; 447 in `collect_visible` | `target/profiling/issue-48/tui_streaming_simulation.sample.txt` | many small activity deltas clone and rebuild visible activity state | High | P1 |
| provider_sse_parser | `src/providers/stream.rs` | `StreamParser::push_chunk_outcome` / `serde_json::de::from_trait` | macOS `sample` stack | 376 samples in `push_chunk_outcome`; 369 in serde JSON parse beneath it | `target/profiling/issue-48/provider_sse_parser.sample.txt` | parser overhead is real but smaller than rendering/state hotspots in synthetic run | High | P2 |
| startup_discovery | `src/sessions/mod.rs` + `src/skills/mod.rs` | `SessionManager::list` / `discover_skills` / `discover_agents` | macOS `sample` stack | 5750 samples in `SessionManager::list`; 303 + 301 in `discover_skills`; 83 + 79 in `discover_agents` | `target/profiling/issue-48/startup_discovery.sample.txt` | startup scales with local session count first, then skill/instruction discovery | High | P2 |
| tool_timeout_cleanup | `src/tools/process.rs` | `terminate_child_tree_and_wait` / `poll_process_group_exit` / `signal_process_group` | macOS `sample` stack | 4525 samples in `terminate_child_tree_and_wait`; 3772 in `poll_process_group_exit`; 693 in `signal_process_group` | `target/profiling/issue-48/tool_timeout_cleanup.sample.txt` | timeout cleanup cost dominated by sleep/process polling and subprocess probes | High | P3 |
## CPU-reduction priority order
1. P1: reduce repeated transcript markdown/render projection in heavy transcript and streaming paths. Expected user impact: smoother long-answer TUI and lower redraw CPU.
2. P1: reduce activity visible-node clone/rebuild churn in streaming paths. Expected user impact: lower CPU during tool-heavy runs.
3. P2: inspect startup session listing cost when many JSONL files exist; keep session semantics unchanged.
4. P2: keep idle loop under observation with optional physical-terminal wakeup sample; PTY evidence shows low CPU and no P0 idle blocker.
5. P3: process cleanup polling changes are riskier because they touch safety semantics; optimize only with process-tree regression coverage.
## Before/after comparison method
1. Use same machine, same branch base, same Rust profile, same terminal size for TUI checks.
2. Run:
```sh
python3 scripts/profile_cpu.py --repo . --all --min-seconds 10
cargo test --release profile_cpu -- --ignored --nocapture
```
3. For changed area, add platform profiler:
```sh
sample <pid> 10 -file target/profiling/issue-48/<scenario>.sample.txt
# or Linux:
perf record -F 99 -g -- cargo test --release <scenario> -- --ignored --nocapture
```
4. Compare baseline table fields: `scenario`, `date/time`, `machine/OS/arch`, `Rust version/profile`, `command`, `duration/iterations`, `primary metric`, `result`, `artifact path`, `notes`.
5. Compare hotspot table fields: `scenario`, `source file`, `function/symbol`, `metric type`, `metric value`, `profiler artifact`, `impact`, `confidence`, `follow-up priority`.
6. Report delta as `(after - before) / before`, plus whether profiler top symbols moved away from target function.
## Optional live-provider profiling
Live provider profiling is non-default. Skip it unless credentials and network are intentionally available. Do not store provider captures, bearer-style headers, provider secret strings, account identifiers, authorization codes, or secret payloads in fixtures or artifacts.