# Rendering
A UI plugin paints *surfaces*: regions the viewer places (a sidebar card, a
pane footer, the stage) and sizes. Each surface is in one of two models,
fixed by the manifest:
| `cells` | a terminal cell | a grapheme, foreground, background, attributes | `Surface<Cells>` |
| `pixels` | a pixel | RGBA8 (straight alpha) | `Surface<Pixels>` |
Examples: [`clock_card.rs`](../examples/clock_card.rs)
(cells) and [`shapes_stage.rs`](../examples/shapes_stage.rs)
(pixels and the shape helpers).
## The shared buffer
A surface's pixels or cells live in the plugin's own linear memory. The
viewer reads them directly; nothing is serialized per frame.
```text
offset 0 header, 8 little-endian u32 words:
seq, model (0 cells, 1 pixels), cols, rows,
px-w, px-h, cell-px-w, cell-px-h
offset 32 slot 0
offset 32+slot-len slot 1
```
- A cell is 16 bytes: `grapheme: u32` (0 empty, else a Unicode scalar),
`fg: u32`, `bg: u32`, `attrs: u16` (bold 1, dim 2, italic 4, underline 8,
reverse 16, strikethrough 32), two bytes of padding.
- A colour word's top byte is its model: `0x00______` default (theme text,
or a transparent background), `0x01RRGGBB` RGB, `0x020000II` terminal
palette index, `0x030000TT` theme slot (0 fg, 1 bg, 2 recede-fg,
3 recede-bg, 4 accent). The SDK's `Colour` encodes these; prefer theme
slots (`Colour::FG`, `Colour::ACCENT`, `Colour::RECEDE_FG`, ...) so the
plugin follows the user's terminal colours.
- A pixel is 4 bytes, `r, g, b, a`, rows top to bottom;
`px-w = cols * cell-px-w`.
The buffer is double-buffered. The plugin paints one slot and commits it
with the rectangles that changed; the viewer samples only committed slots,
only the changed rectangles, and only when a frame is due and the surface
is visible. A half-painted slot is never shown.
Successful `commit()` publishes the buffer; it does not acknowledge that a
viewer has already displayed it.
The viewer fills a pane header or footer's reserved space with the pane's
solid background before its first commit and beyond a smaller retained
frame during resize. Plugins do not need to commit a blank frame to hide
the workspace texture.
The SDK does all of this for you:
- `Surface::new(id)` asks the host for the layout, allocates the region,
writes the header and attaches it.
- Every drawing call records the rectangle it touched. `commit()` sends the
merged rectangles (collapsed to their bounding box past 16), bumps the
header's sequence number, and switches slots. With nothing drawn,
`commit()` returns `Ok(false)` and makes no host call: no frame.
- The next slot starts as a copy of what was just committed, copied lazily
and only where it changed, so drawing is always incremental: change one
cell, commit one cell.
- To repaint everything (an animation), skip that copy: `Surface<Pixels>::
repaint_all()` hands out the whole slot with stale contents and marks it
all dirty, and `clear()` does the same for either model.
## Cells
```text
put(x, y, char, style) -> columns text(x, y, &str, style) -> columns
fill(rect, char, style) clear() / clear_with(style)
cell(x, y) -> Option<Cell> row_text(y)
```
`text` walks the string by grapheme cluster and writes each cluster's first
character: a cell holds one scalar (the contract reserves a grapheme pool
for longer clusters; it is not implemented), so combining marks and
variation selectors are dropped. Widths follow the viewer's rule, the
scalar's East Asian width:
- A wide character takes two cells: the character, then an empty cell.
One that would not fit before the right edge leaves one empty cell
instead of half a glyph.
- Overwriting half of a wide character empties the other half and marks
it dirty, so the viewer never shows a stale half.
- Control, bidi-control and zero-width characters are never written.
`Style` carries `fg`, `bg` and `attrs`: `Style::fg(Colour::ACCENT).bold()`.
Glyphs: write Nerd Font icons at their standard codepoints. The viewer
installs its own symbols font (Standard Code Symbols, drawn from Symbols
Nerd Font) and moves Nerd codepoints in plugin cells to that font's own
copies, so plugin icons match the interface's in every terminal that can
reach the font. Where one cannot (Warp, which draws only its bundled fonts),
the viewer shows Unicode symbols instead. Write Legacy Computing characters
(U+1FB00 to U+1FBFF) at their real codepoints too: in a terminal that
cannot draw one, the native viewer draws its fallback in that cell (the
checkerboard U+1FB95 becomes the closest pattern that terminal can draw).
Ship no font and choose no codepoint by terminal.
Put a space cell after every Nerd icon: write `"\u{f062} 2"`, not
`"\u{f062}2"`, and separate two icons with a space. Ghostty and kitty draw a
private-use glyph at its full size only when a plain space (not a no-break
space) follows it. Before text or another icon, Ghostty shrinks it to one
cell, so the same icon shows at two sizes. The rule holds at a surface's
right edge too: the cell past the last column belongs to the host's frame,
so an icon there has no space after it. Keep the last column free of an
icon. The interface follows the same rule.
Colours follow the viewer. `Colour::Theme(slot)` (`FG`, `BG`, `RECEDE_FG`,
`RECEDE_BG`, `ACCENT`) and `Colour::Indexed(i)` resolve to the viewer's
theme and terminal palette when it paints; `view::theme()` reports them as
RGB, `palette` included (the 16 ANSI colours as this viewer shows them),
for plugins that blend colours themselves. "Grayed out" means receded
toward the viewer's background, never a fixed gray:
`theme.recede(rgb, GRAYED_OUT_PERCENT)` (or `recede(rgb, toward, percent)`)
blends as the viewer does, keeping the colour's hue on light, dark and
tinted themes. `view::surface_tint(id)` is the identity tint of what an
instance surface belongs to (a machine row's machine, a project row's
project, a pane footer's project, else its machine) and `view::identity_tint(id)` any machine's or
project's, so a row can carry the same colour the sidebar gives its owner.
Motion follows the viewer too. `view::reduced_motion()` is true while the
user asked for reduced motion: a plugin's own transitions then reach their
end state in the next frame and its ambient motion (idle drift, sparkle,
looping decoration) does not run; a progress spinner may keep turning.
`Event::ThemeChanged` comes when it changes. The host's own motion (a
column sliding open, the stage) obeys the setting without the plugin's
help.
The host carries a surface with its containing list or pane while it
scrolls. A plugin paints in surface coordinates during that motion.
Scrolling alone does not require a repaint.
## Pixels
```text
set_pixel(x, y, rgba) blend_pixel(x, y, rgba) pixel(x, y)
clear(rgba) blit(x, y, w, h, &rgba8)
blit_clipped(x, y, Image::new(w, h, &rgba8), clip, blend)
pixels_mut() pixels_untracked() + mark_dirty(rect)
repaint_all()
```
`blit` copies an RGBA8 image, clipped to the surface. `blit_clipped` also
clips to an `Area` (a sprite drawn into a viewport, a scrolling strip) and,
with `blend`, draws each pixel over what is there with its alpha; only the
pixels it writes are marked dirty.
`pixels_mut()` gives the whole slot (holding the last frame) and marks it
all dirty; for small changes use the drawing calls, or write through
`pixels_untracked()` and `mark_dirty` what you changed.
Each commit's dirty area is what the viewer sends to the terminal, so an
animation's cost follows the area it changes every frame, not the size of
the surface. Something tall that moves sideways by a pixel dirties its full
height at both edges. Some terminals pay for each separate image the viewer
sends as well as for its pixels: iTerm2 went from choppy to smooth at
60 fps when full-height moving shapes were sent as fewer, taller images.
Mark only what changed, and prefer motion that changes a small area.
Whether pixels reach the terminal as graphics depends on the viewer:
`capabilities::get().graphics` is true when it paints them as real images
(the kitty graphics protocol, `kitty`); without it the viewer shows a
pixels surface as half-block cells at one by two pixels per cell, so a
pixel plugin still works everywhere, at lower resolution. Where the
viewer's own menus or another surface's window, popover or tooltip cover part of
your surface, a viewer that paints images shows the rest of the image
around them; a dialog that recedes the whole screen hides it until the
dialog closes. The surface keeps its size, and nothing changes for the
plugin.
`capabilities::get()` also reports the paced `frame_rate` and the cell
pixel size; `Event::CapabilitiesChanged` says when any of them changed.
A pixels surface can host an embedded program that speaks pixel's
embedded host transport, such as terminal-browser:
`surface.bind_embed()`. The viewer opens a private socket and a private
terminal for it and sends `EmbedEvent::Ready` (`standard_plugin::embed`)
with both paths. Start the program from your companion with `PIXEL_EMBED`
set to the socket and `PIXEL_TTY` to the terminal. The viewer then gives
the program the surface's size, the theme's colours, focus, visibility,
keys, pointer, wheel ticks and pastes, and draws the frames it sends; none
of it passes through the UI plugin. `EmbedEvent::Title` names the
program's content, `EmbedEvent::Clipboard` reports text it copied (the
viewer has already put it on the clipboard), and `EmbedEvent::Closed` says
it left. `unbind_embed()` ends the connection. The web app answers
`unavailable`.
When an agent works in your surface (an agent driving an embedded page,
say), show where it points and clicks with
`surface.set_agent_pointer(Some(AgentPointer { x_px, y_px, press }))` (or
`cx.set_agent_pointer(id, ...)`). The place is in device pixels from the
surface's top left: a pixels surface's own pixels, a cells surface's at
the viewer's cell pixel size. The viewer draws its own agent cursor there,
an arrow with a name tag in Standard's agent colour, over the surface and
clipped to it. Each new place glides from the last one (it jumps under
reduced motion), and `press: true` plays a click once the pointer gets
there. `None` removes the pointer. A place outside the surface is
`Invalid`; while the viewer has no cell pixel size, and in the web app, the
call is `Unavailable`.
## Shapes
`Shapes` is implemented by both surface types, with an `Ink` of `Style`
for cells and `Rgba` for pixels:
| `stroke_rect`, `fill_rect` | box-drawing border; full blocks in the style's foreground | one-pixel border; filled (blended when not opaque) |
| `line` | `─ │ ╲ ╱` along a Bresenham line | antialiased (Wu) |
| `bar(area, fraction, fill, track)` | full blocks, one eighth block, `░` track | filled with a coverage edge |
| `rounded_rect`, `fill_rounded_rect` | `╭╮╰╯` corners; quarter blocks when filled | antialiased corners of the radius |
| `circle`, `fill_circle` | `•` outline or full blocks; the radius counts rows and spans twice as many columns | antialiased |
| `fill_rect_at(x, y, w, h, colour)` (pixels only) | | a rectangle at fractional coordinates, edges antialiased |
| `label(x, y, text, ink)` | grapheme text, clipped on both sides | the built-in 5x7 ASCII font; `label_scaled` for integer sizes |
Coordinates are `i32` and may lie outside the surface; sizes may exceed
it. Circles take `f32` centres and radii in continuous coordinates (the
unit at `(x, y)` spans `x..x + 1`), so a moving ball sits between pixels
and antialiases there; cells round to the nearest cell. Everything is clipped, work is bounded by the visible part, and only
what is drawn is marked dirty.
The 5x7 font `label` draws on pixels is public as `standard_plugin::font`:
`font::glyph(c)` is a character's seven rows of five bits (top row first,
the leftmost column in bit 4), `None` outside printable ASCII;
`font::lit(&rows, x, y)` tests one pixel; `GLYPH_WIDTH`, `GLYPH_HEIGHT`
and `GLYPH_ADVANCE` size it. Draw the same letters in cells, as a mask, or
at any scale with them.
## Resize
When the viewer resizes a surface it sends `Event::Resize`. Before the
plugin sees the event the SDK has already asked for the new layout,
allocated a new zeroed region and attached it. The next commit is the
first at the new size and is sent fully dirty; the old region stays
allocated until that commit lands, because the viewer keeps showing it
until then. Repaint everything after a resize (the new buffer is empty):
the examples reset their "what I last painted" state in `event`.
## Scrolling
A cells surface can be taller than where it shows. Call
`surface.set_scroll(Some(Scroll::rows(n).fixed(top, bottom)))` (or
`cx.set_scroll(id, ...)`) with the rows your content needs, and paint the
whole surface as usual. The viewer sends `Event::Resize` with `n` rows (or
the rows the surface shows, when more) and scrolls the rows between the
`top` and `bottom` fixed rows the way it scrolls its own lists:
- wheel and trackpad ticks, followed smoothly;
- a finger's drag, its glide after release, and the rubber band at each
end;
- pixel motion where the terminal can move text by pixels, and whole rows
elsewhere.
The fixed rows stay in place: a title or toolbar on top, a status line or
key hints at the bottom of where the surface shows. Pointer events arrive
in the surface's own rows, so a row's hit test does not change as it
scrolls. The viewer keeps the vertical wheel over the scrolling rows; the
plugin does not receive it there.
When a key moves a selection, call `surface.reveal(first, last)` with the
selected rows. The viewer brings them into view the way its own lists
follow a selection, and leaves rows that already show where they are. For
Page Up and Page Down, `surface.viewport()` returns the first scrolling row
in view and how many show. Call `set_scroll` again when the content's
height changes; `set_scroll(None)` returns the surface to the size it shows
at. A scrolling surface is at most `Scroll::MAX_ROWS` rows. A pixels surface
paints what shows and takes no `set_scroll`: to scroll pictures, lay them over
a scrolling cells surface ([Graphics](#graphics)). A complete example is
[`scroll_list.rs`](../examples/scroll_list.rs).
## Graphics
Treat real graphics as a progressive enhancement over a working cells
rendering. Paint your content in cells first, for every viewer, then draw a
picture over those cells for a viewer that can show it. A terminal without
Kitty graphics, or a viewer that loses them, keeps a surface that works, and
you write one paint for both.
```rust
pane.text(1, top, "Requests", Style::fg(Colour::FG));
paint_bars(&mut pane, place, &samples); // block characters, for every viewer
});
pane.commit()?;
```
`Surface<Cells>::draw_graphic(key, place, draw)` lays a picture over the
cells in `place`. `draw` receives a transparent `Surface<Pixels>` with every
[Shapes](#shapes) helper, at the viewer's cell pixel size, and runs only where
the viewer shows real graphics. It returns `Ok(false)` when it did not run:
that viewer shows your cells, and you need do nothing more.
A graphic lives on the surface's rows. In a surface with
[`set_scroll`](#scrolling) it moves with its rows under every scroll the
viewer has: the wheel and trackpad, a finger's drag and glide, the rubber band,
and pixel motion. The window cuts it at either edge, and the plugin keeps no
scroll offset for it.
Use a graphic for what cells draw poorly: a chart, a sparkline with a fine
line, a logo, a picture. Keep what people read in cells. Real text stays
terminal text, in the viewer's font, and it keeps working when someone selects
or reads it through a screen reader. A graphic hides the cell glyphs under it,
so give labels a row or column of their own beside the picture. A surface that
is one full picture, such as a game, belongs on a
[pixels surface](#pixels), with `["pixels", "cells"]` in the manifest when it
has a cells form.
The rules that keep an image true:
- **Same content.** The cells under a graphic are its fallback, so they say
the same thing. A viewer whose cell size changed also shows them until your
next paint.
- **One graphic per picture, inside one zone.** A graphic in a scrolling
surface keeps to the fixed rows or to the scrolling rows. One that crosses
between them shows its cells.
- **Paint on an event.** Drawing costs a buffer of the image's size and your
drawing time, and `frame` has a small budget. Call `draw_graphic` when the
data, the size, the theme or the capabilities change: `Event::Resize`,
`Event::ThemeChanged`, `Event::CapabilitiesChanged`. Graphics take no timer,
and scrolling needs no repaint.
- **Pixels match the theme.** Draw with `view::theme()` colours, repaint on
`ThemeChanged`, and leave the background transparent.
- **Cell backgrounds show through.** The cells under a graphic lose their
glyphs and keep their backgrounds, so a transparent picture shows those
backgrounds. Paint those cells with no background where `draw_graphic`
returned `Ok(true)`, or draw the picture opaque.
### Keys, commits and cost
`key` is your number for the graphic. Drawing a key again replaces it. All of
it publishes with the next `commit`, together with the cells of that commit, so
a chart and its labels change in one frame; a commit that changed only graphics
still publishes. `Surface::clear` marks your graphics for removal, and the next
commit removes the ones you did not draw again. `remove_graphic(key)` removes
one. A resize keeps those that still fit the new size at the same cell pixel
size. A section that opens or closes above a chart changes your surface's height,
so call `move_graphic(key, place)` for a chart whose data did not change. It draws
nothing and sends no image, and the chart and its cells move in one frame.
`Surface::fill` blanks cells and keeps your graphics. Use it in place of `clear`
when only cells change, as for a tooltip or a clock, and call `draw_graphic`
only for a picture whose data changed. An event has 50 ms of CPU, so a surface
with many pictures can stop at a time limit and draw the rest on the next
event, with their cells showing until then.
The SDK compares each image with the last for that key. Identical pixels cost
your drawing time and nothing else: no bytes reach the host, and a picture that
only moved sends its new place. The viewer keeps one copy of each image and
uploads only the parts near the view, so a long surface's cost follows what
shows.
A surface holds at most 256 graphics and 64 MiB of images. A chart 88 columns
wide and 6 rows tall takes 1.4 MB at 18 by 36 pixel cells, so a surface holds
about 49 of them there. Past either limit `draw_graphic` returns
`Error::Invalid` and the cells stand, so a longer surface shows its first charts
as pictures and the rest in cells. Draw in the order that matters most.
Graphics show on `pane`, `column`, `stage` and `window` surfaces. Any other
anchor answers `Error::Invalid`; a row in the sidebar or a pane's footer uses a
pixels surface. A complete example is
[`graphics_chart.rs`](../examples/graphics_chart.rs), and
[testing](testing.md#graphics) shows how to test both kinds of viewer.
## Cadence
`UiPlugin::frame(&mut self, frame: &Frame)` is called only when the viewer
paints a paced frame *and* one of the plugin's surfaces is visible *and*
the plugin asked for a frame. What the plugin asks for decides its cost:
| `frame.request_frame()` | Called again on the next paced frame: an animation at the viewer's rate |
| `frame.wake_at(ms)` / `wake_after(ms)` | Called at that instant on the viewer's clock: a clock asks for the next second |
| nothing | Not called again until the surface becomes visible again |
A commit from `event` is not a frame request: it is sampled on the next
frame something else causes. A plugin that repaints in `event` (a value
changed, a plugin event, a key) asks for a frame from there with
`cx.request_frame()` (or `cx.wake_at(ms)`); a card that changes only on
events never asks otherwise
([`together_ui.rs`](../examples/together_ui.rs)).
The viewer asks for a frame itself when a surface becomes visible.
A surface that shows nothing is never visible, so it never gets
`frame()`. A `machine.after`, `project.after` or `project.before`
instance at zero rows (`"height": 0`) stays hidden until the plugin asks
for rows. Ask from `activate` and from the events that change what the
row shows, with `Instances::instance(owner).request_size(0, rows)`; asking
only in `frame()` means no row ever shows
([agent guide](agent-guide.md#1-a-collapsed-anchor-never-gets-frame)).
Project `before` and `after` instances also hide when the user collapses their
project folder. The host retains the instances but stops placing and drawing
them; expanding the folder makes them visible again at their requested size.
`frame.elapsed()` is the time since the previous frame, and zero on the
first frame after every surface of the plugin was hidden.
`now_ms` is the viewer's monotonic clock, for animation and wakes. For
dates and countdowns, `view::wall_ms()` is the wall clock (milliseconds since
the Unix epoch) where the viewer runs, `view::utc_offset_minutes()` its local
offset right now (read it when formatting; it follows daylight saving),
`view::time_zone()` its IANA name when known, and `view::local_ms()` the two
combined (`local_ms() / 86_400_000` is the local day). A browser reports its
own clock and zone.
Nothing else wakes an idle viewer: a hidden surface costs nothing, and a
clock card costs one wake a second. `frame.power()` is `SavePower` when the
machine is saving power; slow animations down then (the shapes example
drops to one frame a second). `frame()` has half the frame interval of CPU
time (wall time in a browser). A plugin with a visible pixels surface that
keeps overrunning is degraded first: its frame rate is halved at each
overrun down to 15 fps, then its pixel resolution is halved (the viewer
scales the image up; `capabilities::get().pixel_scale` is 2 and the cell
pixel size halves). Every step arrives as `Event::CapabilitiesChanged`, and
the resolution step as a resize. Only then do further overruns restart and
finally pause it (three in total, restarts included, in every viewer).
A cells-only plugin skips straight to that.
## Two models
A surface whose manifest lists both models (`"model": ["pixels", "cells"]`)
is painted in the first one the viewer can show: pixels where it has real
graphics, else cells. The viewer may switch while the plugin runs (the
terminal changes); the plugin sees a resize. `AnySurface::new(id)` follows
it and hands out the surface in the current model:
```rust,ignore
match self.court.get() {
AnySurfaceMut::Pixels(pixels) => { /* draw pixels */ }
AnySurfaceMut::Cells(cells) => { /* draw cells */ }
}
```
Cell editors may use `surface.copy_text(selected)` during a key, paste or
press event to copy the user's selection. The call accepts at most 64 KiB
of UTF-8 text, requires a known surface and offers no clipboard read
access. Treat an error as a failed copy; only remove cut text after success.
The viewer colors a text caret with its opener's machine identity and a
project popover's frame with its project identity. Shift+Enter and Alt+Enter
are available to multiline fields; plain Enter can submit.
For embedded content, convert its coordinates into surface device pixels,
including any chrome inset, before setting an agent pointer. Clear it with
`None` when the agent finishes or the content ends. If you use live values,
check the agent's current account state before displaying a replay; the
last value can outlive the turn that produced it.