Phosphene

Watch it work

In the replay, a cell that has just been written lights on the grid and travels across the pipeline row: into the cell lattice, through its role, and out as a message. Grey diagonal hatching marks cells the gateway has not matched yet; those go out as a Terminal component carrying styled runs, so nothing is lost while the model decides. When the timeline turns black the screen is locked and frames are filled from the template without calling the model.

…but why?

Every terminal client today does the same work. It parses an escape-code stream, keeps a cell grid, and draws characters. The result is faithful and opaque. A screen reader sees a wall of characters, a phone gets a grid it cannot reflow, and an agent has to read box-drawing characters to find out which row is selected.

The structure is there. People see it at a glance: a title, a menu, a highlighted row, a status bar with key hints. It is simply never written down. phosphene writes it down once, close to the program, and sends it in a form any client can render natively.

Native clients

A2UI is a declarative interface stream: components bound to a data model, sent as small JSON messages. A web, mobile or desktop renderer draws its own widgets for it. Nothing on the client parses VT.

Agents

An agent reads a typed surface instead of characters. The selected row is a data value. F10 is a Button with an action, and pressing it sends the keystroke back to the program.

Faithful when wanted

Every component also carries a style binding with its cell rectangle and ANSI colours. A client that wants the terminal look can lay the components out on the grid, as the right half of the player does.

Anatomy

Keyframes

Programs repaint in bursts. A keyframe is a screen that has stopped changing, with a damage mask of the cells written since the last one. The model only ever sees keyframes.

Roles

Fifteen: blank, text, prompt, input, border, title, status bar, menu item, selected, table, progress, code, log, error, key hint. They are chosen so a compiler can act on them: a border means a box, a selected cell means a list with a cursor, a key hint means a button.

Templates

A template is the compiled layout plus a map of slots: which cell ranges feed which data paths. It is cached by grid size and a layout signature that ignores letters and digits, so the next htop screen, or the next htop session, matches without the model. Regions can lock separately: a stable title bar and key hints lock while a log pane is still moving.

Reconciler

The client keeps the last UI. When a frame changes only content, the gateway sends updateDataModel patches by JSON pointer: one table row, one percentage, one keystroke. updateComponents goes out only when the structure changes.

Actions

Buttons carry their key, lists carry a select event, inputs an edit event. The gateway turns them into bytes for the program: F10, the arrow presses that move a selection to row 7, or backspaces and the new text.

The model

The network is deliberately small: a convolutional stem over per-cell features (character class, colours, attributes, cursor), then four blocks of axial attention that alternate between rows and columns, then a role head per cell and an app head per screen. 1.26M parameters, 5.0 MB in float32, exportable to ONNX for WebGPU. It is a pure function of one screen; all the state lives in the gateway.

Training data comes from three places. A synthetic TUI generator renders boxes, menus, tables and logs through the real emulator, so its 14,223 frames have exact labels. Heuristic labelling functions cover every real frame for free and badly: they agree with reviewed labels at mIoU 0.20. And 600 real frames, chosen from 44,168 de-duplicated keyframes for disagreement and diversity, were labelled as region rectangles by Claude Sonnet subagents. 90 of the hardest were reviewed by Claude Opus, which put the Sonnet labeller at mIoU 0.74.

An mIoU of 0.51 sounds low and is less bad than it looks. The macro average weighs rare roles such as progress and error the same as text, and most of the errors are at region edges: a table that starts one column late, an input that runs past the typed text. The compiler absorbs a lot of this, because it acts on regions rather than single cells, and the gateway only needs the model to recognise a layout once before the template takes over.

Measured

The template cache does what it is for. On less, dialog and vim, whose screens keep one layout while the content moves, three to nine screens in ten never reach the model. On top, htop and nano, where meters, scroll positions and the edited text keep changing what the layout signature sees, most screens go back to it. Across 193 random recordings from the corpus (13,967 settled screens, one shared cache), 40% of screens were served from a locked template and the model ran 0.70 times per screen.

The bytes are the honest surprise. The A2UI stream is larger than the VT stream it replaces: 25× over the same 193 recordings. VT is a very compact format for a grid: “move here, write these six characters”. JSON components and pointer patches are not, and the unmatched regions still go out as styled runs. Incremental updates bring it down to 43% of re-sending everything, which is the reconciler working. The claim this prototype supports is about the client, not the wire: the receiving side draws widgets and never runs a terminal emulator.

Limits

• The model is a first round: 600 agent-labelled real frames. Its errors are visible in the replay as regions that flicker between roles on screens the template cache has not locked.
• The wire format is not yet compact. A binary or delta-coded encoding of the data patches, and fewer styled runs for unmatched regions, are the obvious next steps.
• The demos are recordings, so the action path (A2UI events back to keystrokes) is tested in the repository but not shown here.
• The demos were found by searching the recordings for each app’s signature and start at its first screen; the shell session before it is skipped, and the gateway starts cold there.

Evidence boundaryMeasured: the label-quality and training figures (docs/RESULTS.md) and the per-demo gateway figures, all generated on a Colab T4 on 8 October 2026 from public asciinema recordings (4,000 of a 79k-recording archive). Not measured: rendering cost on a client, latency over a network, or accessibility outcomes. The replay is pre-generated; no model runs in this page.

1. A2UI: agent-to-user interface protocol, v0.9
   context · public
2. asciinema terminal recordings (Hugging Face dataset)
   measured · public
3. phosphene source, labels, results log and demo generator
   measured · public · docs/RESULTS.md · packages/vtm/src/vtm/demos.py
4. Ho et al., Axial Attention in Multidimensional Transformers (2019)
   context · public