computer-use-mcp

An MCP server that gives AI agents a computer. Any MCP client, such as Claude Code or OpenCode, can see and control one shared Linux desktop that runs in Docker. You can watch and use the same desktop in a browser or with any VNC client.

Status: early. The MCP server starts the computer and hands out sessions (start_computer, end_session), shows each session its own screen (computer_observe), and lets it click, type, and scroll (computer_act), runs shell commands (shell, set_cwd), lists, reads, and writes files (list_files, read_file, write_file), copies files and folders between your machine and the computer (file_transfer), opens pages, files, and applications on the screen (open_path, launch_app), and inspects the screen's browser with Chrome DevTools (start_chrome_devtools, stop_chrome_devtools).

Design

• There is one computer. It is a Docker container with a Docker volume for its home folder.
• When an agent starts the MCP server, the server starts the container if it is stopped, or creates it if it does not exist. The server never stops the container. The only time it removes one is a stopped container on an older image than the binary's, which it creates again on the newer image with the same home volume and the same ports. It never touches a running container and never moves the computer to an older image. To upgrade, run docker stop on the computer and call start_computer. computer-use-mcp info shows whether an upgrade is pending.
• Every agent connected through the MCP server sees and controls the same desktop.
• Files in the volume survive container restarts and container deletion. Only you stop or delete the computer, with docker stop or docker rm.
• The computer is Debian 13 with Python 3, uv, Node.js 24 (LTS), Git, gh, the AWS CLI v2, Ruby, fish (bash stays the default shell), build tools, ripgrep, ImageMagick, LibreOffice (Writer, Calc, Impress), clipboard tools, and fonts for Latin, CJK, and emoji.
• The user computer has passwordless sudo, so agents can sudo apt-get install more. Only home survives when the container is recreated, so system packages are lost then. npm install -g and pip install go to ~/.local (first on PATH) and survive.
• The container logs print the viewer link and the VNC password, so you can reopen a closed view.

The project has three Rust crates:

Install

With Rust 1.99 or newer, run cargo install computer-use-mcp. It builds the same release binary, which uses the image of its own version.

Or use a prebuilt binary:

1. Download the archive for your platform from the releases page. Builds exist for Windows x64 (x86_64-pc-windows-msvc) and macOS Apple Silicon (aarch64-apple-darwin).
2. Put computer-use-mcp on your PATH.
3. On macOS, the binary is not signed. If you downloaded it with a browser, remove the quarantine flag with xattr -d com.apple.quarantine computer-use-mcp.
4. The image lives in a private GitHub registry. Log in once with a token that has the read:packages scope, using docker login ghcr.io -u <github-user>.

Add it to your agent host as an MCP server that runs computer-use-mcp with no arguments. Check the install with computer-use-mcp info. It prints the version, the image the binary uses, and the state, image version, and pending upgrade of the computer.

Configuration

Release builds pull their image when it is missing. Dev builds never pull, so a dev host binary is never paired with an old image by accident. Build the dev image with just image.

The server makes no Docker calls until an agent calls start_computer. That tool takes a title (1 to 80 characters) and returns a session id. The computer gets the host timezone and the en_US.UTF-8 locale when it is created.

computer_observe takes the session id and returns a PNG of that session's own screen plus the frame id, capture time, size, cursor position, and active window title. The first call opens the screen, which is one Xvnc and Fluxbox inside the computer. The computer has 16 screens. A session that never calls it gets none, and ending the session closes its screen. When nothing changed since the session's previous screenshot, the image is left out and the text says so.

computer_act takes the session id and up to 24 ordered actions: click, move, down, up, type, key, scroll, wait, and focus. Positions are pixels on the session's screen. A double click counts as two actions. Waits and the settle time are capped at 5 s, and a scroll takes 1 to 20 steps. type handles any Unicode text. key takes names such as enter, esc, and f5 with the modifiers ctrl, alt, shift, and super (also cmd, option, meta, win). focus raises a window by its class or title, or starts the app when no window matches. The batch ends with a screenshot by default, taken settle_ms (default 300) after the last action, and the unchanged-frame rule applies to it. Set observe to false to skip it. The 4th identical batch of scroll, pointer, or key actions in a row that leaves the screen unchanged is refused.

shell takes the session id, a command, and an optional timeout in seconds. It runs bash -lc as the user computer inside the computer, in the session's working folder (home until set_cwd changes it), with no input. DISPLAY points at the session's screen when it has one. The result gives the exit code, the duration, and stdout and stderr separately. A failing command is a normal result. A command that runs past its timeout is killed with everything it started, and the result says so and keeps the output printed until then. Each stream keeps its first and last 15000 bytes with a [... N bytes omitted ...] marker between them. The call returns when the command exits, so start long-running jobs in the background with their output redirected, for example setsid nohup server >log 2>&1 &. Shell calls never wait for desktop actions, and one session may run several at once.

set_cwd takes the session id and a path, resolves a relative path from the current working folder (~ is home), and fails when the folder does not exist. It returns the new absolute path.

list_files, read_file, and write_file take the session id and a path. Relative paths start at the session's working folder, ~ is home, and absolute paths work anywhere the user computer can reach. All sessions see the same files. list_files lists one folder (the working folder by default), folders first, with type, size, and modified time, and caps at 1000 entries. read_file returns UTF-8 text as text and PNG or JPEG files (up to 1 MB, found by their first bytes) as images. It refuses other binary files with a hint to use shell or file_transfer. Long text keeps its first and last 15000 bytes with a marker between them and a note, and the optional offset and limit (in lines, from 1) read a range. write_file replaces a file with UTF-8 content of up to 10 MB, creates missing folders, and writes atomically while keeping an existing file's permissions. It refuses a path that is a folder.

file_transfer copies a file or a whole folder between the host (your machine) and the computer. It takes the session id, direction (to_computer or from_computer), host_path, computer_path, and overwrite (default false). A host path is absolute, starts with ~ for your home folder, or is relative to the MCP server's working folder, and any path the server process can reach works. A computer path resolves like the other file tools. To put a file the agent made on the computer into your downloads, the agent calls it with direction: from_computer, computer_path: ~/report.pdf, and host_path: ~/Downloads. To give the agent a folder to work on, it uses to_computer with that folder as host_path. Any file works, binary or text, with no size limit, and folders keep their contents, empty folders, executable bits, and symlinks. Data streams through tar in both directions, so memory stays small however big the files are. The destination follows cp -r. When the destination is an existing folder, the source goes inside it under its own name (report.pdf into ~/Downloads becomes ~/Downloads/report.pdf). Otherwise the destination is the new name, and missing parent folders are created. With overwrite off, the call refuses when the final path already exists and names it, before it writes anything. With overwrite on, files are replaced and folders merge, replacing same-named files and keeping the others. A file never replaces a folder or the other way round. With overwrite on, a conflict found partway through a folder stops the copy. Files already copied stay, and the error says how far it got. Each file is written under a temporary name and renamed into place. On a Windows host, names Windows cannot hold (reserved names such as CON, and the characters : ? * " < > |) and symlinks Windows will not create are skipped, and the reply lists them together with pipes, sockets, and devices. The reply gives the final path, files, folders, bytes, and time. A transfer has no total time limit and fails when no data moves for 60 seconds, or when the computer does not answer within 60 seconds after the last data was sent. A refusal to write to the destination is reported before any data moves. Cancelling the call or ending the session stops it and removes its temporary files.

Sessions end on their own, so a crashed or forgotten agent does not hold a screen. Each MCP server process sends a heartbeat every 10 s for all its sessions, and its sessions end 30 s after the last one, which covers a killed process. A session also ends after its idle time with no agent call. A running shell command counts as activity, so a long command never makes its own session idle. When the MCP server exits on stdin close, Ctrl+C, or SIGTERM, it ends its sessions at once, waiting at most 3 s. Ending a session closes its screen and frees its number. Files in home are never touched. A call on a session that ended on its own says why and asks the agent to call start_computer again.

Docker endpoint

The server talks to the same Docker as your docker command. It picks the endpoint in this order.

1. DOCKER_HOST.
2. The context named by DOCKER_CONTEXT.
3. currentContext in ~/.docker/config.json (the folder moves with DOCKER_CONFIG).
4. The platform default, a named pipe on Windows and /var/run/docker.sock elsewhere.

The context default means step 4. Colima, OrbStack, and Rancher Desktop work after docker context use <name> (for example colima, orbstack, or rancher-desktop). Run computer-use-mcp info to see the endpoint in use and why it was chosen.

Supported endpoints are unix://, npipe://, tcp://, and http://. A context that needs TLS or uses ssh:// is refused with a message that names it. Use a context with a socket, or set DOCKER_HOST.

Watching and using the screens

start_computer returns a viewer link such as http://127.0.0.1:20900/#key=ab3d5fgh, and computer-use-mcp info prints it while the computer runs. The container logs print it at every start (docker logs <name>). Open it in a browser. The page lists every live session in a sidebar with its title, screen number, start time, and number of viewers, and updates as sessions start and end. Click a screen to connect to it. A bar above the screen shows its number, title, and size. Every connection starts locked (View only), so watching never sends a stray click or keypress. Press Unlock to take control and Lock to hand back. Nothing coordinates you with the agent while you are unlocked. While the page is unlocked, copy and paste work between your computer and the screen. Press Ctrl+V (Cmd+V on macOS) on the screen to paste the text on your clipboard, and text copied in an app on the screen lands on your clipboard. Locked pages move nothing, and only text is copied. Your browser may ask once to allow clipboard access, and the top bar shows Clipboard blocked by the browser if you refuse. The lock lives in the page only and does not affect native VNC clients, which sync the clipboard on their own. The browser keeps the key in local storage and removes it from the address bar. The address bar keeps #screen=<n> so a link can open the page on one screen.

Native VNC clients connect to 127.0.0.1 on the base port plus the screen number (20901 to 20916 by default) and use the key as the password. macOS Screen Sharing works with open vnc://:<key>@127.0.0.1:20901. The key is 8 characters because classic VNC password authentication ignores everything after the 8th. computerd makes it on the first start with a fresh home and keeps it in home, so links stay valid across restarts.

Ports published on 127.0.0.1 only:

The API port is picked by Docker when the container is created. Port 7071 (the launcher for the Fluxbox menu), 9221 + N (DevTools), and 5900 + N (Xvnc, behind the viewer) exist inside the container only.

Every viewer goes through computerd, which checks the key, refuses page requests whose Host is not 127.0.0.1 or localhost on that port, and checks Origin on WebSocket upgrades. While a viewer is attached to a screen, the session's idle timer does not run. When a session ends while someone watches, it ends for the agent at once, but its screen stays open until the last viewer disconnects.

The base port is a setting of the computer, read when the computer is created. Two computers on the same machine need different bases, for example COMPUTER_USE_NAME=other COMPUTER_USE_PORT_BASE=21900. If a port is taken, start_computer says which one and names the setting. A computer that already exists keeps its ports. Remove it with docker rm (home stays) to create it again with another base.

Apps and pages

• open_path opens an http(s) URL or a file on the session's screen and returns a screenshot. Pages, and HTML, PDF, image, text, JSON, and XML files, open in the screen's Chromium. Other files open with their default application (xdg-open).
• launch_app starts an application on the session's screen, or raises its window when one is open, then returns a screenshot. It takes browser, terminal, the name of an installed application (a .desktop entry), or a program on PATH. A uri is opened in the application, and the browser opens it in a new tab. The focus action of computer_act does the same when no window matches.
• Every screen has its own Chromium with uBlock Origin Lite, which installs itself from the Chrome Web Store within a minute of the first start. The user can disable it. The Chromium starts the first time something needs a browser on that screen, and closes with the screen so its profile is saved. Each Chromium runs on a fresh profile cloned from a template profile in home.
• Logins are shared across screens. About every 2 seconds computerd reads the cookies of every running Chromium over DevTools, merges the changes into one cookie jar in home (~/.local/share/computer-use/cookies.json, mode 0600), and writes the differences into the other Chromiums. Logging out spreads the same way. A new Chromium loads the jar before it opens its page. Sign in once on any screen and every other screen, and every screen opened later, is signed in a few seconds after a reload.
• Bookmarks, saved passwords, autofill data, and browser settings (such as page zoom) live in the template profile. A Chromium copies back the ones it changed when it closes cleanly. A running Chromium does not see changes made on another screen until it is closed and the next one starts. When two Chromiums change the same file, the one that closes last wins.
• Known gaps. Logins kept in localStorage or IndexedDB do not carry over. The list of installed extensions and their settings do not carry over, so uBlock installs again at every start. The "Sign in to Chromium" button is cosmetic and does not sync anything. Chromium caps cookie expiry at 400 days, so a longer expiry shows as the capped one. Cookies in opaque partitions stay on their screen. The jar keeps the 5000 most recently changed cookies, and a jar that cannot be read is kept as cookies.json.bad before a new one starts.
• The Chromium runs without its own sandbox, because Docker's default seccomp profile blocks the user namespaces that sandbox needs. The container is the boundary. DevTools listens on 127.0.0.1 inside the container only, on port 9221 plus the screen number.
• start_chrome_devtools runs chrome-devtools-mcp against the screen's Chromium and bridges it with mcpc, so the agent calls its tools through the shell tool: mcpc @cdt-N tools-call <tool> '<json>', where N is the screen number. The reply explains this. Tools include list_pages, take_snapshot, navigate_page, evaluate_script, list_console_messages, list_network_requests, and take_screenshot, and every tool needs a pageId from list_pages. It starts the screen's Chromium when none runs. Usage statistics and the CrUX lookup of chrome-devtools-mcp are off. A running session uses about 330 MB. It stops by itself after COMPUTER_USE_DEVTOOLS_IDLE without calls, and the next mcpc call starts it again with the page kept. stop_chrome_devtools stops it at once, and end_session stops it too. The two npm packages are pinned in the Dockerfile and bumped by hand.
• The right-click menu of a screen offers Browser and Terminal. Both open on that screen. xdg-open of a web link inside the computer opens it in that screen's Chromium too.

Opening the viewer

The viewer opens by itself the first time a session's screen opens, so the user sees the agent work without any step. COMPUTER_USE_OPEN picks how, per MCP server process.

• browser (default). If a viewer page is already open in a browser, it switches to the new screen and highlights it in the sidebar, and no tab opens. Otherwise a tab opens on the new screen. Several screens opening at once open one tab. The page count is a best guess. If no page is really there, the viewer link from start_computer still works.
• vnc on macOS runs open vnc://:<key>@127.0.0.1:<port>, which opens Screen Sharing.
• vnc on Windows runs a VNC viewer directly, because Windows viewers do not read a password from a vnc:// link. It needs TigerVNC. The server uses COMPUTER_USE_VNC_VIEWER (the path of vncviewer.exe, run as vncviewer.exe -passwd <file> 127.0.0.1::<port>), (it must be a file), or else vncviewer.exe from PATH or from Program Files\TigerVNC. The password goes into a private temp file that is removed after 15 s or when the server exits. With no viewer found, the server opens the browser and logs the reason to stderr.
• none opens nothing.

Opening never delays or fails a tool call. Problems go to stderr. vnc on Linux hosts opens the browser.

Development

Requirements: Rust (the toolchain version comes from rust-toolchain.toml), just, Docker, and gh. On Windows, just runs recipes with Git Bash's sh.

just            # list recipes
just check      # fmt check, clippy with -D warnings, tests
just info       # run the host binary's info command
just image      # build the computer-use-mcp:dev image
just image-check  # build the image, check every tool runs, and call the health endpoint

Commits follow Conventional Commits. feat: and fix: decide the next version and the changelog entry.

CI and releases

• Every branch push runs just check and builds the Windows and macOS binaries. The binaries are kept as workflow artifacts for 14 days.
• Merges to master update a release PR managed by release-plz. Merging that PR tags vX.Y.Z and creates the GitHub release.
• Every v* tag builds the host binaries and the linux/amd64 + linux/arm64 image. It uploads the binaries to the GitHub release and pushes the image to GHCR.
• Pre-releases come from any branch with just tag-prerelease 0.2.0-beta.1.

See RELEASE.md for the full flow and the one-time GitHub App setup.

License

MIT. See LICENSE.