Computer use MCP server for the Mac: let any AI agent operate your apps
What a computer use MCP server does, what to check before you pick one, where the built-in options from Claude and Codex stop, and how ShadowCursor works in Claude Code, Codex, Cursor and VS Code.
Short answer: A computer use MCP server is a program on your Mac that gives an AI agent tools to see and operate apps: read a window as a screenshot plus its accessibility elements, then click, type, scroll and choose menu items. Because it speaks MCP, it works in any client that supports MCP, such as Claude Code, Codex CLI, Cursor, VS Code or opencode. The built-in computer use of Claude and Codex is tied to their own apps and plans. ShadowCursor is a computer use MCP server that gives every agent its own cursor, keyboard and clipboard in background windows, with several agents at once.
What a computer use MCP server does
MCP (Model Context Protocol) is the open standard AI clients use to call outside tools. A computer use MCP server exposes the graphical interface of your Mac as such tools. The model decides what to do; the server carries it out and reports what happened.
A typical round looks like this:
- The agent asks for the state of a window (
get_app_state). It gets a screenshot and a list of accessibility elements: buttons, text fields, checkboxes and menus, each with its label and an index. - The model picks an element and calls an action such as
click,type_text,set_valueorselect_menu_item. - The server performs the action and returns the window's new state, and the loop continues until the task is done.
Both halves of that state matter. The screenshot shows what a person would see, including canvases, images and apps that expose little to accessibility. The accessibility elements give exact targets with names, so the model can click "Save" by its index instead of guessing pixel coordinates. A good server returns both with every step.
The tools you can expect
Tool names differ between servers, but the set is similar. ShadowCursor offers these:
| Tool | What it does |
|---|---|
list_apps, launch_app | See running apps and their windows, start an app or open a file |
get_app_state | Screenshot and accessibility elements of one window |
click, hover, drag, scroll | Pointer actions, by element index or by position in the screenshot |
type_text, press_key | Typing and keyboard shortcuts into that window |
set_value, select_text | Fill a field, move a slider, pick a dropdown entry, select text |
select_menu_item, perform_secondary_action | Choose from the app's menus, trigger an element's other actions |
paste | Insert text through the agent's own clipboard |
Where computer use fits next to other tools
Computer use is the broadest way for an agent to reach an app, and the slowest. Claude Code's documentation describes the order it follows: an MCP server for the service if you have one, the shell for commands, Claude in Chrome for browser work, and only then computer use (source). Browser automation such as Playwright only reaches web pages. A computer use server covers what is left: native Mac apps, file dialogs, simulators, desktop software without an API, and websites that should see a real browser window.
What to check before you pick one
Most computer use MCP servers can click and type. The differences that decide whether you can use one every day are these:
| Question | Why it matters |
|---|---|
| Does it work in the background? | Servers that drive the real pointer (for example through PyAutoGUI or cliclick) move the one cursor your Mac has and bring windows to the front. While they run, the screen belongs to the agent. |
| Can several agents use it at once? | With one shared pointer, two agents interrupt each other. Parallel agents need a cursor and keyboard each. |
| What happens to the clipboard? | If copy and paste go through the system clipboard, an agent's copy replaces whatever was there, and two agents overwrite each other. |
| Does input reach only the target window? | Keys sent to "whatever has focus" can land in the wrong app. |
| What do screenshots contain? | A capture of the whole screen shows the model everything on it. A capture of one window shows only that window. |
| Does input look human? | Hover menus, sliders, date pickers and bot checks react to pointer movement and typing rhythm, not to a pointer that jumps. |
| How hard is setup? | Every tool on a Mac needs Accessibility and Screen Recording. Beyond that, some need configuration files, a Python environment or a build. |
The built-in options and their limits
Claude and Codex ship computer use of their own. Claude Code's version is itself a built-in MCP server called computer-use. As documented in October 2026:
| Option | Where you can use it | While it works | Agents at once | Plan |
|---|---|---|---|---|
Claude Code CLI (computer-use) | Claude Code in the terminal, macOS | Hides your other visible apps until the turn ends | One session holds a lock | Pro or Max, not Team, Enterprise, Bedrock, Vertex or Foundry, not in claude -p |
| Claude Desktop and Cowork | The Claude desktop app, macOS and Windows | Background mode on macOS | Not documented | Pro or Max |
| Codex in the ChatGPT desktop app | The ChatGPT desktop app | Background on macOS, foreground on Windows | Not documented | Supported regions, no plan named |
| Open-source MCP servers | Any MCP client | Many drive the real pointer, a few work in the background | Check each | Open source |
Some details matter in practice. In Claude Desktop, browsers are view-only and terminals and IDEs are click-only. Codex computer use cannot automate terminal apps. Sources: Claude Code computer use, Claude Desktop, ChatGPT computer use.
If you work in one of these apps, on a plan that includes it, with one agent at a time, the built-in option may be enough. A separate MCP server is worth it when you want computer use in the client you already use (Codex CLI, Cursor, VS Code, opencode), on a plan the built-in feature excludes, or for several agents at once. For a broader look at foreground and background operation, see background computer use on the Mac.
ShadowCursor: a computer use MCP server for background work
ShadowCursor is a Mac app and MCP server. When an MCP client connects, its agent gets a full set of its own:
- Its own cursor, drawn on screen in the agent's color with a name tag.
- Its own keyboard. Keys arrive only in the agent's window, in one app.
- Its own clipboard. Copy and paste never touch your clipboard or another agent's.
- Its own windows. The agent works in background windows, even when yours cover them, and screenshots capture only the window it works in.
Several agents can work at the same time, each with its own cursor. Every action returns the window's new state, so the model sees the result of each step without asking again. Input moves like a person: curved pointer paths with natural speed, a slight overshoot and correction, and typing at about 200 words per minute with a human rhythm. Hover menus in Safari, sliders, signature fields and date pickers react as they do to a person, and an agent can sign a PDF in Preview with a real pen stroke.
ShadowCursor runs locally and needs macOS 15 Sequoia or later.
Setup without configuration files
- Open ShadowCursor and allow Accessibility and Screen Recording when your Mac asks.
- Connect your client. The app finds Claude, Codex, Cursor and VS Code on your Mac and connects them with one click. Other clients that speak MCP can use ShadowCursor as well.
- Ask for a task that needs an app, for example: "Open the exported CSV in Numbers and add a total row."
Limits to know
- Minimized and hidden windows cannot be seen or operated. Covered windows can.
- System drag and drop of files, tabs or text follows the real mouse, so ShadowCursor cancels such drags instead of dropping.
- Hover without a pressed button works in Safari, not in Chrome or Electron apps.
- An agent cannot use a background tab of your browser window; it opens its own window.
- LibreOffice ignores posted clicks; its menus and keyboard work.
- Code editors such as VS Code do not expose their text through accessibility, so agents read them from the screenshot.
FAQ
Is a computer use MCP server the same as a browser automation MCP server?
No. Browser automation servers, for example ones built on Playwright, control a browser through its developer interface and only reach web pages. A computer use server operates apps through their buttons, fields and menus, so it also works in native Mac apps, file dialogs and desktop software.
Do I need one if Claude Code already has computer use?
Not always. The built-in computer-use server works in Claude Code on a Pro or Max plan, hides your other apps while Claude works and allows one session at a time. You need a separate server for Codex CLI, Cursor or VS Code, for Team, Enterprise, API key or Bedrock setups, or for several agents at once.
Which clients can use a computer use MCP server?
Any client that supports MCP servers, including Claude Code, Codex CLI, Cursor, VS Code and opencode. ShadowCursor connects Claude, Codex, Cursor and VS Code with one click.
Do screenshots leave my Mac?
The server runs locally, but the model has to see the window to work in it, so screenshots go to the model your client uses, as with any computer use. With ShadowCursor, those screenshots contain only the window the agent works in.
What does ShadowCursor cost?
A 7-day free trial without a card, then $12.99 per month or $69.99 per year, taxes included, with unlimited agents on up to three devices.