What is computer use in AI? Screenshots, accessibility trees and how agents click
A plain explainer of AI computer use: the screenshot loop, accessibility trees, coordinates or element indexes, foreground or background, API tool, desktop apps and MCP servers, and safety.
Short answer: Computer use is the ability of an AI model to operate a computer through its graphical interface, the way a person does: it receives a screenshot of the screen or a window, decides what to do, and sends back actions such as clicks, keystrokes and scrolling, then looks at the new screenshot and continues. The model only decides; a separate tool on the computer carries out the actions. That tool can be an API environment you run, a feature built into an app such as Claude Desktop or the ChatGPT desktop app, or an MCP server that any AI client can use.
Where the term comes from
Anthropic introduced computer use as a public beta on October 22, 2024, with Claude 3.5 Sonnet: developers could direct Claude to use computers "the way people do", by "looking at a screen, moving a cursor, clicking buttons, and typing text" (source). Since then the term has become generic. OpenAI uses it for Codex in the ChatGPT desktop app, and many open-source tools call themselves computer use servers.
Computer use matters because most software has no API an agent can call: a form behind a login, a native Mac app, a file dialog, a simulator. With computer use, an agent reaches them through the same buttons and fields you use. Browser automation such as Playwright is faster for pure web tasks, but it only reaches web pages.
The basic loop
Every computer use system runs the same cycle:
- Observe. The tool captures the screen or one window and sends it to the model, often together with a description of the interface elements.
- Decide. The model reads the image, compares it with the task and picks the next action: click this button, type this text, press Return, scroll down.
- Act. The tool performs the action on the computer.
- Check. The tool captures the result, and the loop starts again until the task is done or the model asks you something.
This is why computer use is slower than an API call. Every step is a round trip with an image. Anthropic calls it "the broadest and slowest" way for Claude to reach an app, and Claude Code tries an MCP server for the service, the shell and Claude in Chrome before it falls back to the screen (source).
How an agent sees an app
There are two sources of information, and good tools use both.
| Aspect | Screenshot | Accessibility tree |
|---|---|---|
| What it is | An image of the screen or one window | A structured list of the interface: buttons, fields, menus, checkboxes, with their labels and values |
| Where it comes from | Screen capture (on a Mac, the Screen Recording permission) | The operating system's accessibility interface (on a Mac, the Accessibility permission) |
| Strengths | Shows everything a person sees, including images, charts and canvases | Exact names and states, such as "Save, button, enabled" or the text in a field |
| Weaknesses | Costs tokens; small text can be hard to read after downscaling | Some apps expose little or nothing, for example canvases, games or editors that draw their own text |
The accessibility tree exists for screen readers such as VoiceOver. It describes an app by meaning rather than pixels, so an agent can find "the Send button" without guessing where it is.
How an agent points: coordinates or element indexes
Once the model knows what to do, it has to say where. There are two ways.
Coordinates are pixel positions in the screenshot, such as "click at x 512, y 300". They work in any app, because every app has pixels. But they depend on the screenshot's scale: Anthropic's API documentation notes that Retina screenshots on a Mac are captured at twice the pixel density, so either the image or the coordinates must be scaled, and that Claude "might make mistakes or hallucinate when outputting specific coordinates" (source).
Element indexes or references point at an item in the accessibility tree, such as "click element 14", where 14 is the Save button in the list the tool just sent. They cannot miss by a few pixels, but they only exist where the app exposes its elements.
Anthropic's own tools show both approaches. Its API computer use tool "works through screenshots and coordinates alone", while its browser use tool tells Claude to "prefer references where the page has a usable accessibility tree" and to use pixels only for content the tree does not describe (source). Tools that offer both let the model use an index where one exists and coordinates where it does not.
Foreground and background
The tool that carries out the actions also decides what happens to your screen. Foreground computer use drives the one pointer and keyboard focus your computer has and brings windows to the front, so you wait while it works. Background computer use gives the agent its own input and lets it work in windows behind yours. On the Mac, Claude Desktop and Codex in the ChatGPT app now have background modes; the Claude Code CLI hides your other apps while it works. The differences are explained in background computer use on the Mac.
Three forms of computer use
"Computer use" can mean three different products, and they are easy to mix up:
| Form | Who performs the actions | Typical user | Examples |
|---|---|---|---|
| API tool | Your own code, in an environment you run, usually a VM or container | Developers building a product or pipeline | Anthropic's computer use tool in the Claude API |
| Feature of a desktop app | The vendor's app on your computer | People working in that app | Claude Desktop and Cowork, the Claude Code CLI, Codex in the ChatGPT desktop app |
| MCP server | A separate program on your computer that any MCP client can call | People who want computer use in the client they already use | Claude Code's built-in computer-use server, open-source servers, ShadowCursor |
With the API tool, Claude never touches a computer directly. It returns actions, your application performs them and sends back screenshots, and Anthropic recommends a dedicated virtual machine or container with minimal privileges. You pay for tokens and run the environment; the pricing guide shows what that costs.
Desktop features are the quickest start, but tied to the vendor's app and plans. Claude's requires Pro or Max; the setup is in Claude computer use on the Mac.
An MCP server exposes the computer as tools over MCP (Model Context Protocol), the open standard AI clients use for outside tools. Any MCP client, such as Claude Code, the Codex CLI, Cursor or VS Code, can then see and operate apps. What to look for in one is covered in computer use MCP servers for the Mac.
Safety
An agent with computer use acts with your apps and your logins, on your real desktop. The main risk is prompt injection: text on a web page, in an email or a document that is written as an instruction to the AI. Anthropic's safety guidance says plainly that "computer use has no sandbox between Claude and your applications" (source). Approve only the apps a task needs, keep password managers and sensitive apps closed, and watch steps that send, buy or delete. The guide to letting an AI control your Mac safely goes through the rules.
Where ShadowCursor fits
ShadowCursor is an MCP server and Mac app of the third kind. It returns both halves of a window's state with every action, a screenshot and the accessibility elements, so the model can click by element index and fall back to a position in the screenshot. It gives every agent its own cursor, drawn in its color with a name tag, its own keyboard, clipboard and windows, so agents work in background windows and several can run at once. It needs macOS 15 or later, and it cannot operate minimized or hidden windows.
FAQ
Is computer use the same as an AI agent?
No. An agent is a model that works through a task in steps with tools. Computer use is one of those tools: the one that lets the agent see and operate a graphical interface. Many agents work without it, through files, the shell and APIs.
What is computer use in Claude?
Anthropic's name for Claude operating a computer through screenshots, clicks and typing. It exists as a tool in the Claude API and as a research preview in Claude Desktop, Cowork and the Claude Code CLI on Pro and Max plans.
What is computer use in Codex?
OpenAI's computer use for Codex runs in the ChatGPT desktop app. On macOS it can run a task in the background; on Windows it works on the active desktop. It cannot automate terminal apps (source).
What is a computer use MCP server?
A program on your computer that offers computer use as MCP tools, such as taking a window's state, clicking, typing and choosing menu items. Any client that supports MCP can use it, independent of the AI vendor.
Does computer use send my screen to the cloud?
The screenshots the agent receives go to the model it uses, which for Claude and Codex runs in the provider's cloud. Tools differ in how much they capture: the whole screen, approved apps or a single window.