VS Code and GitHub Copilot computer use on the Mac, through an MCP server
VS Code's agent has browser tools but no tool for desktop apps. How MCP works across Copilot, Local, Claude and Codex sessions in VS Code, and how to let the agent operate Mac apps in its own windows.
Short answer: GitHub Copilot in VS Code has no built-in computer use for desktop apps. Its built-in browser tools operate web pages in VS Code's integrated browser, and everything else outside the editor needs an MCP server. Add a computer use MCP server, and the agent in an Agent session can see a Mac app's window, click, type and choose menu items. ShadowCursor is such a server and connects VS Code with one click.
What the agent in VS Code can do on its own
VS Code's agent sessions come with built-in tools for files, search, the terminal and the browser (tools reference). The browser tools are the closest thing to computer use: the agent can start your app, open it in the integrated browser, navigate, interact with the page, read console errors and take screenshots, without an extra MCP server (browser tools).
They stop at the edge of the integrated browser. VS Code documents no built-in tool for native Mac apps, the iOS simulator, file dialogs, Preview, Numbers or an outside browser such as Safari. VS Code's own documentation says to add an MCP server "when your agent needs a capability its existing tools don't provide" (source).
| Task | Built-in browser tools | A computer use MCP server |
|---|---|---|
| Test the web app you are building | Yes | Yes, in a Safari window |
| Read console errors | Yes | No, it sees what is on screen |
| Click through a native macOS app | No | Yes |
| Check an exported PDF in Preview or a sheet in Numbers | No | Yes |
| Work through file dialogs and app menus | No | Yes |
For web development, start with the built-in browser tools. Reach for computer use when the task leaves the browser.
Harnesses decide where MCP configuration lives
VS Code now runs agent sessions in several harnesses, and each reads MCP servers from its own place (agent harnesses, MCP configuration):
| Harness or client | Where its MCP servers come from |
|---|---|
| Copilot (the default) | The portable Copilot format (~/.copilot/mcp-config.json or a workspace .mcp.json), plus eligible servers VS Code forwards from its own mcp.json |
| Local | VS Code's mcp.json, in the workspace (.vscode/mcp.json) or your user profile |
| Claude | The provider's own configuration: Claude Code's |
| Codex | The provider's own configuration: Codex's |
| GitHub Copilot CLI, outside VS Code | ~/.copilot/mcp-config.json and project files, added with /mcp add (source) |
VS Code forwards servers from its own format to Copilot sessions except those that need interactive input, such as ${input:...} variables. A server without such inputs is available in Copilot and Local sessions alike.
How MCP tools work in an agent session
- Use the Agent role. In the Copilot harness, Ask only uses read-only tools and cannot make changes. Computer use belongs in an Agent session.
- Trust. VS Code's documentation asks you to review any server trust prompt before you accept it and to trust only configurations you have reviewed.
- Choose tools. Chat: Open Customizations > Tools controls which tools a Copilot session has; Local sessions use Configure Tools in the chat input. A request can have at most 128 tools enabled, so turn off servers you do not need (source).
- Approvals. With the default Manual permissions, tools that are not auto-approved ask before they run. Chat: Manage Tool Approval groups tools by MCP server, so you can trust a whole server or single tools. Allow all and Autopilot skip confirmations, including for external tool calls (source).
- Organization policy. Your organization can restrict which MCP servers are available, for example through an allowlist.
ShadowCursor: computer use for the agent in VS Code
ShadowCursor is a Mac app and MCP server. When VS Code connects, the agent gets a full set of its own:
- Its own cursor, drawn on screen in the agent's color with a name tag.
- Its own keyboard. Keys arrive only in the agent's window, in one app.
- Its own clipboard. Copy and paste never touch your clipboard or another agent's.
- Its own windows. The agent works in background windows, even when yours cover them, and screenshots capture only the window it works in.
Every action returns the window's new state, a screenshot plus accessibility elements, so the agent sees what each click did. Several agents can work at once, each with its own cursor: a Copilot session in VS Code next to Claude Code or Codex in a terminal. Input moves like a person, with curved pointer paths and typing at about 200 words per minute, so hover menus in Safari, sliders and date pickers respond as they do to a person. It runs locally and needs macOS 15 Sequoia or later.
Setup
- Open ShadowCursor and allow Accessibility and Screen Recording when your Mac asks.
- Connect VS Code with one click. The app finds VS Code and registers itself in VS Code's
mcp.json. It needs no interactive inputs, so VS Code can forward it to Copilot sessions as well. If you use the Claude or Codex harness, connect Claude Code or Codex in the same step; those harnesses read their own configuration. - Review the trust prompt if VS Code shows one for the server, then accept it.
- Start an Agent session and ask for a task in an app. Check the tool call in the chat to confirm it came from ShadowCursor.
For the GitHub Copilot CLI outside VS Code, add ShadowCursor as a local (stdio) server with /mcp add, using the command shown under Set up manually in ShadowCursor's setup.
Example tasks
- "Build the macOS app, launch it, open the new export sheet and check that every format option is listed."
- "Export the report as PDF, open it in Preview and check that the table fits on one page."
- "Open the staging site in a Safari window of your own, sign up with the test account and check the welcome email link."
Name the app and what counts as done. Tell the agent to stop and ask before sending, buying or deleting anything; whether it is safe to let an AI agent control your Mac covers the other rules.
Limits to know
- Code editors such as VS Code do not expose their text through accessibility, so an agent reads VS Code windows, including an Extension Development Host, from the screenshot.
- Hover without a pressed button works in Safari, not in Chrome or Electron apps. VS Code itself is an Electron app.
- Minimized and hidden windows cannot be seen or operated. Covered windows can.
- System drag and drop of files, tabs or text follows the real mouse, so ShadowCursor cancels such drags instead of dropping.
- An agent cannot use a background tab of your browser window; it opens its own window.
- ShadowCursor is a separate subscription from Copilot: a 7-day free trial without a card, then $12.99 per month or $69.99 per year.
FAQ
Does GitHub Copilot have computer use?
Not for desktop apps. Copilot in VS Code has built-in browser tools for web pages in the integrated browser. For native Mac apps, it needs a computer use MCP server. Searches for "Copilot computer use" often find Microsoft 365 Copilot, which is a different product.
Does computer use work in Ask mode?
No. In the Copilot harness, Ask only uses read-only tools. Use an Agent session.
Which Copilot plan do I need?
VS Code's MCP documentation names no plan. If your Copilot seat belongs to an organization, it can restrict which MCP servers are available, so an admin may have to allow ShadowCursor first.
Why does VS Code say I have more than 128 tools?
A request can have at most 128 tools enabled. Deselect servers or tools you do not need in the tools picker, then try again.
Can the Copilot agent and Claude Code operate apps at the same time?
Yes, with ShadowCursor. Each agent connects on its own and gets its own cursor, keyboard and clipboard. Give each one its own window or document. See running several AI agents at once.