Claude Code GUI testing for Mac apps: build, launch, click through, screenshot

Short answer: Yes, Claude Code can test a GUI: it builds your Mac app in the terminal, launches it, and with a computer use tool clicks through the windows, reads screenshots and accessibility elements, and reports or fixes what is broken. Claude Code's built-in computer use hides your other apps while it works and allows one session at a time. With ShadowCursor, Claude Code, Codex or Cursor test your app in a background window, each agent with its own cursor, keyboard and clipboard. For checks that must run on every commit, have the agent write XCUITests instead.

Why let an agent click through your app

Unit tests tell you the logic works. They do not tell you that the Save button is disabled after a rename, that a sheet opens off-screen, or that a label is cut off in German. Those are the bugs you find by using the app, and they are what an agent with computer use finds too: it launches the build, does what a user would do and looks at the result.

The value grows when the agent can do it while you work. A coding agent that changes a view, rebuilds, opens the app and checks the change itself closes the loop without you switching windows to look.

The options

Approach	What it does well	Where it stops
XCUITest (`XCUIApplication`)	Repeatable UI tests that run in CI on every commit	Only checks what you wrote a test for; tests take time to write and maintain
Claude Code built-in computer use	Exploratory testing from the CLI, no extra install	Hides your other apps while Claude works; one session at a time; Pro or Max plan
A computer use MCP server such as ShadowCursor	Exploratory testing in a background window with the agent's own cursor and keyboard, in any MCP client, several agents at once	Not a CI tool; needs your Mac and a visible (not minimized) window

XCUITest's XCUIApplication is "a proxy that can launch, monitor, and terminate a test application" and supports macOS (Apple's documentation). That is the right tool for regressions. An agent with computer use is the right tool for "go and try it", and a good one can turn what it found into XCUITests afterwards. Claude Code's built-in limits are documented in the Claude Code docs.

How it works with ShadowCursor

ShadowCursor is a Mac app and MCP server that gives AI agents their own cursor, keyboard and clipboard. The agent operates your app's window in the background, even when your editor covers it. Its keys arrive only in that window, and the screenshots it gets show only that window. Its cursor is drawn in its color with a name tag, so when you glance at the app you see the agent working in it. Several agents can do this at the same time, each with its own cursor, so one can test while another implements.

The loop

Build. The agent runs your usual build in its terminal: xcodebuild for an Xcode project, swift build for a Swift package, or your build script.
Launch. It quits the previous instance and opens the fresh build, for example with open path/to/YourApp.app.
Look. It reads the window: every action returns a screenshot and the window's accessibility elements, so it knows buttons, fields, lists and their labels.
Act. It clicks, types, scrolls, opens menus with menu commands and works sliders and canvases, at human speed and with human pointer movement.
Check. After each step it compares the new state with what should have happened.
Fix or report. In a coding session it changes the code and goes back to step 1; otherwise it writes up what it found, with steps to reproduce.

Example prompts

Build the app with ./Scripts/build.sh, launch the dev build and test the new
rename flow: create a document, rename it twice, undo, close and reopen the
window. Tell me what fails and fix it, then repeat the test.

Launch the app and go through every item in the Settings window. Check that
each control does what its label says and that nothing is cut off. Report
problems with steps to reproduce; do not change code.

Switch the app's language to German with the launch argument
-AppleLanguages "(de)", open each main window and list labels that are
truncated or still in English.

You just tested the import flow by hand. Turn the steps that worked into an
XCUITest in AppUITests so we keep them as a regression test.

Make your app easy to test

Give controls accessibility labels and identifiers. The agent reads the accessibility tree first. A button with a label is found by name; an icon without one has to be found in the screenshot. In SwiftUI, .accessibilityIdentifier(_:) "uses the string you specify to identify the view" (Apple's documentation)), and the same identifiers serve XCUITest later.
Use a separate dev build. A distinct bundle identifier and data directory for the dev build keep the agent away from your real data and your installed copy.
Make state resettable. A launch argument or environment variable that starts with empty data lets the agent repeat a test from the same starting point.
Tell it how to quit and relaunch. Rebuilding while the old instance runs is a common source of confusion; say which command to use.

Pitfalls

Custom-drawn views

Views that draw their own content without accessibility elements, such as canvases, games or code editors, are only visible in the screenshot. The agent can still work from it, but clicks by coordinates are less precise than clicks on named elements. Code editors such as VS Code do not expose their text through accessibility, so agents read them from the screenshot.

Drag and drop

Drags inside your app, such as sliders, canvases and selection rectangles, work. Dragging files, tabs or text between windows or apps is a system drag that follows the real mouse, and ShadowCursor cancels it instead of dropping. Test file import through your app's menu or open panel, or open files with open -a YourApp file.

Hover

Hover without a pressed button reaches Safari and WebKit content, not Chrome or Electron apps. If you test an Electron app, hover effects need the real mouse.

System prompts

The first launch of a new build can trigger macOS permission prompts for your app. Grant those yourself once, so the agent's test run does not start with a system dialog.

Window state

Minimized and hidden windows cannot be seen or operated; covered windows can. Leave the app's window open behind your editor rather than minimizing it.

Typing in the same app

If you click into the app while the agent tests it, the agent waits for your pauses. That is useful for a quick look, but for a clean test run, leave the window to the agent.

FAQ

Can Claude Code test a Mac app's GUI?

Yes. It builds and launches the app from its terminal, and with a computer use tool it clicks through the windows and reads screenshots and accessibility elements. Its built-in computer use hides your other apps during a turn; with an MCP server such as ShadowCursor it works in a background window.

Does this replace XCUITest?

No. An agent finds problems by exploring; XCUITest makes sure they stay fixed, in CI, without an agent. Use the agent to explore and to write the XCUITests for what it found.

Does it work with SwiftUI and AppKit apps?

Both expose accessibility elements, which is what the agent reads, so standard controls work in either. Custom-drawn views are read from the screenshot.

Can I keep coding while it tests?

Yes, with ShadowCursor. The agent gets its own cursor and keyboard and works in your app's window in the background; its keys arrive only in that window, not in your editor.

Can Codex or Cursor do the same?

Yes. ShadowCursor works in any MCP client, including Codex, Cursor and VS Code, and several agents can test at the same time, each with its own cursor.

What does ShadowCursor cost?

There is a 7-day free trial without a card, then $12.99 per month or $69.99 per year, with unlimited agents on up to three devices. It needs macOS 15 Sequoia or later.