Computer-Use Models Can Click Any App, but Pixels Slow Them Down

A five-step invoice entry exposes the difference. The worker opens a vendor record in an old web application, types an amount, selects a date from a custom calendar, attaches a PDF, and presses Submit. There is no supported API, and the calendar is drawn on a canvas rather than built from normal browser controls.

A computer-use model can attempt the whole job because it receives screenshots and returns actions such as moving the pointer, clicking coordinates, typing text, or pressing a key. Anthropic and OpenAI document versions of this screenshot-action loop: the developer runs the software, sends the current screen to the model, executes the requested action, captures the result, and repeats.

That makes automation without a dedicated integration newly practical. It does not make every click equally reliable.

In the invoice fixture, the amount field is easy for both pixel and structure-aware systems. The calendar favors pixels because its days are visible but absent as individual page elements. Submit is the dangerous step: a screenshot can show two similar buttons, while a browser tool reading the page structure can identify the element whose accessible name is Submit and verify that it is enabled before acting.

One screenshot can replace an integration

Traditional automation needs an interface it can address. An API exposes named operations, while a browser automation framework such as Playwright can inspect the Document Object Model, or DOM, the browser’s structured representation of elements and attributes on a page. Both approaches let software request a specific thing rather than estimate where that thing appears.

Computer use removes that prerequisite. If a person can see an application through a remote desktop and operate it with a mouse and keyboard, a model can potentially operate it through the same surface. The application might be a legacy purchasing portal, a Windows utility reached through virtualization, or an internal tool whose original developers are gone.

For the invoice job, that means the calendar no longer blocks the entire workflow. A DOM controller may see one canvas element where the operator sees a grid of dates. The pixel model sees the rendered numbers, chooses the requested day, and clicks its apparent position. Rebuilding the calendar or persuading the vendor to add an API is no longer the first setup step.

The developer still has work. The model needs an isolated browser or virtual machine, a defined screen size, permission boundaries, screenshot capture, action execution, and a loop that reports each changed screen. Anthropic’s documentation tells implementers to run computer use in a sandboxed environment and limit access to sensitive data. OpenAI’s documentation similarly describes executing the model’s requested computer actions and returning an updated screenshot, with human confirmation around consequential operations.

This is broader compatibility, not zero integration. The integration has moved outward, from each application’s internal interface to a controlled desktop around the application.

Pixels pay for reach with extra turns

A structure-aware browser tool can often fill the invoice form through direct element operations. It locates the vendor input, sets the value, finds the amount field, uploads the file through the page’s file input, and targets the named submit control. The page may still react between operations, but the tool does not need a new image merely to discover where every field moved.

The pixel loop is more expensive in sequence. It captures a screen, sends image data to a model, waits for an action, executes that action, and captures another screen whenever the model needs to inspect the result. A five-step business task can require more than five model turns because opening a menu, waiting for a modal, correcting a mistyped field, or dismissing an overlay each creates another observation and decision.

No single latency figure applies across vendors, models, network locations, and screen resolutions. The mechanism is enough to make the direction clear: repeated image processing adds model calls and image input, while a DOM script can issue several deterministic operations inside one browser session. Computer use therefore makes the unsupported invoice portal automatable, but it is a poor replacement for a stable API when throughput matters.

Screenshots also enlarge the ambiguity budget. The model has to infer whether a gray button is disabled, whether a tiny icon means download or delete, and whether a loading overlay has disappeared. Coordinate errors can arise after the window resizes, a banner pushes content downward, or a remote desktop scales the image differently from the action surface.

The invoice calendar demonstrates the favorable case because the target date has a visible label. A row of unlabeled toolbar icons is worse. So is a table with two Submit buttons, one for the invoice and another for a side panel. DOM access can distinguish the buttons by parent container, attributes, or associated text; pixels must infer the intended target from layout and context.

The accessibility tree is the useful middle layer

The accessibility tree is the semantic view an operating system or browser exposes to assistive technology, with roles and names such as button, checkbox, or Invoice amount. It can give an agent more meaning than a screenshot without requiring the application to publish a business API.

On a conventional web form, the tree may expose the invoice amount as a text box with a label and Submit as a button with a name. That lets the controller address controls semantically, even when their CSS layout changes. Desktop applications can expose comparable information through platform accessibility frameworks.

Coverage depends on the software. A canvas calendar is often opaque unless its developer added accessible descendants or a parallel control. A custom component may expose the wrong role, omit its name, or combine an entire interactive region into one node. The tree reflects what the application declares, not everything a sighted operator can infer from the rendered screen.

This changes the sensible order of attack. Try the most structured interface available, then fall back toward pixels only where structure ends. For the invoice fixture, a hybrid controller can use DOM or accessibility selectors for the vendor, amount, file input, and Submit button, while reserving screenshot control for the canvas date picker. That keeps the one capability pixels add without paying the screenshot loop across every field.

The same rule separates interface classes. DOM control is strongest on well-formed web pages. Accessibility access can cover browser chrome and desktop software that expose useful semantics. Pixel control earns its cost on remote desktops, streamed applications, canvas-heavy design tools, charts, maps, and old interfaces whose controls exist visually but not structurally.

Treat the final click differently

Target ambiguity matters most when an action cannot be quietly retried. Typing the wrong amount can be detected by reading the field back. Pressing Submit may create a duplicate invoice, send a message, place an order, or commit a record that requires another workflow to reverse.

Vendor guidance for computer-use systems emphasizes isolation, allowlists, and human confirmation for high-impact actions. Those controls should attach to the action, not to a vague judgment that the whole application is risky. The invoice agent can navigate, populate fields, and attach the PDF autonomously, then stop before submission and present the extracted vendor, date, amount, file name, and intended button for approval.

A structured tool can add stronger checks before that handoff. It can confirm that the amount field’s current value matches the source document, that the chosen vendor identifier is the expected one, and that the button belongs to the invoice form rather than a neighboring panel. A pixel-only agent can inspect the screen, but its evidence remains another model interpretation of an image.

Logs should preserve both sides of the loop: the screenshot or structured state the model received and the action the executor performed. Coordinate clicks without their source image are hard to audit, while screenshots alone do not prove where the click landed. For sensitive workflows, record the model request, action coordinates or selector, resulting screen, and approval event, subject to the organization’s retention and privacy rules.

The adoption decision is therefore narrower than whether computer use works. Use it when the invoice portal’s missing integration is the binding constraint, especially at low volume or during a migration. Keep DOM automation or an API for stable, repetitive work where speed and deterministic targeting justify the setup. If one canvas widget blocks an otherwise structured flow, do not hand the entire desktop to pixels.

Questions people ask

Can computer-use AI operate software with no API?

Yes, if the software can run in an environment where the developer can capture its screen and execute mouse or keyboard actions. The model still needs a desktop harness, permissions, and safety boundaries, so it removes the application-specific API requirement rather than all integration work.

Is

DOM access always more reliable than screenshots?

No. DOM access is usually more precise for standard web controls because it addresses named elements instead of coordinates, but it cannot inspect visual objects drawn inside an opaque canvas. Broken markup, changing selectors, cross-origin frames, and custom components can also force a fallback to accessibility data or pixels.

When should a human approve an agent’s click?

Require approval when a click commits a purchase, submits a record, sends external communication, changes permissions, or triggers another hard-to-reverse action. In the invoice workflow, the agent can prepare the form autonomously, but the approval screen should show the vendor, amount, date, attachment, and exact action awaiting execution.