Computer Use Agents

Work

Extracting data trapped in a legacy application

Using an agent as a reluctant API for software that never had one.

What happens

  1. Navigate to the dashboard or report nobody can export from
  2. Read the values off the screen
  3. Page through until the dataset is complete
  4. Return the result as structured text or JSON

A neat inversion of the usual framing. The agent is not automating a workflow here — it is being used as an interface to software that was never given one. Read-only, which makes it the safest of the work cases: the worst outcome is a wrong answer rather than a wrong action.

Where it breaks

Pagination, and anything requiring a scroll to reveal the rest of the data. It is easy to get a confident, complete-looking answer that in fact covers only the first page — always check the row count against something independent.

Tools that suit this

Copilot Studio computer use Computer use tool inside Copilot Studio. Automates Windows desktop applications and websites for agents, billed per step. Claude Computer Use Screenshot-and-control toolset exposed through the Anthropic API. Moves the mouse, types, and reads the screen of a full desktop environment you host yourself. Skyvern Vision-driven browser automation aimed at workflows that break conventional selector-based scripts.

More work tasks

Work Invoice processing The canonical business case. Every organisation has a version of it, and the target system almost never has an API. Work Data entry into a system with no API Where computer use most clearly beats the alternative, because the alternative is a person doing it by hand.