There is a category confusion baked into the phrase "AI agent that uses a computer." Most of the tools people reach for when they say that — browser-use, Stagehand, Playwright MCP — don't use a computer. They use a browser, and specifically they use the browser's DOM: a structured, queryable tree where every button and text field has an address. When one of those agents decides to "click Save," it resolves that intent to a CSS selector and dispatches a synthetic event. The page hands it a map.
Computer-use agents don't get a map. They get a photograph.
Two families, split by what they're allowed to see#
The distinction the site has drawn conceptually before is worth making concrete, because it dictates the entire shape of the open-source stack. A pixel-driven agent captures a screenshot, feeds it to a vision-language model, and the model has to answer a genuinely hard question: where, in this bitmap, is the thing I should click? The output isn't a selector. It's an (x, y) coordinate, which the agent then hands to the OS to move the real mouse.
Everything that makes computer-use agents powerful and everything that makes them hard flows from that one fact. Powerful, because a photograph is universal — you can screenshot a native settings panel, a desktop installer, a terminal, a fifteen-year-old line-of-business app with no API, and the agent can drive all of them identically. There is no DOM for any of those, which is exactly why browser agents can't touch them. Hard, because turning "click the blue Submit button" into a coordinate accurate enough to actually land on it is a research problem with a name: visual grounding.
A browser agent is handed the page's structure for free. A computer-use agent has to reconstruct that structure from pixels every single frame. Grounding is the tax it pays for going everywhere.
Read the serious repos in this space with grounding in mind and they stop looking like six competitors. They look like four different jobs on one assembly line.
The model that grounds#
The heaviest bet is to train a model whose native job is grounding, and ByteDance's UI-TARS line is the biggest of them.
The important word is native. UI-TARS doesn't bolt grounding onto a general chat model with prompting; the model is trained end-to-end to take a screenshot plus an instruction and emit an action with coordinates. The desktop app is the shrink-wrap — a thing you can install and watch drive your machine — but the reason it's the anchor of the category is the model underneath, and the fact that you can self-host an open vision-language model to do this at all is the whole point of the open-source stack existing.
The parser that makes grounding cheap#
The opposite bet is: don't retrain a model to see coordinates — pre-process the screenshot so any model can. That is the entire reason OmniParser exists.
OmniParser is not an agent. It's the perception layer you slot underneath an agent. It runs a detection model to find every interactable region and a captioning model to describe what each one does, then hands the vision-language model a labeled map instead of a raw bitmap — turning "find the Save button in this image" back into "pick element #17," the easy version of the problem. If UI-TARS makes the model better at grounding, OmniParser makes the grounding problem smaller before the model ever sees it. Both are valid; they're just opposite ends of the same lever.
The loops that plan and act#
Above the perception layer sit the full agent frameworks — the ones that plan a multi-step task, ground each step, act, observe the new screenshot, and correct.
Agent-S and self-operating-computer bracket the design space. Agent-S is the research-grade framework with planning and memory, benchmarked against OSWorld and its peers; self-operating-computer is the ~200-line loop you read in one sitting to understand what "operate a computer" actually compiles down to. If you're learning the category, start with the small one, then graduate to the framework once you feel where the naïve loop breaks.
The floor you run it on#
There's a fourth job, and it's the one people skip until the first bad click. A pixel-driven agent moves the real mouse. A mis-grounded coordinate doesn't throw an exception — it clicks whatever is genuinely at that spot, which might be a "delete account" button or a system dialog. You do not want that agent loose on your actual desktop.
Cua is to computer-use agents what E2B and Daytona are to code-execution agents: the disposable environment that bounds the blast radius. It isn't an agent and doesn't want to be — it's the OS-level sandbox you point the others at. Treat it as mandatory, not optional; the container-isn't-a-sandbox lesson applies with extra force when the untrusted actor controls your mouse.
One that refuses the premise#
Worth ending on the outlier, because it inverts the whole model.
OpenAdapt's bet is that most desktop automation is the same task repeated, so paying a vision-language model to re-ground every element on every run is wasteful and flaky. Record the workflow once, compile it to something deterministic, and only invoke the model when the UI has actually drifted enough to break the replay. It's a quiet rebuke to the whole live-grounding stack: the cheapest grounding is the grounding you don't redo.
So the decision isn't "which of these six is best." It's "which of the four jobs am I missing." Need a model that grounds? UI-TARS. Need to make grounding cheap for the model you already have? OmniParser. Need the planning loop around it? Agent-S, or self-operating-computer if you want to read the whole thing. Need somewhere safe for it to click? Cua, always. And if your task is the same every time, OpenAdapt argues you shouldn't be grounding live at all. Star counts sort these repos by popularity. The assembly line sorts them by what they're for.
