What is computer use, actually?
Computers have been driving other computers for a very long time. In 2004 a guy at ThoughtWorks named Jason Huggins got tired of clicking through the same web forms to test them and wrote a script that did the clicking for him. That became Selenium, and for the next fifteen years "a program that uses a website like a person" mostly meant a test suite. Then browsers learned to run without a screen. PhantomJS in 2011, headless Chrome in 2017, then Puppeteer and Playwright, and suddenly you could rent a hundred invisible browsers and have them fill out forms, scrape prices, and log into things at 3am. None of that was "AI." It was scripts. Every click was written by a human in advance, and the moment a button moved, the script broke.
I've been in that world for a while. I ran headless browsers for scraping and testing before it was cool, I've maintained more Playwright scripts than I want to admit, and when the first open-source personal agents showed up I had my own OpenClaw boxes running around the clock, poking at my email and calendar and breaking in new and creative ways. So when Anthropic shipped a Claude that could take a screenshot and click in October 2024 and called the feature "computer use," I was not a skeptic about the idea. The idea was obvious. The scripts were always the problem, and a model that looks at the screen and decides for itself is exactly what you'd want instead of a script.
I just didn't like the thing itself. That first version scored about 15 percent on OSWorld, the benchmark of real desktop tasks, where humans get around 72. It was slow, because every step was a full model call with an image attached. It was expensive for the same reason. It misclicked constantly, because it was guessing pixel coordinates off a screenshot with no idea what the buttons were actually called. It got lost in dropdowns. It fell for cookie banners. A task I could script in Playwright in ten minutes and run in two seconds took the agent three minutes and failed a third of the time. OpenAI's Operator came a few months later and had the same shape and the same problems, just in a browser. I remember thinking: this is the right idea implemented at the wrong moment. The models couldn't see well enough yet.
That has changed, and it changed faster than I expected. The models got dramatically better at looking at a screen and knowing where things are, partly because ByteDance released UI-TARS in early 2025 as open weights trained specifically for this, and everyone else had to keep up. The harnesses stopped relying on screenshots alone and started reading the accessibility tree, driving the browser through code, and typing shell commands whenever a terminal was in reach. Simular's Agent S crossed the human score on OSWorld in late 2025. And the computers the agents run on stopped being throwaway VMs and started persisting, which matters more than it sounds. The result is that in 2026 I've gone from "fun demo, not for real work" to actually handing an agent errands on my own Mac and walking away.
Which means it's finally worth explaining properly. The usual definition, an AI that controls a computer the way a human does, is true and unhelpful, because it describes the whole thing as one blob. The way to actually understand it is to split the blob into parts. So let's start.
It's three things, not one
A computer-use agent is a brain, a loop, and a computer. Three separate pieces. You can swap any one of them without touching the other two, and once you see that, most of the confusion goes away.
The brain is the model. It gets an image of the screen and some text about the task, and it produces a decision: click here, type this, scroll down, run this command. Claude does this. The latest GPT does this. ByteDance trained a model called UI-TARS to do exactly this and nothing else. You can even run a smaller open model like Qwen on your own hardware. The brain doesn't know or care what computer it's driving. It only sees pixels and returns actions.
The loop is the boring code that wraps the brain. Take a screenshot. Send it to the model. Get an action back. Do the action. Wait a beat. Take another screenshot. Repeat until the task is done or something goes wrong. That's it. People give this part fancy names like "agent harness" or "orchestrator," but it's a while loop with some error handling. Agent S, UI-TARS Desktop, Microsoft's UFO, and Open Interpreter are all open-source versions of this loop. You could write a crappy one in an afternoon.
The computer is whatever the loop is pointed at. This is the part people forget is a separate choice. It could be your actual MacBook. It could be a Windows laptop in an office. It could be a Linux virtual machine in someone's cloud. Companies like Orgo, E2B, Blaxel, and Daytona sell you this layer: a computer that appears in seconds, has a browser installed, and can be thrown away when you're done. They are not the brain. They are not the loop. They are the desk the agent sits at.
So when a company says "we're a computer-use company," the right question is: which layer? OpenAI and Anthropic mostly sell the brain, and both let you bring your own computer. Orgo and friends sell the computer. Agent S sells the loop for free. And some products bundle all three so tightly you can't see the seams, which is exactly why the phrase is confusing.
How the loop actually works
Let me make this concrete, because "screenshot, decide, act" sounds simpler than it is in practice. Say I tell an agent: open Chrome, download this month's invoice from Stripe, and drop it in the finance folder.
The loop takes a screenshot of the desktop. It sends that picture to the model along with my instruction. The model looks at the picture, spots the Chrome icon, and says "click at these coordinates." The loop moves the mouse there and clicks. Now it waits a moment, because Chrome takes a second to open, and screenshots again. The model sees the Chrome window, says "click the address bar, type stripe.com, press enter." The loop does that. Screenshot. Model sees the Stripe login page, or maybe it sees the dashboard because I was already logged in. And so on, one screenshot and one action at a time, until the file lands in the folder.
Two things about this are worth sitting with. First, each step costs a model call, and a model call with an image in it is slow and not cheap. A task that takes a human forty seconds might take an agent thirty steps and a few minutes. Second, the model only knows what's in the picture. If a popup appears that it wasn't expecting, it has to notice the popup and deal with it. If a page loads slowly and the screenshot catches it half-rendered, the model might click the wrong thing. The whole system is exactly as reliable as the model's ability to look at a screenshot and understand what's going on, which in 2026 is pretty good and still not great.
Not every loop is purely visual, by the way. The smarter harnesses cheat wherever they can. Instead of squinting at a screenshot of Excel, Microsoft's UFO talks to Excel directly through Windows' built-in accessibility hooks, so it knows the exact cell names and doesn't have to guess where a button is. Instead of clicking through a browser pixel by pixel, a lot of agents drive it through Playwright, which is a tool for controlling a browser from code. And when there's a terminal available, the agent will usually just type a command rather than click through six menus, because a command either works or it doesn't. Screenshots are the fallback for when nothing better exists, which is still most of the time on a normal desktop.
Your computer or a rented one
This is the choice I care about most, and it's the one most explainers skip. The loop has to point at some computer. There are really two options, and they trade off against each other in an annoying way.
Point it at your own machine. Run UI-TARS Desktop or Agent S on your Mac, and the agent sees your real screen and clicks your real apps. The upside is huge: all your stuff is already there. You're logged into Slack, Gmail, your bank, your company's internal tools. Your files are on disk. Your browser has fifteen years of cookies. The agent can just start working. The downside is also huge: it's operating your actual computer. If it misreads a screenshot and hits delete in the wrong place, that's your file. If it gets confused and starts typing into the wrong window, that's your Slack message to your boss. And you can't really use the computer while it's working, because you're both fighting over the same mouse.
Point it at a rented computer. Spin up a Linux desktop from Orgo or E2B, hand the agent the connection, and let it go. Now the upside and downside flip. It's safe, because the worst it can do is wreck a disposable VM. It's parallel, because you can spin up ten of them. It's repeatable, because you can reset to a clean state. But it's empty. No logins, no files, no cookies, no history. Every task that depends on "the state of Sahil's computer" now starts with a long, fragile setup phase where you try to recreate that state on a blank machine, and some of it (MFA, device trust, that one internal tool with the weird VPN) you may not be able to recreate at all.
Most of the interesting work in this space right now is really about this tension. The agent wants to be on your computer because that's where your life is. You want the agent on a different computer because you don't trust it with your life. Everything from browser extensions that reuse your logged-in Chrome, to snapshot systems that clone an already-configured machine, to persistent cloud desktops you log into once and keep forever, is an attempt to get both sides at the same time. I'll write more about that one separately, because it turns out to be the actual hard problem.
A word on the benchmark
OSWorld is a few hundred real tasks on a real Linux desktop: edit this spreadsheet, change this VLC setting, find this file, fix this thing in GIMP. It went from the teens in 2024 to past the human number by late 2025, and that jump is the whole reason this post exists. But the tasks are short and well-defined, and real work is long and messy. An agent that nails a two-minute task on a clean VM is a different animal from one that can run for an hour on your actual laptop without doing something you regret. Passing the benchmark is necessary. It is nowhere near sufficient.
What I'd tell someone starting out
If you want to actually understand this space, run one. Install Agent S or UI-TARS Desktop on your Mac, point it at whichever model you have an API key for, and give it a real errand. Something like: open my calendar, find next week's flight, and add the confirmation number to my notes. Watch it work. You'll learn more from watching it get confused by a cookie banner than from any blog post, including this one.
Then do the same task in a cloud desktop and notice everything you have to set up before the agent can even start. That gap, between what the agent could do on your machine and what it can do on a blank one, is the thing I keep coming back to. The brain is getting good fast. The loop is a solved problem. The computer is where all the unsolved stuff lives.