Human-level on the benchmark, can’t click OK on a MacBook

Oct 4, 2026 · Technical

Agent S is one of the best open-source computer-use agents you can run today. The latest version scores 66% on the OSWorld benchmark on its own and 72.6% with its best-of-N trick, which is around human level. I spent a week inside its codebase and shipped two fixes for issues that had been open for months. Neither touched a model or a prompt. Both were in the small amount of plain Python that sits between the models and the screen, and both were invisible on the benchmark.

That last part is what I want to write about. A benchmark is a very specific environment, and code that is correct inside it can be quietly wrong everywhere else. Here's how I found the two bugs, what caused them, how I fixed them, and what the benchmark was hiding. This is a companion to last week's post on three fixes to other computer-use tooling, and the shape of the lesson is similar.

What Agent S is and how it works

A computer-use agent is a program that gets things done the way a person does: look at the screen, decide what to do, move the mouse, type. I've written before about how these agents are really three separate things, and Agent S is a clean example. Under the hood it is a loop with two models in it.

• The main model is a big general-purpose model that can see images. It looks at the screenshot and the task and writes one high-level action, like "click the Firefox icon in the dock." It never produces coordinates.
• The grounding model is a smaller model trained specifically to look at a screenshot and a description and answer with a point. Agent S recommends UI-TARS-1.5-7B from ByteDance, usually self-hosted on something like a Hugging Face Inference Endpoint.
• Between the two sits a small amount of plain Python that encodes the screenshot, sends it to each model, and converts the grounding model's answer into real screen pixels.

Both of my fixes live in that glue. That is part of why the bugs were still open: the code looked too simple to be wrong.

Picking the issues

I wanted an issue that was high impact, affected real users, and had nobody working on it. The repository had 22 open issues with no assignee, but "no assignee" turned out to be a weak filter. Almost every meaningful one already had a pull request open from the person who reported it.

The two in bold stood out. Issue #140 was the most-discussed open issue in the repository, with nine comments from four separate people on Linux and Windows, a maintainer guessing at the cause, and nobody fixing it. Issue #175 had a PR against it, but that PR ran the grounding model several times and took the median, which smooths over noise rather than fixing a bug. Nobody had asked why the coordinates were off.

Bug 1: the request that was too big to send

PR: simular-ai/Agent-S#228 · Issue: #140

The symptom

People running UI-TARS on their own server got this on the very first step of every task:

🔄 Step 1/15: Getting next action from agent...
Response success!
...
(Grounded Action)
agent.click("The Firefox application icon ... at the very top of the left-side dock", 1, "left")

Attempt 1 failed: Failed to buffer the request body: length limit exceeded

Notice the order. The main model answered fine ("Response success!"). The failure happened on the next call, the one to the grounding model. One user said it also happened on the main model call when that model was self-hosted too.

A maintainer replied that it sounded like a problem with the user's UI-TARS setup and suggested trying a hosted provider instead. Later they added: "it might be screenshot being too big for request body?" That was the right instinct, but the thread went quiet.

Finding the cause

The error text is very specific, so I searched for it. "Failed to buffer the request body: length limit exceeded" is the exact response that Hugging Face's text-generation-inference server (TGI, the program that serves the model) returns, with HTTP status 413, when a request body is larger than its PAYLOAD_LIMIT. That limit defaults to 2,000,000 bytes. Hugging Face Inference Endpoints run that server, and so do many self-hosted setups.

So the question became: how big is the request Agent S sends?

Agent S captures a screenshot, resizes it so neither side is longer than 2400 pixels, saves it as PNG, base64-encodes it, and puts it inside the JSON body. Base64 is the standard way to put binary data inside text, and it makes anything about a third bigger. I couldn't take a real screenshot from my terminal, so I generated a desktop-like image: a 1920x1080 canvas full of text with scattered noise, which is roughly how a busy desktop compresses.

1920x1080 desktop-like PNG      1.90 MB
after base64                    2.53 MB
inside the JSON request body    2.53 MB   ← over the 2 MB limit

That's it. A normal 1080p screenshot of a normal desktop produces a request body that is over the limit before any model sees it. Clean, sparse screens compress better and squeeze through, which is why the bug felt random to the people hitting it.

The request flow, before and after

The fix

I wanted something that was correct, did not change behaviour for anyone who was not affected, and gave people a clear path when they hit the error.

A size budget for images. I added a helper, compress_image_bytes(image_bytes, max_bytes), that takes an encoded image and a byte budget. If the image already fits, it is returned untouched, byte for byte. If it doesn't fit, the helper tries progressively cheaper encodings until one does.

The order matters. Lowering JPEG quality costs very little visual fidelity on UI screenshots, but shrinking the image costs real detail, and the grounding model needs detail to find small icons. So quality goes first and resizing is a last resort.

Applying it everywhere images are sent. The class that builds messages for every model, LMMAgent, now reads an optional max_image_bytes from the engine parameters and runs every image through the helper before base64-encoding it. That covers single images, lists of images, and messages that get replaced later in the trajectory. Because it lives in one place, it applies to both the main model and the grounding model.

Honest MIME types. While doing this I noticed the data URLs were hardcoded as data:image/png;base64,... regardless of what the bytes actually were. Once the helper can produce a JPEG, that lie would matter. The code now detects the type from the bytes and labels it correctly, both for OpenAI-style data URLs and for Anthropic's media_type field.

A flag and a hint. The command line gets --max_image_mb, which is off by default so existing users see no change at all. And the retry loop, which previously printed "Attempt 1 failed" three times and gave up, now recognises this particular error and tells you what to do:

Attempt 1 failed: Failed to buffer the request body: length limit exceeded
Hint: The endpoint rejected the request body as too large. This usually means
the screenshot exceeds the server's payload limit (HuggingFace TGI defaults
to 2 MB). Re-run with --max_image_mb (e.g. --max_image_mb 1.0) or raise the
server's payload limit.

Why not just always compress?

I thought about making compression the default. I decided against it. The maintainers benchmark against hosted providers with no body limit, and silently switching everyone's screenshots from PNG to JPEG would change the inputs to their benchmark numbers. An opt-in flag with a loud hint fixes the problem for the people who have it and changes nothing for everyone else. That is an easier change to accept.

Tests

The helper is a pure function, so it is easy to test. The test file builds PNGs filled with random noise, because noise doesn't compress and so stays reliably over any budget I pick. It checks that:

• an image under budget is returned as the exact same object
• a None budget disables compression entirely
• an oversized image comes back under budget and still decodes
• quality reduction is tried before any resizing, by asserting the output dimensions are unchanged at a moderate budget
• resizing happens when it has to, with the aspect ratio preserved
• an impossible budget returns the smallest attempt rather than crashing
• LMMAgent emits PNG data URLs without a budget and JPEG ones with it, for single images, lists, and replacements
• the error hint fires on the TGI message and on a generic 413, and not on a 404

Bug 2: the clicks that landed in the wrong place

PR: simular-ai/Agent-S#231 · Issue: #175

The symptom

The issue is short and polite: "I found that the process of using Agent S was not very precise. Especially when dealing with smaller buttons." Someone else replied with an analysis blaming model randomness and proposed running the grounding model several times and taking the median.

That would help with noise. But a model that is consistently off in one direction does not get fixed by averaging. I wanted to know whether the offset was random or systematic.

How coordinates are supposed to work

The grounding model Agent S recommends, UI-TARS-1.5, is built on Qwen2.5-VL. Models in that family do not see your image at its original size. The image processor first runs a step called smart_resize, which rounds both sides to multiples of 28 pixels and scales the whole image down if it has too many pixels. The model then answers with absolute pixel coordinates in that resized image.

ByteDance documents this precisely, and ships the conversion back to your original image:

new_x = int(model_x / resized_width  * original_width)
new_y = int(model_y / resized_height * original_height)

The key point: the coordinate space depends on the image you sent. Send a 1470x956 screenshot and the answer is in roughly 1470x956 space. Send 2400x1350 and it is in roughly 2400x1350 space.

What Agent S did instead

Agent S asks you to pass --grounding_width 1920 --grounding_height 1080 for UI-TARS-1.5, and the conversion was:

x_screen = model_x * screen_width  / grounding_width    # grounding_width  = 1920
y_screen = model_y * screen_height / grounding_height   # grounding_height = 1080

This is only right when the screenshot you sent was 1920x1080. On OSWorld, the benchmark, the virtual machine is always 1920x1080, so the formula happens to be correct there, and every benchmark number ever published for Agent S was produced under exactly that condition. On a real laptop the screenshot is whatever the screen is.

Measuring the damage

I simulated an element at 90% across and 95% down the screen, which is where "OK", "Save", and dock icons tend to live, and worked out what UI-TARS-1.5 would answer for each screen size and where Agent S would then click. The "after" column comes from running the real OSWorldACI class with a mocked model response, not a reimplementation.

Three things jumped out.

1. On a MacBook, clicks were about 300 pixels off horizontally. The agent was aiming at the wrong window.
2. On 1440p and 4K monitors the click was off the screen entirely. pyautogui clamps to the screen edge, so the agent clicked the corner of the display every time it aimed at the bottom right.
3. Even on the benchmark it was 11 pixels off. The 28-pixel rounding means the model answers in 1932x1092 space, not 1920x1080. Eleven pixels is nothing for a big button and everything for a 16-pixel icon. That is the "smaller buttons" the issue was about.

The offset was not noise. It was a fixed scaling error that grew with distance from the top-left corner.

The fix

The principle is simple: convert coordinates back from the space the model actually answered in, not from a space we assumed.

Concretely:

Port the resize rule. I copied smart_resize and its helpers from the UI-TARS reference into Agent S, with the same constants (factor 28, minimum 100×28×28 pixels, maximum 16384×28×28 pixels). Now Agent S can compute the exact space the model answered in from the size of the image it sent.

Remember what was sent. generate_coords, the method that calls the grounding model, now records the size of the screenshot it is about to send. resize_coordinates uses that, passed through smart_resize, as the source space.

Handle both kinds of model. Not every grounding model answers in image pixels. UI-TARS-72B answers in a fixed 0 to 1000 space regardless of the image. So there are two modes:

• image: the model answers in pixels of the screenshot it saw. The grounding box is ignored for scaling.
• fixed: the model answers in a fixed grounding_width x grounding_height space. This is the old behaviour.

Pick the mode automatically. I did not want to force a new flag on every user. The grounding box people already pass contains enough information: a pixel coordinate space always has the same shape as the screenshot, so a square box like 1000x1000 for a non-square screen cannot be a pixel space. It must be a normalized one.

So auto, the default, treats a square box as fixed and anything else as image. Both configurations in the README resolve correctly with no changes, and a new --grounding_coordinate_space image|fixed flag exists for anything unusual.

Fix the text-highlight action too. While reading the code I found a sibling bug. The highlight_text_span action runs OCR on the screenshot and uses the word boxes it gets back as screen coordinates directly, with no scaling at all. That is correct only when the screenshot is exactly the screen size. On a 1440p monitor, where the CLI caps screenshots at 2400 pixels wide, every text drag was in the wrong place. I added screenshot_to_screen and routed those coordinates through it.

A note on the benchmark

This change moves OSWorld coordinates by up to 11 pixels near the right and bottom edges. That is the 28-pixel rounding being handled correctly for the first time, and I called it out plainly in the PR because the maintainers will want to see a benchmark run before trusting it. Being upfront about a small, explainable change to benchmark inputs is better than having them discover it.

Tests

tests/test_grounding_coordinates.py has fourteen tests. They build an OSWorldACI with a fake API key, mock the grounding model call, and check the end-to-end conversion for:

• smart_resize against the reference value for 1080p (1920x1080 → 1932x1092), the multiple-of-28 rule, and the upscale and downscale branches
• a 1080p screen with a 1080p box returning the centre exactly
• a MacBook screen where the model points at the bottom-right corner of its image, which must map to the bottom-right corner of the screen
• a 1440p screen whose screenshot was downscaled by the CLI
• a square 1000x1000 box being read as a normalized space
• each explicit mode overriding the detection
• an invalid mode being rejected
• the fallback before any grounding call has happened
• OCR coordinates scaling from screenshot to screen, and staying unchanged when the sizes already match

What both bugs have in common

Neither bug is visible on OSWorld. The benchmark runs on a 1920x1080 virtual machine and talks to models through hosted providers with generous limits. Under those conditions the screenshot is small enough to send and the coordinate formula happens to be right. Every published result is correct. The code just was not general.

Both bugs are in glue code, not in models or prompts. Request encoding and coordinate scaling are the kind of code that gets written once, works in the test environment, and is never looked at again because the interesting work is elsewhere.

And both had been diagnosed as something else. The payload error was blamed on users' endpoint setups. The coordinate offset was blamed on model randomness. In both cases the real cause was a sentence in someone else's documentation: Hugging Face's default payload limit, and ByteDance's description of their coordinate space.

How I worked

Read the issue thread, then verify every claim in it. The maintainer's guess that the screenshot was too big was right, but a guess in a comment is not a fix. Measuring the actual request body turned the guess into a number.

Reproduce before changing anything. For the payload bug, that meant building a realistic screenshot and measuring the encoded size. For the coordinate bug, it meant simulating what the model would return for five screen sizes and running it through the real conversion code. The before/after tables in both PRs came straight out of those scripts.

Write the test first. Each PR started with a failing test file. For the coordinate work in particular, writing the tests forced me to be precise about what "correct" meant for each screen size before I touched the conversion code.

Small commits with reasons in them. Each PR is five commits: helper, wiring, CLI flag, hint or related fix, docs. Every commit message says what changed and why, and references the issue. A reviewer can read the history and follow the reasoning without the PR description.

Change nothing for people who are not affected. Both fixes default to the existing behaviour wherever that behaviour was correct. The compression flag is off by default. The coordinate detection resolves the README's two documented configurations to the right mode with no flag changes. That is what makes a change to a popular repository easy to say yes to.

Say the uncomfortable thing in the PR. The coordinate fix shifts benchmark inputs by 11 pixels. I wrote that in a section called "Things worth knowing" rather than hoping nobody would notice.

The bigger lesson, for me, is about what a benchmark number means. Agent S at 66% on OSWorld is real and impressive. But the number describes one room: one screen size, one class of model host. The moment you take the agent out of that room and put it on a MacBook talking to a model you host yourself, two separate pieces of glue break before the first click lands. Most of the work of making agents useful is making them work outside the room, and a lot of that work looks exactly like this: unglamorous, in plain Python, and hiding behind an issue that everyone assumed was something else.

Appendix: numbers, commands, and references

Measured screenshot sizes

Desktop-like 1920x1080 image, text plus noise:

Hugging Face text-generation-inference default PAYLOAD_LIMIT: 2,000,000 bytes.

UI-TARS smart_resize constants

IMAGE_FACTOR = 28
MIN_PIXELS   = 100   * 28 * 28  =     78,400
MAX_PIXELS   = 16384 * 28 * 28  = 12,845,056

Examples of what the model actually sees:

Using the fixes

# Payload limit on a self-hosted endpoint
agent_s --provider openai --model gpt-5-2025-08-07 \
        --ground_provider huggingface --ground_url https://<endpoint>/v1 \
        --ground_model ui-tars-1.5-7b \
        --grounding_width 1920 --grounding_height 1080 \
        --max_image_mb 1.0

# Coordinate mapping is automatic. To force a mode:
agent_s ... --grounding_coordinate_space image   # pixels of the screenshot (UI-TARS-1.5)
agent_s ... --grounding_coordinate_space fixed   # fixed box, e.g. 0-1000 (UI-TARS-72B)

Files touched

References

• Agent S repository: github.com/simular-ai/Agent-S
• PR #228, payload size: github.com/simular-ai/Agent-S/pull/228
• PR #231, coordinate mapping: github.com/simular-ai/Agent-S/pull/231
• Issue #140: github.com/simular-ai/Agent-S/issues/140
• Issue #175: github.com/simular-ai/Agent-S/issues/175
• UI-TARS coordinate processing guide: README_coordinates.md
• UI-TARS-1.5-7B model card: huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B
• Hugging Face discussion of the 2 MB payload limit: discuss.huggingface.co