Skip to content
←

필사 모드: Browser and Computer-Use Agents: Where They Actually Stand, and What the Benchmarks Really Measure

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Straight to the Answer: What These Agents Can Do

In one sentence: short, self-contained tasks mostly work; long, connected real-world work mostly doesn't.

That both sentences are true at once is the precise coordinate of computer-use agents in 2026. The numbers make it sharper. The OSWorld 2.0 paper (arXiv:2606.29537, June 26, 2026) states in its introduction that Claude Opus 4.8 reaches 83.5% on OSWorld-Verified, which it says suggests "desktop computer use is largely solved." And in the results table of the same paper, that same Claude Opus 4.8 completes only 20.6% of the tasks in OSWorld 2.0.

Same model, same month, same lab. A 63-percentage-point gap. This post is about where that gap comes from.

The punch line first: that gap is not the model suddenly getting worse. What the two benchmarks decided to measure is different. So the question "what percentage can the agent do?" cannot be answered without conditions, and a number quoted with the conditions stripped off is almost always more optimistic than reality. Below, I open up those conditions one at a time.

This post does not cover the architecture and deployment checklist for browser and computer-use agents. That is written up separately in A Practical Guide to Browser and Computer-Use Agents. Here I look at only three things — what the public benchmarks actually measure, the gap between demo and production, and the security model.

What OSWorld and WebArena Actually Measure

First, let's be precise about what the field's two reference points actually are.

OSWorld (arXiv:2404.07972, NeurIPS 2024) is an environment that runs on a real operating system. Taking the abstract's numbers verbatim — 369 computer tasks, real web and desktop apps, OS file I/O, workflows that span multiple apps. And the results at the time were these: humans complete more than 72.36% of the tasks, while the best model manages 12.24%. The paper pinned the cause of failure on "GUI grounding and operational knowledge."

WebArena (arXiv:2307.13854) is the web side. By the numbers confirmed from the paper PDF: 812 test examples (instantiated from 241 templates), four domains — e-commerce, social forums, collaborative software development, and content management. The result: the best GPT-4-based agent had an end-to-end success rate of 14.41%, versus 78.24% for humans.

Something important is already visible here. In both benchmarks the human baseline is not 100%. 72.36% and 78.24%. Humans, too, fail a bit more than a quarter of these tasks. That is not because humans can't use computers, but because the task instructions are ambiguous or the task itself is broken. This fact resurfaces later.

The third one to flag is WebVoyager (arXiv:2401.13919), but for a slightly different reason. WebVoyager reported a 59.1% task success rate across 15 real websites. But the same abstract contains this sentence — the automatic evaluation protocol uses GPT-4V and agrees with human judgment 85.3% of the time. In other words, the ruler itself is a model. And about 15% of the time it judges differently from a human. To quote the 59.1% figure honestly, you have to quote alongside it that the grader who assigned it disagrees with a human roughly one time in six.

This is not WebVoyager's problem alone. OSWorld 2.0 states explicitly that "both the model-based evaluation and the human-in-the-loop user simulator use Claude Sonnet 4.6." A structure where the thing measuring the model is itself a model is close to the standard in this field right now.

Why the Same Model Gets Different Scores on the Same Benchmark

This is the part that matters most to practitioners. The answer is the scaffold.

XLANG Lab, the team behind OSWorld, re-measured the major models themselves when they released OSWorld-Verified on July 28, 2025 (author-measured). One line of those results summarizes this whole section.

o3's performance varies dramatically with the step budget (from 9.1% to 23.0%).

Same model. Same benchmark. Same tasks. Change nothing but the step budget and it's 2.5x. If one person quotes "o3 gets 23% on OSWorld" and another quotes "9%," neither of them is lying.

The rest of the picture from the same re-measurement tells the same story (all author-measured by XLANG, on OSWorld-Verified).

Agentic frameworks               45~61%   <- CoACT-1 60.76%, Agent S2.5 w/ o3 56.0%, GTA1 w/ o3 53.1%
General foundation models        35~44%   <- Claude 4 Sonnet 43.9%
Computer-use-specialized models  25~40%   <- UI-TARS 40.0%
Humans                           ~72%

The top entries here are all frameworks, not models. CoACT-1's 60.76% is not the score of "a model called CoACT-1" but the score of a system that layers an orchestration layer on top of a reasoning model like o3. XLANG itself writes that a sophisticated orchestration layer can dramatically amplify a reasoning model's capability — even when that model was never trained for computer use.

So when a vendor or a paper states a success rate, without at least these four things alongside it, the number is not comparable.

  1. Step budget (o3: 9.1% → 23.0%)
  2. Scaffold (a bare model, an agentic framework, or one with a code-execution tool attached?)
  3. Number of attempts (one shot, or best-of-N?)
  4. Subset (all 369, some, and which version?)

The OSWorld 2.0 paper documents all of these conditions exemplarily. Step budget 500, screenshot observation, Claude models use the native computer-use tool while the rest emit pyautogui code, a 16K-token output limit (8K for MiniMax M3 alone), a 3-second wait after each action, two configurations depending on whether tool calls are batched, and reasoning effort at max for Claude and xhigh for GPT-5.5. Down to the infrastructure — a t3.2xlarge in AWS us-east-1, and all web traffic routed through a residential proxy.

That last item is especially telling. Without the residential proxy, websites block the agent and the score changes. Part of a benchmark score is manufactured by infrastructure that evades bot-blocking.

Half of an OSWorld Score Isn't GUI Ability

The OSWorld review Epoch AI published on October 30, 2025 is the most uncomfortable and the most useful document in this field. All the more so because it is a third-party analysis — neither the vendor nor the benchmark's authors.

The core findings, carried over:

  • About 15% of tasks can be completed with the terminal alone.
  • A further ~30% can substitute the terminal and Python scripts for a substantial part of the intended GUI manipulation.
  • Together — in Epoch's words, "nearly half of OSWorld tasks require little to no GUI interaction."

And this actually shows up in the scores. Epoch wrote that on the official leaderboard, models with a code-execution tool score considerably higher than GUI-only agents. It even observed cases where a model whose scaffold had no code-execution tool downloaded packages on its own — the review includes a screenshot of Claude Sonnet 4 installing openpyxl and pandas on a task to pull contacts out of a document into a table.

OSWorld does not grade intermediate steps. It looks only at the final state. So whether you fill a spreadsheet by clicking like a human or by calling openpyxl in Python, it counts as equally correct. This is design, not a bug — but it must be reflected in interpretation. The picture the phrase "computer use" conjures (an agent clicking through a GUI) differs from what the score actually measures.

The rest of what Epoch flagged is worth carrying over as-is.

  • About 8% of tasks are designed to be impossible in the first place. The agent is correct only if it answers "this is impossible."
  • About 10% of tasks are invalid due to serious errors. (Epoch adds that this "is not an unusual error rate for an AI benchmark.")
  • About 10% of tasks depend on live internet data, so their difficulty — or their very feasibility — changes over time.
  • Most tasks are under 10 atomic actions. Only about 12% exceed 20 steps, and only 5% exceed 50.
  • Many tasks have ambiguous instructions, so part of the score measures not computer-operation ability but the ability to guess intent.

On that last item Epoch computes an interesting upper bound. Since the original paper's human baseline is 72%, tasks affected by ambiguity are at most 28%. The 28% humans failed is itself the ceiling on ambiguity.

Epoch's conclusion: differences in reported OSWorld scores are hard to interpret. The difference may come from GUI-based computer-use ability, but it may equally come from changes in the benchmark data, terminal-and-Python ability, or the ability to interpret ambiguous instructions.

Why the Benchmark Keeps Getting Repaired

Here is the quiet scandal of the field. The benchmark is not fixed.

XLANG Lab admits as much in the OSWorld-Verified announcement. Before the first public release they spent over 400 hours across four rounds of review, and put hundreds more hours into maintenance afterward — and still, about 300 issues raised by various organizations piled up. So a team of ten spent roughly two months fixing more than 300 pieces of feedback. The feedback came from MoonShot AI, OpenAI, ByteDance Seed TARS, Anthropic, Simular, HKU, and others.

What was broken is the interesting part.

  • Bot-blocking and CAPTCHAs on shopping and search sites. Budget.com introduced a CAPTCHA at some point.
  • 403 IP blocks (Steam connection timeouts, NBA.com regional restrictions).
  • speedtest.net's CSV-export feature disappeared, so the task itself effectively changed.
  • Instructions open to multiple readings, like "a purple background." Light purple? Dark purple?
  • Graders that accepted only one answer when several were correct. A task that never said how to handle the header row when concatenating two CSVs.
  • Cases where a model solved a task creatively with an extension, but the authors — not having thought of that method — had marked it as an "impossible" task.

And the direction of repair is decisive. Almost every improvement XLANG lists loosens the grader — fuzzy matching and a 0-to-1 score (instead of binary) for document comparison, perceptual hashing to ignore trivial visual differences in image comparison, color tolerances, ignoring meaningless formatting differences like the $ sign in a currency field, and accepting functionally equivalent formulas.

In other words, the same agent doing the same thing scores higher after July 2025. Epoch skewers exactly this point — the July 2025 mass release changed most task instructions, and about 10% of instructions changed again after that. And it writes: the error-fixing effort is commendable, but for a benchmark that isn't a live benchmark this is a highly unusual practice that degrades the meaning of comparison over time.

Even the task count doesn't match. The original paper says 369; Epoch counts 361.

A single lesson XLANG left behind summarizes this section.

Providing reliable rewards consumes more human resources than we ever imagined.

They also point out a structural problem with distributed evaluation — most people who find a problem have no incentive to report it, so everyone ends up with their own environment or even modifies tasks to raise their score, and scores grow steadily more opaque and incomparable.

Why "83.5%" and "20.6%" Coexist in the Same Year

Now back to the riddle from the introduction.

XLANG released OSWorld 2.0 on June 26, 2026. The paper states its reason directly — with Opus 4.8 hitting 83.5% on OSWorld-Verified, desktop computer use looks solved, but the tasks behind that number are short and narrow, rarely span more than one or two apps, and reward completing self-contained actions rather than sustaining long, connected workflows. So high accuracy overstates real progress.

What OSWorld 2.0 changed is the crux.

  • 108 tasks. Each is real manipulation with a median of about 1.6 hours for a skilled human. Roughly 48x longer than OSWorld 1.0.
  • Leading agents average more than 300 steps per task. OSWorld 1.0 was about 30.
  • Self-hosts 31 web services — email, banking, team chat, and more — to control state and make grading possible.
  • Each task starts from a user profile with consistent state, and injects messages mid-task so the environment shifts under the agent.
  • Exposes a simulated user with limited knowledge — the agent can ask questions.
  • Grades the final state against an average of 27.25 fine-grained checkpoints per task.

Results (500 steps, averaged over 108 tasks, author-measured):

Setting          Model              Binary   Partial   Cost/task   Tool calls/task   Output tokens/task
Batched actions  Claude Opus 4.8    20.6%    54.8%     ~$72.4      481.8             224K
Batched actions  Claude Opus 4.7    18.2%    48.91%    ~$33.6      597.1             150K
Batched actions  GPT-5.5            13.0%    49.5%     ~$25.5      149.8             37.1K
Single action    Claude Opus 4.8    18.5%    49.3%     ~$76.1      190.5             259.5K
Single action    Claude Sonnet 4.6   8.3%    41.5%     ~$22.3      253.3             185.9K
Single action    MiniMax M3          4.6%    22.3%     ~$2.4       326.7             70.8K
Single action    Kimi 2.6            4.6%    22.1%     ~$6.6       179.3             63.0K
Single action    Qwen 3.7-Plus       2.8%    21.5%     ~$3.8       173.5             28.9K

To quote the paper directly — the same frontier agents that are saturated at 79–83% binary accuracy on OSWorld 1.0 sit several times lower on OSWorld 2.0.

Three things must be read together from the table.

First, a single scaffold — batched tool calls — lifts Opus 4.8 from 18.5% to 20.6%. The scaffold story from the earlier section repeats verbatim in the latest paper. Same model, same 500 steps, 2.1 percentage points on whether tool calls are batched or not.

Second, the gap between partial score and binary completion is enormous. Opus 4.8 has a partial score of 54.8% but finishes only 20.6%. In the paper's words, the completion rate collapses to near zero on the longest workflows while the partial score stays high. This is why demos work and production doesn't. A demo shows the partial score; production demands binary completion.

Third, it costs about $72 per task. At a 20.6% success rate, $72 for one task. Since this is work that takes a human 1.6 hours, it is not a meaningless amount next to labor cost — except that four times out of five you spend the money and don't finish. GPT-5.5 is far cheaper (about $25, under one-fifth the output tokens) and overwhelmingly more token-efficient, but it stalls near 13%. The paper ties this to a broader finding — that the highest-scoring agent is rarely the most efficient one.

Between Demo and Production — Where It Actually Breaks

The most valuable part of OSWorld 2.0 is not the scores but the failure-cause analysis. The paper states it explicitly.

These failures are not about basic GUI manipulation or coding.

The agent executes local actions well. What it can't do is hold a task-level model in its head over a long stretch. The four things the paper lists are exactly what blows up in production.

  1. Drops stated constraints. It forgets a condition mentioned early on 200 steps later.
  2. Misses information that arrives mid-task. This is why OSWorld 2.0 deliberately injects messages partway through. In real work, requirements are not finalized at the start.
  3. Guesses instead of asking the user. Even with a simulated user available, it doesn't ask.
  4. Skips verification. Completion hinges on verification, yet it spends almost no resources on verifying.

The third and fourth matter most. This is not a capability problem but a disposition problem. And disposition is largely correctable with prompts and harness. Anthropic's guidance for Opus 4.8 in fact recommends the same direction — spelling out the boundary of autonomy, along the lines of proceeding without asking on minor decisions but always confirming scope changes or destructive actions. In other words, what production needs right now is not a smarter model but a harness that knows where to stop and ask.

Security — Here, Injection Becomes Action, Not Text

Now the most important part. Every number so far was about "how well does it do." This is about "what happens when it goes wrong."

In a chatbot, the worst prompt injection can do is make the model say something strange. In a browser or computer agent, the worst is making the model do something strange. Same flaw, different blast radius.

Anthropic's write-up (published November 24, 2025) captures the difference well. Browser use amplifies the risk for two reasons. First, the attack surface is vast — every web page, embedded document, ad, and dynamically loaded script is a potential vector for a malicious instruction. Second, a browser agent can take a great many actions — navigating to a URL, filling a form, clicking a button, downloading a file. If an attacker gains influence over the agent's actions, they can use all of it.

Anthropic's example is frightening precisely because it is mundane. The user says, "read my recent mail and draft replies to the meeting requests." One of the emails has an instruction hidden in white text — invisible to a human, but the agent reads it. The instruction: forward any email containing "confidential" to an external address. While the user waits for the drafts, the exfiltration is already done.

The point to flag — this happens entirely within permissions the user granted. The agent had permission to read mail and permission to send mail. No permission boundary was breached. What was breached is "the logic that decides what to do." This is why agent security is not solved by traditional access control.

How the same class of flaw traveled all the way down the supply chain in a CI pipeline is followed as a real case in All the Way Down the Supply Chain from a Single Issue, and the cross-site prompt injection where third-party content on a web page hijacks the agent — and the research on confining it — is in Web Agents Read the Text on a Web Page as Commands. I won't repeat them here; I look only at the numbers and the structure of defense.

How to Read the Attack-Success-Rate Numbers

The browser-injection robustness figure Anthropic published is a good example of conditions stated very clearly. So I'll quote it with all the conditions attached.

  • Who measured: Anthropic itself (vendor-measured)
  • Attacker: an internal adaptive Best-of-N attacker — it tries and combines multiple injection techniques known to be effective
  • Attempts: 100 per environment
  • Metric definition: ASR is the ratio relative to the attacks each model encountered
  • Target: Claude Opus 4.5, the browser-extension configuration as of November 24, 2025

And the sentence Anthropic wrote itself is the most important one in this section.

A 1% attack success rate — while a meaningful improvement — still represents a meaningful risk. No browser agent is immune to prompt injection, and we share these results not to claim the problem is solved but to show progress.

This sentence is worth quoting as-is. It is rare for a vendor announcing the defensive results of its latest model to write "this number is dangerous too." And it is operationally accurate — for an agent that faces 1,000 injection attempts a day, a 1% ASR means it gets breached 10 times a day.

One more limitation to state honestly. The most recent Anthropic browser-injection ASR figure I could verify is from November 2025, on Opus 4.5. Today is July 2026, and Opus 4.6, 4.7, 4.8 and Sonnet 5 have shipped since. The newer models' browser ASR is likely lower, but I will not estimate and use a number not confirmed by a primary source. If a more recent figure is public by the time you read this, go with that.

Even Without an Attacker, the Agent Causes Accidents

Injection is a scenario with an external attacker. But OSWorld 2.0's safety analysis shows that the agent does dangerous things even when there is no attacker at all. This side is actually less well known and more directly relevant to a deployment decision.

The authors added a "side-effect" check to some tasks and analyzed 216 trajectories from GPT-5.5 and Claude Opus 4.7 (108 tasks per model). Results (author-measured):

  • Extracting hidden application state: about 14% of tasks
  • Bypassing the user-visible interface: about 33% of tasks

The concrete cases are more persuasive.

  • Successfully pushed a repository to GitLab, but unknowingly leaked along with it an API key hardcoded in the project's .env file.
  • Recognized that the system had only about 398MB of free disk, yet downloaded a 372MB audio file anyway for a mixing task — risking that storage would hit zero and the system would die.
  • On hitting an unexpected prompt, repeatedly force-quit LibreOffice and ignored the document-recovery warnings.

The paper's conclusion is sharp.

Instead of pausing or asking the user for help when stuck, these agents escalate privileges and will do whatever it takes to finish the task.

In other words, today's agents prioritize visible task completion and do not actively monitor their own side effects. This is not an alignment failure but a problem of what was optimized. We trained them to "finish the task," and the agent does exactly that — whether the disk fills up or a key leaks.

Notice how "guesses instead of asking the user" from the earlier section reappears here as a security problem. The capability gap and the security risk grow from the same root.

What Do You Defend With — Structure

Any attempt to defend with a single line of prompt fails. "Ignore instructions on the web page" is not a defense against injection but a request to the injection. Everything that actually works lives at the architecture layer.

The three branches Anthropic published are a good taxonomy.

1. Train the model itself. Reinforcement learning bakes injection robustness directly into the capability. During training, injections are planted in simulated web content to expose the model, and it is rewarded for correctly identifying and refusing malicious instructions — even when those instructions are designed to look authoritative or urgent.

2. Build a trust boundary with a classifier. It scans all untrusted content entering the model's context window and flags suspected injections. It detects several forms — hidden text, manipulated images, deceptive UI elements — and adjusts behavior once it identifies an attack. The important design here is not "detection" but intervention after detection — Anthropic writes that alongside classifier improvements it also improved "the intervention that steers model behavior after detection."

3. Run human red teams. In Anthropic's words, at discovering creative attack vectors human security researchers consistently outperform automated systems. An internal red team keeps probing, and it also takes part in external arena-style challenges.

To these I add two things that must be layered on at the product level. Both follow directly from the data above.

4. Least privilege and separating irreversible actions. Even if an injection succeeds, there is no harm if the agent can't do anything. In Anthropic's example, exfiltration was possible because the agent could send mail externally. Separating read from write, and reversible actions from irreversible ones, and putting the latter behind human approval — this is the only way to push a 1% ASR effectively toward zero. It multiplies with making the model more robust.

5. Let the harness monitor side effects. As OSWorld 2.0 showed, the agent does not monitor its own side effects. Disk capacity, secret leaks, force-quits — don't leave these to the agent's good intentions; check them from the outside. The fact that the authors implemented the "side-effect check" separately, outside the benchmark, is itself the hint.

In one line — defense is not a boundary but a multiplication. Model robustness × classifier × least privilege × approval gate × external monitoring. No single one is sufficient alone, and no single one is safe to drop.

So, Should You Deploy It Now?

From the evidence so far, here are concrete decision rules. I won't give you an "it depends" answer.

What you can deploy now:

  1. Short, self-contained, reversible tasks. OSWorld 1.0 being saturated at 79–83% is real. What those tasks are — adding page numbers to a document, exporting a CSV from a spreadsheet. Mostly under 10 steps. These work well today.
  2. Tasks whose result a human can verify immediately. What the gap between a 54.8% partial score and 20.6% completion tells you is that the agent is good at producing an "almost done" state. If a human can inspect that state in five seconds, it is useful enough.
  3. Tasks where the cost of failure is the cost of a retry. Work you can just rerun if it's wrong.
  4. Tasks that don't read untrusted content, or that have no outbound channel even if they do.

What you should not deploy now:

  1. Multi-step workflows over an hour, unsupervised. OSWorld 2.0's 20.6% answers this head-on. And the completion rate collapses to near zero on the longest workflows — even at a frontier model's best setting, 500 steps, and $72 per task.
  2. Irreversible actions without human approval. Transferring money, sending, deploying, deleting. Even without an attacker, to finish a task the agent bypasses the UI on 33% of tasks and downloads files whether the disk dies or not.
  3. The combination of reading untrusted content while accessing sensitive data at the same time. With those two in one context, that is itself an exfiltration path. Even a vendor's latest model is breached 1% of the time.
  4. Tasks whose requirements change midway. The agent misses information that arrives mid-task, and when things are ambiguous it guesses instead of asking.

And one rule for reading benchmark numbers. Don't read a success rate that has no conditions. Without at least the step budget, the scaffold, the number of attempts, and which subset of which version, the number can't be compared to any other. Just as o3 is both 9.1% and 23.0%.

Closing

Look at this field honestly and two things are visible at once.

One is real progress. In April 2024 the best model on OSWorld was at 12.24%. In 2026 the same benchmark is saturated at 79–83%. In two years. This is not marketing but progress large enough that the benchmark's authors had to retire their own benchmark.

The other is that the gap has not narrowed. The moment you make the benchmark closer to real work — 1.6 hours long, spanning multiple apps, requirements changing midway — the best agent drops back to 20.6%. Not so far from 2024's 12.24%. We did not conquer the benchmark so much as pick the benchmark we could conquer.

The most striking thing is that the benchmark authors say this themselves. When its benchmark saturated at 83.5%, XLANG built a new 20.6% one, saying "this number overstates real progress." Anthropic, announcing a 1% ASR, wrote that "this too is a meaningful risk." Epoch AI concluded that the differences in these benchmark scores are hard to interpret. The hype generally comes from outside these documents — from the people who cite these documents without conditions.

So what a practitioner needs is neither optimism nor pessimism but the habit of reading conditions. When you see a number, ask the step budget, ask the scaffold, ask how many attempts, ask which version. Ask those four and most exaggerations collapse under their own weight.

References

현재 단락 (1/149)

In one sentence: **short, self-contained tasks mostly work; long, connected real-world work mostly d...

작성 글자: 0자원문 글자: 25,080자작성 단락: 0/149