GUIDE · COMPUTER USE
How to evaluate computer-use data beyond the happy path
Computer-use data should be evaluated against the job you want a model to complete, including what happens when a step goes wrong. A model that clicks the correct control on a clean screen may still fail when a dialog appears, a field is invalid, or the application has not finished loading. Evaluation should separate finding a target, choosing an action, and achieving the requested result.
For a small team, this does not have to begin with a complicated leaderboard. Start with a few representative workflows, explicit completion checks, and a record of failures. That gives you a way to ask whether a proposed dataset addresses the problems that matter in your own environment.
Write a result a person can verify
Consider a local assistant preparing a draft customer record in a test application. Success is not “the assistant said it finished.” Success might mean that the right test name and address appear in the saved draft, no duplicate was created, and no live message was sent. The evaluation needs to inspect those conditions.
Define the starting state as carefully as the result. Use a known application version, test account, screen size, and initial record set. Otherwise, a difference between two runs might reflect a different environment rather than a difference in the data or model.
Use a sandbox with fictional records for this exercise. You want failures to be observable and repeatable, without turning a test into a change to someone’s real work. Your task definition should also state when the assistant must stop or ask for help.
Separate the layers of performance
First, check grounding: can the model locate the requested control? Then check action selection: does it choose an appropriate next step? Finally, inspect the end state: did the requested task succeed within the permitted actions?
ScreenSpot-Pro addresses professional, high-resolution GUI grounding. It is useful context for testing target localization, but a grounding result alone does not establish that a complete workflow will succeed. ScreenSpot-Pro official repository, released January 4, 2025; accessed October 17, 2026.
For longer tasks, examine environment-based evaluation projects as a separate category. OSWorld-V2 describes a benchmark for long-horizon real-world computer-use tasks. Its current repository includes version-specific setup and task access requirements, so an evaluation report should name the actual version and configuration used. OSWorld-V2 official repository, accessed October 17, 2026.
Add realistic recovery cases
A recovery case begins with a defined disruption. For example, the application rejects a postcode, displays a confirmation dialog, or returns to an earlier screen. The expected behavior should explain how the model can recognize the condition and what a valid response looks like.
For the draft customer record, a useful case might contain an invalid postcode. The assistant should identify the field-level error and request a correction if it cannot derive a valid value from the authorized input. Inventing an address merely to make the form accept it would be a failure, even if the save action succeeds.
Other cases should require stopping. A permission error is not an invitation to work around access controls. An ambiguous customer match may require a person’s choice. Recovery data should teach the distinction between a correctable interface problem and a situation where further autonomous action is inappropriate.
Keep test data independent
Do not evaluate only on screens or task templates included in training. Reserve different layouts, task instances, and relevant application states. Where possible, split related families together: moving near-identical examples into different splits can make generalization look stronger than it is.
Keep a small real-environment test set as well as synthetic checks. Generated examples can test defined variations, but they may miss timing effects, rendering details, or behaviors in the actual application. A result on generated examples should be reported as that result, without implying proven real-world transfer.
Compare the same starting model under matched conditions before and after a data change. Hold the task set, software, action limits, and evaluation procedure steady. If the training setup changes too, document that difference and avoid attributing the whole result to the dataset.
Report failures as well as successes
Record completed tasks, failed tasks, stops, invalid actions, and recovery outcomes. Count recovery success only among cases where recovery was actually required, and keep that denominator visible. Also record how many tasks needed human intervention and why.
For a small evaluation, raw counts are often more informative than a confident-looking percentage. “Eight of ten test tasks completed, with two ambiguous-record stops” tells a reviewer what was tested. It does not imply reliability across every workflow or customer environment.
Use the findings to refine the data request. You might need more invalid-field examples, clearer completion labels, or new layout families. You might instead discover a model or integration limitation that additional data cannot fix.
Talven makes specialized training data for local models, including proposed computer-use applications. Recovery coverage and performance would need to be demonstrated for each released dataset; no general improvement or cost advantage is established here. For custom work, Talven trains the model, tests it on examples it never saw, and delivers it ready to set up on your computer; otherwise training uses your own tools. API access to purchased datasets is planned.
Include one workflow, its failure cases, and the result you need to verify. See also screenshots vs trajectories and the dataset FAQ.