GUIDE · COMPUTER USE
Computer-use datasets: when do you need screenshots or trajectories?
A screenshot dataset teaches a model about a visible screen. A trajectory dataset connects observations, actions, and outcomes across a task. If you want a local model to find a button, screenshots with the right labels may address the immediate problem. If you want it to complete a workflow, you also need to consider what happens before and after that button is clicked.
For a small team, the first decision is therefore practical: what exactly should the model do? “Help with our software” is a useful starting conversation, but it is too broad to specify training data. “Find the export control on this screen” and “export the correct report and confirm it was saved” describe different jobs.
Start with a familiar workflow
Imagine a team that prepares a weekly inventory report. A person opens an application, selects a date range, chooses a warehouse, exports a file, and checks the result. A screenshot of the export menu can show where the export control is. It cannot, on its own, establish whether the selected warehouse is correct or whether the saved file contains the requested dates.
This is the distinction between grounding and task completion. Grounding means connecting a request such as “choose the warehouse menu” to an element in the current image. UGround is an existing research project specifically focused on visual grounding for GUI agents, with released training data and evaluation resources. That existing work is a reason to define a specialized need precisely, rather than assume GUI data is generally unavailable. UGround official repository, accessed October 17, 2026.
What belongs in screenshot data?
A useful screenshot example needs more than an image and a vague instruction. It should connect the instruction to a valid target, describe the coordinate system, and preserve the image dimensions. If a label identifies a clickable point, someone using the data needs to know whether its coordinates refer to the original image or a resized version.
For the inventory example, an instruction might be “Open the warehouse selector.” The associated label could identify the selector’s bounding box. Separate examples could cover a collapsed menu, an expanded menu, and a disabled control. Each screen should have a clearly defined interpretation; a disabled button should not silently be treated as an available next action.
Include variation that matters to the intended environment: screen size, theme, label wording, and relevant application states. More variation is useful only when labels remain correct and the scope stays connected to the actual task. Thousands of nearly identical screens may add less coverage than a smaller set of carefully chosen cases.
What does a trajectory add?
A trajectory records a sequence: the initial task, the visible state, an action, the resulting state, and subsequent steps toward completion. Depending on the intended format, it may also include timing, action arguments, and an explicit completion check. Those fields must match the model and training software that will consume them.
For the report workflow, a sequence could show the date selection, warehouse selection, export action, and confirmation. An unsuccessful export should be labeled as unsuccessful. A recovery sequence might show a missing required field, a correction, and a fresh confirmation. These are proposed example structures, not a description of an available Talven dataset.
Longer sequences create additional review questions. Does each action follow from the previous observation? Does the next image actually reflect that action? Is the final result checked independently of the model’s own statement? A plausible-looking sequence is insufficient if its state transitions are inconsistent.
Choose data around the failure
If your model repeatedly clicks the wrong small icon, start by examining grounding labels, image resolution, and confusing nearby targets. If it finds the icon but exports the wrong report, investigate task interpretation, state tracking, and outcome verification. Buying more screenshots would not automatically solve the second problem.
You can make this decision without designing an entire training pipeline. Collect a short list of representative failures, write what a successful result would look like, and note which information was missing. A technical collaborator can then connect that description to an appropriate data format and evaluation plan.
Local use also needs a compatibility check. The model must accept the intended visual inputs, the software must understand the labels, and the available hardware must support the chosen workflow. A dataset download does not establish any of those requirements by itself.
Define a first dataset request
Describe one application family, one kind of work, a few important variations, and a concrete pass condition. Specify whether you need target locations, action sequences, recovery examples, or a combination. Keep a separate set of realistic test cases so you can check whether the data helps beyond the examples used for training.
Talven makes specialized training data for local models and exploring custom dataset requests. The computer-use examples described here are proposed use cases; availability, formats, rights, and validation must be confirmed for a specific release. For custom work, Talven trains the model, tests it on examples it never saw, and delivers it ready to set up on your computer; otherwise training uses your own tools. API access to purchased datasets is planned.
Start with the workflow and where your model gets stuck; a technical specification can follow. See also how to evaluate computer-use data beyond the happy path and the dataset FAQ.