Computer-use agent datasets: a buyer’s guide

A computer-use agent dataset is a collection of task trajectories. Each trajectory shows an interface being operated, step by step, to accomplish an explicit goal. For example, “update the shipping address on this order” or “export the report as a PDF.” The dataset is what teaches a model the observation → action → next-observation loop that a browser or desktop agent runs at inference time.

The distinction that matters for buyers is between footage and trajectories. A screen recording alone shows what happened; a trajectory pairs that recording with structured, timestamped UI action logs and the page state behind each frame, anchored to a goal and a reviewed outcome. The structured form is easier to use for supervised action prediction, reward modeling, and step-level evaluation than footage alone.

Usable trajectories combine several aligned streams. Missing any one of them limits what a model can learn or how you can evaluate it.

Screen recordings

Continuous video of the desktop or browser viewport, timestamped so every frame aligns with the action and observation streams. Recordings preserve the visual context an agent must reproduce: window layout, modal dialogs, loading states, and transient toasts.

Screenshots and screen states

Discrete screen states let a model learn the observation → action → next-observation loop without decoding video. Paired before/after frames make the effect of an action explicit and support supervised action-prediction objectives.

UI action logs

Machine-readable action records form the input side of the trajectory. Each event carries an action type, a target (coordinates and/or an element reference), and a timestamp, so trajectories can be replayed, filtered, and audited rather than only watched.

DOM and accessibility-tree snapshots

Serialized DOM and platform accessibility trees give a text-grounded view of the interface: roles, names, values, and hierarchy. Accessibility-tree training data lets agents target elements semantically instead of guessing pixel coordinates, which transfers far better across resolutions and themes.

Task and goal metadata

Every trajectory is anchored to an explicit goal, a task category, and an environment descriptor (operating system, application, locale, viewport). This is what turns a screen recording into a supervised example rather than raw footage.

Success labels and QA review

Outcome labels such as success, partial, and failure can support reward modeling and evaluation in addition to imitation. Programs can include reviewer notes, QA fields, and PII-handling requirements defined in the collection specification.

Screenshots and UI action logs are the minimum. DOM and accessibility-tree snapshots are what let an agent target elements semantically. Accessibility-tree context supports structured targeting across resolutions and themes alongside pixel-based supervision.

Compare each source’s published footprint with the environments you deploy on, the locales your users work in, and the difficulty range your agent will face. Include long multi-step and cross-application tasks when those are part of your product.

Under the hood, a trajectory record anchors a goal to an ordered array of steps. Each step should carry a typed action (click, type, scroll, keypress), a resolvable target expressed as both coordinates and, where available, an element reference, a timestamp, and a pointer to the observation captured at that step. That structure is what makes a trajectory replayable and lets you score an agent step by step rather than only on final success.

A published schema is a strong signal. It lets your team validate a sample programmatically instead of eyeballing recordings. Our reference schema and an in-browser inspector are linked at the end of this guide.

Labels turn footage into training signal. Look for human-verified outcome and success labels, recorded QA tiers, and a documented review methodology, ideally with a way to spot-check a sample against the stated quality bar. Failure and recovery trajectories are valuable, not defects: they are what reward models and robust policies learn from, so a dataset that only contains clean successes is quietly narrower than it looks.

On-screen PII redaction should be applied and recorded. Ask how redaction is verified, because unredacted personal data in screenshots is both a privacy risk and a training-data hazard.

Provenance questions often surface during procurement or model-release review, so they are worth asking first. For each record you want documented contributor consent, commercial-use terms, collection dates, and environment provenance. Data collected or licensed for training is very different from recordings scraped without terms, even when the raw modalities look identical.

Datoric’s licensing review includes chain-of-custody and contributor consent documentation, commercial license scope, and privacy records for the selected delivery. See the Computer-Use Traces Dataset for the current specification.

Review the published data rights and provenance, quality methodology, and licensing pages for the controls referenced during dataset review.

Open browser-agent trajectory releases are a good way to prototype and to calibrate expectations. Production programs usually need coverage control, schema fit, documented licensing, and labels defined to an acceptance bar. That is where a build-to-spec commercial program earns its cost. We cover the tradeoffs neutrally in a dedicated guide.

Commercial vs open GUI-agent data→

These primary sources show how prominent open computer-use datasets and benchmarks define task coverage, environments, and evaluation. Source details were last verified on July 21, 2026.

• Mind2Web: Towards a Generalist Agent for the WebNeurIPS 2023 Datasets and Benchmarks Track, 2023. An open dataset of more than 2,000 tasks across 137 websites and 31 domains, with crowdsourced action sequences.
• WebArena: A Realistic Web Environment for Building Autonomous AgentsWebArena authors, 2023. A reproducible web environment and benchmark for long-horizon tasks across realistic self-hosted websites.
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer EnvironmentsNeurIPS 2024 Datasets and Benchmarks Track, 2024. A benchmark of 369 tasks spanning real web and desktop applications, operating-system file I/O, and multi-application workflows.

The lowest-risk path is to validate before you commit: get a representative sample, check it against your loader and an evaluation checklist, run a bounded pilot with labels and QA applied, and measure downstream lift on your own tasks before scaling.

1. 01Scope. We align on task categories, environments, locales, modalities, and the trajectory schema your training or evaluation pipeline expects.
2. 02Sample. You receive a representative review package to validate against your loader and the evaluation checklist before any volume commitment.
3. 03Pilot. A bounded batch is collected to the agreed specification, with labels, QA, and redaction requirements applied so you can evaluate it on your own tasks.
4. 04Scale. Once the pilot clears your acceptance bar, collection scales to the target volume and cadence with the agreed QA and provenance records.