DatoricDataset evaluation checklist & schema inspector
Two tools for evaluating computer-use agent data before you commit: a buyer checklist covering schema, coverage, annotation, and licensing; and an in-browser inspector that checks a trajectory sample against a reference schema. The inspector runs entirely in your browser. Nothing you paste is uploaded.
By Datoric · Last updated July 23, 2026
Buyer checklist
What to verify in any sample
Schema and structure
- Each trajectory anchors to an explicit goal and task category.
- Steps are an ordered array with a typed action and a resolvable target.
- Actions carry both coordinates and an element reference where available.
- Observation streams (screenshot, DOM, accessibility) align to steps by timestamp.
- A published schema or JSON Schema is available to validate against.
Coverage and difficulty
- Environments span the applications and operating systems you deploy on.
- Locales and languages match your target markets.
- Trajectories include long, multi-step, and cross-application tasks.
- Failure and recovery trajectories are present, not only clean successes.
- Task categories are balanced enough to slice and sample.
Annotation and QA
- Outcome and success labels are human-verified, with QA tiers recorded.
- Inter-reviewer agreement or a QA methodology is documented.
- A sample can be spot-checked against the stated quality bar.
- Redaction of on-screen PII is applied and recorded.
Licensing and provenance
- Contributor consent and commercial-use terms are documented.
- Collection dates and environment provenance are available per record.
- Data is collected or licensed for training, not scraped without terms.
- Redaction and privacy handling meet your compliance requirements.
Reference schema
Fields a trajectory record should carry
A neutral reference for a single computer-use trajectory. Required fields are the minimum for supervised training and step-level evaluation; recommended fields enable grounding, reward modeling, and audit.
| Field | Scope | Importance | Purpose |
|---|---|---|---|
task_id | record | required | Stable identifier for the trajectory, used for dedup and joins. |
goal | record | required | The natural-language instruction the trajectory fulfils. |
task_category | record | required | Category label enabling balanced sampling and slicing. |
environment | record | recommended | Descriptor of OS, application, locale, and viewport so results are reproducible. |
steps[] | record | required | Ordered array of observation/action records for the task. |
outcome | record | required | Final result of the trajectory (e.g. success / partial / failure). |
success_label | record | recommended | Human-verified boolean or score for reward modeling and eval. |
qa | record | recommended | QA tier, reviewer, and notes documenting how the label was reached. |
provenance | record | recommended | Consent, licensing, collection date, and redaction status for the record. |
steps[].action | step | required | Action type per step (click, type, scroll, keypress, …). |
steps[].target | step | required | Coordinates and/or element reference the action applies to. |
steps[].timestamp | step | recommended | Timestamp aligning the action with observation streams. |
steps[].screenshot | step | recommended | Reference to the screen state captured at the step. |
steps[].accessibility | step | recommended | DOM or accessibility-tree snapshot enabling semantic, resolution-independent grounding. |
Schema inspector
Check a sample against the schema
Paste a single trajectory record or a JSON array. The inspector reports which reference fields are present on the first record and first step. It is a structural aid, not an acceptance decision, and it never leaves your browser.
Runs in your browser. Nothing is uploaded.
Want a sample to run through this?
Request a representative computer-use trajectory sample and check it against the schema and checklist above.