Learning signal from real, open-ended work

RealCUA is a rapidly growing dataset of open-ended computer-use trajectories with interwoven workflows spanning multiple apps, contexts, and thousands of steps towards real, useful work.

At Fig, we’re developing a new approach to environments. Instead of hand-building one scripted task at a time, we start from real, open-ended work and turn it into signal agents can learn from: real starting states, overlapping objectives, and outcomes that can be checked.

request data 11h+ continuous trajectory sample

Academic or open-source research group?
‍We’d love to support your work. Reach out for access.

Desktop activity decomposed into workspace state, actions, and workflow timelines.
600+
hours of
open-ended work
27,000+
app
transitions
12h+
longest continuous
trajectory
87%
of hours from
recordings 1h+
35
apps covered

RealCUA v0.1

Trajectory viewer

Loading the trajectory viewer…

Browsers
Brave Browser, Google Chrome, Dia, Firefox, Chromium
Communication
Slack, Mail, Superhuman, Messages, Discord, Zoom
Coding & AI
Cursor, Terminal, Ghostty, Claude, ChatGPT, Zo
Knowledge & docs
Notion, Notes, TextEdit, Pages, Granola, Numbers, Preview
Design & media
Figma, CleanShot X, QuickTime Player, Spotify
Desktop & planning
WindowManager, Notion Calendar, Finder, KeePassXC, Activity Monitor, Calculator, Todoist
Recording
H.264 video at 30 fps, with multi-monitor support
Input alignment
Frame-aligned raw input events with sub-millisecond timestamps
Step-level actions
Discrete step-level actions reduced from raw input events in JSON
Window context
Active application and window metadata derived from the accessibility tree
State
Accessibility trees, pruned HTML, and DOM snapshots
Task segmentation
Decomposition of trajectories into segmented workflows and atomic tasks

what's in RealCUA

interwoven workflows

Multiple independent goals executed together using the shared context on screen, with 45+ app transitions per hour. Natural task sequencing, interruption, resumption, and completion with visible cues like notifications and load states.

full workspace states

Entire desktop provenance on live desktops that never reset, with multi-monitor support spanning a variety of resolutions, windows, and production apps. Demonstrated information transformations across app and window boundaries.

open-ended trajectories

Operators carrying out their own unscripted work: multi-thousand-step workflows spanning tens of hours without synthetic data, scripted instructions, or app clones.

RealCUA in context

UNITMultiple Interwoven Tasks
ScriptedOpen-ended, self-directedGOAL
Single Task

Work with Fig

RealCUA v0.1

Over 600 hours of real, open-ended computer work for training, evaluation, and research. Tell us what you're building and how RealCUA can help.

606.5 hfootage
512recordings
274,984 actions27,700 app transitions
Over 35 apps covered154 GB compressed
Download the RealCUA v0.1 Dataset Card

Custom data & environments

We’re developing new scalable approaches that turn real open-ended, multi-objective knowledge work into data and environments with consumable signals to train and augment frontier AI systems.

Tell us where your agent is failing, what it needs to do, and we’ll work with you to scope and build a solution.

  • Human demonstrations: Step-level actions in domains you specify.
  • Verifiable tasks: Tasks from real workflows, with success criteria and verifiers.
  • RL environments: Resettable environments with tasks and rewards.
  • Agent traces: Verified agent rollouts for fine-tuning and analysis.

What’s next for RealCUA

We're sharing early previews with select teams ahead of our upcoming releases. If you're deploying models into complex, open-ended environments, we want to hear from you.

  1. v0.1

    RealCUA

    Real, open-ended work

  2. next

    Roles

    Comprehensive occupation mapping signals for domain-specialized agents

  3. planned

    Projects

    Multi-workflow collaboration signals for role-based agent orchestration

Faq

Who can access RealCUA v0.1?

Academic teams, frontier labs, researchers, open-source contributors, and companies. Access is currently provided on request. Use the Request data form to describe your intended use; We'll be in touch as soon as possible, our current average is within 24 hours.

Why do I have to request access?

We want to better understand each team’s intended use of RealCUA to shape how we plan and release future versions, data products, and broader access.

What can I use RealCUA for?

Example uses include training computer-use agents, designing evaluations and benchmarks, and researching how people perform open-ended computer work. Recorded actions can be paired with preceding desktop states and interaction history. Continuous recordings also support studying how humans naturally switch, interrupt, and resume tasks.

Will RealCUA be released more broadly?

We plan to broaden access to RealCUA as we learn from teams using the v0.1 release. Their requirements will help shape future releases and how we share the data more broadly.

How is RealCUA different from other computer-use datasets?

Recordings capture operators performing real work without assigned tasks or scripted instructions. RealCUA preserves multiple interwoven workflows, evolving desktop states, notifications, pauses, and interruptions to unfinished work. Sessions last over 12 hours, with actions captured alongside their surrounding context rather than only within an assigned task’s boundaries.

How can I collaborate or contribute?

You can propose a research collaboration, share findings from using RealCUA, or request new domains, workflows, and agent failure cases that could inform future releases. Use the request form to outline the project and contribution or collaboration you have in mind.

Do you build custom datasets or other data products?

Yes. Custom projects are scoped around your required capabilities, environments, volume, and delivery needs. Products include human-recorded trajectories for specified applications and domains, verifiable tasks derived from real workflows, RL environments with tasks and rewards, and verified agent traces. We can already generate thousands of hours per month with the goal of reaching tens of thousands.

Who recorded the work in RealCUA?

US-based experts who are seniors in their field, working in across engineering, operations, product, and research roles at US companies. Every operator consented to the recording, use and distribution of their work.

Build with RealCUA

Access a growing dataset of real, open-ended computer work for training, evaluation, and research.
We'll be in touch as soon as possible, our current average is within 24 hours.