Robot learning needs more scalable ways to turn real-world experience into usable training data. Video offers an abundant record of human-object interaction, task structure, and physical context, but it is usually unstructured and difficult to connect to simulation or policy learning.

From Human Videos to Dexterous Robot Capability is a half-day, hands-on CoRL 2026 tutorial on converting unconstrained human-object videos into validated robot-training data. The tutorial motivates Video to Data as a practical bridge from raw demonstrations to actionable robot learning assets, showing how modern vision-language models, reconstruction tools, and robotic grounding can help convert everyday video into structured data for learning generalizable robot skills.

How can raw human video become robot-usable data?

We teach the pipeline as a sequence of validation gates: queryable video segments, plausible reconstructions, simulation-ready task assets, retargeted motions, and policy-training artifacts. Participants will leave with runnable scripts or notebooks, precomputed artifacts, and an adaptation checklist for deciding whether their own video data is ready for robot learning.

Tutorial repository: github.com/nvidia-isaac/video_to_data

By the end of the tutorial, participants will be able to:

• Make video queryable: segment long videos, build entity graphs, store frame embeddings, and retrieve task-relevant clips with natural-language queries.
• Inspect reconstructions: evaluate depth, object masks, meshes, hand/body reconstructions, 6D object poses, and temporal alignment for geometric plausibility.
• Convert to robot-facing data: map reconstructed interactions into motion files, object assets, URDF/USD paths, support surfaces, contact fields, and quality checks.
• Connect to dexterous policy learning: run or inspect a small Isaac Lab training/evaluation example and understand how tracking, contact rewards, and rollouts use demonstration-derived data.
• Diagnose failure modes: identify poor video quality, mask errors, mesh-scale mistakes, drifting poses, invalid retargeting, infeasible support geometry, and weak contact grounding.

The tutorial is a guided coding session: short conceptual introductions followed by runnable exercises. Because full reconstruction and policy training can require GPUs, every exercise has three paths: laptop-only inspection, optional live GPU execution, and precomputed artifacts so every attendee can complete the objectives.

• Code-first pacing: each module begins with a brief explanation, then moves quickly to an exercise.
• Paired debugging: participants compare retrieved clips, overlays, pose tracks, motion-schema fields, and simulation rollouts.
• Validity decisions: groups classify outputs as ready, questionable, or unsuitable for robot learning.
• Live demos: instructors run selected reconstruction overlays and a short Isaac Lab dexterity run.
• Exercise checkpoints: each exercise produces a visible artifact such as a JSONL clip list, retrieved clip, mesh/pose overlay, retargeted motion file, scene viewer, or policy rollout.