Green Challenge

WBC Dataset (Whole-Body Control)

Overview

A dataset in LeRobot v2.1 format. Designed for training VLA (Vision-Language-Action) policies for mobile manipulation tasks.

Robot	Green
Control mode	Absolute joint positions (joint control)
Cameras	3 × RGB 448×448 (head, left wrist, right wrist)
Tasks
Format	LeRobot v2.1 (Parquet + H.264 video + JSON metadata)

Directory Structure

dataset/
├── meta/
│   ├── info.json                          # Data schema, dimension descriptions, statistics, joint names
│   ├── tasks.jsonl                        # task_index → task name
│   ├── subtasks.jsonl                     # Subtask segments per episode (start/end frames, skill, description)
│   ├── episodes.jsonl                     # Episode metadata (index, length, task, variation, labels, masks)
│   └──episodes_stats.jsonl               # Per-episode normalization statistics (min/max/mean/std)
├── data/
│   └── chunk-{000..NNN}/
│       └── episode_{000000..NNNNNN}.parquet   # Trajectory frames (state + action + metadata)
└── videos/
    └── chunk-{000..NNN}/
        ├── observation.images.cam_head/
        ├── observation.images.cam_left_wrist/
        └── observation.images.cam_right_wrist/
            └── episode_{000000..NNNNNN}.mp4   # H.264 video, 448×448

Metadata Files

info.json

Top-level dataset descriptor. Contains:

• codebase_version — LeRobot format version (e.g. "v2.1")
• robot_type — robot model identifier
• total_episodes, total_frames, total_tasks, total_videos, total_chunks
• fps — video frame rate
• splits — train/val split ranges (e.g. {"train": "0:6"})
• data_path, video_path — path templates with {episode_chunk} and {episode_index} placeholders
• features — dict of all data columns with dtype, shape, names; video features include video_info sub-object
• joint_names — mapping of feature key → list of joint names (for state/action features)

episodes.jsonl

One JSON object per line, each describing an episode:

Field	Type	Description
episode_index	int	Sequential index (0-based)
episode_id	int	Globally unique episode identifier
tasks	[str]	List of task descriptions
task_index	int	Index into tasks.jsonl
length	int	Number of frames
zarr_id	int	Source zarr file identifier
zarr_start_idx, zarr_end_idx	int	Frame range in source zarr
is_problem_episode	bool	Flag for problematic episodes
variation	str	Variation identifier
variation_name	str	Human-readable variation name
labels	[str]	Configuration labels (e.g. wrist type, camera position)
action_config	[obj]	Action segments with start_frame, end_frame, action_text, skill, is_negative_episode, is_mirrored, is_reversed, uses_task_text
initial_scene_text	[str]	Initial scene description(s)
state_valid_mask	[bool] × 51	Which state dimensions are valid for training
action_valid_mask	[bool] × 52	Which action dimensions are valid for training

episodes_stats.jsonl

Per-episode normalization statistics. Each line contains episode_index and a stats dict. For each feature key, stats include:

• min, max, mean, std — computed over all frames of the episode
• count — number of frames (per-dimension for vector features)

subtasks.jsonl

One JSON object per episode, containing episode_index and a subtasks array. Each subtask has:

• start_frame, end_frame — frame range
• descriptions — list of natural language descriptions
• skill — skill name
• is_negative_episode, is_mirrored, is_reversed — augmentation flags
• uses_task_text — whether task text conditioning is used

Data Schema

Each .parquet file contains one episode as a table where each row is one frame.

Core Features

observation.state — 51-dim float32

Proprioceptive state of the full humanoid body.

Index	Range	Group	Joints
0–11	[0:12]	Legs	L_hip_pitch, L_hip_roll, L_hip_yaw, L_knee, L_ankle_pitch, L_ankle_roll (×2 for right)
12	[12]	Torso yaw	torso_yaw_joint
13–17	[13:18]	Left arm	L_shoulder_pitch, L_shoulder_roll, L_shoulder_yaw, L_elbow_pitch, L_elbow_yaw
18–22	[18:23]	Right arm	R_shoulder_pitch, R_shoulder_roll, R_shoulder_yaw, R_elbow_pitch, R_elbow_yaw
23–24	[23:25]	Neck	neck_yaw, neck_pitch
25–28	[25:29]	Wrists	L_wrist_roll, L_wrist_pitch, R_wrist_roll, R_wrist_pitch
29–40	[29:41]	Fingers	L/R pinky, ring, middle, index, thumb_pitch, thumb_yaw (6 per hand)
41	[41]	Body roll	torso_roll (from odometry)
42	[42]	Body pitch	torso_pitch (from odometry)
43	[43]	Root height	torso_z (meters)
44	[44]	Torso yaw (dup)	torso_yaw (redundant copy)
45–50	[45:51]	Velocities	lin_vel_x, lin_vel_y, lin_vel_z, ang_vel_x, ang_vel_y, ang_vel_z

action — 52-dim float32

Absolute joint position targets. Dimensions 0–50 match observation.state, with one extra entry:

Index	Value
0–50	Same as observation.state dimensions 0–50
51	episode_progress — float [0, 1], progress within the episode

observation.images.{cam_head, cam_left_wrist, cam_right_wrist}

H.264-encoded MP4 video, 448×448 RGB. Frame rate is dataset-dependent (defined in info.json → fps).

Auxiliary Features

Additional scalar/index features stored per frame in the parquet files:

Feature	Type	Description
observation.robot_command_35x.position	float32 (35,)	Robot command position vector (dimension varies by robot type)
timestamp	float32 (1,)	Frame timestamp in seconds
frame_index	int64 (1,)	Frame index within the episode
episode_index	int64 (1,)	Episode index
episode_id	int64 (1,)	Globally unique episode identifier
index	int64 (1,)	Global frame index across all episodes
task_index	int64 (1,)	Task index (→ tasks.jsonl)
task_name_id	int64 (1,)	Task name identifier
init_scene_text_id	int64 (1,)	Initial scene text identifier
skill_id	int64 (1,)	Skill identifier
action_text_id	int64 (1,)	Action text identifier
loss_mask_index	int64 (1,)	Loss mask configuration index
rosbag_timestamp	float32 (1,)	Original rosbag timestamp

Note: The exact set of auxiliary features may vary between datasets. Refer to info.json → features for the authoritative schema.

task

Natural-language instruction per episode, e.g. "Box carry", "Box carry onehand". Loaded from tasks.jsonl via task index.

Quick Start

import pyarrow.parquet as pq
import numpy as np

# Load a single episode
pq_path = "dataset/wbc/data/chunk-000/episode_000000.parquet"
table = pq.read_table(pq_path)
df = table.to_pandas()

state = np.array(df["observation.state"].tolist())   # (N, 51) float32
action = np.array(df["action"].tolist())              # (N, 52) float32

# Extract joint groups
legs = state[:, 0:12]       # legs
torso = state[:, 12:25]     # torso + arms + neck
fingers = state[:, 29:41]   # fingers
wrists = state[:, 25:29]    # wrists
velocities = state[:, 45:51] # velocities

# Access auxiliary data
timestamps = df["timestamp"].values
episode_ids = df["episode_id"].values