A humanoid learns to catch thrown boxes using only its own camera

Reflex gets a full-size humanoid to catch boxes thrown by people, using only the RGB-D camera on its head: no motion capture, no markers, no reflective objects. The trick is to split the problem into three separately trained stages and pass a compact 8-number “box latent” between them, so the vision model never has to learn to move a body. In simulation the robot catches 85.7% of throws. On a real Unitree G1 it catches 65% at 2.5 m, compared with 85% for human catchers.

• 85.7% vs. 87.9%: catch rate from the camera vs. from perfect, privileged box state, so vision costs only 2.2 points
• 65% / 40% / 15% real-robot catch rate at 2.5 / 5 / 7 m, which is 76% and 59% of human success at the first two distances
• 35.6 points lost if the vision model sees one frame instead of a short history
• 69 ms median camera-to-control latency, against a box that arrives about one second after release
• 25% end-to-end success on catch, carry 7.8 m, hand to a person, and walk back

Three stages, each with its own training signal

Catching a thrown object needs three things at once: whole-body coordination, prediction under latency, and perception from a moving egocentric camera. Training all three end-to-end with reinforcement learning through a renderer is slow and hard to learn. Reflex trains them one at a time and freezes each before moving on.

Stage 1 trains a state-based teacher policy with PPO on a 29-joint G1. It sees the box’s position and velocity, but corrupted with noise, dropouts, and delay, then cleaned up by a Kalman filter. Rewards for approaching, holding, and keeping the box are combined with tracking of retargeted human motion and an adversarial motion prior, so the robot catches the way a person would rather than flailing.

Stage 2 swaps the raw box state for a learned 8-dimensional latent. An encoder maps a 26-dim delayed box observation (position, velocity, size, orientation, spin, visibility, age) into that latent, and the policy is fine-tuned to act on it. The latent is where the model’s sense of “where the box is going” lives.

Stage 3 is the only part that touches pixels. A small CNN finds the box in a 192×144 RGB-D frame, and a GRU reads the last 25 frames plus the robot’s own head motion and regresses the latent. The body controller is never retrained; vision just has to produce the right 8 numbers.

This split pays off in compute too. Without rendering, Stage 2 PPO runs at about 28k environment steps per second on one RTX 3090, finishing 59M steps in roughly 35 minutes. With RGB-D rendering in the loop it would take about 11 hours.

What makes the vision stage work

The ablations show that the vision model is solving a prediction problem, not a detection problem. Every component that helps it track the box over time matters:

Two results stand out. First, predicting the latent beats predicting box state by almost 19 points. The latent was trained to be what the controller needs, so errors in directions the controller ignores cost nothing. The regression loss is even weighted by how sensitive the decoder is to each latent dimension. Second, the gap widens on harder throws. At a 0.6 m lateral offset, dropping temporal history falls from 66.6% to 26.8%, and dropping ego-motion falls to 44.0%.

The learned pipeline also beats the obvious engineering baseline. A YOLO detector feeding the state-based policy, which is the team’s previously deployed perception stack, averages 32.3% across box sizes and 67.2% on centered throws. Excluding the 5 and 7 m throws that lie beyond the vision training data, the visual latent beats the state-based detector pipeline in all 13 simulated conditions, by 5.0 to 15.6 points.

Latency randomization is the other quiet win. With 40 ms of added delay, the visual controller still catches 83.0% of throws; at 80 ms, 60.9%. A version trained without latency randomization drops to 71.7% and 50.8% at the same delays.

On the real robot

On hardware, with 20 throws per condition, the G1’s success falls off with distance much faster than a person’s:

Box properties matter too. The robot catches 65% of regular-size boxes but only 20% of small ones and 45% of large ones; by payload it gets 45% at 0.5 kg, 60% at 1 kg, and 30% at 2 kg. It also caught 8 of 20 throws aimed to its left or right.

Once a catch settles against the chest, a supervisor hands control to a pretrained walking controller (SONIC). In 20 full trials, the robot caught the box 65% of the time, carried it 7.8 m 35% of the time, handed it to a recognized person 30% of the time, and walked back 25% of the time.

Caveats

• Long throws are outside the vision training data. Stage 3 was trained on 3.0 to 3.5 m throws. At 5 m in simulation, vision gets 43.0% vs. 79.7% with privileged input. Adding 2.5 to 7 m throws to Stage 3 raises simulated 5 m success to 75.8%, but that version was not tested on hardware.
• Small real-world samples. Twenty trials per condition means Wilson intervals up to about ±20 points. Some simulation ablations (for example, removing DAgger: 86.2% vs. 85.2%) are within seed noise.
• The controller is size-blind. Boxes outside the reference size are often intercepted (86.1% for large boxes in simulation) but not held (39.7%).
• Latency was measured on the old pipeline. The 69 ms median (p95 90 ms) comes from the detector-based stack, which already exceeds the 20 to 60 ms randomized in training.
• Sim rendering is idealized. Motion blur, auto-exposure, and structured depth artifacts are not modeled.
• One task. Everything is catching human-thrown boxes on one robot.

The broader lesson is about where to put the interface between perception and control. Reflex never asks vision to output a physically meaningful box state. It asks vision to output whatever the controller already learned to act on. That makes the hardest, slowest-to-train piece (the renderer-in-the-loop stage) a supervised regression problem rather than an RL problem, and it is why a 2.2-point vision tax in simulation was achievable at all.