One Brain, Many Bodies: Humanoid Autonomy Stack
A self-hosted vision-language model plans; a compliant whole-body controller walks the Unitree G1 to a bench and fetches a block.
- Walk, pick and place
- 17 of 20 runs
- Falls while walking
- 0 in 30
- Plan and control rate
- 1 / 500 Hz
Read the full write-up
Cross-embodiment humanoid autonomy stack in MuJoCo, driven by a self-hosted vision-language model (Qwen2.5-VL-3B, no cloud APIs) and a compliant whole-body controller. The VLM plans at ~1 Hz via tool-calling with Set-of-Mark visual grounding; the controller executes at ~500 Hz. The same brain drives Unitree G1 and H1 through one tool interface (only a per-robot config file changes); 22-DoF five-finger hands composed onto the H1 at load time via MjSpec. The G1 walks about 3 m to a bench, stops, picks a block and places it in a bin in 17 of 20 runs, using GPU-trained PPO walk-and-stop and stance policies trained on the states and arm motions the task really produces. Every policy is scored against human motion: torso lean while manipulating fell from 36° to 1.7°, foot lifts while standing by 97%, the gait runs at about 2 steps/s (human 1.9) with 0 falls in 30 walks, and the arm follows minimum-jerk paths.
Built with Python, PyTorch, MuJoCo, Qwen2.5-VL, PPO, tool use, Set-of-Mark, admittance control, differential IK, Unitree G1 and Unitree H1.
