so101_bench_quickstart

This repository packages an existing research project, SO-101 Bench, into a ready to run Docker setup so it can be launched with a single command instead of manually configuring a complicated simulation environment. SO-101 Bench itself studies how well AI models that combine vision, language, and robotic action, sometimes called vision language action models, can understand spoken instructions and then carry them out with real physical precision using a small robot arm called the SO-101.

The underlying benchmark recreates a tabletop scene with a plastic bin and dozens of household objects inside Isaac Lab, a physics simulator, so that the same four tasks used in a real world robot study can be tested safely and repeatably in software instead. The four tasks are placing an object in a bin, placing an object next to another one, placing an object between two others, and moving an object in a stated direction. Objects are split into ones the AI model has already seen during training, ones from a familiar category but a new specific object, and ones from a completely new category, to test how well the model generalizes.

According to the original paper's real robot results included here for background, the AI model performs well on simple, familiar tasks but struggles noticeably once tasks require precise spatial reasoning, multiple objects, or unfamiliar items, suggesting that these models currently understand language better than they can physically execute on what they understand.

This particular repository adds the Docker packaging, evaluation scripts for running the GR00T and MolmoAct2 AI models against the simulation, tools for controlling the robot with an Xbox controller, replaying recorded data, and scoring results, along with a document listing the setup fixes needed to get the original project running cleanly from a fresh clone.