Nvidia's 'dreamDojo' Robotics Code Is Riddled With Bugs
The official code for Nvidia's acclaimed dreamDojo robotics paper contains significant bugs that question the validity of its published results.
The released code for Nvidia's dreamDojo paper is broken. Researchers digging into the implementation found significant bugs across the entire pipeline, casting serious doubt on the paper's reported results, which were already marginal. This is another case of a high-profile paper failing the basic test of reproducibility.
What's broken in the dreamDojo code?
According to a detailed Reddit post, the issues aren't minor typos. The bugs affect the core stages of the model's development: pre-training, post-training, and even the final evaluation code. With the entire pipeline compromised, the reported metrics are likely not reproducible from the provided implementation.
Specific errors were found in the data loading and augmentation for pre-training, and the evaluation protocol apparently doesn't match the paper's description. When the core components for training and testing a model are flawed, the entire foundation of the research becomes unstable. You can't trust the results if the measurement system is wrong.
How does this happen?
Nvidia's dreamDojo was a spotlight paper at ICML, one of the top machine learning conferences. This incident shows that peer review primarily vets a paper's ideas, not its implementation. Reviewers rarely have the time or resources to run and validate complex research code, especially for a model trained with massive data and compute budgets.
The pressure to ship papers for major conferences is immense, and code is often cleaned up and released as an afterthought. Releasing the code at all is a step toward transparency, but it also opens the work up to scrutiny that the original review process missed. That public validation is exactly what happened to dreamDojo.
Should you build on this research?
No. Not in its current state. The dreamDojo paper claims to achieve improved results on robotics tasks using a world model, but the bugs in the code make those claims unverifiable. Until Nvidia addresses the issues, patches the repository, and provides a clear path to replicating their original results, building on dreamDojo is a liability.
This is a stark reminder to always validate research code before building on top of it, no matter who published it. Trust, but verify with `git clone`.