NVIDIA Cosmos-H-Dreams: Bringing Real-Time Generative Simulation to Surgical Robotics
huggingface ·
NVIDIA introduced Cosmos-H-Dreams, a real-time, action-conditioned generative simulator for surgical robotics, built by distilling the Cosmos-H-Surgical-Simulator world model into a causal, few-step student model. The simulator runs on a single NVIDIA RTX PRO 6000 GPU and is served through NVIDIA's FlashDreams inference library, achieving interactive operation at roughly 160 frames per second. It accepts an initial RGB frame and a stream of robot kinematics, generating surgical video frames autoregressively; interfaces include browser, Meta Quest, and learned policy controls. Training involved fine-tuning a teacher on dVRK tabletop suturing data that includes failures and out-of-distribution episodes, followed by causal warmup and self-forcing distillation to stabilize long rollouts. The system is designed to be extensible to user-specific embodiments through provided recipes. Future directions include closed-loop benchmarks for action accuracy and stability, as well as applications such as latency-aware telesurgery and surgical rehearsal, though it is not intended as a diagnostic or physical robot controller.
huggingface