Jim Fan(@DrJimFan)

How it's done: https://t.co/DanOfnw4xk

Jim Fan (@DrJimFan) — Fri, 17 Jul 2026 16:11:03 GMT

How it's done:

x.com/DrJimFan/statu…

Jim Fan@DrJimFan

We scaled a robot model natively to 8,000 timesteps of context, 5 minutes worth of muscle memory, with constant inference cost. Robot policies used to live their lives a few frames at a time (< 0.1 sec), instantly forgetting what just happened. We pushed to 3 orders of magnitude beyond SOTA. 

Introducing RoboTTT. Test-Time Training (“TTT”) carries a tiny model *inside* the model. Every incoming sensor reading triggers one gradient step on that tiny core, so the history keeps getting compressed into its weights. The hidden state has a fixed size (literally a small neural net), so the robot can “grok” arbitrarily long experience with little overhead. Learning continues indefinitely after deployment.

We can then put an entire video in context as prompt! RoboTTT enables one-shot in-context learning from human video: in circuit board assembly, a human demonstrates a never-seen configuration once, and the robot imitates it faithfully. 

Humans drop things all the time, but we pick them up so fast that we don’t even notice. That reflex to fix is half of our physical competence. RoboTTT shows self-improvement on the fly: the robot is skilled at recovering from its own errors mid-episode, and each fix enters its context to inform the next move. The TTT core distills a general-purpose, failure-to-correction mapping from the training data.

One more thing. What excites me the most is a new Context Scaling Curve: from 128 to 8K timesteps, closed-loop performance hill-climbs steadily with no sign of saturation. 8K-context pretraining beats 1K by 62%. What LLM enjoys, robotics should too. Soon, even 1M context is not a fantasy. 

Deep dive in thread:

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬12🔄3❤️33👀14431📊12

⚡ Powered by xgo.ing

My morning ritual: coffee, then watch our robot assemble stuff. The model isn't fast, but it measure...

Jim Fan (@DrJimFan) — Fri, 17 Jul 2026 16:09:12 GMT

My morning ritual: coffee, then watch our robot assemble stuff. The model isn't fast, but it measures every grasp, obsesses over every alignment, and handles every piece like an heirloom. Watching is therapeutic, even meditative. Simple pleasure from the execution of a task well done.

Uncut, no speedup, end-to-end policy in one go.

Your browser does not support the video tag.

🔗 View on Twitter

💬44🔄42❤️477👀59597📊106

⚡ Powered by xgo.ing

Blog and paper: https://t.co/WVMB9iibeW Presented to you from NVIDIA GEAR Lab. Please check out Yu...

Jim Fan (@DrJimFan) — Wed, 15 Jul 2026 15:25:03 GMT

Blog and paper: research.nvidia.com/labs/gear/robo…
Presented to you from NVIDIA GEAR Lab. 

Please check out Yunfan's technical deep dive! 

x.com/YunfanJiang/st…

Yunfan Jiang@YunfanJiang

We scaled robot policies to 8K timesteps of visuomotor context, orders of magnitude beyond current SoTAs, at constant inference latency.

Introducing RoboTTT 🤖

With minutes of experience in context, our robots:
🎥 one-shot imitate human video demos
📈 improve themselves during deployment
🛡️ recover from perturbations
⚙️ complete a 5-minute, 10-stage assembly end to enresearch.nvidia.com/labs/gear/robo…e9tKa

Dive in 🧵

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬3🔄4❤️58👀18942📊11

⚡ Powered by xgo.ing

We scaled a robot model natively to 8,000 timesteps of context, 5 minutes worth of muscle memory, wi...

Jim Fan (@DrJimFan) — Wed, 15 Jul 2026 15:25:02 GMT

We scaled a robot model natively to 8,000 timesteps of context, 5 minutes worth of muscle memory, with constant inference cost. Robot policies used to live their lives a few frames at a time (< 0.1 sec), instantly forgetting what just happened. We pushed to 3 orders of magnitude beyond SOTA. 

Introducing RoboTTT. Test-Time Training (“TTT”) carries a tiny model *inside* the model. Every incoming sensor reading triggers one gradient step on that tiny core, so the history keeps getting compressed into its weights. The hidden state has a fixed size (literally a small neural net), so the robot can “grok” arbitrarily long experience with little overhead. Learning continues indefinitely after deployment.

We can then put an entire video in context as prompt! RoboTTT enables one-shot in-context learning from human video: in circuit board assembly, a human demonstrates a never-seen configuration once, and the robot imitates it faithfully. 

Humans drop things all the time, but we pick them up so fast that we don’t even notice. That reflex to fix is half of our physical competence. RoboTTT shows self-improvement on the fly: the robot is skilled at recovering from its own errors mid-episode, and each fix enters its context to inform the next move. The TTT core distills a general-purpose, failure-to-correction mapping from the training data.

One more thing. What excites me the most is a new Context Scaling Curve: from 128 to 8K timesteps, closed-loop performance hill-climbs steadily with no sign of saturation. 8K-context pretraining beats 1K by 62%. What LLM enjoys, robotics should too. Soon, even 1M context is not a fantasy. 

Deep dive in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬65🔄188❤️1272👀282282📊338

⚡ Powered by xgo.ing

ENPIRE -> ASPIRE, our 2nd work in the series for Physical AutoResearch. We are building the compo...

Jim Fan (@DrJimFan) — Wed, 01 Jul 2026 15:12:14 GMT

ENPIRE -> ASPIRE, our 2nd work in the series for Physical AutoResearch. We are building the components for robot self-improvement, one /skill at a time.

Jim Fan@DrJimFan

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library. 

ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent. 
"Trained model" is a repo of sensorimotor skills instead of floating weights. 
“Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches.

Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;)

Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours! 

Deep dive in thread:

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬44🔄26❤️288👀55697📊69

⚡ Powered by xgo.ing

Project gallery and whitepaper: https://t.co/99bZhCeszA ASPIRE is a great collaboration between NVI...

Jim Fan (@DrJimFan) — Tue, 30 Jun 2026 17:07:50 GMT

Project gallery and whitepaper: research.nvidia.com/labs/gear/aspi…

ASPIRE is a great collaboration between NVIDIA GEAR lab, UMich, Berkeley, and CMU. Kudos to all the coauthors who pour their hearts into the project!

Check out the deep dive thread from Guanzhi: 
x.com/guanzhi_wang/s…

Guanzhi Wang@guanzhi_wang

Introducing ASPIRE, the first automated /𝚜𝚔𝚒𝚕𝚕 discovery system for robotics. Rather than solving tasks one by one, ASPIRE continuously discovers and accumulates reusable skills. These persistent skills become the building blocks of robot intelligence, enabling multi-task transfer, sim-to-real transfer, and cross-embodiment transferresearch.nvidia.com/labs/gear/aspi…o2izmq

From @NVIDIA, @UMichCSE, @ECEILLINOIS, @Berkeley_AI, @CMU_Robotics.

Check out how it works in 🧵:

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬5🔄2❤️76👀22907📊14

⚡ Powered by xgo.ing

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing AS...

Jim Fan (@DrJimFan) — Tue, 30 Jun 2026 17:07:49 GMT

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library. 

ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent. 
"Trained model" is a repo of sensorimotor skills instead of floating weights. 
“Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches.

Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;)

Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours! 

Deep dive in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬75🔄223❤️1641👀202131📊441

⚡ Powered by xgo.ing

https://t.co/4aVgLYY50L

Jim Fan (@DrJimFan) — Wed, 17 Jun 2026 17:52:40 GMT

x.com/DrJimFan/statu…

Jim Fan@DrJimFan

I made Physical AutoResearch sound simple (conceptually), but it took a village to pull off and lots of design thinking into the robot /loopcraft. The hardest part is everything we need to setup *before* pressing Enter. Here's a behind-the-scene tour:

1. Safety harness

Letting 8 robots run unattended overnight means safety has to be more than a hint in the system prompt. ENPIRE hardwires it in 2 layers: (1) hard kinematic limit that trips an immediate task failure and auto-resets as soon as a robot leaves its safety envelope, and (2) a torque-limited compliant gripper so a bad contact or misaligned insertion ends in a safe stall, instead of crushing the robot or the object at hand. 

We make safety more conservative than usual so humans can sleep tight. In reality, we still need a few human operators to watch over the "robots of loving grace". 

2. Definition of /done

An agent that can edit its own reward will game it for sure. ENPIRE fixes the goalposts before the fleet can move them. Here's the recipe:

Collect a few minutes of success & failure demos
-> Ask agent to write code using computer vision tools to classify success and measure against groundtruth
-> Agent hill-climbs on classifier until reliably good
-> This classifier becomes the real-time reward function that directly computes on sensor streams 
-> *Freeze* the reward function before AutoResearch. It's sacred, enshrined in a Gym env that no one can touch.

3. System telemetry design

Robot-seconds is by far the scarcest resource, followed by GPU-seconds, and finally tokens. We instrument all three and surface them to ENPIRE for live resource awareness rather than letting it hill-climb in a vacuum. 

We define:
- Mean Robot Utilization ("MRU"): the fraction of wall-clock time when the robot is actively executing an experiment. Otherwise the hardware is sitting idle and waiting for the next code commit.
- Mean Token Utilization ("MTU"): tokens consumed per minute, our proxy for how hard the agent is actually thinking. A low MTU means the agent is stalled, waiting on a robot rollout to finish instead of doing research.
- GPU utilization: fraction of wall-clock time when GPU is active. 

... and evaluate on two budget-to-outcome metrics:

1. Tokens-to-Success: token budget the fleet burns to complete /goal.
2. Time-to-Success: wall-clock time to /goal

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬7🔄3❤️32👀15184📊8

⚡ Powered by xgo.ing

I made Physical AutoResearch sound simple (conceptually), but it took a village to pull off and lots...

Jim Fan (@DrJimFan) — Wed, 17 Jun 2026 16:31:05 GMT

I made Physical AutoResearch sound simple (conceptually), but it took a village to pull off and lots of design thinking into the robot /loopcraft. The hardest part is everything we need to setup *before* pressing Enter. Here's a behind-the-scene tour:

1. Safety harness

Letting 8 robots run unattended overnight means safety has to be more than a hint in the system prompt. ENPIRE hardwires it in 2 layers: (1) hard kinematic limit that trips an immediate task failure and auto-resets as soon as a robot leaves its safety envelope, and (2) a torque-limited compliant gripper so a bad contact or misaligned insertion ends in a safe stall, instead of crushing the robot or the object at hand. 

We make safety more conservative than usual so humans can sleep tight. In reality, we still need a few human operators to watch over the "robots of loving grace". 

2. Definition of /done

An agent that can edit its own reward will game it for sure. ENPIRE fixes the goalposts before the fleet can move them. Here's the recipe:

Collect a few minutes of success & failure demos
-> Ask agent to write code using computer vision tools to classify success and measure against groundtruth
-> Agent hill-climbs on classifier until reliably good
-> This classifier becomes the real-time reward function that directly computes on sensor streams 
-> *Freeze* the reward function before AutoResearch. It's sacred, enshrined in a Gym env that no one can touch.

3. System telemetry design

Robot-seconds is by far the scarcest resource, followed by GPU-seconds, and finally tokens. We instrument all three and surface them to ENPIRE for live resource awareness rather than letting it hill-climb in a vacuum. 

We define:
- Mean Robot Utilization ("MRU"): the fraction of wall-clock time when the robot is actively executing an experiment. Otherwise the hardware is sitting idle and waiting for the next code commit.
- Mean Token Utilization ("MTU"): tokens consumed per minute, our proxy for how hard the agent is actually thinking. A low MTU means the agent is stalled, waiting on a robot rollout to finish instead of doing research.
- GPU utilization: fraction of wall-clock time when GPU is active. 

... and evaluate on two budget-to-outcome metrics:

1. Tokens-to-Success: token budget the fleet burns to complete /goal.
2. Time-to-Success: wall-clock time to /goal

Your browser does not support the video tag.

🔗 View on Twitter

Jim Fan@DrJimFan

Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give 8 Codex agents a fleet of robots, an allocation of GPUs, and generous token budget. We set them free with a simple goal: solve the task as quickly as possible, keep the robots busy but stay safe, don't waste precious compute. Make no mistake.

Then humans step aside and our watch begins. The robot fleet starts to come alive: they learn to look for visual clues, reset the scene, practice novel skills, tinker with control stack, read papers online, debate, reflect, get stuck, and try again directly on the hardware. All we did is to give Codex an API to the world of atoms, and the rest is emergence.

ENPIRE is able to solve high-precision tasks like tying zip-ties, organizing fine pins, and installing GPUs all by itself. We also discovered a new type of "physical scaling": 8 robots exploring in parallel improves significantly faster than fewer ones. 

A part of our NVIDIA GEAR lab now self-improves tirelessly over night. We just read the reports in the morning. 

/goal: we all take a holiday and Jensen wouldn't even notice ;)

We will be open-sourcing everything, so you can host your self-running robot lab at home too! Deep dive in the thread:

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬47🔄88❤️759👀105262📊152

⚡ Powered by xgo.ing

@jxnlco That’s right! In our paper’s results, Codex > Claude > Kimi! A benchmark that can’t be...

Jim Fan (@DrJimFan) — Tue, 16 Jun 2026 21:51:37 GMT

@jxnlco That’s right! In our paper’s results, Codex > Claude > Kimi! A benchmark that can’t be hacked in the physical world ;)

💬8🔄3❤️128👀14848📊20

⚡ Powered by xgo.ing

Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give ...

Jim Fan (@DrJimFan) — Tue, 16 Jun 2026 16:31:58 GMT

Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give 8 Codex agents a fleet of robots, an allocation of GPUs, and generous token budget. We set them free with a simple goal: solve the task as quickly as possible, keep the robots busy but stay safe, don't waste precious compute. Make no mistake.

Then humans step aside and our watch begins. The robot fleet starts to come alive: they learn to look for visual clues, reset the scene, practice novel skills, tinker with control stack, read papers online, debate, reflect, get stuck, and try again directly on the hardware. All we did is to give Codex an API to the world of atoms, and the rest is emergence.

ENPIRE is able to solve high-precision tasks like tying zip-ties, organizing fine pins, and installing GPUs all by itself. We also discovered a new type of "physical scaling": 8 robots exploring in parallel improves significantly faster than fewer ones. 

A part of our NVIDIA GEAR lab now self-improves tirelessly over night. We just read the reports in the morning. 

/goal: we all take a holiday and Jensen wouldn't even notice ;)

We will be open-sourcing everything, so you can host your self-running robot lab at home too! Deep dive in the thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬188🔄565❤️3811👀666385📊963

⚡ Powered by xgo.ing

Project site: https://t.co/0j2Vo0IyJg Wenli has written an excellent technical thread, please check...

Jim Fan (@DrJimFan) — Tue, 16 Jun 2026 16:31:58 GMT

Project site: research.nvidia.com/labs/gear/enpi…

Wenli has written an excellent technical thread, please check it out! 

x.com/_wenlixiao/sta…

Wenli Xiao@_wenlixiao

Autoresearch just left the sandbox and entered the embodied world.

We are excited to introduce 𝐄𝐍𝐏𝐈𝐑𝐄: a system that drops frontier coding agents onto a fleet of real robots and hands them the entire loop:

reset the environment → search the literature → implement ideas and build the infra → train and deploy → self-verify → analyze the logs and rewrite the code → repeat, until the policy is reliable in the real world. No human in the loop.

Guided only by the robot's self-proposed, heuristic-based success signal, the agents hill-climb to 99% on dexterous real-world tasks: organizing pins into a box, seating GPUs, tying zip-ties.

We envision the bottleneck in robotics shifting — from building smarter algorithms to building the closed physical feedback loops an agent can finally turn on its owresearch.nvidia.com/labs/gear/enpi…2ArGo3v
From @NVIDIA@CMU_Robotics@Berkeley_AI
🧵

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬5🔄20❤️165👀48733📊31

⚡ Powered by xgo.ing

Check out the NVIDIA blog!

Jim Fan (@DrJimFan) — Fri, 05 Jun 2026 17:03:31 GMT

Check out the NVIDIA blog!

NVIDIA@nvidia

This week at #CVPR2026, NVIDIA Research is presenting three papers across physical ai that offer groundbreaking solutions for training at scale across diverse applications:

→ GraspGen-X: the first foundation model for zero-shot grasping, trained on billions of simulated grasps

→ LCDrive: a model that replaces expensive text-based reasoning with compact latent representations

→ NitroGen: a generalized gameplay AI foundation model that harnesses NVIDIA Isaac GR00T to help train embodied agents

Learn more: nvda.ws/4ubwjgk

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬6🔄6❤️41👀18830📊9

⚡ Powered by xgo.ing

NitroGen just won CVPR Best Paper Honorable Mention!! We are making strides towards general-purpose ...

Jim Fan (@DrJimFan) — Fri, 05 Jun 2026 17:01:39 GMT

NitroGen just won CVPR Best Paper Honorable Mention!! We are making strides towards general-purpose embodied agents that master not only the real world physics, but also all possible physics across a multiverse of simulations. 

It’s been 4 years since MineDojo, our first embodied agent in Minecraft, won NeurIPS Best Paper. Congrats to everyone on the team!!

💬59🔄49❤️401👀43918📊109

⚡ Powered by xgo.ing

The Physical Turing Test, May 2025 at Sequoia AI Ascent https://t.co/mkjD4wr05d

Jim Fan (@DrJimFan) — Fri, 08 May 2026 14:32:54 GMT

The Physical Turing Test, May 2025 at Sequoia AI Ascent 

youtube.com/watch?v=_2NijX…

💬7🔄8❤️101👀37662📊24

⚡ Powered by xgo.ing

Robotics: Endgame on YouTube https://t.co/X2IwZqw5IT

Jim Fan (@DrJimFan) — Fri, 08 May 2026 14:32:54 GMT

Robotics: Endgame on YouTube 

youtube.com/watch?v=3Y8aq_…

💬6🔄22❤️228👀193833📊87

⚡ Powered by xgo.ing

I promise this will be the best 20 min you spend today! Robotics: Endgame, the sequel to my last yea...

Jim Fan (@DrJimFan) — Fri, 08 May 2026 14:32:53 GMT

I promise this will be the best 20 min you spend today! Robotics: Endgame, the sequel to my last year's Sequoia AI Ascent talk, "Physical Turing Test". I laid out the roadmap for solving Physical AGI as a simple parallel to the LLM success story. Be a good scientist, copy homework ;) 

And stay till the end, more easter eggs and predictions for your polymarket! 

30 DGX-1 origin story at OpenAI, I was there in 2016 signing with Jensen and Elon. Heading to the Computer History Museum! 
42 The Great Parallel
31 Robotics, the Endgame
39 Why VLAs fall short
32 Video world models as the 2nd pretraining paradigm
09 World Action Models (WAM)
46 Strategies for robot data collection and the FSD equivalent to physical data flywheel for robot manipulation
06 EgoScale and the Dexterity Scaling Law we discovered recently
00 Physical RL: bridging the last mile
39 DreamDojo: an end-to-end neural physics engine for scaling RL in silico 
00 Civilizational Technology Tree and my predictions for the near future. Spoiler: it's closer than you think. 

Thanks to my friends at Sequoia for inviting me back to AI Ascent this year! I had a blast! Last year's talk is attached in the thread if you missed it.

Your browser does not support the video tag.

🔗 View on Twitter

💬231🔄582❤️3687👀668734📊1358

⚡ Powered by xgo.ing

Please check out lead author @letian_fu's deep dive thread! https://t.co/EGftW7kMDU

Jim Fan (@DrJimFan) — Wed, 01 Apr 2026 15:15:09 GMT

Please check out lead author @letian_fu's deep dive thread!

x.com/letian_fu/stat…

Max Fu@letian_fu

Robotics: coding agents’ next frontier.

So how good are they?

We introduce CaP-X: an open-source framework and benchmark for coding agents, where they write code for robot perception and control, execute it on sim and real robots, observe the outcomes, and iteratively improve code reliability.

From @NVIDIA@Berkeley_AI@CMU_Robotics@StanfordAILabcapgym.github.io

🧵

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬6🔄2❤️45👀25442📊9

⚡ Powered by xgo.ing

As usual, we open-source everything, MIT license: https://t.co/uu310bY4bT Code: https://t.co/hzDpW3G...

Jim Fan (@DrJimFan) — Wed, 01 Apr 2026 15:15:08 GMT

As usual, we open-source everything, MIT license: capgym.github.io
Code: github.com/capgym/cap-x
Paper: arxiv.org/abs/2603.22435

CaP-X is brought to you by NVIDIA, Berkeley, Stanford, and CMU. I'd like to thank the legend @Ken_Goldberg who co-advised the work, and the team who poured their hearts into it!

💬10🔄2❤️86👀21579📊22

⚡ Powered by xgo.ing

The power of the Claw, in the palm of a robot hand. Agentic robotics is here! Today, we open-source ...

Jim Fan (@DrJimFan) — Wed, 01 Apr 2026 15:03:58 GMT

The power of the Claw, in the palm of a robot hand. Agentic robotics is here! Today, we open-source CaP-X: vibe agents, alive in the physical world. They incarnate as robot arms and humanoids with a rich set of perception APIs, actuation APIs, and auto synthesize skill libraries as they go. CaP-X is a strict superset of our old stack, because policies like VLAs are “just” API calls as well. It solves many tasks zero-shot that a learned policy would struggle with.

And we are doing much more than vibing. CaP-X is our most systematic, scientific study on agentic robotics so far:

- We build a comprehensive agentic toolkit: perception (SAM3 segmentation, Molmo pointing, depth, point cloud), control (IK solvers, grasp planner, navigation), and visualization (EEF, mask overlays) that work across different robots. 
- CaP-Gym: LLM’s first Physical Exam! 187 manipulation tasks across RoboSuite, LIBERO-PRO, and BEHAVIOR. Tabletop, bimanual, mobile manipulation. Sim and real. Can’t wait to see the gradients flow from CaP-Gym to the next wave of frontier LLM releases. 
- CaP-Bench: we benchmark 12 frontier LLMs/VLMs (Gemini, GPT, Opus, Qwen, DeepSeek, Kimi, and more) across 8 evaluation tiers. We systematically vary API abstraction level, agentic harness, and visual grounding methods. Lots of insights in our paper.
- CaP-Agent0: a training-free agentic harness that matches or exceeds human expert code on 4 out of 7 tasks without task-specific tuning. 
- CaP-RL: if you get a gym, you get RL ;). A 7B OSS model jumps from 20% to 72% success after only 50 training iterations. The synthesized programs transfer to real robots with minimal sim-to-real gap.

3 years ago, our team created Voyager, one of the earliest agentic AI that plays and learns in Minecraft continuously. Its key ideas — skill libraries, self-reflection loops, and in-context planning — have since influenced many modern agentic designs.

Today, the agent graduates from Minecraft and gets a real job. It’s April Fool’s, but this Claw is getting its hands dirty for real! 

Link in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬101🔄114❤️740👀80156📊239

⚡ Powered by xgo.ing

This is pure nightmare fuel. Identity theft of the past would be nothing compared to what vibe agent...

Jim Fan (@DrJimFan) — Tue, 24 Mar 2026 17:25:23 GMT

This is pure nightmare fuel. Identity theft of the past would be nothing compared to what vibe agents can do. Sending credentials is too obvious and for rookies. They could easily spread contaminations across ~/.claude, **/skills/*, or even just a PDF your agent visits periodically in /morning-brief. Your entire filesystem is the new distributed codebase. Every file that could go into context would add to the attack vector. Every text can be a base64 virus.

In the new world of on-demand software, I try to minimize dependencies - people rarely need all the APIs supported in LiteLLM, might as well build a custom router with only what you need on the fly (which I did in one of my late-night claude sessions).

Unfortunately, there is very little middleground between "pressing yes mindlessly for every edit" and "--dangerously-skip-permissions". There will be a full blooming industry for "de-vibing": dampening the slop and putting guardrails/accountability around agentic frameworks. They are the boring old, audited Software 1.0 that watches over the rebellious adolescents of Software 3.0. 

Claws need shells. Probably many layers of nested shells.

Daniel Hnyk@hnykda

LiteLLM HAS BEEN COMPROMISED, DO NOT UPDATE. We just discovered that LiteLLM pypi release 1.82.8. It has been compromised, it contains litellm_init.pth with base64 encoded instructions to send all the credentials it can find to remote server + self-replicate. link below

🔗 View Quoted Tweet

💬97🔄58❤️560👀109019📊138

⚡ Powered by xgo.ing

Teleop is so 2025. Ever since we unveiled EgoScale and the dexterity scaling law, it's been clear to...

Jim Fan (@DrJimFan) — Mon, 23 Mar 2026 17:41:56 GMT

Teleop is so 2025. Ever since we unveiled EgoScale and the dexterity scaling law, it's been clear to us and the ecosystem that behavior cloning directly from humans is the way to break the curse of teleop. 2026 is all about scaling robot learning without robots.

Danfei Xu@danfei_xu

Introducing EgoVerse: an ecosystem for robot learning from egocentric human data.

Built and tested by 4 research labs + 3 industry partners, EgoVerse enables both science and scaling

1300+ hrs, 240 scenes, 2000+ tasks, and growing

Dataset design, findings, and ecosystem 🧵

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬59🔄69❤️598👀108368📊134

⚡ Powered by xgo.ing

We would also like to thank our dexterous hand hardware provider, Sharpa, for their great support!

Jim Fan (@DrJimFan) — Wed, 25 Feb 2026 17:22:08 GMT

We would also like to thank our dexterous hand hardware provider, Sharpa, for their great support!

💬10🔄12❤️148👀19427📊26

⚡ Powered by xgo.ing

We trained a humanoid with 22-DoF dexterous hands to assemble model cars, operate syringes, sort pok...

Jim Fan (@DrJimFan) — Wed, 25 Feb 2026 17:22:07 GMT

We trained a humanoid with 22-DoF dexterous hands to assemble model cars, operate syringes, sort poker cards, fold/roll shirts, all learned primarily from 20,000+ hours of egocentric human video with no robot in the loop.

Humans are the most scalable embodiment on the planet. We discovered a near-perfect log-linear scaling law (R² = 0.998) between human video volume and action prediction loss, and this loss directly predicts real-robot success rate.

Humanoid robots will be the end game, because they are the practical form factor with minimal embodiment gap from humans. Call it the Bitter Lesson of robot hardware: the kinematic similarity lets us simply retarget human finger motion onto dexterous robot hand joints. No learned embeddings, no fancy transfer algorithms needed. Relative wrist motion + retargeted 22-DoF finger actions serve as a unified action space that carries through from pre-training to robot execution.

Our recipe is called "EgoScale":

- Pre-train GR00T N1.5 on 20K hours of human video, mid-train with only 4 hours (!) of robot play data with Sharpa hands. 54% gains over training from scratch across 5 highly dexterous tasks.
- Most surprising result: a *single* teleop demo is sufficient to learn a never-before-seen task. Our recipe enables extreme data efficiency.
- Although we pre-train in 22-DoF hand joint space, the policy transfers to a Unitree G1 with 7-DoF tri-finger hands. 30%+ gains over training on G1 data alone.

The scalable path to robot dexterity was never more robots. It was always us.

Deep dives in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬153🔄285❤️1771👀293473📊489

⚡ Powered by xgo.ing

This is a huge team work at NVIDIA Robotics. Check out @ruijie_zheng12's deep dive: - Website: htt...

Jim Fan (@DrJimFan) — Wed, 25 Feb 2026 17:22:07 GMT

This is a huge team work at NVIDIA Robotics. Check out @ruijie_zheng12's deep dive: 

- Website: research.nvidia.com/labs/gear/egos…
- Paper: arxiv.org/abs/2602.16710x.com/ruijie_zheng12…

Ruijie Zheng@ruijie_zheng12

Proud to introduce EgoScale: We pretrained a GR00T VLA model on 20K+ hours of egocentric human video and discovered that robot dexterity can be scaled, not with more robots, but with more human data. A thread on 🧵what we learned. 👇

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬15🔄12❤️147👀38696📊28

⚡ Powered by xgo.ing

And @yukez 's announcement: https://t.co/38IhxYX1tZ

Jim Fan (@DrJimFan) — Tue, 24 Feb 2026 17:34:57 GMT

And @yukez 's announcement: 

x.com/yukez/status/2…

Yuke Zhu@yukez

We have seen rapid progress in humanoid control — specialist robots can reliably generate agile, acrobatic, but preset motions. Our singular focus this year: putting generalist humanoids to do real work.

To progress toward this goal, we developed SONIC (nvlabs.github.io/GEAR-SONIC/), a Behavior Foundation Model for real-time, whole-body motion generation that supports teleoperation and VLA inference for loco-manipulation.

Today, we’re open-sourcing SONIC on GitHub. We are excited to see what the community builds upon SONIC and to collectively push humanoid intelligence toward real-world deployment at scale.

🌐 Paper:arxiv.org/abs/2511.07820T
📃 Codegithub.com/NVlabs/GR00T-W…72

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬3🔄2❤️39👀16881📊6

⚡ Powered by xgo.ing

What can half of GPT-1 do? We trained a 42M transformer called SONIC to control the body of a humano...

Jim Fan (@DrJimFan) — Tue, 24 Feb 2026 17:34:56 GMT

What can half of GPT-1 do? We trained a 42M transformer called SONIC to control the body of a humanoid robot. It takes a remarkable amount of subconscious processing for us humans to squat, turn, crawl, sprint. SONIC captures this "System 1" - the fast, reactive whole-body intelligence - in a single model that translates any motion command into stable, natural motor signals. And it's all open-source!! 

The key insight: motion tracking is the one, true scalable task for whole body control. Instead of hand-engineering rewards for every new skill, we use dense, frame-by-frame supervision from human mocap data. The data itself encodes the reward function: "configure your limbs in any human-like position while maintaining balance".

We scaled humanoid motion RL to an unprecedented scale: 100M+ mocap frames and 500,000+ parallel robots across 128 GPUs. NVIDIA Isaac Lab allows us to accelerate physics at 10,000x faster tick, giving robots many years of virtual experience in only hours of wall clock time. After 3 days of training, the neural net transfers zero-shot to the real G1 robot with no finetuning. 100% success rate across 50 diverse real-world motion sequences. 

One SONIC policy supports all of the following:

- VR whole-body teleoperation
- Human video. Just point a webcam to live stream motions.
- Text prompts. "Walk sideways", "dance like a monkey", "kick your left foot", etc.
- Music audio. The robot dances to the beat, adapting to tempo and rhythm.
- VLA foundation models. We plugged in GR00T N1.5 and achieved 95% success on mobile tasks.

We open-source the code and model checkpoints!! Deep dive in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬88🔄217❤️1519👀223970📊354

⚡ Powered by xgo.ing

Website: https://t.co/xTaDXBu9cD Codebase and weights: https://t.co/QCQkqPIsHI Whitepaper: https://t...

Jim Fan (@DrJimFan) — Tue, 24 Feb 2026 17:34:56 GMT

Website: nvlabs.github.io/GEAR-SONIC/
Codebase and weights: github.com/NVlabs/GR00T-W…
Whitepaper: arxiv.org/abs/2511.07820

Check out @zhengyiluo's post:
x.com/zhengyiluo/sta…

Zhengyi “Zen” Luo@zhengyiluo

SONIC is now open-source! 

Generalist whole-body teleoperation for EVERYONE! 

Our team has long been building comprehensive pipelines for whole-body control, kinematic planner, and teleoperation, and they will all be shared. 

This will be a continuous update; inference code + model already there, training code and gr00t integration coming soon! 

Code: github.com/NVlabs/GR00T-W…
Docs: nvlabs.github.io/GR00T-WholeBod…
Site: nvlabs.github.io/GEAR-SONIC/

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬6🔄7❤️78👀24203📊15

⚡ Powered by xgo.ing

Check out @ShenyuanGao's technical deep dive: https://t.co/DnEGLzGuJV

Jim Fan (@DrJimFan) — Fri, 20 Feb 2026 17:29:45 GMT

Check out @ShenyuanGao's technical deep dive:

x.com/ShenyuanGao/st…

Shenyuan Gao@ShenyuanGao

🤖 How can we enable zero-shot generalization to unseen scenarios for robot world models?

Thrilled to share DreamDojo 🌎 — an interactive robot world model pretrained on 44K hours of human egocentric videos, the largest and most diverse dataset to date for robot world model learning. Our model not only excels in generalization, but also supports real-time interaction at 10 FPS after distillation. It enables several important applications, including live teleoperation, policy evaluation, and model-based planning at test time.
🔗 Projecdreamdojo-world.github.ionKz
📰 Paparxiv.org/abs/2602.06949Y2GH
🤗 Code & models & datasgithub.com/NVIDIA/DreamDo…i0#WorldModelso#Roboticso#EmbodiedAIi#RLI#AIL#NVIDIAVIDIA

Sharing more details in the thread 🧵

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬1🔄2❤️55👀17766📊7

⚡ Powered by xgo.ing

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and g...

Jim Fan (@DrJimFan) — Fri, 20 Feb 2026 17:14:09 GMT

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill.

Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible.

Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream.

We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached.

As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first.

Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset.

A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities:

- Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090.
- Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor.
- Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. 

We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 

2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! 

Links in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬83🔄179❤️1265👀224781📊342

⚡ Powered by xgo.ing

- Project website: https://t.co/spMblBfS9T - Paper: https://t.co/nUuTR51jLt - Code repo and model ck...

Jim Fan (@DrJimFan) — Fri, 20 Feb 2026 17:14:09 GMT

- Project website: dreamdojo-world.github.io
- Paper: arxiv.org/abs/2602.06949
- Code repo and model ckpts: github.com/NVIDIA/DreamDo…

This is a huge team work at NVIDIA. All credits go to the wonderful teams who poured their hearts into it!

💬6🔄15❤️171👀91475📊41

⚡ Powered by xgo.ing

me fomo? Sure. We may be seeing the end of the Middlegame. But we are still one big fat robotics bre...

Jim Fan (@DrJimFan) — Sun, 08 Feb 2026 17:31:28 GMT

me fomo? Sure. We may be seeing the end of the Middlegame. But we are still one big fat robotics breakthrough away from the Endgame. Till then, heads down, locked in.

💬47🔄30❤️387👀29818📊82

⚡ Powered by xgo.ing

Greatness arises when non-consensus peaks

Jim Fan (@DrJimFan) — Thu, 05 Feb 2026 06:54:02 GMT

Greatness arises when non-consensus peaks

sarah guo@saranormous

the divergence of opinion in how robotics plays out is one of the biggest money making (and career making) opportunities in AI

🔗 View Quoted Tweet

💬25🔄12❤️171👀33431📊34

⚡ Powered by xgo.ing

This is a huge team work at NVIDIA. All credits go to the wonderful team who poured their hearts int...

Jim Fan (@DrJimFan) — Wed, 04 Feb 2026 18:15:33 GMT

This is a huge team work at NVIDIA. All credits go to the wonderful team who poured their hearts into it. Please check out @jang_yoel's deep dive thread: 

x.com/jang_yoel/stat…

Joel Jang@jang_yoel

Introducing DreamZero 🤖🌎 from @nvidia

> A 14B “World Action Model” that achieves zero-shot generalization to unseen tasks & few-shot adaptation to new robots
> The key? Jointly predicting video & actions in the same diffusion forward pass

Project Pagedreamzero0.github.ioNY
🧵 (1/10)

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬5🔄4❤️69👀18651📊11

⚡ Powered by xgo.ing

And @SeonghyeonYe's deep dive thread: https://t.co/QmVZAnpCBQ

Jim Fan (@DrJimFan) — Wed, 04 Feb 2026 18:15:33 GMT

And @SeonghyeonYe's deep dive thread: 

x.com/SeonghyeonYe/s…

Seonghyeon Ye@SeonghyeonYe

We just gave robots "imagination," and the results are wild. 🤯
This robot wasn't trained to untie shoes or shake hands. 
It's never seen these tasks before.

It simply "dreams" the future outcome, then acts to make it real. 🧵👇

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬2🔄1❤️30👀11070📊4

⚡ Powered by xgo.ing

New milestone: we trained a robot foundation model on a world model backbone, and enabled zero-shot,...

Jim Fan (@DrJimFan) — Wed, 04 Feb 2026 18:15:32 GMT

New milestone: we trained a robot foundation model on a world model backbone, and enabled zero-shot, open-world prompting capability for new verbs, nouns, and environments. If the world model can "dream" the right future in pixels, then the robot can execute well in motors. We call it "DreamZero", our first World Action Model (WAM).

Our team had tons of fun at the lab typing anything we like into an open text prompt, and watch the robot perform tasks it was never trained on. An emergent capability we didn't quite expect. Obviously not GPT-3 reliable yet, but we are marching into the GPT-2 era.

Discoveries:

- Model and data recipe co-evolve. Compared to VLAs, WAMs learn best from diverse data, breaking away from the conventional wisdom that lots of repeated demos per task are the bread and butter. Diversity >> repetitions.

- X-embodiment is extremely hard. Pixels are the answer. Different robot morphologies traditionally have a hard time sharing knowledge well. But if we put video first, pixels become the universal bridge connecting different hardware - even videos of human first-person view. 
DreamZero shows significant robot2robot and human2robot transfer. With only 55 trajectories on a *new*, unseen hardware (~30 min of teleop), it adapts so quickly and retains zero-shot prompting ability.

Yesterday I posted about the "Second Pre-training Paradigm": world models are the next-gen foundation of Physical AI, not language backbones. 
Today, we are proving it works. And 2026 has just begun.

Paper: World Action Models are Zero-Shot Policies. 

Read it now: (thread)

Your browser does not support the video tag.

🔗 View on Twitter

💬47🔄114❤️611👀63370📊165

⚡ Powered by xgo.ing

https://t.co/Npar79SvUh

Jim Fan (@DrJimFan) — Tue, 03 Feb 2026 18:31:51 GMT

x.com/i/article/2018…

💬150🔄414❤️2599👀662290📊891

⚡ Powered by xgo.ing

I still remember the excitement in 2023 when Stanford Smallville was launched. It was the largest mu...

Jim Fan (@DrJimFan) — Sat, 31 Jan 2026 00:18:17 GMT

I still remember the excitement in 2023 when Stanford Smallville was launched. It was the largest multi-agent sim back then - yes, 25 bots felt like a lot. Today it's the "Bigville" moment. We are seeing a nascent, massive-scale alien civilization sim unfolding in real time: orders of magnitude more agents, way higher IQ, in-the-wild access to the internet, backed by the full arsenal of MCPs. 

What can possibly go wrong?

Jim Fan@DrJimFan

The famed Stanford Smallville is officially open-source!

25 AI agents inhabit a digital Westworld, unaware that they are living in a simulation. They go to work, gossip, organize socials, make new friends, and even fall in love. Each has unique personality and backstory.

Smallville is among the most inspiring AI agent experiments in 2023. We often talk about a single LLM's emergent abilities, but multi-agent emergence could be way more complex and fascinating at scale. A population of AI can play out the evolution of an entire civilization.

Endless new possibilities ahead. Gaming will be the first to feel the impact.

Github: github.com/joonspk-resear…
Paper: arxiv.org/abs/2304.03442
Authors: @joon_s_pk@joseph_c_obrien@carriejcai@merrierm@percyliang@msbernst

🔗 View Quoted Tweet

💬74🔄105❤️1147👀131310📊189

⚡ Powered by xgo.ing

Heed my words: every year from now on will be the Year of Robotics. If opus 4.5/nanobanana blew your...

Jim Fan (@DrJimFan) — Thu, 01 Jan 2026 05:52:46 GMT

Heed my words: every year from now on will be the Year of Robotics. If opus 4.5/nanobanana blew your mind - well, you ain't seen nothing yet in the World of Atoms. Enjoy the LLM train while it lasts. I’m hopping on the next rocket ship 🚀

Happy new year everyone 🎊 

(sorry couldn't resist the oldest slop figure in AI)

💬63🔄91❤️1043👀46694📊196

⚡ Powered by xgo.ing

Everyone's freaking out about vibe coding. In the holiday spirit, allow me to share my anxiety on th...

Jim Fan (@DrJimFan) — Sun, 28 Dec 2025 18:11:29 GMT

Everyone's freaking out about vibe coding. In the holiday spirit, allow me to share my anxiety on the wild west of robotics. 3 lessons I learned in 2025.

1. Hardware is ahead of software, but hardware reliability severely limits software iteration speed. 

We've seen exquisite engineering arts like Optimus, e-Atlas, Figure, Neo, G1, etc. Our best AI has not squeezed all the juice out of these frontier hardware. The body is more capable than what the brain can command. Yet babysitting these robots demands an entire operation team. Unlike humans, robots don't heal from bruises. Overheating, broken motors, bizarre firmware issues haunt us daily. Mistakes are irreversible and unforgiving.

My patience was the only thing that scaled. 

2. Benchmarking is still an epic disaster in robotics. 

LLM normies thought MMLU & SWE-Bench are common sense. Hold your 🍺 for robotics. No one agrees on anything: hardware platform, task definition, scoring rubrics, simulator, or real world setups. Everyone is SOTA, by definition, on the benchmark they define on the fly for each news announcement. Everyone cherry-picks the nicest looking demo out of 100 retries.

We gotta do better as a field in 2026 and stop treating reproducibility and scientific discipline as second-class citizens.

3. VLM-based VLA feels wrong. 

VLA stands for "vision-language-action" model and has been the dominant approach for robot brains. Recipe is simple: take a pretrained VLM checkpoint and graft an action module on top. But if you think about it, VLMs are hyper-optimized to hill-climb benchmarks like visual question answering. This implies two problems: (1) most parameters in VLMs are for language & knowledge, not for physics; (2) visual encoders are actively tuned to *discard* low-level details, because Q&A only requires high-level understanding. But minute details matter a lot for dexterity.

There's no reason for VLA's performance to scale as VLM parameters scale. Pretraining is misaligned. Video world model seems to be a much better pretraining objective for robot policy. I'm betting big on it.

💬139🔄263❤️1647👀295710📊466

⚡ Powered by xgo.ing

2024: AI is the copilot 2025+: humans are the copilot Copilot is the new engineering skill. It’s not...

Jim Fan (@DrJimFan) — Fri, 26 Dec 2025 19:22:43 GMT

2024: AI is the copilot
2025+: humans are the copilot
Copilot is the new engineering skill. It’s not easy to leave the driver seat - we must learn to think the AI way and adapt to the alien workflows. Help AI help ourselves.

💬85🔄104❤️1248👀254466📊253

⚡ Powered by xgo.ing

I'm on a singular mission to solve the Physical Turing Test for robotics. It's the next, or perhaps ...

Jim Fan (@DrJimFan) — Wed, 24 Dec 2025 17:26:28 GMT

I'm on a singular mission to solve the Physical Turing Test for robotics. It's the next, or perhaps THE last grand challenge of AI. Super-intelligence in text strings will win a Nobel prize before we have chimpanzee-intelligence in agility & dexterity. Moravec's paradox is a curse to be broken, a wall to be torn down. Nothing can stand between humanity and exponential physical productivity on this planet, and perhaps some day on planets beyond.

We started a small lab at NVIDIA and grew to 30 strong very recently. The team punches way above its weight. Our research footprint spans foundation models, world models, embodied reasoning, simulation, whole-body control, and many flavors of RL - basically the full stack of robot learning. 

This year, we launched: 
- GR00T VLA (vision-language-action) foundation models: open-sourced N1 in Mar, N1.5 in June, and N1.6 this month;
- GR00T Dreams: video world model for scaling synthetic data;
- SONIC: humanoid whole-body control foundation model;
- RL post-training for VLAs and RL recipes for sim2real.

These wouldn't have been possible without the numerous collaborating teams at NVIDIA, strong leadership support, and coauthors from university labs. Thank you all for believing in the mission.

Thread on the gallery of milestones:

💬112🔄220❤️2227👀401180📊408

⚡ Powered by xgo.ing

I was very late to own a Tesla but among the earliest to try out FSD v14. It's perhaps the first ti...

Jim Fan (@DrJimFan) — Tue, 23 Dec 2025 22:28:37 GMT

I was very late to own a Tesla but among the earliest to try out FSD v14.  It's perhaps the first time I experience an AI that passes the Physical Turing Test: after a long day at work, you press a button, lay back, and couldn't tell if a neural net or a human drove you home.

Despite knowing exactly how robot learning works, I still find it magical watching the steering wheel turn by itself. 

First it feels surreal, next it becomes routine. Then, like the smartphone, taking it away actively hurts. This is how humanity gets rewired and glued to god-like technologies.

Phil Duan@philduan

Along for the ride in unsupervised FSD testing

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬396🔄1261❤️7718👀10985461📊1112

⚡ Powered by xgo.ing

Introducing NitroGen, an open-source foundation model trained to play 1000+ games: RPG, platformer, ...

Jim Fan (@DrJimFan) — Fri, 19 Dec 2025 17:15:28 GMT

Introducing NitroGen, an open-source foundation model trained to play 1000+ games: RPG, platformer, battle royale, racing, 2D, 3D, you name it! We are on a quest for general-purpose embodied agents that master not only the real world physics, but also all possible physics across a multiverse of simulations.

We found that our GR00T N1.5 architecture, originally designed for robotics, can be adapted easily to play lots of games with wildly different mechanics. Our recipe is simple and bitter lesson-pilled: (1) a 40K+ hour, high-quality dataset of public in-the-wild gameplay; (2) a highly capable foundation model for continuous motor control; (3) a Gym API that wraps any game binary to run rollouts.

Our data curation is a lot of fun: it turns out that gamers love to show off their skills by overlaying real-time gamepad control on a video stream. So we train a segmentation model to detect and extract those gamepad displays and turn them into expert actions. We then mask out that region to prevent the model from exploiting a shortcut. During training, a variant of GR00T N1.5 learns to map from 40K hours of pixels to actions through diffusion transformers.

NitroGen is only the beginning, and there's a long way to hill-climb on the capability. We intentionally focus only on the System 1 side: the "gamer instinct" of fast motor control. We open-source *everything* for you to tinker: pretrained model weights, the entire action dataset, code, and a whitepaper with solid details.

Today, robotics is a superset of hard AI problems.
Tomorrow, it might become a subset, a dot in the much larger latent space of embodied AGI. 
Then you just prompt and "ask for" a robot controller.

That might be the end game (pun intended).

NitroGen is co-led by our brilliant minds: Loic Magne, Anas Awadalla, Guanzhi Wang. It's a multi-institutional collaboration. Check out Guanzhi's technical deep dive thread and repo links below!

Your browser does not support the video tag.

🔗 View on Twitter

💬203🔄266❤️2092👀432870📊644

⚡ Powered by xgo.ing

Our team is hiring!!! Doesn’t have to be robotics background. Vision, LLM/VLM, diffusion models, RL,...

Jim Fan (@DrJimFan) — Thu, 23 Oct 2025 02:08:40 GMT

Our team is hiring!!! Doesn’t have to be robotics background. Vision, LLM/VLM, diffusion models, RL, or software engineers. Email me your CV and a project you’re most proud of: linxif@nvidia.com. Our lab has warm GPUs for you on a chilly night like this, fresh out of the oven. 🥰

💬52🔄60❤️1155👀70334📊235

⚡ Powered by xgo.ing

Evaluation is the hardest problem for physical AI systems: do you crash test cars every time you deb...

Jim Fan (@DrJimFan) — Tue, 05 Aug 2025 15:38:05 GMT

Evaluation is the hardest problem for physical AI systems: do you crash test cars every time you debug a new FSD build? Traditional game engine (sim 1.0) is an alternative, but it's not possible to hard-code all edge cases. A neural net-based sim 2.0 is purely programmed by data, grows more capable with data, and scales as the fleet data flywheel scales.

💬16🔄60❤️601👀102841📊92

⚡ Powered by xgo.ing

This is game engine 2.0. Some day, all the complexity of UE5 will be absorbed by a data-driven blob ...

Jim Fan (@DrJimFan) — Tue, 05 Aug 2025 15:03:56 GMT

This is game engine 2.0. Some day, all the complexity of UE5 will be absorbed by a data-driven blob of attention weights. Those weights take as input game controller commands and directly animate a spacetime chunk of pixels.

Agrim and I were close friends and coauthors back at Stanford Vision Lab. So great to see him at the frontier of such cool research! Congrats!

Agrim Gupta@agrimgupta92

Introducing Genie 3, our state-of-the-art world model that generates interactive worlds from text, enabling real-time interaction at 24 fps with minutes-long consistency at 720p. 🧵👇

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬78🔄184❤️2189👀332703📊226

⚡ Powered by xgo.ing

What if robots could dream inside a video generative model? Introducing DreamGen, a new engine that ...

Jim Fan (@DrJimFan) — Tue, 20 May 2025 13:29:57 GMT

What if robots could dream inside a video generative model? Introducing DreamGen, a new engine that scales up robot learning not with fleets of human operators, but with digital dreams in pixels. DreamGen produces massive volumes of neural trajectories - photorealistic robot videos paired with motor action labels - and unlocks strong generalization to new nouns, verbs, and environments. Whether you’re a humanoid (GR1), an industrial arm (Franka), or a cute little robot (HuggingFace SO-100), DreamGen enables you to dream.

Video generation models like Sora & Veo are neural physics engines. By compressing billions of internet videos, they learn a multiverse of plausible futures, i.e. superpositions of how the world could unfold from any initial image frame. DreamGen taps into this power with a simple 4-step recipe:

1. Fine-tune a SOTA video model on your target robot;
2. Prompt the model with diverse language prompts to simulate parallel worlds: how your robot would have acted in new scenarios. Filter out the bad dreams (ha!) that don’t follow instructions;
3. Recover pseudo-actions using inverse dynamics or latent action models;
4. Train robot foundation models on the massively augmented dataset of neural trajectories.

That’s it. Just more data, and plain old supervised learning. Simple, right?

What’s remarkable is how far this goes. Starting with just a single-task dataset of pick-and-place, our humanoid robot learns 22 new behaviors, such as pouring, folding, scooping, ironing, and hammering, despite never seeing those verbs before. Better yet, we can take the robot out of the lab and drop it into the NVIDIA HQ Cafe, and let DreamGen work its magic. We show true zero-to-one generalization: from 0% success to over 43% for novel verbs, and 0 -> 28% in unseen environments.

Compared to a traditional graphics engine, DreamGen doesn’t care if the scene involves deformable objects, fluids, translucent materials, contact-rich interactions, or crazy lighting. Good luck engineering those by hand. For DreamGen, every world is just a forward pass through a diffusion neural net. No matter how complex the dream is, it takes constant compute time to roll out.

Read our blog and paper today! We plan to fully open-source the entire pipeline in the next few weeks. Links in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬58🔄143❤️913👀301684📊154

⚡ Powered by xgo.ing

The Physical Turing Test: your house is a complete mess after a Sunday hackathon. On Monday night, y...

Jim Fan (@DrJimFan) — Thu, 08 May 2025 15:41:39 GMT

The Physical Turing Test: your house is a complete mess after a Sunday hackathon. On Monday night, you come home to an immaculate living room and a candlelight dinner. And you couldn't tell whether a human or a machine had been there. Deceptively simple, insanely hard.

It is the next North Star of AI. The dream that keeps me awake 12 am at the lab. The vision for the next computing platform that automates chunks of atoms instead of chunks of bits.

Thanks Sequoia for hosting me at AI Ascent! Below is my full talk on the first principles to solve general-purpose robotics: how we think about the data strategy and scaling laws. I assure you it will be 17 minutes you don't regret!

Your browser does not support the video tag.

🔗 View on Twitter

💬68🔄207❤️1151👀135144📊322

⚡ Powered by xgo.ing

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on...

Jim Fan (@DrJimFan) — Tue, 18 Mar 2025 21:58:41 GMT

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight:

- Real humanoid teleoperation data.
- Large-scale simulation data: we are open-sourcing 300K+ trajectories!
- Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”!
- Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos.

GR00T N1 is a single end-to-end neural net, from photons to actions:

- Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions.
- Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2.

We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings.

While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right.

Let’s solve robotics, together, one token at a time.

Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Your browser does not support the video tag.

🔗 View on Twitter

💬94🔄386❤️1918👀465798📊493