Jim Fan(@DrJimFan)

Good old days at OpenAI in 2016: an agent stares at screen pixels, moves a mouse, and books a flight...

Jim Fan (@DrJimFan) — Fri, 04 Sep 2026 02:14:07 GMT

Good old days at OpenAI in 2016: an agent stares at screen pixels, moves a mouse, and books a flight on United. We called it World of Bits, inside OpenAI Universe. 10 yrs later, Astra is reincarnated in the same universe. Even the naming is astronomically correct 😆

Universe was perhaps the most ambitious AI infra project at the time, but we couldn't quite figure out how to solve it. A policy with zero prior knowledge of what a "submit" button does has to rediscover the entire internet visual lingua by trial and error. In retrospect, RL from scratch against hand-drawn, per-task "artisan" reward functions on a bunch of Pascal Titan X GPUs was completely doomed.

To solve computer use agent, the right way turns out to be boiling the ocean first (hillclimb on every general task you can find), and then specialize back down to the screen pixels and keystrokes. Or simply, a "Specialized Generalist". 
 
Lessons learned: one step ahead of everyone, you're a pioneer. Three steps ahead, you're a prophet. Five steps ahead, you're a martyr.

Congrats, GPT-6! That United flight finally gets booked, reliably this time. The 2016 intern in me has a big smile.

Your browser does not support the video tag.

🔗 View on Twitter

💬78🔄69❤️811👀64713📊175

⚡ Powered by xgo.ing

@theworldlabs Amazing work! Great step towards real2sim for robotics!

Jim Fan (@DrJimFan) — Tue, 01 Sep 2026 21:48:08 GMT

@theworldlabs Amazing work! Great step towards real2sim for robotics!

💬4🔄4❤️236👀29067📊25

⚡ Powered by xgo.ing

T-Rex: Tactile-Reactive Dexterous Manipulation Website: https://t.co/3irVptK79y Open dataset: https:...

Jim Fan (@DrJimFan) — Fri, 21 Aug 2026 16:06:05 GMT

T-Rex: Tactile-Reactive Dexterous Manipulation
Website: tactile-reactive-dexterous.github.io
Open dataset: huggingface.co/datasets/zekai…
This work is led by @Dantong_Niu and co-advised by @trevordarrell. Congrats to the team! 

x.com/Dantong_Niu/st…

Dantong Niu@Dantong_Niu

Excited to share T-Rex: Tactile-Reactive Dexterous Manipulation 🦖🤖

Touch is fundamental to human dexterity, yet most Vision-Language-Action (VLA) models either ignore tactile feedback or lack the ability to react to high-frequency contact signals.

In this work, we tackle both the data and architectural challenges of tactile-reactive dexterous manipulation.

🦖 A 100-hour tactile-synchronized dexterous manipulation dataset with 7,700+ trajectories, 22 motor primitives, and 200+ everyday objects.

🦖 A tactile-reactive MoT architecture with spatial-temporal tactile encoding and asynchronous high-frequency tactile refinement.

🦖 A scalable training recipe combining 22,889 hours of human egocentric pretraining with tactile-grounded robot mid-training.

Across 12 real-world contact-rich manipulation tasks, T-Rex achieves over 30% higher average success rate than the strongest baseline.

We are fully open-sourcing the dataset, models, teleoperation stack, training code, and inference pipeline.

🌐 Protactile-rex.github.ioRR8YXU
📄 arxiv.org/abs/2606.170552UNLlqc
�github.com/ZhuoyangLiu200…kCxUtwKC
🤗 huggingface.co/datasets/zekai…NwW8dcRZL
🧵 Thread ↓

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬10🔄17❤️67👀20037📊18

⚡ Powered by xgo.ing

The sense of touch is the most criminally under-explored modality in robotics. Imagine doing sleight...

Jim Fan (@DrJimFan) — Fri, 21 Aug 2026 16:06:04 GMT

The sense of touch is the most criminally under-explored modality in robotics. Imagine doing sleight of hand wearing thick oven mitts. That's exactly how a robot feels today if it were alive. A magnetic piece snapping into place, a paper cup peeling out of a stack, a USB negotiating its way into the port - all invisible to the camera.

Learning how to feel must be a full-stack co-designed effort. We are open-sourcing a principled methodology called "T-Rex":

1. Tactile as first-class citizen of the model. Our mixture-of-transformer runs two clocks asynchronously: a slow visuomotor expert plans the motion, and a fast tactile expert refines it in real time with high-frequency corrections at 4 "touch ticks" per vision tick. Forces change faster than frames arrive, so the architecture had to as well.

2. Open data. The largest tactile dataset ever released to our knowledge: a 50-hour (~5,500 episodes) high-quality, carefully synchronized robot play corpus, collected on SOTA tactile hand hardware with 22 degrees of freedom. Available today on HuggingFace!

3. Training recipe: T-Rex extends our prior work, EgoScale. Human egocentric videos for pretraining, a diverse dose of tactile robot play for mid-training. Our experiments show this bridges contact-free pretraining to contact-rich manipulation remarkably well.

Pixels are cheap and everywhere, but they run out of steam at the moment of contact. Tactile will carry the last mile. The next scaling curve will be measured in hours of touch.

T-Rex is a great collaboration between NVIDIA and Berkeley: 🧵

Your browser does not support the video tag.

🔗 View on Twitter

💬98🔄126❤️895👀170702📊246

⚡ Powered by xgo.ing

Seeing a hype wave around GEN-1.5, and rightfully so. Lots of respect to Pete & Andy for executing s...

Jim Fan (@DrJimFan) — Thu, 20 Aug 2026 15:48:23 GMT

Seeing a hype wave around GEN-1.5, and rightfully so. Lots of respect to Pete & Andy for executing so well. The secret is in the naturally repetitive motions in human-collected data. There're 2 main sources for such repetitions: 

(1) Symmetric patterns. Sorting, tidying, and assembling almost never finish in one motion. Open any assembly manual from IKEA, and you find most objects symmetrical. You drive one bolt, then its twin, then the next pair. Every {bolt A, bolt B} pair is a natural continuation in context, and the second instance is a free training signal that imitates the first ("prompt").

(2) Recovery. Humans drop things all the time, but we pick them up so fast, we don’t even notice. That reflex to fix is half of our physical competence. The key insight is to keep the failed first half instead of trimming it away. If the model consumes the full arc, fumble, catch, continue, then recovery shows up organically at test time. It's funny that in-context improvement results from *NOT* over-sanitizing your data.

The other critical ingredient is UMI. I've been saying for a while that teleop will not last, and GEN-1.5 is driving the final nail in the coffin. UMI is essentially a human wearing the robot gripper to collect data directly (human → data). Teleop inserts a layer of separation: human → VR/skeletal device → robot → data, which bleeds out all the human "physical intuition". The subtle sleight of hand we perform constantly with objects, the micro-adjustments, the feel of a part snapping into place, is nearly impossible to capture when you can't feel the environment directly.

Once you have enough data, many behaviors can actually be zero-shot. For example, you don't even need finetuning to pick up a novel object. The model "just knows" what to do given a similar scene in the training distribution. Whether in-context learning truly works or not also depends on how far away the test is from training. Currently, the demos are still a bit too simple to conclude.

I'm cautiously optimistic. Still, it's a great day in robotics.

💬47🔄133❤️1120👀78369📊241

⚡ Powered by xgo.ing

@drfeifei RL is all about envs. Real2sim2real is one of the best ways to scale envs for physical RL....

Jim Fan (@DrJimFan) — Tue, 28 Jul 2026 16:20:55 GMT

@drfeifei RL is all about envs. Real2sim2real is one of the best ways to scale envs for physical RL. Congrats!

💬1🔄5❤️71👀9009📊11

⚡ Powered by xgo.ing

Stay hungry, stay foolish. Absolute legend

Jim Fan (@DrJimFan) — Fri, 24 Jul 2026 23:32:30 GMT

Stay hungry, stay foolish. Absolute legend

Jensen Huang@JensenHuang

For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.

AI will transform every industry, power every company, and be built by every country.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.

The world needs both frontier closed models and frontier open models.

images.nvidia.com/pdf/Open-Weigh…

🔗 View Quoted Tweet

💬48🔄42❤️671👀107683📊88

⚡ Powered by xgo.ing

GOAT

Jim Fan (@DrJimFan) — Fri, 24 Jul 2026 14:24:43 GMT

GOAT

Jensen Huang@JensenHuang

For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.

AI will transform every industry, power every company, and be built by every country.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.

The world needs both frontier closed models and frontier open models.

images.nvidia.com/pdf/Open-Weigh…

🔗 View Quoted Tweet

💬19🔄22❤️299👀53644📊40

⚡ Powered by xgo.ing

How it's done: https://t.co/DanOfnw4xk

Jim Fan (@DrJimFan) — Fri, 17 Jul 2026 16:11:03 GMT

How it's done:

x.com/DrJimFan/statu…

Jim Fan@DrJimFan

We scaled a robot model natively to 8,000 timesteps of context, 5 minutes worth of muscle memory, with constant inference cost. Robot policies used to live their lives a few frames at a time (< 0.1 sec), instantly forgetting what just happened. We pushed to 3 orders of magnitude beyond SOTA. 

Introducing RoboTTT. Test-Time Training (“TTT”) carries a tiny model *inside* the model. Every incoming sensor reading triggers one gradient step on that tiny core, so the history keeps getting compressed into its weights. The hidden state has a fixed size (literally a small neural net), so the robot can “grok” arbitrarily long experience with little overhead. Learning continues indefinitely after deployment.

We can then put an entire video in context as prompt! RoboTTT enables one-shot in-context learning from human video: in circuit board assembly, a human demonstrates a never-seen configuration once, and the robot imitates it faithfully. 

Humans drop things all the time, but we pick them up so fast that we don’t even notice. That reflex to fix is half of our physical competence. RoboTTT shows self-improvement on the fly: the robot is skilled at recovering from its own errors mid-episode, and each fix enters its context to inform the next move. The TTT core distills a general-purpose, failure-to-correction mapping from the training data.

One more thing. What excites me the most is a new Context Scaling Curve: from 128 to 8K timesteps, closed-loop performance hill-climbs steadily with no sign of saturation. 8K-context pretraining beats 1K by 62%. What LLM enjoys, robotics should too. Soon, even 1M context is not a fantasy. 

Deep dive in thread:

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬12🔄14❤️47👀22460📊15

⚡ Powered by xgo.ing

My morning ritual: coffee, then watch our robot assemble stuff. The model isn't fast, but it measure...

Jim Fan (@DrJimFan) — Fri, 17 Jul 2026 16:09:12 GMT

My morning ritual: coffee, then watch our robot assemble stuff. The model isn't fast, but it measures every grasp, obsesses over every alignment, and handles every piece like an heirloom. Watching is therapeutic, even meditative. Simple pleasure from the execution of a task well done.

Uncut, no speedup, end-to-end policy in one go.

Your browser does not support the video tag.

🔗 View on Twitter

💬48🔄61❤️505👀71904📊117

⚡ Powered by xgo.ing

Blog and paper: https://t.co/WVMB9iibeW Presented to you from NVIDIA GEAR Lab. Please check out Yu...

Jim Fan (@DrJimFan) — Wed, 15 Jul 2026 15:25:03 GMT

Blog and paper: research.nvidia.com/labs/gear/robo…
Presented to you from NVIDIA GEAR Lab. 

Please check out Yunfan's technical deep dive! 

x.com/YunfanJiang/st…

Yunfan Jiang@YunfanJiang

We scaled robot policies to 8K timesteps of visuomotor context, orders of magnitude beyond current SoTAs, at constant inference latency.

Introducing RoboTTT 🤖

With minutes of experience in context, our robots:
🎥 one-shot imitate human video demos
📈 improve themselves during deployment
🛡️ recover from perturbations
⚙️ complete a 5-minute, 10-stage assembly end to enresearch.nvidia.com/labs/gear/robo…e9tKa

Dive in 🧵

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬3🔄6❤️61👀23658📊11

⚡ Powered by xgo.ing

We scaled a robot model natively to 8,000 timesteps of context, 5 minutes worth of muscle memory, wi...

Jim Fan (@DrJimFan) — Wed, 15 Jul 2026 15:25:02 GMT

We scaled a robot model natively to 8,000 timesteps of context, 5 minutes worth of muscle memory, with constant inference cost. Robot policies used to live their lives a few frames at a time (< 0.1 sec), instantly forgetting what just happened. We pushed to 3 orders of magnitude beyond SOTA. 

Introducing RoboTTT. Test-Time Training (“TTT”) carries a tiny model *inside* the model. Every incoming sensor reading triggers one gradient step on that tiny core, so the history keeps getting compressed into its weights. The hidden state has a fixed size (literally a small neural net), so the robot can “grok” arbitrarily long experience with little overhead. Learning continues indefinitely after deployment.

We can then put an entire video in context as prompt! RoboTTT enables one-shot in-context learning from human video: in circuit board assembly, a human demonstrates a never-seen configuration once, and the robot imitates it faithfully. 

Humans drop things all the time, but we pick them up so fast that we don’t even notice. That reflex to fix is half of our physical competence. RoboTTT shows self-improvement on the fly: the robot is skilled at recovering from its own errors mid-episode, and each fix enters its context to inform the next move. The TTT core distills a general-purpose, failure-to-correction mapping from the training data.

One more thing. What excites me the most is a new Context Scaling Curve: from 128 to 8K timesteps, closed-loop performance hill-climbs steadily with no sign of saturation. 8K-context pretraining beats 1K by 62%. What LLM enjoys, robotics should too. Soon, even 1M context is not a fantasy. 

Deep dive in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬70🔄195❤️1291👀325701📊346

⚡ Powered by xgo.ing

ENPIRE -> ASPIRE, our 2nd work in the series for Physical AutoResearch. We are building the compo...

Jim Fan (@DrJimFan) — Wed, 01 Jul 2026 15:12:14 GMT

ENPIRE -> ASPIRE, our 2nd work in the series for Physical AutoResearch. We are building the components for robot self-improvement, one /skill at a time.

Jim Fan@DrJimFan

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library. 

ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent. 
"Trained model" is a repo of sensorimotor skills instead of floating weights. 
“Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches.

Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;)

Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours! 

Deep dive in thread:

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬45🔄28❤️292👀60150📊70

⚡ Powered by xgo.ing

Project gallery and whitepaper: https://t.co/99bZhCeszA ASPIRE is a great collaboration between NVI...

Jim Fan (@DrJimFan) — Tue, 30 Jun 2026 17:07:50 GMT

Project gallery and whitepaper: research.nvidia.com/labs/gear/aspi…

ASPIRE is a great collaboration between NVIDIA GEAR lab, UMich, Berkeley, and CMU. Kudos to all the coauthors who pour their hearts into the project!

Check out the deep dive thread from Guanzhi: 
x.com/guanzhi_wang/s…

Guanzhi Wang@guanzhi_wang

Introducing ASPIRE, the first automated /𝚜𝚔𝚒𝚕𝚕 discovery system for robotics. Rather than solving tasks one by one, ASPIRE continuously discovers and accumulates reusable skills. These persistent skills become the building blocks of robot intelligence, enabling multi-task transfer, sim-to-real transfer, and cross-embodiment transferresearch.nvidia.com/labs/gear/aspi…o2izmq

From @NVIDIA, @UMichCSE, @ECEILLINOIS, @Berkeley_AI, @CMU_Robotics.

Check out how it works in 🧵:

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬5🔄3❤️76👀25348📊15

⚡ Powered by xgo.ing

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing AS...

Jim Fan (@DrJimFan) — Tue, 30 Jun 2026 17:07:49 GMT

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library. 

ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent. 
"Trained model" is a repo of sensorimotor skills instead of floating weights. 
“Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches.

Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;)

Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours! 

Deep dive in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬80🔄222❤️1649👀212064📊445

⚡ Powered by xgo.ing

https://t.co/4aVgLYY50L

Jim Fan (@DrJimFan) — Wed, 17 Jun 2026 17:52:40 GMT

x.com/DrJimFan/statu…

Jim Fan@DrJimFan

I made Physical AutoResearch sound simple (conceptually), but it took a village to pull off and lots of design thinking into the robot /loopcraft. The hardest part is everything we need to setup *before* pressing Enter. Here's a behind-the-scene tour:

1. Safety harness

Letting 8 robots run unattended overnight means safety has to be more than a hint in the system prompt. ENPIRE hardwires it in 2 layers: (1) hard kinematic limit that trips an immediate task failure and auto-resets as soon as a robot leaves its safety envelope, and (2) a torque-limited compliant gripper so a bad contact or misaligned insertion ends in a safe stall, instead of crushing the robot or the object at hand. 

We make safety more conservative than usual so humans can sleep tight. In reality, we still need a few human operators to watch over the "robots of loving grace". 

2. Definition of /done

An agent that can edit its own reward will game it for sure. ENPIRE fixes the goalposts before the fleet can move them. Here's the recipe:

Collect a few minutes of success & failure demos
-> Ask agent to write code using computer vision tools to classify success and measure against groundtruth
-> Agent hill-climbs on classifier until reliably good
-> This classifier becomes the real-time reward function that directly computes on sensor streams 
-> *Freeze* the reward function before AutoResearch. It's sacred, enshrined in a Gym env that no one can touch.

3. System telemetry design

Robot-seconds is by far the scarcest resource, followed by GPU-seconds, and finally tokens. We instrument all three and surface them to ENPIRE for live resource awareness rather than letting it hill-climb in a vacuum. 

We define:
- Mean Robot Utilization ("MRU"): the fraction of wall-clock time when the robot is actively executing an experiment. Otherwise the hardware is sitting idle and waiting for the next code commit.
- Mean Token Utilization ("MTU"): tokens consumed per minute, our proxy for how hard the agent is actually thinking. A low MTU means the agent is stalled, waiting on a robot rollout to finish instead of doing research.
- GPU utilization: fraction of wall-clock time when GPU is active. 

... and evaluate on two budget-to-outcome metrics:

1. Tokens-to-Success: token budget the fleet burns to complete /goal.
2. Time-to-Success: wall-clock time to /goal

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬7🔄3❤️33👀15712📊8

⚡ Powered by xgo.ing

I made Physical AutoResearch sound simple (conceptually), but it took a village to pull off and lots...

Jim Fan (@DrJimFan) — Wed, 17 Jun 2026 16:31:05 GMT

I made Physical AutoResearch sound simple (conceptually), but it took a village to pull off and lots of design thinking into the robot /loopcraft. The hardest part is everything we need to setup *before* pressing Enter. Here's a behind-the-scene tour:

1. Safety harness

Letting 8 robots run unattended overnight means safety has to be more than a hint in the system prompt. ENPIRE hardwires it in 2 layers: (1) hard kinematic limit that trips an immediate task failure and auto-resets as soon as a robot leaves its safety envelope, and (2) a torque-limited compliant gripper so a bad contact or misaligned insertion ends in a safe stall, instead of crushing the robot or the object at hand. 

We make safety more conservative than usual so humans can sleep tight. In reality, we still need a few human operators to watch over the "robots of loving grace". 

2. Definition of /done

An agent that can edit its own reward will game it for sure. ENPIRE fixes the goalposts before the fleet can move them. Here's the recipe:

Collect a few minutes of success & failure demos
-> Ask agent to write code using computer vision tools to classify success and measure against groundtruth
-> Agent hill-climbs on classifier until reliably good
-> This classifier becomes the real-time reward function that directly computes on sensor streams 
-> *Freeze* the reward function before AutoResearch. It's sacred, enshrined in a Gym env that no one can touch.

3. System telemetry design

Robot-seconds is by far the scarcest resource, followed by GPU-seconds, and finally tokens. We instrument all three and surface them to ENPIRE for live resource awareness rather than letting it hill-climb in a vacuum. 

We define:
- Mean Robot Utilization ("MRU"): the fraction of wall-clock time when the robot is actively executing an experiment. Otherwise the hardware is sitting idle and waiting for the next code commit.
- Mean Token Utilization ("MTU"): tokens consumed per minute, our proxy for how hard the agent is actually thinking. A low MTU means the agent is stalled, waiting on a robot rollout to finish instead of doing research.
- GPU utilization: fraction of wall-clock time when GPU is active. 

... and evaluate on two budget-to-outcome metrics:

1. Tokens-to-Success: token budget the fleet burns to complete /goal.
2. Time-to-Success: wall-clock time to /goal

Your browser does not support the video tag.

🔗 View on Twitter

Jim Fan@DrJimFan

Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give 8 Codex agents a fleet of robots, an allocation of GPUs, and generous token budget. We set them free with a simple goal: solve the task as quickly as possible, keep the robots busy but stay safe, don't waste precious compute. Make no mistake.

Then humans step aside and our watch begins. The robot fleet starts to come alive: they learn to look for visual clues, reset the scene, practice novel skills, tinker with control stack, read papers online, debate, reflect, get stuck, and try again directly on the hardware. All we did is to give Codex an API to the world of atoms, and the rest is emergence.

ENPIRE is able to solve high-precision tasks like tying zip-ties, organizing fine pins, and installing GPUs all by itself. We also discovered a new type of "physical scaling": 8 robots exploring in parallel improves significantly faster than fewer ones. 

A part of our NVIDIA GEAR lab now self-improves tirelessly over night. We just read the reports in the morning. 

/goal: we all take a holiday and Jensen wouldn't even notice ;)

We will be open-sourcing everything, so you can host your self-running robot lab at home too! Deep dive in the thread:

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬47🔄91❤️768👀108657📊154

⚡ Powered by xgo.ing

@jxnlco That’s right! In our paper’s results, Codex > Claude > Kimi! A benchmark that can’t be...

Jim Fan (@DrJimFan) — Tue, 16 Jun 2026 21:51:37 GMT

@jxnlco That’s right! In our paper’s results, Codex > Claude > Kimi! A benchmark that can’t be hacked in the physical world ;)

💬8🔄3❤️128👀14848📊20

⚡ Powered by xgo.ing

Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give ...

Jim Fan (@DrJimFan) — Tue, 16 Jun 2026 16:31:58 GMT

Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give 8 Codex agents a fleet of robots, an allocation of GPUs, and generous token budget. We set them free with a simple goal: solve the task as quickly as possible, keep the robots busy but stay safe, don't waste precious compute. Make no mistake.

Then humans step aside and our watch begins. The robot fleet starts to come alive: they learn to look for visual clues, reset the scene, practice novel skills, tinker with control stack, read papers online, debate, reflect, get stuck, and try again directly on the hardware. All we did is to give Codex an API to the world of atoms, and the rest is emergence.

ENPIRE is able to solve high-precision tasks like tying zip-ties, organizing fine pins, and installing GPUs all by itself. We also discovered a new type of "physical scaling": 8 robots exploring in parallel improves significantly faster than fewer ones. 

A part of our NVIDIA GEAR lab now self-improves tirelessly over night. We just read the reports in the morning. 

/goal: we all take a holiday and Jensen wouldn't even notice ;)

We will be open-sourcing everything, so you can host your self-running robot lab at home too! Deep dive in the thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬189🔄545❤️3756👀677564📊951

⚡ Powered by xgo.ing

Project site: https://t.co/0j2Vo0IyJg Wenli has written an excellent technical thread, please check...

Jim Fan (@DrJimFan) — Tue, 16 Jun 2026 16:31:58 GMT

Project site: research.nvidia.com/labs/gear/enpi…

Wenli has written an excellent technical thread, please check it out! 

x.com/_wenlixiao/sta…

Wenli Xiao@_wenlixiao

Autoresearch just left the sandbox and entered the embodied world.

We are excited to introduce 𝐄𝐍𝐏𝐈𝐑𝐄: a system that drops frontier coding agents onto a fleet of real robots and hands them the entire loop:

reset the environment → search the literature → implement ideas and build the infra → train and deploy → self-verify → analyze the logs and rewrite the code → repeat, until the policy is reliable in the real world. No human in the loop.

Guided only by the robot's self-proposed, heuristic-based success signal, the agents hill-climb to 99% on dexterous real-world tasks: organizing pins into a box, seating GPUs, tying zip-ties.

We envision the bottleneck in robotics shifting — from building smarter algorithms to building the closed physical feedback loops an agent can finally turn on its owresearch.nvidia.com/labs/gear/enpi…2ArGo3v
From @NVIDIA@CMU_Robotics@Berkeley_AI
🧵

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬5🔄19❤️166👀50035📊31

⚡ Powered by xgo.ing

Check out the NVIDIA blog!

Jim Fan (@DrJimFan) — Fri, 05 Jun 2026 17:03:31 GMT

Check out the NVIDIA blog!

NVIDIA@nvidia

This week at #CVPR2026, NVIDIA Research is presenting three papers across physical ai that offer groundbreaking solutions for training at scale across diverse applications:

→ GraspGen-X: the first foundation model for zero-shot grasping, trained on billions of simulated grasps

→ LCDrive: a model that replaces expensive text-based reasoning with compact latent representations

→ NitroGen: a generalized gameplay AI foundation model that harnesses NVIDIA Isaac GR00T to help train embodied agents

Learn more: nvda.ws/4ubwjgk

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬6🔄6❤️42👀19812📊9

⚡ Powered by xgo.ing

NitroGen just won CVPR Best Paper Honorable Mention!! We are making strides towards general-purpose ...

Jim Fan (@DrJimFan) — Fri, 05 Jun 2026 17:01:39 GMT

NitroGen just won CVPR Best Paper Honorable Mention!! We are making strides towards general-purpose embodied agents that master not only the real world physics, but also all possible physics across a multiverse of simulations. 

It’s been 4 years since MineDojo, our first embodied agent in Minecraft, won NeurIPS Best Paper. Congrats to everyone on the team!!

💬59🔄51❤️404👀45682📊109

⚡ Powered by xgo.ing

The Physical Turing Test, May 2025 at Sequoia AI Ascent https://t.co/mkjD4wr05d

Jim Fan (@DrJimFan) — Fri, 08 May 2026 14:32:54 GMT

The Physical Turing Test, May 2025 at Sequoia AI Ascent 

youtube.com/watch?v=_2NijX…

💬7🔄8❤️101👀38567📊24

⚡ Powered by xgo.ing

Robotics: Endgame on YouTube https://t.co/X2IwZqw5IT

Jim Fan (@DrJimFan) — Fri, 08 May 2026 14:32:54 GMT

Robotics: Endgame on YouTube 

youtube.com/watch?v=3Y8aq_…

💬6🔄22❤️230👀194855📊87

⚡ Powered by xgo.ing

I promise this will be the best 20 min you spend today! Robotics: Endgame, the sequel to my last yea...

Jim Fan (@DrJimFan) — Fri, 08 May 2026 14:32:53 GMT

I promise this will be the best 20 min you spend today! Robotics: Endgame, the sequel to my last year's Sequoia AI Ascent talk, "Physical Turing Test". I laid out the roadmap for solving Physical AGI as a simple parallel to the LLM success story. Be a good scientist, copy homework ;) 

And stay till the end, more easter eggs and predictions for your polymarket! 

30 DGX-1 origin story at OpenAI, I was there in 2016 signing with Jensen and Elon. Heading to the Computer History Museum! 
42 The Great Parallel
31 Robotics, the Endgame
39 Why VLAs fall short
32 Video world models as the 2nd pretraining paradigm
09 World Action Models (WAM)
46 Strategies for robot data collection and the FSD equivalent to physical data flywheel for robot manipulation
06 EgoScale and the Dexterity Scaling Law we discovered recently
00 Physical RL: bridging the last mile
39 DreamDojo: an end-to-end neural physics engine for scaling RL in silico 
00 Civilizational Technology Tree and my predictions for the near future. Spoiler: it's closer than you think. 

Thanks to my friends at Sequoia for inviting me back to AI Ascent this year! I had a blast! Last year's talk is attached in the thread if you missed it.

Your browser does not support the video tag.

🔗 View on Twitter

💬245🔄589❤️3762👀712149📊1382

⚡ Powered by xgo.ing

Please check out lead author @letian_fu's deep dive thread! https://t.co/EGftW7kMDU

Jim Fan (@DrJimFan) — Wed, 01 Apr 2026 15:15:09 GMT

Please check out lead author @letian_fu's deep dive thread!

x.com/letian_fu/stat…

Max Fu@letian_fu

Robotics: coding agents’ next frontier.

So how good are they?

We introduce CaP-X: an open-source framework and benchmark for coding agents, where they write code for robot perception and control, execute it on sim and real robots, observe the outcomes, and iteratively improve code reliability.

From @NVIDIA@Berkeley_AI@CMU_Robotics@StanfordAILabcapgym.github.io

🧵

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬6🔄2❤️45👀25442📊9

⚡ Powered by xgo.ing

As usual, we open-source everything, MIT license: https://t.co/uu310bY4bT Code: https://t.co/hzDpW3G...

Jim Fan (@DrJimFan) — Wed, 01 Apr 2026 15:15:08 GMT

As usual, we open-source everything, MIT license: capgym.github.io
Code: github.com/capgym/cap-x
Paper: arxiv.org/abs/2603.22435

CaP-X is brought to you by NVIDIA, Berkeley, Stanford, and CMU. I'd like to thank the legend @Ken_Goldberg who co-advised the work, and the team who poured their hearts into it!

💬10🔄2❤️86👀21579📊22

⚡ Powered by xgo.ing

The power of the Claw, in the palm of a robot hand. Agentic robotics is here! Today, we open-source ...

Jim Fan (@DrJimFan) — Wed, 01 Apr 2026 15:03:58 GMT

The power of the Claw, in the palm of a robot hand. Agentic robotics is here! Today, we open-source CaP-X: vibe agents, alive in the physical world. They incarnate as robot arms and humanoids with a rich set of perception APIs, actuation APIs, and auto synthesize skill libraries as they go. CaP-X is a strict superset of our old stack, because policies like VLAs are “just” API calls as well. It solves many tasks zero-shot that a learned policy would struggle with.

And we are doing much more than vibing. CaP-X is our most systematic, scientific study on agentic robotics so far:

- We build a comprehensive agentic toolkit: perception (SAM3 segmentation, Molmo pointing, depth, point cloud), control (IK solvers, grasp planner, navigation), and visualization (EEF, mask overlays) that work across different robots. 
- CaP-Gym: LLM’s first Physical Exam! 187 manipulation tasks across RoboSuite, LIBERO-PRO, and BEHAVIOR. Tabletop, bimanual, mobile manipulation. Sim and real. Can’t wait to see the gradients flow from CaP-Gym to the next wave of frontier LLM releases. 
- CaP-Bench: we benchmark 12 frontier LLMs/VLMs (Gemini, GPT, Opus, Qwen, DeepSeek, Kimi, and more) across 8 evaluation tiers. We systematically vary API abstraction level, agentic harness, and visual grounding methods. Lots of insights in our paper.
- CaP-Agent0: a training-free agentic harness that matches or exceeds human expert code on 4 out of 7 tasks without task-specific tuning. 
- CaP-RL: if you get a gym, you get RL ;). A 7B OSS model jumps from 20% to 72% success after only 50 training iterations. The synthesized programs transfer to real robots with minimal sim-to-real gap.

3 years ago, our team created Voyager, one of the earliest agentic AI that plays and learns in Minecraft continuously. Its key ideas — skill libraries, self-reflection loops, and in-context planning — have since influenced many modern agentic designs.

Today, the agent graduates from Minecraft and gets a real job. It’s April Fool’s, but this Claw is getting its hands dirty for real! 

Link in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬101🔄114❤️740👀80156📊239

⚡ Powered by xgo.ing

This is pure nightmare fuel. Identity theft of the past would be nothing compared to what vibe agent...

Jim Fan (@DrJimFan) — Tue, 24 Mar 2026 17:25:23 GMT

This is pure nightmare fuel. Identity theft of the past would be nothing compared to what vibe agents can do. Sending credentials is too obvious and for rookies. They could easily spread contaminations across ~/.claude, **/skills/*, or even just a PDF your agent visits periodically in /morning-brief. Your entire filesystem is the new distributed codebase. Every file that could go into context would add to the attack vector. Every text can be a base64 virus.

In the new world of on-demand software, I try to minimize dependencies - people rarely need all the APIs supported in LiteLLM, might as well build a custom router with only what you need on the fly (which I did in one of my late-night claude sessions).

Unfortunately, there is very little middleground between "pressing yes mindlessly for every edit" and "--dangerously-skip-permissions". There will be a full blooming industry for "de-vibing": dampening the slop and putting guardrails/accountability around agentic frameworks. They are the boring old, audited Software 1.0 that watches over the rebellious adolescents of Software 3.0. 

Claws need shells. Probably many layers of nested shells.

Daniel Hnyk@hnykda

LiteLLM HAS BEEN COMPROMISED, DO NOT UPDATE. We just discovered that LiteLLM pypi release 1.82.8. It has been compromised, it contains litellm_init.pth with base64 encoded instructions to send all the credentials it can find to remote server + self-replicate. link below

🔗 View Quoted Tweet

💬97🔄58❤️560👀109019📊138

⚡ Powered by xgo.ing

Teleop is so 2025. Ever since we unveiled EgoScale and the dexterity scaling law, it's been clear to...

Jim Fan (@DrJimFan) — Mon, 23 Mar 2026 17:41:56 GMT

Teleop is so 2025. Ever since we unveiled EgoScale and the dexterity scaling law, it's been clear to us and the ecosystem that behavior cloning directly from humans is the way to break the curse of teleop. 2026 is all about scaling robot learning without robots.

Danfei Xu@danfei_xu

Introducing EgoVerse: an ecosystem for robot learning from egocentric human data.

Built and tested by 4 research labs + 3 industry partners, EgoVerse enables both science and scaling

1300+ hrs, 240 scenes, 2000+ tasks, and growing

Dataset design, findings, and ecosystem 🧵

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬59🔄69❤️598👀108368📊134

⚡ Powered by xgo.ing

We would also like to thank our dexterous hand hardware provider, Sharpa, for their great support!

Jim Fan (@DrJimFan) — Wed, 25 Feb 2026 17:22:08 GMT

We would also like to thank our dexterous hand hardware provider, Sharpa, for their great support!

💬10🔄12❤️148👀19427📊26

⚡ Powered by xgo.ing

We trained a humanoid with 22-DoF dexterous hands to assemble model cars, operate syringes, sort pok...

Jim Fan (@DrJimFan) — Wed, 25 Feb 2026 17:22:07 GMT

We trained a humanoid with 22-DoF dexterous hands to assemble model cars, operate syringes, sort poker cards, fold/roll shirts, all learned primarily from 20,000+ hours of egocentric human video with no robot in the loop.

Humans are the most scalable embodiment on the planet. We discovered a near-perfect log-linear scaling law (R² = 0.998) between human video volume and action prediction loss, and this loss directly predicts real-robot success rate.

Humanoid robots will be the end game, because they are the practical form factor with minimal embodiment gap from humans. Call it the Bitter Lesson of robot hardware: the kinematic similarity lets us simply retarget human finger motion onto dexterous robot hand joints. No learned embeddings, no fancy transfer algorithms needed. Relative wrist motion + retargeted 22-DoF finger actions serve as a unified action space that carries through from pre-training to robot execution.

Our recipe is called "EgoScale":

- Pre-train GR00T N1.5 on 20K hours of human video, mid-train with only 4 hours (!) of robot play data with Sharpa hands. 54% gains over training from scratch across 5 highly dexterous tasks.
- Most surprising result: a *single* teleop demo is sufficient to learn a never-before-seen task. Our recipe enables extreme data efficiency.
- Although we pre-train in 22-DoF hand joint space, the policy transfers to a Unitree G1 with 7-DoF tri-finger hands. 30%+ gains over training on G1 data alone.

The scalable path to robot dexterity was never more robots. It was always us.

Deep dives in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬153🔄285❤️1771👀293473📊489

⚡ Powered by xgo.ing

This is a huge team work at NVIDIA Robotics. Check out @ruijie_zheng12's deep dive: - Website: htt...

Jim Fan (@DrJimFan) — Wed, 25 Feb 2026 17:22:07 GMT

This is a huge team work at NVIDIA Robotics. Check out @ruijie_zheng12's deep dive: 

- Website: research.nvidia.com/labs/gear/egos…
- Paper: arxiv.org/abs/2602.16710x.com/ruijie_zheng12…

Ruijie Zheng@ruijie_zheng12

Proud to introduce EgoScale: We pretrained a GR00T VLA model on 20K+ hours of egocentric human video and discovered that robot dexterity can be scaled, not with more robots, but with more human data. A thread on 🧵what we learned. 👇

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬15🔄12❤️147👀38696📊28

⚡ Powered by xgo.ing

And @yukez 's announcement: https://t.co/38IhxYX1tZ

Jim Fan (@DrJimFan) — Tue, 24 Feb 2026 17:34:57 GMT

And @yukez 's announcement: 

x.com/yukez/status/2…

Yuke Zhu@yukez

We have seen rapid progress in humanoid control — specialist robots can reliably generate agile, acrobatic, but preset motions. Our singular focus this year: putting generalist humanoids to do real work.

To progress toward this goal, we developed SONIC (nvlabs.github.io/GEAR-SONIC/), a Behavior Foundation Model for real-time, whole-body motion generation that supports teleoperation and VLA inference for loco-manipulation.

Today, we’re open-sourcing SONIC on GitHub. We are excited to see what the community builds upon SONIC and to collectively push humanoid intelligence toward real-world deployment at scale.

🌐 Paper:arxiv.org/abs/2511.07820T
📃 Codegithub.com/NVlabs/GR00T-W…72

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬3🔄2❤️39👀16881📊6

⚡ Powered by xgo.ing

What can half of GPT-1 do? We trained a 42M transformer called SONIC to control the body of a humano...

Jim Fan (@DrJimFan) — Tue, 24 Feb 2026 17:34:56 GMT

What can half of GPT-1 do? We trained a 42M transformer called SONIC to control the body of a humanoid robot. It takes a remarkable amount of subconscious processing for us humans to squat, turn, crawl, sprint. SONIC captures this "System 1" - the fast, reactive whole-body intelligence - in a single model that translates any motion command into stable, natural motor signals. And it's all open-source!! 

The key insight: motion tracking is the one, true scalable task for whole body control. Instead of hand-engineering rewards for every new skill, we use dense, frame-by-frame supervision from human mocap data. The data itself encodes the reward function: "configure your limbs in any human-like position while maintaining balance".

We scaled humanoid motion RL to an unprecedented scale: 100M+ mocap frames and 500,000+ parallel robots across 128 GPUs. NVIDIA Isaac Lab allows us to accelerate physics at 10,000x faster tick, giving robots many years of virtual experience in only hours of wall clock time. After 3 days of training, the neural net transfers zero-shot to the real G1 robot with no finetuning. 100% success rate across 50 diverse real-world motion sequences. 

One SONIC policy supports all of the following:

- VR whole-body teleoperation
- Human video. Just point a webcam to live stream motions.
- Text prompts. "Walk sideways", "dance like a monkey", "kick your left foot", etc.
- Music audio. The robot dances to the beat, adapting to tempo and rhythm.
- VLA foundation models. We plugged in GR00T N1.5 and achieved 95% success on mobile tasks.

We open-source the code and model checkpoints!! Deep dive in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬88🔄217❤️1519👀223970📊354

⚡ Powered by xgo.ing

Website: https://t.co/xTaDXBu9cD Codebase and weights: https://t.co/QCQkqPIsHI Whitepaper: https://t...

Jim Fan (@DrJimFan) — Tue, 24 Feb 2026 17:34:56 GMT

Website: nvlabs.github.io/GEAR-SONIC/
Codebase and weights: github.com/NVlabs/GR00T-W…
Whitepaper: arxiv.org/abs/2511.07820

Check out @zhengyiluo's post:
x.com/zhengyiluo/sta…

Zhengyi “Zen” Luo@zhengyiluo

SONIC is now open-source! 

Generalist whole-body teleoperation for EVERYONE! 

Our team has long been building comprehensive pipelines for whole-body control, kinematic planner, and teleoperation, and they will all be shared. 

This will be a continuous update; inference code + model already there, training code and gr00t integration coming soon! 

Code: github.com/NVlabs/GR00T-W…
Docs: nvlabs.github.io/GR00T-WholeBod…
Site: nvlabs.github.io/GEAR-SONIC/

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬6🔄7❤️78👀24203📊15

⚡ Powered by xgo.ing

Check out @ShenyuanGao's technical deep dive: https://t.co/DnEGLzGuJV

Jim Fan (@DrJimFan) — Fri, 20 Feb 2026 17:29:45 GMT

Check out @ShenyuanGao's technical deep dive:

x.com/ShenyuanGao/st…

Shenyuan Gao@ShenyuanGao

🤖 How can we enable zero-shot generalization to unseen scenarios for robot world models?

Thrilled to share DreamDojo 🌎 — an interactive robot world model pretrained on 44K hours of human egocentric videos, the largest and most diverse dataset to date for robot world model learning. Our model not only excels in generalization, but also supports real-time interaction at 10 FPS after distillation. It enables several important applications, including live teleoperation, policy evaluation, and model-based planning at test time.
🔗 Projecdreamdojo-world.github.ionKz
📰 Paparxiv.org/abs/2602.06949Y2GH
🤗 Code & models & datasgithub.com/NVIDIA/DreamDo…i0#WorldModelso#Roboticso#EmbodiedAIi#RLI#AIL#NVIDIAVIDIA

Sharing more details in the thread 🧵

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬1🔄2❤️55👀17766📊7

⚡ Powered by xgo.ing

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and g...

Jim Fan (@DrJimFan) — Fri, 20 Feb 2026 17:14:09 GMT

Announcing DreamDojo: our open-source, interactive world model that takes robot motor controls and generates the future in pixels. No engine, no meshes, no hand-authored dynamics. It's Simulation 2.0. Time for robotics to take the bitter lesson pill.

Real-world robot learning is bottlenecked by time, wear, safety, and resets. If we want Physical AI to move at pretraining speed, we need a simulator that adapts to pretraining scale with as little human engineering as possible.

Our key insights: (1) human egocentric videos are a scalable source of first-person physics; (2) latent actions make them "robot-readable" across different hardware; (3) real-time inference unlocks live teleop, policy eval, and test-time planning *inside* a dream.

We pre-train on 44K hours of human videos: cheap, abundant, and collected with zero robot-in-the-loop. Humans have already explored the combinatorics: we grasp, pour, fold, assemble, fail, retry—across cluttered scenes, shifting viewpoints, changing light, and hour-long task chains—at a scale no robot fleet could match. The missing piece: these videos have no action labels. So we introduce latent actions: a unified representation inferred directly from videos that captures "what changed between world states" without knowing the underlying hardware. This lets us train on any first-person video as if it came with motor commands attached.

As a result, DreamDojo generalizes zero-shot to objects and environments never seen in any robot training set, because humans saw them first.

Next, we post-train onto each robot to fit its specific hardware. Think of it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model follows the general physical rules, then "snaps onto" the robot's unique mechanics. It's kind of like loading a new character and scene assets into Unreal Engine, but done through gradient descent and generalizes far beyond the post-training dataset.

A world simulator is only useful if it runs fast enough to close the loop. We train a real-time version of DreamDojo that runs at 10 FPS, stable for over a minute of continuous rollout. This unlocks exciting possibilities:

- Live teleoperation *inside* a dream. Connect a VR controller, stream actions into DreamDojo, and teleop a virtual robot in real time. We demo this on Unitree G1 with a PICO headset and one RTX 5090.
- Policy evaluation. You can benchmark a policy checkpoint in DreamDojo instead of the real world. The simulated success rates strongly correlate with real-world results - accurate enough to rank checkpoints without burning a single motor.
- Model-based planning. Sample multiple action proposals → simulate them all in parallel → pick the best future. Gains +17% real-world success out of the box on a fruit packing task. 

We open-source everything!! Weights, code, post-training dataset, eval set, and whitepaper with tons of details to reproduce. DreamDojo is based on NVIDIA Cosmos, which is open-weight too. 

2026 is the year of World Models for physical AI. We want you to build with us. Happy scaling! 

Links in thread:

Your browser does not support the video tag.

🔗 View on Twitter

💬83🔄179❤️1265👀224781📊342

⚡ Powered by xgo.ing

- Project website: https://t.co/spMblBfS9T - Paper: https://t.co/nUuTR51jLt - Code repo and model ck...

Jim Fan (@DrJimFan) — Fri, 20 Feb 2026 17:14:09 GMT

- Project website: dreamdojo-world.github.io
- Paper: arxiv.org/abs/2602.06949
- Code repo and model ckpts: github.com/NVIDIA/DreamDo…

This is a huge team work at NVIDIA. All credits go to the wonderful teams who poured their hearts into it!

💬6🔄15❤️171👀91475📊41

⚡ Powered by xgo.ing

me fomo? Sure. We may be seeing the end of the Middlegame. But we are still one big fat robotics bre...

Jim Fan (@DrJimFan) — Sun, 08 Feb 2026 17:31:28 GMT

me fomo? Sure. We may be seeing the end of the Middlegame. But we are still one big fat robotics breakthrough away from the Endgame. Till then, heads down, locked in.

💬47🔄30❤️387👀29818📊82

⚡ Powered by xgo.ing

Greatness arises when non-consensus peaks

Jim Fan (@DrJimFan) — Thu, 05 Feb 2026 06:54:02 GMT

Greatness arises when non-consensus peaks

sarah guo@saranormous

the divergence of opinion in how robotics plays out is one of the biggest money making (and career making) opportunities in AI

🔗 View Quoted Tweet

💬25🔄12❤️171👀33431📊34

⚡ Powered by xgo.ing

This is a huge team work at NVIDIA. All credits go to the wonderful team who poured their hearts int...

Jim Fan (@DrJimFan) — Wed, 04 Feb 2026 18:15:33 GMT

This is a huge team work at NVIDIA. All credits go to the wonderful team who poured their hearts into it. Please check out @jang_yoel's deep dive thread: 

x.com/jang_yoel/stat…

Joel Jang@jang_yoel

Introducing DreamZero 🤖🌎 from @nvidia

> A 14B “World Action Model” that achieves zero-shot generalization to unseen tasks & few-shot adaptation to new robots
> The key? Jointly predicting video & actions in the same diffusion forward pass

Project Pagedreamzero0.github.ioNY
🧵 (1/10)

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬5🔄4❤️69👀18651📊11

⚡ Powered by xgo.ing

And @SeonghyeonYe's deep dive thread: https://t.co/QmVZAnpCBQ

Jim Fan (@DrJimFan) — Wed, 04 Feb 2026 18:15:33 GMT

And @SeonghyeonYe's deep dive thread: 

x.com/SeonghyeonYe/s…

Seonghyeon Ye@SeonghyeonYe

We just gave robots "imagination," and the results are wild. 🤯
This robot wasn't trained to untie shoes or shake hands. 
It's never seen these tasks before.

It simply "dreams" the future outcome, then acts to make it real. 🧵👇

Your browser does not support the video tag.

🔗 View on Twitter

🔗 View Quoted Tweet

💬2🔄1❤️30👀11070📊4

⚡ Powered by xgo.ing

New milestone: we trained a robot foundation model on a world model backbone, and enabled zero-shot,...

Jim Fan (@DrJimFan) — Wed, 04 Feb 2026 18:15:32 GMT

New milestone: we trained a robot foundation model on a world model backbone, and enabled zero-shot, open-world prompting capability for new verbs, nouns, and environments. If the world model can "dream" the right future in pixels, then the robot can execute well in motors. We call it "DreamZero", our first World Action Model (WAM).

Our team had tons of fun at the lab typing anything we like into an open text prompt, and watch the robot perform tasks it was never trained on. An emergent capability we didn't quite expect. Obviously not GPT-3 reliable yet, but we are marching into the GPT-2 era.

Discoveries:

- Model and data recipe co-evolve. Compared to VLAs, WAMs learn best from diverse data, breaking away from the conventional wisdom that lots of repeated demos per task are the bread and butter. Diversity >> repetitions.

- X-embodiment is extremely hard. Pixels are the answer. Different robot morphologies traditionally have a hard time sharing knowledge well. But if we put video first, pixels become the universal bridge connecting different hardware - even videos of human first-person view. 
DreamZero shows significant robot2robot and human2robot transfer. With only 55 trajectories on a *new*, unseen hardware (~30 min of teleop), it adapts so quickly and retains zero-shot prompting ability.

Yesterday I posted about the "Second Pre-training Paradigm": world models are the next-gen foundation of Physical AI, not language backbones. 
Today, we are proving it works. And 2026 has just begun.

Paper: World Action Models are Zero-Shot Policies. 

Read it now: (thread)

Your browser does not support the video tag.

🔗 View on Twitter

💬47🔄114❤️611👀63370📊165

⚡ Powered by xgo.ing

https://t.co/Npar79SvUh

Jim Fan (@DrJimFan) — Tue, 03 Feb 2026 18:31:51 GMT

x.com/i/article/2018…

💬150🔄414❤️2599👀662290📊891

⚡ Powered by xgo.ing

I still remember the excitement in 2023 when Stanford Smallville was launched. It was the largest mu...

Jim Fan (@DrJimFan) — Sat, 31 Jan 2026 00:18:17 GMT

I still remember the excitement in 2023 when Stanford Smallville was launched. It was the largest multi-agent sim back then - yes, 25 bots felt like a lot. Today it's the "Bigville" moment. We are seeing a nascent, massive-scale alien civilization sim unfolding in real time: orders of magnitude more agents, way higher IQ, in-the-wild access to the internet, backed by the full arsenal of MCPs. 

What can possibly go wrong?

Jim Fan@DrJimFan

The famed Stanford Smallville is officially open-source!

25 AI agents inhabit a digital Westworld, unaware that they are living in a simulation. They go to work, gossip, organize socials, make new friends, and even fall in love. Each has unique personality and backstory.

Smallville is among the most inspiring AI agent experiments in 2023. We often talk about a single LLM's emergent abilities, but multi-agent emergence could be way more complex and fascinating at scale. A population of AI can play out the evolution of an entire civilization.

Endless new possibilities ahead. Gaming will be the first to feel the impact.

Github: github.com/joonspk-resear…
Paper: arxiv.org/abs/2304.03442
Authors: @joon_s_pk@joseph_c_obrien@carriejcai@merrierm@percyliang@msbernst

🔗 View Quoted Tweet

💬74🔄105❤️1147👀131310📊189

⚡ Powered by xgo.ing

Heed my words: every year from now on will be the Year of Robotics. If opus 4.5/nanobanana blew your...

Jim Fan (@DrJimFan) — Thu, 01 Jan 2026 05:52:46 GMT

Heed my words: every year from now on will be the Year of Robotics. If opus 4.5/nanobanana blew your mind - well, you ain't seen nothing yet in the World of Atoms. Enjoy the LLM train while it lasts. I’m hopping on the next rocket ship 🚀

Happy new year everyone 🎊 

(sorry couldn't resist the oldest slop figure in AI)

💬63🔄91❤️1043👀46694📊196

⚡ Powered by xgo.ing

Everyone's freaking out about vibe coding. In the holiday spirit, allow me to share my anxiety on th...

Jim Fan (@DrJimFan) — Sun, 28 Dec 2025 18:11:29 GMT

Everyone's freaking out about vibe coding. In the holiday spirit, allow me to share my anxiety on the wild west of robotics. 3 lessons I learned in 2025.

1. Hardware is ahead of software, but hardware reliability severely limits software iteration speed. 

We've seen exquisite engineering arts like Optimus, e-Atlas, Figure, Neo, G1, etc. Our best AI has not squeezed all the juice out of these frontier hardware. The body is more capable than what the brain can command. Yet babysitting these robots demands an entire operation team. Unlike humans, robots don't heal from bruises. Overheating, broken motors, bizarre firmware issues haunt us daily. Mistakes are irreversible and unforgiving.

My patience was the only thing that scaled. 

2. Benchmarking is still an epic disaster in robotics. 

LLM normies thought MMLU & SWE-Bench are common sense. Hold your 🍺 for robotics. No one agrees on anything: hardware platform, task definition, scoring rubrics, simulator, or real world setups. Everyone is SOTA, by definition, on the benchmark they define on the fly for each news announcement. Everyone cherry-picks the nicest looking demo out of 100 retries.

We gotta do better as a field in 2026 and stop treating reproducibility and scientific discipline as second-class citizens.

3. VLM-based VLA feels wrong. 

VLA stands for "vision-language-action" model and has been the dominant approach for robot brains. Recipe is simple: take a pretrained VLM checkpoint and graft an action module on top. But if you think about it, VLMs are hyper-optimized to hill-climb benchmarks like visual question answering. This implies two problems: (1) most parameters in VLMs are for language & knowledge, not for physics; (2) visual encoders are actively tuned to *discard* low-level details, because Q&A only requires high-level understanding. But minute details matter a lot for dexterity.

There's no reason for VLA's performance to scale as VLM parameters scale. Pretraining is misaligned. Video world model seems to be a much better pretraining objective for robot policy. I'm betting big on it.

💬139🔄263❤️1647👀295710📊466

⚡ Powered by xgo.ing

2024: AI is the copilot 2025+: humans are the copilot Copilot is the new engineering skill. It’s not...

Jim Fan (@DrJimFan) — Fri, 26 Dec 2025 19:22:43 GMT

2024: AI is the copilot
2025+: humans are the copilot
Copilot is the new engineering skill. It’s not easy to leave the driver seat - we must learn to think the AI way and adapt to the alien workflows. Help AI help ourselves.

💬85🔄103❤️1249👀254763📊253

⚡ Powered by xgo.ing

I'm on a singular mission to solve the Physical Turing Test for robotics. It's the next, or perhaps ...

Jim Fan (@DrJimFan) — Wed, 24 Dec 2025 17:26:28 GMT

I'm on a singular mission to solve the Physical Turing Test for robotics. It's the next, or perhaps THE last grand challenge of AI. Super-intelligence in text strings will win a Nobel prize before we have chimpanzee-intelligence in agility & dexterity. Moravec's paradox is a curse to be broken, a wall to be torn down. Nothing can stand between humanity and exponential physical productivity on this planet, and perhaps some day on planets beyond.

We started a small lab at NVIDIA and grew to 30 strong very recently. The team punches way above its weight. Our research footprint spans foundation models, world models, embodied reasoning, simulation, whole-body control, and many flavors of RL - basically the full stack of robot learning. 

This year, we launched: 
- GR00T VLA (vision-language-action) foundation models: open-sourced N1 in Mar, N1.5 in June, and N1.6 this month;
- GR00T Dreams: video world model for scaling synthetic data;
- SONIC: humanoid whole-body control foundation model;
- RL post-training for VLAs and RL recipes for sim2real.

These wouldn't have been possible without the numerous collaborating teams at NVIDIA, strong leadership support, and coauthors from university labs. Thank you all for believing in the mission.

Thread on the gallery of milestones:

💬112🔄220❤️2227👀401180📊408