Nate’s Substack
Nate B. Jones Podcast
Inside NVIDIA: What a world model actually is, and how it connects to the LLM you already use.
Preview
0:00
-0:27

Inside NVIDIA: What a world model actually is, and how it connects to the LLM you already use.

A conversation with NVIDIA’s Cosmos lead about the next expansion of AI: what world models do, how they connect to LLMs, and the strategic choices that matter as AI learns to act.

NVIDIA builds tools for developers.

That is the place to start if you want to understand why a company most people associate with chips is building models that can help simulate a factory, predict what happens when a robot moves, or generate actions for a machine.

I went to NVIDIA last week to talk with Ming-Yu Liu, the vice president of Cosmos Lab. Cosmos is NVIDIA’s world-model effort. Early in our conversation, Ming-Yu explained the relationship between NVIDIA’s computers and everything the company builds around them. Developers need ways to program the hardware, libraries that make difficult work easier, and models they can use to build applications. NVIDIA supplies those tools. Building them also teaches NVIDIA what its next generation of computers needs to do.

That explanation matters at a moment when so much of the conversation about AI assumes developers are on their way out. The tools are getting better, the range of things people can build is expanding, and more of the work involves deciding what a system should do and making that behavior reliable.

Developers create our worlds. Developers also have to keep them safe.

This becomes especially consequential when the system can affect physical space. Someone chooses the information it receives, the actions it can take, the situations in which it can operate, and the evidence required before deployment. Those choices help determine the experience everyone else has with the technology.

World models expand what developers can attempt. Understanding them also gives you a better way to think about AI generally: why related techniques can produce an essay, generate a video, or help control a machine; why NVIDIA wants to encourage that variety; and where useful expertise remains as the underlying models improve.

Here’s what’s inside:

  • What a world model actually models. The scope depends on the job, and nothing here needs a complete model of the universe.

  • What Cosmos 3 is built out of. A reasoner and a generator, and what each one is for.

  • There is a lot of data in the world, and the missing parts matter. You may see a change without seeing the action that caused it.

  • Why a thousand simulated kitchens could change development. Computation can buy more opportunities to practice and compare possibilities.

  • Why the robot might fold laundry before it cooks your eggs. A task can be hard to execute and still offer a clear way to judge the result.

You can follow all of this from a laptop. The one thing it asks is that you stop picturing AI as a box that produces text.

Subscribers get the full deep dive plus access to Nate’s Library MCP!

Listen to this episode with a 7-day free trial

Subscribe to Nate’s Substack to listen to this post and get 7 days of free access to the full post archives.