# World Models: Enabling the next Ai revolution
- Transcript ID: `724ea6112b01efbc`
- Source tier: `publisher_caption`
- Original language: `en-US`
- Translation status: `agent_translated_reviewed` (134/134 chunks)
- Original source: https://video.ethz.ch/!v/evEaF22xCg_Ge
- Raw evidence: `data/transcripts/official/yann-lecun-eth-2026-evEaF22xCg_Ge.json`
> 中文是派生翻译，英文是留存的源文本。中文未复核时，以英文和原始音视频为准。
## 00:00:00
**中文**
我将谈谈世界模型如何可能成为下一场 AI 革命的推动因素。我想在座或许有很多机器学习研究者，但我要告诉大家一个坏消息：机器学习很糟糕。基本上，当我们把机器的学习能力与人类和动物相比时，差距显然还非常大。
**English / Original**
Yeah, I'll talk about what models possibly the enabler for the next AI revolution. So there's a lot of machine learning people I think in the room perhaps have bad news for you. Machine learning sucks. You know, basically if when we compare the learning abilities of machines with humans and animals, clearly there is a big gap.
## 00:00:31
**中文**
人和动物只需极少的尝试、极少的样本，就能极快学会新任务。人有常识，动物也有物理常识。许多任务即使此前从未遇到，我们也能零样本完成。那么机器怎样才能做到？我们拥有人人都在使用的强大 AI 技术，但它们并不能真正处理现实世界。
**English / Original**
You know, people and animals can learn new tasks extremely quickly and with very few trials, very few samples. People have common sense, animals too physical common sense. There's a lot of tasks that we can accomplish zero shot even if we've never faced them before. And how how do we do this with machines? And you know, we have very powerful AI techniques that everybody is using, but they don't really handle the real world.
## 00:01:01
**中文**
它们应付不了连续、高维且含噪的数据。相比之下，语言很容易，现实世界却很凌乱。语言简单——这与 Vladlen 和 Jitendra 刚才所说的莫拉维克悖论相呼应：人觉得简单的事情，对计算机很难；而人觉得复杂的事情，对计算机反倒没那么难，例如下棋、符号积分、解方程和证明数学定理。
**English / Original**
They don't handle continuous, high dimensional, noisy data. Language is easy by comparison. And so the real world is messy. Language is simple that and and you know disconnects with what you know London said earlier and Jitendra as well. The more avid paradox that you know things that are simple are difficult for computers and things that are complicated for humans turn out to not be that difficult for computers like playing chess, computing integral symbolically and you know solving equations, proving math theorems etcetera.
## 00:01:40
**中文**
一个 10 岁孩子基本能完成我们希望家用机器人完成的事情，而且多数任务无需专门训练：第一次让他做，他就会做——他也许不愿意，但确实能做。为什么任何青少年练习几小时就能学会开车，而自动驾驶公司却拥有数百万小时的训练数据？
**English / Original**
So you know, how is it that a 10 year old can basically do what you would like a domestic robot to do and do most of those tasks without actually being trained to do them? The first time you ask them, they can do it. They may not want to do it, but they can. How come, you know, any teenager can want to drive a car in a few hours of practice, yet the self driving car companies have literally millions of hours of training data.
## 00:02:08
**中文**
即使有数百万小时的数据，它们也无法仅靠模仿人类，让机器达到人类驾驶的可靠程度；否则我们早已有 L5 级自动驾驶汽车了，但事实并非如此。消费汽车领域至多只有 L2 或 L3；机器人出租车则是经过大量工程设计、配备各种传感器等设备的系统。
**English / Original**
And Despite that, they can't use these millions of hours of training data to get a machine to just imitating humans to to drive at the at the same level of reliability. Otherwise we'll have level 5 start driving cars and we don't we don't have them at best in the consumer car business. We have level two or three and the the robotax, ER, engineered, you know, very heavily engineered with various sensors and other things.
## 00:02:39
**中文**
我们不断撞上莫拉维克悖论。如果你认为智能需要扎根于现实，就必须超越它。当然，一些哲学家、尤其一些语言研究者并不认为这是必需的，但我认为是。我们身在瑞士，这是 Jean Piaget 的故乡；他对我影响很大。
**English / Original**
So, you know, we keep bumping into this Moravic paradox and we really have to go beyond this if you believe that intelligence requires grounding. Of course, some philosophers and certainly some language people don't believe that's necessary, but I think it is. So like Vlad Land, where in Switzerland, outside Jean Piaget, he was a big influence on me.
## 00:03:01
**中文**
20 世纪 70 年代末，他在法国与 Noam Chomsky 辩论语言究竟是先天的还是后天学习的。那场辩论留下了与会者发言的文字记录。其中一位曾与 Jean Piaget 共事的 MIT 教授谈到了感知机，大意是：这些简单的机器学习模型竟能学会出乎意料地复杂的任务。
**English / Original**
He had a debate with Noam Chomsky in France in the late 1970s where, you know, they were debating whether language was in it or learned. And they trans, you know, those transcriptions of that debate with, you know, people participating in it. And one of them was, was, was a a guy who had worked with Jean Piaget, who was a professor at MIT and was talking about the perceptron kind of saying, you know, there's a simple machine learning models that I keep, you know, capable of learning surprisingly complex tasks.
## 00:03:45
**中文**
这或许可以作为学习确实可能发生的证据，回应 Chomsky 的观点。这个人是 Seymour Papert，MIT 教授。而在十年前，他合著的一本书通过指出感知机的局限，基本扼杀了整个神经网络领域。
**English / Original**
And, you know, that may be kind of evidence for the fact that learning is possible contribute to what Chomsky was saying. This guy was Seymour Pepper. He was professor at MIT. And 10 years before that, he had written a book that basically killed the entire field of neural net, including the basically pointing out the limitation of the perceptron.
## 00:04:07
**中文**
但十年后，他又在论证这些东西其实值得研究。无论如何，常被归于 Jean Piaget 的一句话是：“智能不是你知道什么，而是你不知道时会做什么。”其实他从未真正说过，这是后人附会；只是其他心理学家用这句话概括了他的思想。
**English / Original**
But here it was 10 years later, arguing for the fact that those things were actually interesting to study. But anyway, so Jean Pierre says intelligence is not what you know. It's what you do when you don't know. In fact, he never actually said this. This is apocryphal. But there are other psychologists who basically kind of distilled his thinking into this sentence, which he never said.
## 00:04:31
**中文**
因此人们常这样引用他。智能不是陈述性知识的积累。LLM 的确积累了陈述性知识——不止于此，但它们之所以有用，主要就是因为能积累大量陈述性知识。智能也不是技能集合。只要投入足够资源，你大概可以造出完成几乎任何特定任务的机器，包括自动驾驶。
**English / Original**
So he's kind of quoted as as saying that. So intelligence is not an accumulation of declarative knowledge. LL ends are an accumulation of declarative knowledge. Not just, but the main reason they're useful is because they can accumulate a lot of declarative knowledge. Intelligence is not a collection of skills. You can probably build a machine to accomplish sort of any task if you spend enough resources on it, including things like self self driving.
## 00:05:00
**中文**
但那并不是真正的智能。智能是大约 20 小时就学会驾驶，或只经少量训练就能学会任何新任务，乃至零样本完成新任务。这才是智能，也是 Jean Piaget 那句话真正表达的意思。这意味着我们不会有简单的智能度量：因为对任何特定任务，只要投入足够的精力和时间，总能攻克它。
**English / Original**
That's not really what intelligence is. Intelligence is the ability to launch a drive in about 20 hours or to, you know, learn any new task with very little training or accomplish new task zero shot. That's really what intelligence is, and that's really what Jean Piaget means. And what that means is that we're not going to have any simple measures of intelligence because, because again, any particular task, you can always, you know, if you spend enough effort and, and time, you can always kind of crack it.
## 00:05:31
**中文**
关键更多在于适应能力。这也与 Vladlen 所说的相连：AGI 这个概念完全没有意义。人类智能是专门化的；人类智能的特征是适应得非常快，能够学习新任务。我们每个人掌握的知识集合和技能都不同。
**English / Original**
So it's more, you know, how adaptive you are. And you know, this, this connects to something that Bladden said, like the notion of AGI is complete nonsense. Human intelligence is specialised with the characterisation of human intelligence. It's that it's very quickly adaptive and we can learn new task. We all, all of us know different sets of knowledge and have different skills.
## 00:06:00
**中文**
这是因为我们接触过不同环境、解决过不同问题。我们能够适应，这才是真正的智能。那么人类——以及动物——怎样学习？生命最初几个月会发生大量学习，而且主要通过观察完成。
**English / Original**
And so it's because we've been exposed to, you know, different environments and we've we've had to solve different problems. We're adaptive. That's really what intelligence is. OK, so how do how do humans learn and animals for that matter? There's a lot of learning that takes place in the early months of life, mostly by observation.
## 00:06:23
**中文**
两个月大的婴儿会挥动四肢，能够形成自身肢体的动力学模型，但基本还无法影响外界：不能移动物体之类。尽管如此，他仍能了解世界的许多事情。婴儿很快就能学会的一点是：世界是三维的。
**English / Original**
So, you know, two months old baby, you know, we can gesticulate so can develop a dynamical model of its own limbs, but basically cannot affect the world. You know, it can't move an object or anything, but it does it, you know, can learn a lot of things about the world. One thing a baby can learn really quickly is that the world is 3 dimensional.
## 00:06:47
**中文**
为什么？因为世界中每一点都与我们有距离；当我们移动头部时，用这一事实最能解释视野为何改变。当然，婴儿未必主动转头，但会被抱着移动，因此他们看到视差，并由此推断世界是三维的。
**English / Original**
Why? Because the fact that an object has a distance every every point in the world has a distance from us is the best way to explain how or view of the world changes when we move our head. And of course babies don't move necessarily their head, but they are being moved. So they they they see paradox and sort of derive from this the fact that the world is 3 dimensional.
## 00:07:13
**中文**
今天的学习机器也能做到这一点：仅被动观看视频，就能学到世界是三维的。这很有意思——我先把手机静音。物体恒存、稳定性、刚性等基本概念也能很快学会。
**English / Original**
We can do this with learning machines today. They wonder that the world is 3 dimensional only by being exposed passively to videos. So that's that's an interesting thing and I'm going to silence my phone because it's actually OK. So, so basic concepts like object permanence is this is one really quickly notions of stability, rigidity and things like that.
## 00:07:51
**中文**
但我们所谓的直觉物理，例如惯性和重力，人类婴儿要到九个月左右才掌握；多数动物更快。把八九个月的婴儿放在高脚椅上，再放上一堆玩具，他很可能会系统地把所有玩具扔到地上，并观察结果。
**English / Original**
But then what we would consider intuitive physics, things like inertia, gravity, that actually takes nine months for human infants, shorter for most animals. And if you put a 8 month old on a high chair and you put a bunch of toys eight or nine months old, the child will most likely systematically take all the toys and throw them on the floor and watch the result.
## 00:08:23
**中文**
他们是在做实验，验证重力是否适用于一切，因此这需要很长时间。这种学习如何发生？其中包含什么类型的学习？他们会做实验，但也能仅靠观察学习重力。比如画面底部这个场景：一辆小车位于平台上，被推离平台后却悬浮在空中。
**English / Original**
They're doing the experiment that gravity actually applies to everything, so that takes a long time. How does that happen? What type of learning is taking place there? They're doing the experiment, but they can learn about gravity just for observation as well. So if you if you show the the scenario here at the bottom where a car is on a platform, you push it off the platform, it appears to float in the air.
## 00:08:46
**中文**
六个月大的婴儿几乎不会在意，因为还没学会重力；十个月大的婴儿则会像图中的小女孩一样非常惊讶。心理学家正是通过“预期违背”来判断婴儿是否学会了某个世界概念。我们也可以用这类技术测试机器学习系统是否获得了某种常识。
**English / Original**
A six month old will barely pay attention, hasn't learned about gravity yet. A 10 month old will go very surprised, like the little girl. And that's actually how psychologists measure whether a baby has learned a particular concept about the world, which is the violation of expectation. And we actually, we can actually use those techniques to test whether machine learning system has have acquired some, you know, some notion of common sense.
## 00:09:14
**中文**
关于这一点还有很多可说。Jitendra 和我合作过一篇论文，实际上主要由 Emmanuel Dupoux 撰写，我和 Jitendra 的贡献很少，讨论的就是这一整组问题。但如果智能既不是技能积累，也不是陈述性知识积累，那它到底是什么？正如我所说，是无需事先训练便能完成新任务、解决新问题的能力。
**English / Original**
So there, there's a lot that can be said about this. Jitendra and I collaborated on on a paper here, mostly written by Emmanuel Dupu actually. And Jitendra have very little contributions to it on, on, on this, this whole kind of set of questions. OK, but what is really, what is intelligence really if it's not an accumulation of, of skills nor an accumulation of declarative knowledge is the ability to accomplish new task, as I said, solve new problems with, with without prior training.
## 00:09:48
**中文**
再次强调，“AGI”作为一个词组没有意义。人类智能是专门化的。问题不是“你是否什么都会”，而是“你能否迅速学会几乎任何事情，或至少很广泛的一组事情”。下方是一篇稍带哲学意味、由我的一些年轻同事撰写的论文。
**English / Original**
And again, AGI has no, makes no sense as a, as a, as a phrase, human intelligence is specialised and the the question is, is not do you know how to do everything? Is it, can you learn quickly how to do anything or a wide spectrum of of things? It's a little kind of somewhat philosophical paper here at the bottom, written by some of my young colleagues.
## 00:10:15
**中文**
这里有个简单计算。仍有许多人，尤其美国西海岸的人，相信只要扩大 LLM 的规模——也许用合成数据训练，再在后训练和强化学习中加一些技巧——就能达到他们所谓的 AGI。我认为这是不可能的。
**English / Original**
So here is a simple calculation. There's still a lot of people, particularly on the West Coast of the US, who believes that we're going to reach what they call AGI, OK, by by scaling up LL Ms. maybe training them on synthetic data, maybe, you know, using a few tricks in post training and reinforcement learning. And I think that's impossible.
## 00:10:42
**中文**
如果你愿意，可以说我是“扎根智能”的信奉者。做个简单计算：今天典型的 LLM 大约用 20 万亿个词训练，对应约 30 万亿个 token；每个 token 大约 3 字节，因此数据量约为 10 的 14 次方字节。
**English / Original**
I'm a believer in sort of grounded intelligence if you want. But but you can do this simple calculation and LLM, you know, typical LLM of today is trained on something like 20 trillion words. That corresponds to about 30 trillion tokens, and each token is 3 bytes, something like that. So the data volume is about 10 to the 14 bytes.
## 00:11:07
**中文**
任何人要读完这些内容，大约需要 40 万年。再比较一个四岁儿童一生中看过的信息：约为每天 16 小时的清醒时间；顺便说，这点视频量只相当于 YouTube 约 30 分钟的上传量。我们有 200 万根视神经纤维，每根每秒传输约 1 字节。
**English / Original**
This would take about 400,000 years for any human to read. Then compare this with what a a four year old has seen during his or her life. That's about 16 hours of wait time and which by the way, the small amount of video is about 30 minutes of YouTube uploads. And we have 2 million optic nerve fibres carrying about 1 byte per second each.
## 00:11:31
**中文**
所以，一个四岁儿童通过视觉——可能还包括触觉——接收的数据量也约为 10 的 14 次方字节。四年视觉经验的数据量，等于 40 万年的文本阅读量。即使把互联网上所有公开的人类文本都用上，仅靠文本训练也不可能得到类人智能。
**English / Original**
So the data volume that a four year old has seen through vision and probably through touch as well is about 10 to the 14 bytes. So 4 year old through vision, same amount of data as 400,000 years through text. With all the human produced text available on publicly on the Internet, we're not going to get to anything like human like intelligence by just training on text.
## 00:11:58
**中文**
这就是不会发生。当然你可以说视频比文本冗余得多，但实际上这是特性，不是缺陷。要训练系统，特别是用自监督学习训练，就需要数据中存在冗余；没有冗余，什么也学不到。
**English / Original**
It's just not going to happen. So of course you get to say, well, video is much more redundant than than text, but in fact that's a feature, not a bug. If you want to train a system, particularly using self supervised learning, you need redundancy in the in the data. If you don't have redundancy, you can learn anything.
## 00:12:23
**中文**
所以冗余是好事，只是也不能过多。接下来是智能系统应具备什么性质的问题。在我看来，一个重要性质是推理方式：它是通过某个神经网络固定层数的前向传播来计算输出吗？
**English / Original**
So redundancy is a good thing. You don't want too much of it though. OK, so then there is another question about what are the right properties of intelligent systems. And in my opinion, an important property of an intelligent system is the mode of inference. Does it compute its output by propagating through a fixed number of layers of some neural net?
## 00:12:47
**中文**
还是采用另一种方式：通过搜索与输入最相容的输出来计算系统输出？也就是说，你观察一个情境，经由感知模块得到某种对当前世界状态的表征。
**English / Original**
Or consider the alternative. The alternative is computing the output of a system by searching for an output that is most compatible, if you want, with the input. OK, so you observe a situation that runs with some perception module that produce some sort of representation of the current state of the world as you observe it.
## 00:13:09
**中文**
你可以直接产生动作，那就是反应式系统。也可以先设想一个动作，再让智能系统判断：对当前观察而言，这是个好动作吗？它能完成我想要的任务吗？这里的目标函数描述了系统想完成的任务是否已经完成。
**English / Original**
You can directly produce an action. OK, that's a reactive system if you want. Or you could imagine an action and then have an intelligent system. The intelligent system figure out is this a good action for this observation? Is this something that you know will accomplish the task I want? So the objective here characterises whether the task the system wants to accomplish has been accomplished or not.
## 00:13:37
**中文**
把它看成成本函数。它不用于学习，而用于推理；可以把它理解为概率推断模型中的负似然，或者像我更愿意的那样，理解为能量函数。推理本质上是在推理时搜索一个使能量函数最小的输出。
**English / Original**
Think of it as a cost function. It's not used for learning, it's used for inference. Think of it as, you know, negative likelihood in a probabilistic inference model, or as I prefer to think of it, an energy function. So basically there the inference is a process by which you search for an output that minimises some energy function at inference time.
## 00:14:00
**中文**
从计算能力上说，这天然比仅通过固定层数传播更强。作个对比：左边的模型类似 LLM，把一个输入窗口送入拥有数千亿参数的大型神经网络的固定层数，产生一个 token；再把该 token 移入输入，产生第二个 token，如此继续。
**English / Original**
OK, that's intrinsically more powerful computationally than just propagation through a fixed number of layers and then contrast. So the model on the on the left is sort of LLM like right, take a window of of inputs, run this through a fixed number of layers of some big neural net with a few 100 billion parameters, produce 1 token, OK, then shift that token on the in the input and then produce the second token, etcetera, etcetera.
## 00:14:34
**中文**
这就是自回归预测。生成每个 token 都只执行固定数量的计算，穿过神经网络固定的层数。这不是一个好模型，也不是推理的好模型。迫使 LLM 推理的方法，是设法诱导它生成更多 token。
**English / Original**
That's autoregressive prediction. And every token involves the computation of a don't get fixed amount of computation running through a fixed number of layers of some neural net. This is not a good model. It's not a good model of reasoning. The way you coerce an LLN to do reasoning is that you trick it into generating more tokens.
## 00:14:53
**中文**
但我们并非这样推理。我们在内部推理，不是在 token 空间、甚至也不是在语言中推理。相比之下看右边的模型，它是前一个模型略微特化的版本：你感知世界或环境，得到对世界当前状态的某种认识。
**English / Original**
But that's not the way we reason. We reason internally. We don't reason in token space. Language even. OK, compare this with the model on the on the right, which is a a slight specialisation of the the previous one. You perceive the world or your environment, you get some idea of the state, the current state of the of the world.
## 00:15:17
**中文**
随后你设想一系列动作，即提出一个动作方案，将其输入系统内部的世界模型。世界模型预测结果，再把结果送入目标函数，衡量任务完成到何种程度。然后通过优化，在推理时搜索使该目标最优、使能量最小的动作序列。
**English / Original**
And then you imagine a sequence of actions, a proposal for an action. You feed it to an internal role model for the system and the role model predicts the outcome and then fix feeds this outcome to an objective that measures to what extent a task has been accomplished or not. OK, Then by optimization, you search for an action sequence that optimizes this objective, minimises this energy at inference time.
## 00:15:42
**中文**
到这里我还没谈学习。在我看来，这是强大得多的模型，但你需要一个世界模型。大约五年前，我逐渐确定了这种理念或架构，并在 2022 年把一篇长论文放到网上，其中描述了总体架构等内容。
**English / Original**
I haven't talked about learning yet. OK, In my opinion, there's so much more powerful model, but you need a role model now if you do have so. So I've I've sort of settled on this kind of idea or architecture about five years ago I wrote a long paper about it that I put online in 2022 with some general architecture, etcetera.
## 00:16:10
**中文**
如果要拍照，屏幕上有二维码可以打开。文章相对容易读，只是比较长。核心思想是：推理和规划至关重要，它们通过能量最小化、而不是前向传播来进行；要实现这一点，就需要某种世界模型。
**English / Original**
If you want to take pictures, your QR codes, you can get to it and it's, you know, relatively easy to read, but kind of long. And, and it's really based on this idea that reasoning and planning are essential and they basically proceed by energy minimisation rather than forward propagation. And that for this to work, you need some more model.
## 00:16:39
**中文**
过程与刚才相同，只多了几个技巧。你观察环境，感知模块产生初始世界状态的表征，但它只代表当前感知到的内容。因此你可能必须把它与记忆中的内容结合，才能得到对世界状态——至少是你所知部分——更完整的认识。
**English / Original**
OK, so same process as I described before. There is a few a few additional tricks You observe. The environment perception module produce a representation of the initial state of the world, but only a representation of what you currently perceive. So you may have to combine this with the content of a memory to get a complete complete idea of the the state of the world, of what you know about it at least.
## 00:17:04
**中文**
然后把该状态与候选动作序列一起输入世界模型，世界模型预测动作序列的结果。再把结果输入目标函数，也就是衡量某个任务完成程度的能量函数。任务完成时输出 0；未完成时输出某个正数，也许还能衡量距完成任务还有多远。
**English / Original**
Then you feed this to your one model together with a proposal for an action sequence and and your one model predicts the outcome of that action sequence. You feed this to an objective, an energy function that measures to what extent a particular task has been accomplished. So this function output 0 if the task is accomplished and some you know positive number is attack is not accomplished and perhaps measure some distance to the task being accomplished.
## 00:17:33
**中文**
此外还可以设置另一组作为护栏的目标，确保系统将世界带过的任何状态序列都不会杀害或伤害任何人，也不会产生任何有害影响。这样构建的系统可以具有内在安全性，因为它产生每个输出时都必须服从并优化护栏目标与任务目标。
**English / Original**
So until you see, you can have another set of objectives that are guard rails that would ensure that whatever state sequences the the system is going to take the world through, it's not going to kill anyone or hurt anyone or, you know, have any kind of Datarius effect. And so a system constructed this way can be made intrinsically safe because it has to obey and, you know, optimise the guardrail objective with every output it produces.
## 00:18:04
**中文**
LLM 并非如此。要让 LLM 安全、无毒或达到你想称呼的状态，唯一方法是微调它；而条件约束总有办法被突破，也就是越狱。这里这种系统无法被越狱，因为它除了优化护栏目标和任务目标之外，什么也做不了。
**English / Original**
This is not the case for an LLM, an LLM. The only way an LLM can be made, you know, safe or non-toxic or whatever you want to call it is by fine tuning it. And there is always a way to break the conditioning. If you want to jailbreak the the system here, you can't jailbreak a system like this. It, it can do nothing but optimise the guardrail objectives and and the task objective.
## 00:18:31
**中文**
当然，有了世界模型——在座肯定有很多机器人系统和最优控制研究者——就可以跨多个时间步反复应用它，把动作序列分解成一系列动作，并在序列的每一步应用护栏。这就是世界模型的用法；通过优化进行规划的方式类似模型预测控制（MPC），是可追溯到 20 世纪 60 年代的经典最优控制方法。
**English / Original**
Of course, if you have a one model, there's certainly a lot of robotic system optimal control people in the room. You can apply this role model multiple time steps and you know each action, an action sequence can be decomposed into a sequence. The guardrails can be applied to all the the steps in the sequence. OK, that that's the way you would use a role model and the way you plan by optimization there is akin to model predictive control MPC, very classical stuff in optimal control going back to the 1960s.
## 00:18:59
**中文**
但我们最终想要的是能做分层规划的东西。我们所有人都会分层规划，动物也会。什么是分层规划？假设我坐在纽约大学办公室里，想在明天到达巴黎。我不可能用肌肉动作来规划整段旅程，每 10 毫秒规划一次——那是人类能执行的基本动作粒度。
**English / Original**
Ultimately what you want, though, is something that can do hierarchical planning. All of us do hierarchical planning. Animals do hierarchical planning. What is hierarchical planning? Let's say that I'm sitting in my office at NYU and I want to be in Paris tomorrow. There's no way I can plan my entire trip to Paris in terms of kind of muscle actions, 10 millisecond by 10 millisecond, which are the elementary actions that humans can can do.
## 00:19:30
**中文**
我做不到，首先因为序列太长，其次因为信息不足：我不知道走到街上后要等多久才有出租车停下。所以不可能把整件事一次规划完，必须做分层规划。高层上，我可以说：不知道去机场到底要多久，但大概一小时到一个半小时。
**English / Original**
I can't do that because first of all, it's too long, but second of all, I don't have the information. I don't know if when I'm going down on the street, how long I'm going to have to wait before a taxi stops, right? So there's no way I can plan the entire thing. I have to do hierarchical planning. So what I have to do is at a high level, I have to say, well, I don't know how long it's going to take me to go to the airport, but you know, maybe maybe roughly an hour, an hour and a half.
## 00:19:54
**中文**
所以我需要先到机场，再赶上飞机。这是一个两步的高层计划，不需要知道许多细节。现在我有了一个子目标：到达机场。我人在纽约，去机场涉及下楼、走到街上、招一辆出租车，再前往机场。
**English / Original**
So I need to get to the airport and catch a plane. OK, that's a 2 step high level. Plan. I don't need to know many details to make that plan. And now I have a sub goal which is being at the airport. I'm in New York, so going to the airport involves going down on the street and catching, hailing a taxi and go to the airport.
## 00:20:15
**中文**
接着还要走到街上：在任何一栋楼里，这都意味着走到电梯、按按钮、下楼并走出门。于是又有一个目标——先到电梯。如此沿层级不断向下，最终会到达某个行动非常简单的层级。
**English / Original**
Now I need to go down on the street. I mean, anyway, you building that involves walking to the elevator, pushing the button, getting down and walking out the door. Now there's a goal, getting to the elevator, right, etcetera. So you you can sort of go down this entire hierarchy and at some point you get to a point where the action you need to take is very simple.
## 00:20:35
**中文**
那是你熟悉的事情，可能不必动用全部脑力来规划。你大概无需思考就能从椅子上站起来，那可以只由策略完成。但最终我们希望系统能做分层规划。怎样解决？这仍是一个未解问题。
**English / Original**
It's something that you are familiar with. You may not have to use your full mental power to kind of plan the action. You can probably stand up from your chair without having to think about it. That could be just a policy. But essentially, ultimately we want systems to do hierarchical training. How do we solve that? This is an unsolved problem.
## 00:20:55
**中文**
如果你从事机器人、机器人 AI 或智能体 AI，并正准备以此开始博士研究，这是个绝佳题目：它完全开放。没人知道怎么做，至少没人证明自己知道。现在的大问题是：怎样训练这些世界模型？先不考虑分层，暂从非分层模型开始。
**English / Original**
If you're a roboticist or an AI for robotics kind of person or agentic AI kind of person, if you're starting APHD on this topic, this is a great topic. It's completely open. Nobody knows how to do this or nobody has proved that they know how to do this. OK, so now the big question is how, how are we going to train those those well models, OK, hierarchical or not, let's say non hierarchical to start.
## 00:21:23
**中文**
首先必须确定采用什么架构。在当下这个时代，人们自然会想到训练生成模型。事实上，我尝试训练类似世界模型的东西已有约 15 年，前 10 年大多失败，因为我一直在尝试生成模型。
**English / Original**
So first of all, we have to figure out what architecture to give them and a natural instinct in these days, there's an age is to train a generative model. And in fact, I've been working on sort of trying to train one model like things for about 15 years, mostly failing for the 1st 10 because I was trying to train generative models.
## 00:21:47
**中文**
什么是生成模型？自监督学习在语言领域取得了难以置信、令人惊叹的成功。取一串词，删除其中一些、破坏输入，再把受损输入送入大型神经网络，训练它恢复缺失部分。
**English / Original**
OK, what's a generative model? So self supervised learning has been incredibly successful, astonishingly successful in the context of language, right? You take a a string of words, you remove some of the words, you corrupt the the input, and then you run the corrupted input through some big neural net and you train it to recover the missing parts.
## 00:22:11
**中文**
这对文本效果惊人。早期的 BERT 等模型就是这样做的。LLM 是其中一个特例：只删除最后一个词，整个系统只尝试生成序列中的下一个词。只要方法正确，它效果极好，而且能够扩展。
**English / Original**
OK, that works amazingly well for text. So there are original models like Bird that that used to do this. And LLM is a special case of this where the only word you remove is the last one, so that the entire system is trying to just produce the next word in a in a sequence. OK, but it works amazingly well if and it it scales if you do it right.
## 00:22:31
**中文**
但把它用于视频就不奏效。给系统一段视频的开头，要求它在像素层面预测接下来发生什么，实际效果并不好；系统由此得到的视频表征也不够好。
**English / Original**
It doesn't work if you apply it to video. So if you take a a video and then you show the initial segment of the video to the system and you ask it to predict what's going to happen next at the pixel level, it doesn't really work. Like the representations you get out of the system for your video are not particularly good.
## 00:22:49
**中文**
原因是你根本无法预测视频里发生的一切，可能发生的事情有无穷多种。文本容易，是因为词的数量有限；系统可以对词典中所有词或 token 输出一个概率分布。但视频做不到。
**English / Original**
And the reason is you simply cannot predict everything that takes place in a video. There's an infinite number of possible things. In text, it's easy because there is only a finite number of words. And so you can get the system to produce a probability distribution over all possible words or tokens in your dictionary. But you can't do this with video, right?
## 00:23:10
**中文**
可能的视频帧数量大得难以置信。举个例子：如果我拍摄这个房间，从这里开始，让系统续写视频，它或许能预测我们在某种教室或礼堂里，房间大小有限。
**English / Original**
There's just a incredibly large number of possible video frames. Let me take an example. If I take a video of this room, I start here and I cannot slow you outside the camera. I start here and as the system continue the video, you know, it's probably going to predict, you know, we are in some sort of classroom auditorium and you know, the room has a finite size.
## 00:23:33
**中文**
它也许能猜到侧面可能有窗户等，但绝不可能预测你们每个人长什么样，或哪些椅子空着，因为信息根本不存在。训练系统做这种预测会把它“逼死”。当然，你会说：可是我们能训练系统生成好看的视频。
**English / Original**
There might be windows on the side and things like that. There's absolutely no way the system can predict what all of you look like or which, you know, chairs are unoccupied. It's just impossible. You just don't have the information. So when you train a system to make this kind of prediction, you kill it. Now of course you're going to tell me, oh, but we can train system to produce cute videos, right?
## 00:23:54
**中文**
视频生成当然可以，但这种预测通常是在表征空间而非像素空间完成的；只在第二阶段才把预测变成高分辨率、高帧率视频。系统只需生成一段看起来不错的视频，不必表示所有合理的视频可能性。
**English / Original**
Video generation, yes, but this prediction usually is done in representation space, not in pixel space. It's only a second stage that actually turns the predictions into high resolution, high frame rate videos. And the system only needs to produce 1 cute looking video. It doesn't need to actually represent all plausible videos.
## 00:24:24
**中文**
所以那是简单得多的问题。正如我说的，过去 15 年的大部分时间里我都在尝试这件事。这是一篇十年前的论文：我们试着训练神经网络，根据 4 帧上下文预测 2 帧短视频，得到的是模糊预测。
**English / Original**
So which is a much simpler problem. OK. And as I said, I've been kind of attempting to work on this for better part of the last 15 years. So this is a 10 years, 10 years old paper. We try to train some neural net to predict, you know, short video clips, you know, 2 frames from from 4 frames of context, you get blurry predictions.
## 00:24:43
**中文**
为什么？因为系统预测了所有可能情况的平均值。当然可以用扩散模型等潜变量模型修正这一点，但当时我们还不知道。我们尝试过 GAN 等方法，不太成功；潜变量模型、尤其扩散模型或许有帮助，它们当然能生成好看的视频。
**English / Original**
Why? Because the system predicts the average of everything that can happen. Of course you can correct that with latent variable models like diffusion models, but we which we didn't know at the time, we tried to use Gans and stuff like that, wasn't too successful, but you know, perhaps using latent variable. Models with with help, diffusion models in particular, which of course produce cute videos.
## 00:25:06
**中文**
但它们真的理解世界吗？证据表明没有。我的解决方案是我称为联合嵌入、更准确说是联合嵌入预测架构（JEPA）的架构，如右图所示。左边是生成式架构：观察 X，也许还观察动作 A，再观察结果 Y；系统尝试以最细微的细节重建 Y。
**English / Original**
Do they actually understand the world? The evidence is no. So here's my solution. My solution is architecture I called joint embedding or more precisely joint embedding predictive architecture JEPA, which is shown on the right. OK, on the left you have generative architecture. You observe X, Maybe you observe A, an action that is taking place and you observe the result Y and the system is trying to reconstruct Y in its most manual details.
## 00:25:37
**中文**
使用 JEPA 时也观察 X、Y 和 A，但会编码 X 与 Y，预测发生在表征空间中。这是关键差别。系统在构建 Y 的表征时，可以从输入中剔除所有根本不可预测的、关于 Y 的信息。
**English / Original**
With Jappa you observe X&Y and A, but you encode both X&Y and the prediction takes place in that representation space. OK, major difference. What the system can do is essentially eliminate from the input by constructing a representation of why. It can eliminate all the information about about why. That is simply not predictable, right?
## 00:26:06
**中文**
这让预测更抽象、细节更少，但在某种意义上更准确。怎样训练生成模型？很容易，因为成本就是重建成本：训练它进行重建。可以将其训练为自编码器，但必须限制编码的信息容量；也可以用去噪自编码器，这是掩码自编码器等许多技术尝试的方式。
**English / Original**
And that makes the prediction more abstract with fewer details, but more accurate in a way. So, you know, how do you train a generative model? It's easy to train a generative model because the cost is just a reconstruction cost. It's just going to, you know, you're just training it to reconstruct. You can train it as an auto encoder, but then you need to restrict the information content in the code or as denoising auto encoder, which is what a lot of techniques have attempted to do like Max auto encoders and things like that.
## 00:26:37
**中文**
也就是取一个输入，以某种方式破坏它，再训练自编码器恢复原始输入。顺便说，扩散模型是这一广义去噪思路的一个特殊情形。坏消息是：用这类系统学习图像表征，得到的表征并不好。
**English / Original**
So that means taking an input, correcting it in some ways, and then training an auto encoder to recover the initial one. And by the way, diffusion models are a bit of a special case of this, of this sort of general, general thing of denoising. So the bad news is when you train systems of this type to learn representations of images, you don't get good representations.
## 00:27:05
**中文**
如果把这样获得的图像表征送入下游任务，并以监督方式训练一个任务头，结果并不出色。要得到好结果，必须使用联合嵌入架构。所有利用自监督学习来训练图像或视频表征的最佳系统都是如此。
**English / Original**
If you use the representation of images obtained this way, you feed it to a downstream task that you train, supervise, OK, you train ahead, supervise. The results you get are not great. To get good results you have to use joint embedding architectures. All the best systems that use self supervised learning to train an image or video representation systems system.
## 00:27:29
**中文**
最佳系统全部使用联合嵌入，没有一个使用重建。以图像为例，要么提供同一场景的两个视图，训练神经网络产生表征，并要求两个表征相同；要么使用输入破坏技术。
**English / Original**
All you joint embedding, none of them uses reconstruction. OK all the best ones. And either you, let's say you apply this to images, either you have two views of the same scene and you train a neural net to produce representations and you tell the system, I want those two representations to be identical. Or you you'd use this corruption technique.
## 00:27:56
**中文**
取一个输入，以某种方式破坏或变换它，再训练 JEPA 架构，根据受损版本的表征预测原始图像的表征。但这里有个大问题：系统可能坍缩。其实生成模型在某种程度上也会坍缩。
**English / Original**
You take an input, you corrupt it or transform it in some ways. And then, you know, train the this Jetpaa architecture to predict the representation of the original image from the representation of the corrupted version. OK, there is a big issue with this, which is that the system can collapse. Now the generative models can actually collapse to some extent.
## 00:28:20
**中文**
如果训练自编码器时不限制编码的信息容量，自编码器只会学到恒等函数，这就是坍缩，并没有学到有用的东西。同样，这种系统也可能坍缩：它可以完全忽略输入、输出恒定表征，于是预测问题变得轻而易举。
**English / Original**
Like if you train an auto encoder without a restricted restriction on the information content of the code, the auto encoder is just going to learn the identity function and that's a collapse. It's not going to learn anything useful. Similarly, a system like this can can collapse and how can you collapse? It can essentially completely ignore the inputs, produce constant representations, and now the prediction problem is trivial.
## 00:28:51
**中文**
所以若只训练这类系统最小化预测误差，它就会坍缩，无法发挥作用。联合嵌入系统做自监督学习的全部诀窍，就是如何防止坍缩。我最喜欢的概念是信息最大化，稍后也会谈其他办法。
**English / Original**
So if you just train a system of this type to minimise the prediction error, it's going to collapse, It's not going to do anything useful for you. So the whole trick of how you do self supervised learning for a joint templating system is how you prevent collapse. And there is my favorite concept for this. I'll talk about other ways to do this later, but my favorite concept to prevent collapse is, is information maximisation.
## 00:29:22
**中文**
基本做法是设计一个目标函数，衡量编码器输出表征的某种信息量，并最大化它，因此成本函数可以是信息量的负值等。过去六七年出现了许多技术，例如 MCR、MCR²、WMSE、SIGReg、VICReg 和 Barlow Twins；Barlow Twins、VICReg 与 SIGReg 来自和我共事的人。
**English / Original**
OK, so you basically come up with some objective function that measures the some sort of information content of the representation that comes out of your encoders and you try to maximise that information content. OK, so your cost function is minus the information or whatever. So there's a bunch of techniques for this, you know, since like the last six or seven years with names like MNCRNCR, Square, WMSE, Sigrag, VC Reg and Ballot Twins, the Ballot Twins VC Reg, Sigrag.
## 00:30:00
**中文**
其他方法来自别的团队：MCR 来自伯克利，MCR² 来自纽约大学的一位同事。神经科学方面还有 Simoncelli。JEPA 思想正在流行起来；Google Scholar 上约有 1,700 篇论文明确提到“joint embedding predictive architecture”。
**English / Original**
Come from people working with me, the other ones from other groups. NCR comes from Berkeley and NCR Squared from a colleague at NYU. In neuroscience there was Simon Cherry. So but this idea of jetpack is getting popularity. There's about 1700 papers that mentioned joint emitting predictive architecture spelled out on Google Scholar.
## 00:30:29
**中文**
这类方法有个问题：怎样衡量信息量？我们需要一个可微的信息量度量作为成本函数，才能反向传播梯度并将其最大化。坏消息首先是，我们其实没有客观的信息量度量，因为所有正规的定义都依赖于知道待测向量等对象的分布。
**English / Original**
OK, so there's an issue with this type of method, which is how do you measure information content? We need to have a cost function that is a differentiable measure of information content so we can back propagate gradient and maximize it. And the bad news is, first of all, we don't actually have objective measures of information content because all the proper definitions are based on knowing the distribution of the the vectors or whatever that you want to measure the information content of.
## 00:30:59
**中文**
但我们不知道分布，只有编码器输出的样本。如何从有限样本计算信息量？这是第一个问题。第二，要最大化某个量，需要它的信息量下界，这样最大化时才会推动真实信息量上升。
**English / Original**
And we don't know the distribution. We only have samples coming out of an encoder. So how do you how do you compute the information content from a finite number of samples? OK, that's the first problem. Second problem is to maximize something, you would need a lower bound on information content so that when you maximize, you push the actual information content up.
## 00:31:21
**中文**
问题是，我们所有经验度量都是上界。怎么办？找一个不错的上界，祈祷它有效，再证明一些定理之类。这项技术以及许多相关方法，都可以用我倡导约 20 年的“基于能量的模型”框架来恰当地解释；它不仅适用于自监督学习系统，实际上适用于所有学习系统。
**English / Original**
Problem is, every empirical measure that we we have are all upper bounds. So what do we do? We come up with a good upper bound and we cross our fingers and we show some theorems and whatever. OK, so this technique and, and many others on on like the way to properly explain how you, you know, how you can train self supervised learning systems and, and every learning system really is a framework I call energy based models that have been advocating for 20 years or so.
## 00:31:56
**中文**
基本思想如下：如果想捕捉两个变量 X 与 Y 之间的依赖，但二者不存在真正的函数关系——也就是说，给定 X 并没有唯一的 Y——那么这只是依赖或某种映射关系，而不是函数。
**English / Original**
It's basically the basic idea is like this. If you want to capture the dependency between two variables, X&Y and there is no real functional relationship between X&Y. So you cannot, there's no single Y for a given X, right? It's just a dependency, but it's not a function. Like it's a relation for some kind of mapping, but not a function.
## 00:32:16
**中文**
右图表示这种情况：许多黑点构成数据，显示 X 与 Y 之间存在某种依赖。既然无法用一个函数根据 X 计算 Y，怎样捕捉这种依赖？一种办法是学习或构造一个对比函数、能量函数，判断 XY 空间中的某点是否靠近训练数据。
**English / Original**
So indicated by the diagram on the right here you have a bunch of data points of all the the black dots and so they indicate some sort of dependency between X&Y. How do you capture this dependency given that you cannot run a function that that computes away from X? So one way to do this is to learn or build a contrast function, energy function that tells you a point in this XY space is near the the, the training data or not.
## 00:32:52
**中文**
把它想成一片地形，黑点处于谷底——在瑞士，那会是一片湖。离开这些区域时，等高线升高，也就是能量升高。现在若给定 X，就能推断出一组与 X 相容的 Y 值。
**English / Original**
OK, So think of it as some, some sort of landscape where the, the the black dots are in the valley. OK, in Switzerland, it would be a lake. And then, you know, you get like, you know, level curves, right? As as you move out, you know, outside of those regions, the altitude goes up. OK, the energy goes up. Right? Now, if I give you a value for X, you can infer, you can give me a bunch of values for Y that are compatible with X.
## 00:33:21
**中文**
它们是使能量最小的 Y 值。这就是前面讲的推理：通过优化而不是前向传播。也可能反过来做：给定 Y，从 Y 推断 X，而且可以给出多个答案。在视频预测这种答案事实上有无穷多种的情形中，训练这类系统的正确方式，是从基于能量的模型角度思考。
**English / Original**
There are values of Y that minimize the energy, right? So it's the kind of inference I was talking about earlier, influenced by optimization, not by forward propagation. But you can also possibly do it the other way around. If I give you AY, you can infer X from Y and you can give me multiple answers. So in situations like like video prediction where there is a basically an infinite number of possible answers, the proper way to train the system of this type is to kind of think of it in terms of energy based models.
## 00:33:50
**中文**
顺便说，概率模型只是一个特殊情形，其中能量具有特定形式，训练采用特定损失函数。因此如果愿意，可以说这是比概率推断和学习稍微更一般的框架。训练基于能量的模型时，必须防止坍缩。
**English / Original**
And by the way, probabilistic models are a special case where where your energy has particular form and the way you train it this particular loss function. So it's slightly more general framework if you want than probabilistic inference and learning. OK So what? To train an energy based model you have to prevent collapse.
## 00:34:12
**中文**
我之前说的坍缩问题，会表现为能量函数处处平坦。你训练系统把一批训练样本的能量降到最低，结果系统给出一个处处为 0 的能量函数。没有信息容量约束、只学会恒等函数的自编码器，做的就是这件事。
**English / Original**
The collapse problem I was telling you about before will be manifested by the energy function being flat everywhere. You train the system to minimise the energy for a bunch of training samples and what the system gives you an energy function that is 0 everywhere. That's what a auto encoder that learns the identity function, that's what it it does to you.
## 00:34:29
**中文**
忽略输入、产生恒定表征的 JEPA，对一切都有零预测误差，这也是坍缩。防止坍缩有两类办法。第一类是对比方法：生成数据区域之外的点，把它们的能量推高；设计成本函数，让数据点能量降低、其他点能量更高。这类方法有很多。
**English / Original**
A jet path that ignores the input and produces a constant representation at 0 prediction error for everything. So it's a collapse. To prevent collapse, you need to do one of two things. One is contrasting methods. You, you generate points outside the region of data and you push the energy up. OK, you come up some cost function that make sure the energy of the data points come down and and the energy of other points is is higher and there's a whole bunch of them.
## 00:34:57
**中文**
另一类是我逐渐更偏好的正则化方法，它通过缩小可具有低能量的空间体积来工作。把某些区域的能量压低后，其他区域就必须升高，因为可分配给低能量的空间体积只有那么小。
**English / Original**
And there is another set of method which I have come to prefer. Regularise method which work by minimising the volume of space. They can take low energy, OK, so if you push down the energy of certain regions, the rest has to go up because there is only a small amount of a small volume of energy to go around of low energy.
## 00:35:18
**中文**
实践中怎样实现这两类方法？让我们回到信息最大化。我想用某种信息度量训练 JEPA。假设把一批样本送入一个编码器，得到一个矩阵，其中每一行是一个样本的表征。
**English / Original**
So in practice, how how how do you sort of reduce this to practice? OK, those one of those two methods. OK, so let's go back to this idea of information maximization. So I want to train this this Jappa some measure of information. Let's say I run a batch of samples through through one of the encoders, I get a matrix where each row is the representation for one sample.
## 00:35:47
**中文**
每一列是所有样本在表征中某个变量的取值。有两种办法让矩阵具有信息：一是确保所有行不同，二是确保所有列不同。需要列彼此不同，因为如果所有列都一样，就意味着表征中的每个变量都携带同样的信息。
**English / Original**
Each column is the value of 1 variable in the representation for all samples. OK, there's two ways to make that matrix informative. One way is to make sure all the rows are different, and other way is to make sure all the columns are different. You want to make sure the columns are different because if all the columns are the same, that means every variable in the representation carries the same information.
## 00:36:13
**中文**
那当然信息量不高。因此希望表征中的每个变量尽可能与其他变量解耦，携带独立于其他变量的信息。这就是所谓维度对比方法的例子，属于正则化方法。
**English / Original**
And of course that's not very informative. So you want to you know each variable in the representation to be maximally disentangled from the other ones to give you an independent information from the other variables. OK, so that would be an example of what we can call dimension contrastive methods, which is a form of regularized method.
## 00:36:38
**中文**
图下方让各行彼此不同的准则，则属于对比方法或样本对比方法。若干对比方法在某些应用中很流行。LLM 中许多感知流水线用 CLIP 训练，它本质上是在图像与文本之间进行联合嵌入的对比方法。
**English / Original**
And then at the bottom the the type of criterion that makes the the rows all different, Those are contrastive method or sample contrastive methods. OK, several contrasting methods are very popular for certain applications. A lot of the perceptual pipelines in LLMS are trained with a technical clip, which basically is contrasting method that does joint embedding between images and text.
## 00:37:04
**中文**
但我更喜欢另一类方法。要作出预测，需要找到输入的抽象表征，这个想法很自然。人类一直这样做，科学家和工程师一直这样做，动物也一样。原则上，我可以在量子场论或粒子物理层面解释或模拟此刻房间里发生的一切。
**English / Original**
But I prefer the other one. So this idea that you need to find an abstract representation of an input to be able to make prediction is actually very natural. We do this all the time as humans. We do this all the time as scientists and engineers. Animals do it too. Let me explain why in principle, I could explain or simulate everything that takes place in this room at the moment at the level of quantum field theory or particle physics, right?
## 00:37:35
**中文**
可以模拟房间里每个粒子的轨迹，甚至一直模拟到我们每个人的大脑过程。原则上运行这种模拟后，我能判断你们是否真的理解我的话，或者此刻是否在睡觉、是否无聊透顶。
**English / Original**
Could simulate the trajectory of every particle in this room. And then we go down to actually simulating all of our brain processes and everything. So in principle, running the simulation, I could figure out if, you know, any of you actually understands what I'm saying or not, OK Or if you are sleeping right now or if you're, like, totally bored, right.
## 00:37:58
**中文**
但这当然完全不切实际。科学所做的是发明抽象来帮助预测，而这些抽象会忽略系统底层状态的大量细节。我们构建了一层层抽象：从量子场到粒子、原子、分子、蛋白质、细胞器、细胞、生物体、个体、社会和生态系统。
**English / Original**
OK. But of course, that's completely impractical. And then, you know, what we do in science is we invent abstractions to allow us to make predictions and those abstractions ignore a lot of details about the state, the underlying state of the system. So we invent those abstractions, you know, from quantum field to particles, atoms, molecules, proteins, organelle cells, organisms, individual societies, ecosystem.
## 00:38:22
**中文**
这个层级中的每一层都是描述世界的特定抽象层次。它忽略下一层的大量细节，从而让我们能够比低层作出更长程的预测。因此，要理解此刻房间里发生什么，心理学层面比粒子物理层面更合适。当然，物理学家总爱取笑别人，说“你们不过是应用物理”。
**English / Original**
Every level in this hierarchy is a particular level of abstraction with which we describe the world, which allows us to make longer range predictions, if you want, than the levels below by ignoring a lot of details about the level below. Which is why the the way to understand what goes on in this room at the moment is more at the level of psychology than at the level of particle physics right now, of course, physicists always make fun of everyone saying like, you know, you just applied physics, right?
## 00:38:58
**中文**
从某种意义上，连心理学也是应用物理。但事实上，化学中的特定知识并不能直接从物理学推导出来。因此，这种抽象实际上包含了下一层并不显现的新知识、新信息或新结构。
**English / Original**
Even psychology is applied physics to some extent. But in fact, you know, there is, you know, specific knowledge about chemistry that does not derive directly from physics, right? So, so this abstraction actually kind of contains new knowledge or information or structure if you want. That was not apparent at the level below.
## 00:39:21
**中文**
JEPA 思想正建立在这一概念上：要作出预测，必须找到抽象。比如要设计飞机，就得设计机翼翼型，使用计算流体动力学来模拟机翼周围的气流。
**English / Original**
So this idea jetpack really kind of constructs on this concept that you need to find an abstraction to be able to make predictions. Let's say you you want to do you want to design an airplane, you need to design the airfoil for the airplane. You do computational fluid dynamics, right? You simulate the flow of air around around the wing.
## 00:39:42
**中文**
你用速度、密度等量来描述机翼周围每个小立方体中的空气状态，再求解 Navier–Stokes 偏微分方程，以模拟气流；但实际上，这忽略了底层机制中的海量细节。
**English / Original**
You, you model the state of the air in every little cube around around the wing by basically the velocity and the density and things like that. And and then you saw Navier Stokes partial differential equations and that simulates the flow of air, but in fact it's ignored. A huge amount of details in the underlying mechanism.
## 00:40:03
**中文**
底层机制是空气分子彼此碰撞、并撞击飞机，但模拟流体时从不在这个层面进行，因为太复杂；而且细节太多会让模拟很快偏离现实。所以，必须忽略细节，才能作出准确的长期预测。
**English / Original**
The underlying mechanism is molecules of air bumping into each other and bumping on the plane. But you never simulate fluids at that level. It's just too complicated. And also it would diverge from reality really quickly because it has too many details. So you have to ignore details to be able to make accurate long term predictions.
## 00:40:25
**中文**
科学一直这样做。因此世界模型不应是模拟器，而应在抽象空间中工作；不应是所谓“数字孪生”，这个说法很糟；绝不应只是生成模型，也不应等同于视频生成。许多人做视频生成，却称之为世界模型。
**English / Original**
And so we do this in science all the time. And so one model should not be simulators, right? You should work in abstract space. You should not be digital twins. You know, that's a best phrase. You should definitely not be generating models that just explained and you should not be video generation. So a lot of people are working on video generation and they call this one models.
## 00:40:47
**中文**
它们不是世界模型，而是视频生成系统。今天演讲的一条重要信息是：若想研究世界模型，就不要做视频生成，那是另一个问题。想生成漂亮视频，可以研究视频生成；但想控制机器人或工业过程、理解世界，就不要研究生成。
**English / Original**
They're not one models, they are video generation systems. So 11 big message from my talk is that if if you want to use one models do not work on video generation. This is a different problem. OK, if you want to produce cute videos, work on video generation. But if you want to like control robots or industrial processes or understand the world, do not work on generation.
## 00:41:17
**中文**
世界模型应控制那些无法通过写一组方程来刻画动力学的复杂系统。如果是人形机器人或其他机器人，你可以写出动力学方程并模拟其动力学，让人形机器人翻跟头、练功夫之类。
**English / Original**
You want one models to control complex systems where you cannot model the dynamics of the system by writing a bunch of equations. OK, if you have a humanoid robot or you know any kind of robot, you can just write down the dynamic of equations and then simulate the dynamics of the robot. And you can get your humanoid robot to do some assaults and, and Kung Fu and whatever, right?
## 00:41:38
**中文**
那很简单。一旦机器人开始与现实世界互动，情况就复杂得多，也更难归约为简单方程。设想涡轮喷气发动机、化工厂、病人或以复杂方式与现实世界互动的机器人等复杂系统。
**English / Original**
That's simple. As soon as a robot starts to interact with the real world, that's a lot more complicated and and that is actually more difficult to reduce to simple equations. But then, you know, think about a complex system, like you say a turbojet or I don't know, a chemical plant or a patient or a robot, but a robot that interacts with the real world in complex ways.
## 00:42:05
**中文**
你无法把它归约为少数几个方程。必须学习整个系统——受控系统及其与环境的交互——的现象学模型，才能作出预测并规划动作序列以达到特定结果。这才是世界模型。
**English / Original**
You can now reduce this to a small number of equations. What you have to do is basically learn a phenomenological model of the whole system, the system you control, and its interaction with the environment, so that you can make predictions and you can plan a sequence of actions to arrive at a particular outcome. So that's one model.
## 00:42:27
**中文**
这个概念非常古老，可追溯至 20 世纪 60 年代，是最优控制的根基。下面介绍一种我非常喜欢的具体技术；我认为未来几个月和几年会扩展它，用来实现之前说的信息最大化。它叫 SIGReg。
**English / Original**
I mean, the concept is very old. It goes back to the 1960s. It's the root of optimal control. And OK, so now I come down to a particular technique that I'm very fond of, which I think we're going to expand over the next few months and years to do this information maximization that was telling you about earlier. And it's called sigreg.
## 00:42:52
**中文**
全称是“草图各向同性高斯正则化”。诀窍如下：把一批样本送入编码器，得到表征空间中的一组点，空间维度取决于表征维度。我们要设法让这些点的分布成为各向同性高斯分布，即所有维度方差相同。
**English / Original**
That means sketch isotropic Gaussian regularization. OK, the trick here is the following. You you run a batch of samples through your encoders and what you get is a bunch of points in the vector space of dimension. Whatever the dimension of your representation space is. We're going to try to make the distribution of those points isotropic Gaussian with the same variance in all dimensions.
## 00:43:17
**中文**
为什么？因为在各向同性高斯分布中，所有变量相互独立，因此每个变量都具有最大的信息性；而且在给定方差下，它也是熵最大的分布，不过这并非我们真正关心之处。关键是它让变量彼此独立。
**English / Original**
Why? Because an isotropic Gaussian is a distribution where all the variables are independent. OK, so they're maximally informative individually. And it's also the distribution that has maximum entropy for a given variance. But we don't really care about that. What's interesting is that it makes the variable independent of each other.
## 00:43:40
**中文**
怎样做到？我们当然不知道分布，只知道空间中的一组点；空间可能有 2,000 维，而点只有几百或几千个。怎样确保它是高斯分布？诀窍是沿某一个方向投影这些点，得到一个边缘分布。
**English / Original**
OK, so how do we do this? Now, of course we don't have the distribution. We just have a bunch of points in that space and it may be a high dimensional space like 2000 dimensions and we may have, you know, a few 100 or a few thousand points like how how can we make sure this is this is Gaussian. So here's a trick. The trick is you project the individual points along a single direction and what you get is a marginal distribution.
## 00:44:09
**中文**
当然，得到的仍是离散点，并非连续密度。一种办法是计算这些点形成的累积分布：在一维中，离散点形成阶梯。然后比较这个阶梯——点的经验累积分布——与某个高斯分布的累积分布之间的距离。
**English / Original**
OK, Now of course you still have discrete points. You don't have a distribute, you have continuous density, you have discrete points. OK, so 1 trick you can do is compute the cumulative distribution that those points give you, right? So it's a staircase, right? Because you have discrete points in one dimension. And then what you can you can you can do is you can ask what is the distance between the staircase, the cumulative empirical cumulative distribution of my points and the cumulative distribution of let's say a Gaussian.
## 00:44:41
**中文**
这是可行的，因为高斯分布的形状已知；对阶梯上的每个点，都能判断它位于理想高斯分布的左侧还是右侧，由此得到梯度，知道在该投影中应把点往哪边移动。对批次中的每个训练样本都可以这样做。通过梯度下降优化该成本函数，就会让分布在这个投影方向上的边缘分布趋于高斯。
**English / Original**
You can do that because you know what the Gaussian looks like. And for every point you can tell on the staircase, you can tell if it's to the left or to the right of the ideal Gaussian. And so that gives you a gradient. Like do I move the point this way or that way in that projection? OK, it gives you a gradient. Now for every training sample in your in your batch, OK, Now if you make the distribution, you know by gradient descent, like optimising this cost function, it's going to make the distribution Gaussian along the marginal of that distribution along this projection.
## 00:45:19
**中文**
有一个定理说，如果沿越来越多的方向这样做，在极限情况下，联合分布就会成为各向同性高斯分布。因此需要进行大量投影，针对每个投影计算梯度，移动这些点，把梯度反向传播进网络并改变权重，使整体分布更接近高斯。
**English / Original**
But now there's a theorem that says if you do this along lots and lots and lots of directions in the limit, your joint distribution is actually isotropic Gaussian. OK. So what we need to do now is do many, many projections for all of those projections, compute those gradients, you know, move the points of ACT propagate through the network, change the weights so that the points move so that the overall distribution gets more Gaussian.
## 00:45:46
**中文**
如果把它用于左上角那种呈 X 形的分布——图中显示的是 1,024 维中的两个维度——再做梯度下降，仅移动这些点而不训练其他内容，就会得到左侧那种近似高斯的结果。
**English / Original**
And if you apply this to a distribution like the one on the top left here, like an X, these are actually 22 dimensions among 1024. And then you do gradient descent. You just move the points here. You don't train and all that you. So the technique I'm advocating for is on the left, you get something that's sort of Gaussian E ish.
## 00:46:10
**中文**
实践证明这确实有效。我们已经用它训练动作条件世界模型，并用于规划，效果相当不错。源代码是公开的，方法非常简单，一块 GPU 就能训练。接下来需要做的是把它扩展到更大规模。
**English / Original**
And this, this really works in, in practice, we've actually applied it to training one models that are action condition and, and we've used them for, for planning. And it works decently. It's a very the, the, the source code is available. It's very simple. You can train it on one GPU. And what we need to do with this technique is scale it up.
## 00:46:32
**中文**
基本就是这样。还有少数其他事情要做，但扩展规模是主要任务。在简单情形中，可以训练这种世界模型，用它在模拟环境中的 Push-T 等简单机器人场景里规划简单动作。它还需要扩大规模，但算是一个不错的起点。
**English / Original**
Basically. There's a few other things that we need to do, but that's the the main one. And so in simple cases, you can train this role model and you can use it to plan simple actions in like, you know, push TE or like simple robotic situation in simulated environments. So that needs to be scaled up, but it's it's sort of a good work.
## 00:46:52
**中文**
几天前我们发表了一篇理论论文：假设数据的底层分布本来就是各向同性高斯分布，又假设从世界得到的观察是这些点经过某种复杂非线性变换的结果，例如这里的螺旋变换。
**English / Original**
There is a theoretical paper that we put out just a few days ago where if you make the hypothesis that the underlying distribution of your data is actually an isotropic Gaussian. If you assume that the observation you get from the world are some sort of complicated non linear transformation of those points, like in this case like some sort of spiral transformation.
## 00:47:20
**中文**
在其上用 SIGReg 训练神经网络，网络会在表征空间恢复原始高斯分布。这不是对所有情形都有效的一般性证明，但它证明：如果原始解释变量是高斯的，系统就能恢复这些变量，至多相差一次旋转。我们可以把这些技术用于自监督学习，训练图像识别系统。
**English / Original**
You apply, you train a neural net with Sigrag on it, it will recover the original Gaussian in the representation space. OK, so it's not a general proof that it works in every case, but it's a proof that if your original explanatory variables are Gaussian, the system will recover those variables up to a rotation. OK, so we, we can use those techniques to, you know, context, the self supervised learning to train an image recognition system.
## 00:47:56
**中文**
还有另一组技术值得一提，因为效果非常好，而且迄今已经实现了大规模扩展。SIGReg 在概念上是我最喜欢的方法，但它很新，我们尚未扩大规模；基于蒸馏的其他方法则已被扩大规模，并通过 I-JEPA 和 V-JEPA 等技术在图像和视频上取得了非常好的结果。
**English / Original**
And there is another set of technique which I should mention because they work really well and they are the ones that have been scaled up so far. Sigrag is conceptually my favorite method, but it's very recent and we haven't scaled it up. Whereas those other methods that are based on distillation, we've scaled them up and we got really good results both for images and video with techniques like AJPA and VIJPA.
## 00:48:18
**中文**
这些蒸馏方法的基本思想是什么？仍有两个编码器，仍是 JEPA 架构。取一个输入，对其变换、破坏或遮蔽，再训练系统在表征空间中预测；但不让梯度穿过右侧编码器传播。
**English / Original**
So what's the basic idea of those distillation methods? You you still have those two encoders. So it's a jet power architecture. You take an input, you transform it or corrupt it or mask it or something. And then you train the system, you know, to predict in representation space. But you don't propagate gradient through the encoder on the right.
## 00:48:39
**中文**
两个编码器架构相同，并以某种方式共享权重。特别之处是，右侧编码器采用左侧编码器权重随时间变化的指数移动平均。左侧编码器接收梯度并持续更新；右侧更新得更慢，本质上共享权重。
**English / Original**
OK, those are two encoders with identical architectures and they kind of share their weights. But The funny thing is that the encoder on the right use a exponential moving average over time of the weights of the anchor on the left. The encoder on the left gets gradient and gets updated all the time. The anchor on the right gets updated slower essentially and share the weights.
## 00:49:05
**中文**
这个思路源于 Google DeepMind 一些人的直觉：他们用类似技术稳定强化学习中的方差，随后意识到可将其用于仅从图像进行自监督学习。他们称之为 BYOL，即“bootstrap your own latent”。此外还有许多方法，尤其来自 Meta，例如 MoCo 等。
**English / Original**
This is derived from some intuitive ideas of some people at Google Deepmine who are using techniques like this to stabilize the variance in reinforcement learning and then realise you could apply this to self supervisor only from images. They call this BYOL bootstrap your own latent and there is like a whole bunch of methods from from meta in particular since you have Moco etcetera.
## 00:49:29
**中文**
它们采用指数移动平均思想。这里展示的一个具体方法 I-JEPA 取得了非常非常好的结果。我们比较了 I-JEPA 与名为 MAE（掩码自编码器）的生成式方法：I-JEPA 不仅效果更好，而且训练快得多。
**English / Original**
They use this exponential moving average idea and a particular method called Ijeta, which I, I, I show here it for just really, really good results. And what we're able to do with IGEPA is compare the results of IGEPA with a generative approach called Mae mask auto encoder. And it, it, it's not only better, but it, it's much faster to train another.
## 00:49:56
**中文**
另一项技术叫 DINO。你们很多人肯定听说过；我知道有人用过，因为机器人演示项目中就使用了它。这是我在巴黎 Meta 的一些前同事完成的，完全采用自监督学习，是联合嵌入架构。
**English / Original**
And another technique is, is called Dino. How many of you? I'm sure I've heard of it. I, I know some of you have used it because there were projects in the robot demos that actually used, you know, so this is done by some of my former colleagues at, at Meta in Paris. And it's completely self supervised. It's a joint embedding architecture.
## 00:50:15
**中文**
它使用蒸馏并加入多种技巧，我不打算逐一解释，背后有大量工程工作。目前，这类系统基本能产生最好的通用图像表征。如果有任何视觉任务要做，这大概是最佳技术——也就是最佳图像编码器。
**English / Original**
It's using distillation, but with various tricks, which I'm not going to explain. There's a lot of engineering that goes behind it. And those systems basically at at this time produce the best generic representations of images. If you have any type of vision task that you want to to do, that's probably the best, the best, the best technique to, I mean, the best encoder for images.
## 00:50:37
**中文**
我们所做的工作之一，是把 DINO 用作编码器，再训练世界模型并进行规划。我展示一段有趣的视频：这里有一个动力学相当复杂的模拟环境初始状态，目标显示在顶部和底部。
**English / Original**
OK. But what we've done is, among other things, is use Dino as an encoder and then train a role model and do planning. Let me show you just a cute video on this if if I can. OK, so you have a initial state here of a kind of simulated environment that has pretty complex dynamics. And you have goals at the top and at the bottom.
## 00:50:56
**中文**
画面显示规划器的一系列动作；规划器利用训练好的世界模型，在不到 25 步内让世界达到尽可能接近目标的配置。这已用于双摆、Push-T 等多种场景。I-JEPA 的效果很好。
**English / Original**
What you see is the sequence of actions of a planner that uses this trained one model to get the world to a configuration as close as possible to the original one in less than 25 steps. And this has been applied to a number of different scenarios like double pendulum and and push T and whatever. Now IJIPA works really well.
## 00:51:26
**中文**
最近我们又把它用于视频：取一段视频，遮蔽很大一部分，再训练 JEPA 产生良好表征，使其能根据部分遮蔽视频的表征，预测完整视频的表征。训练完成后，用编码器提取视频特征，再在其上训练一个任务头来完成某项任务。
**English / Original**
So we recent more recently applied you to video. So there you take a video, you mask a big chunk of it and you train the Jetpa to again produce good representations so that you can predict the representation in full video from the representation of partially masked 1. Once the system is trained, you use the encoder as a way to extract features from the video and you train a head on top of it to accomplish some task.
## 00:51:50
**中文**
它在许多传统视觉任务上都达到顶尖水平，尤其是视频动作识别、动作预测等。不过，我不想用结果表格让大家厌烦，更值得一提的是：这些系统、尤其 V-JEPA，已经学到了一定程度的常识。
**English / Original**
And it it, it works like really well at state-of-the-art for a lot of traditional vision task, particularly from video like action recognition, action prediction and stuff like that. But one interesting thing that I want to mention instead of boring you with a table of results is, is that those systems, VGPA in particular, has, has learned some level of common sense.
## 00:52:11
**中文**
由于训练 V-JEPA 预测视频接下来会发生什么，我们可以训练一个预测器并测量它的内部预测误差。给它看视频，在每个时间步监测内部误差。系统每次处理 16 帧窗口；我们在视频上滑动这些帧，测量未来约 15、16 帧的预测误差。
**English / Original**
So one thing we can do VGPA because we train it to predict what's going to happen next in the video. We can train a predictor to do that. We can measure it's internal prediction error. We can show it a video and monitor the internal prediction error at every, every time step. This system takes a window of 16 frames. So we, we just slide those, those frames right on a, on a video and measure the prediction error with for the next 1516 frames.
## 00:52:38
**中文**
有意思的是，如果给它看一段发生了不可能、违反物理规律事件的视频，预测误差会急剧飙升。这就像开头幻灯片中的小女孩看到小车没有落下一样。例如视频里抛出一个球，球却突然消失。
**English / Original**
And the cool thing is that if you show it a video where something impossible occurs, something unphysical, the prediction error shoots to the roof. So it's like the little girl in the one of the earliest slides, like you know, looks at the scene of the car not falling. Same thing. You have a video of a, a ball being thrown and the ball disappears.
## 00:53:01
**中文**
预测误差就会飙升。这很有意思，因为至少在我看来，这是第一次看到完全自监督的系统获得某种常识，能分辨什么可能、什么不可能。下面这段我略过；它很有趣，但只是说明 V-JEPA 可用于规划。
**English / Original**
Prediction error shoots to the roof. So that's interesting because it's the first time, at least from my point of view, that I've seen completely self supervised system acquire some level of common sense tell you what's possible, what's not possible. Let me skip this. It's cute, but it's it just says VJ pack can be used for planning.
## 00:53:25
**中文**
它还有一些新版，在规划等方面做得更好。这里还有件有意思的事：我说过，婴儿之所以学会世界是三维的，是因为三维结构最能解释移动头部时视野为何变化。我们取了 V-JEPA 2.1 某个版本学到的表征。
**English / Original**
And you know this like new versions of this that do a better job at planning and everything. But here is an interesting thing. Remember I told you the way babies learn that the world is 3 dimensional is because it's the best way to explain how you view of the world changes when you move your head. OK, so we took the representation learned by some version of VJ pack called VJ pack 2.1.
## 00:53:47
**中文**
然后在其上训练一个任务头，根据单张图像预测深度，效果非常好，实际上优于 DINOv3。这表明，系统仅通过在表征层面预测、填补视频中的空缺，就基本掌握了世界是三维的这一点。
**English / Original**
And then we train ahead on top of it to predict depth from a single image. And it does a really good job. It's it's, but it's really good results, in fact, better than the, you know, V3. And what that shows is that this system by just being trained to predict, to miss, you know, fill in the blanks in videos at the representation level, basically understands that the world is 3 dimensional.
## 00:54:15
**中文**
所谓“掌握”要加引号。它也掌握了物体概念：把其表征输入分割系统，效果相当不错；用于其他各种任务也一样。接下来总结。说来也有趣：放弃生成模型——如果你研究 LLM 当然另说，但你本来就不该研究 LLM；至少身在学术界，绝对不该研究 LLM。
**English / Original**
I mean, understands with double quotes, understands the notion of object. If you use the representation as input to a segmentation system, it, it works decently well and for, you know, various other things. OK, then you conclude. So it's funny. Yeah. So abandoned generating models. I mean, if you work on LLM of course, but you should not work on LLM, at least if you are in academia, you should absolutely not work in LLM.
## 00:54:49
**中文**
你没有什么独特价值可以贡献。如果对智能感兴趣，就应放弃生成模型，转向联合嵌入架构；放弃概率模型，转向基于能量的模型。我没时间充分解释原因。我已经为正则化方法、即通过变量而非样本实现信息最大化作了论证。
**English / Original**
There's nothing you can bring to the table. So Evandon generating model in favour of joint emitting architectures. If you are interested in, you know, intelligence. Abandoned probabilistic models in favour of those energy based models. I didn't have time to really kind of explain why I, you know, I made an argument in favour of of those regularized methods or information maximization through through variables instead of samples.
## 00:55:19
**中文**
我也一直不赞成对比方法，尽管它们有许多实际应用。我多年来一直说要“放弃强化学习”，当然并非真的彻底放弃，而是尽量少用，因为它的样本效率糟糕透顶。我知道在座有人研究它，但强化学习是走投无路、别无他法时才用的办法。大部分学习都应通过观察完成，例如学习世界模型等。
**English / Original**
So I've been in contracting methods, which again have a lot of practical applications. I've, I've been saying forever to abandoned reinforcement learning. I don't really mean abandoned, I mean minimise it's use because it's so, so horribly inefficient in terms of sample efficiency. And I know there are people here who work on this, but but like, you know, RL is like what you do when you're desperate and there is nothing else you can do, OK, Like you have to do most of the learning, you know, in by observation, you know, learning one models, blah, blah, blah.
## 00:55:55
**中文**
有了好的表征后，可以在上面使用强化学习，因为表征已经很好，就不再需要太多样本。有时确实无法避免它。如果你想在 AI、尤其扎根现实世界的 AI 方面取得真正进展，也是如此。
**English / Original**
And once you have good representations, you know, you can use RL on top of it because you already have the good representations. You you won't require too many samples. Sometimes you can't you can't avoid it. And certainly if you're interested in making real progress in AI in so grounded, you know, AI for the real world.
## 00:56:13
**中文**
如果想做物理 AI，就不要研究 LLM，也不要研究生成模型。你们大概能猜到，这些观点让我在硅谷不太受欢迎。很多人可能知道，我去年年底离开 Meta，创立了一家名为 AMI Labs 的新公司。AMI Labs 的目标就是面向现实世界的 AI。
**English / Original**
If you want physical AI, don't work on LLNS, don't work on generating models either. So as you can probably guess, this does not make me very popular in Silicon Valley. And so I left Meta, as many of you probably know, at the end of last year and formed a new company called Emilabs. And the purpose of Emilabs is sort of AI for the real world.
## 00:56:40
**中文**
物理 AI 和机器人是用例，但不止于此，还包括控制各种工业过程。凡是连续、高维且含噪、LLM 完全无能为力的问题，都是我们要研究的。我的演讲到此结束，非常感谢。还有很多问题，也许可以回答一两个，但随后必须结束。
**English / Original**
You know, physical AI robotics is a use case, but it's not just that. It's control of industrial processes like anything. That is, how do you mention continuous and noisy for which LLMS are completely helps us. This is the kind of problems we're working on and that's it. Thank you very much. OK, So I know there's many questions, maybe we'll take you know one or two, but then we have to to wrap up.
## 00:57:21
**中文**
请简短提问，我也简短回答。提问：谢谢您的演讲。我想问您之前一张提到 MPC 的幻灯片中的护栏。工程师喜欢 MPC，因为可以加入约束，并在三维空间等状态空间里描述约束。但据我理解，您的系统中一切都在表征空间里运行。
**English / Original**
So and please quick questions and quick answers. Thanks for the talk. I wanted to ask about the guard rails that you mentioned on the one of the earlier slides where you also talked about MPC. Engineers love MPC Cos they can put in their constraints, describe them in state space like 3D space. But from what I understand in your system everything works in representation space.
## 00:57:44
**中文**
那么怎样把“不要撞墙”这样的约束放入表征空间？您设想让系统自行学习约束，还是工程师真的能手工加入？回答：需要在表征上训练一个非常小的任务头，把表征映射到你关心的约束。
**English / Original**
How do I even get the constraint? Like don't bump into the wall into this representation space? Do you envision the system learning the constraints by itself or can engineers really put them in? No, you will have to learn a very small head on top of your representation that maps that to your, you know, the constraint that you're interested in.
## 00:58:07
**中文**
这部分必须训练，但只需极少量样本，因为它基本只是一个很小的投影。提问：但每类约束都需要不同编码器吗？回答：不是，需要针对每个约束使用不同投影器。比如任务是开门——这里先不谈约束，而谈任务目标。
**English / Original**
So that part has to be trained. But you can train it with very small number of samples because it's a tiny basically just a projection. But you need a different encoder for each kind of constraint. I don't might want to put it no. Well, you need a different projector for for each contract, right? So if if your task is to like open a door, I'm not talking about constraint, I'm talking about like a task objective.
## 00:58:29
**中文**
你需要某个成本函数判断门是否打开。它也许要在训练系统完成任务时学得，但基本只需两个样本。主持人：恐怕只能到这里了。非常感谢您，Yann。回答：谢谢。
**English / Original**
You need some cost function to tell you like, is the door open or not, right. And so that might have to be trained when you trained to accomplish the task, but basically that requires 2 samples. OK. I think we'll have to leave it here. Thank you, Jan very much. Right. Thank you.