Richard Sutton recently quit Keen to pursue his own reinforcement learning startup. That reminded of his conversation with Dwarkesh Patel, in which Sutton argues forcefully against the usefulness of training on existing human knowledge. Even though we can apply a lot of compute to it and have done so, we will run out of knowledge and then what. Arguably we have already arrived at this point. Sutton’s answer of course is that you just want to figure out what it really means to learn and then AI can learn everything there is to know.

To make this more concrete, consider the case of mathematics. Humans have been doing math for a few thousand years and we have developed so much of it by now that no one human comes even remotely close to understanding all of it. Most mathematicians are experts on just a tiny fraction of the extant mathematical knowledge. There are projects underway to formalize all the math that humans have already found in LEAN (a language and system for generating and checking proofs). The idea is that we can then use AI models to conjecture new mathematical ideas on top of this human work, express them in LEAN, and then we can check them in an automated fashion giving rise to an explosion in mathematical understanding.

This sounds like a sensible approach but a more Sutton-esque way of going about it would be to set out a bunch of math agents with minimal knowledge and have the set off discovering all of math from the beginning. You would still use something like LEAN to express and check ideas but you wouldn’t bother encoding most of our existing math knowledge (other than say a few axioms). The idea here is that with access to enough compute agents would wind up  rapidly recreating the already existing human math, so why bother encoding that. Also agents might come up with more elegant formulations faster because they are “dimensionally unconstrained” beings. Unlike humans whose brain has been heavily influenced by having to survive in a 3D + time environment, for agents higher dimensional space is just as readily accessible. And as we are beginning to see, there is a huge amount of structure in the world that exists at these higher dimensional levels.

The difference as a first approximation is as between AlphaGo and AlphaZero. The former was trained on human knowledge, the latter entirely on selfplay. And of course the latter was better. AlphaZero was truly “bitter lesson pilled.” Unconstrained by prior human knowledge it came up with new creative moves. Similarly maths agents unconstrained by human math might arrive at better solutions than humans have to date. The challenge of doing this for math problems is unsolved for now, as the search space is orders of magnitude larger than that of Go and solutions need to be precise instead of approximate. So what I am writing about here is speculative. 

For now let’s assume that we come up with a truly bitter-lesson-pilled approach to math. Agents developing math from the ground up with humans having provided just the axioms. It sounds magical and yet there is a major potential problem with this approach: We might not be able to understand any of the math that these agents come up with. There would be no reason for them to use any of the same language that we have invented in the field. We have a mini-example of this possibility. The mathematician Shinichi Mochizuki went away for a few years and came back with a monster 500 page work called “Inter-universal Teichmüller theory”. There was so much new math here that many mathematicians concluded it was too tall a mountain to climb and to this date his results are disputed (with apparently only a couple of people having worked through all of it). Now imagine the same from many machines collaborating. We would be faced with a mountain of math that’s utterly impenetrable for humans.

One might argue that it doesn’t really matter. After all, if the math works, what’s the problem? Wouldn’t it be great to know whether the Poincare conjecture is true (or not)? Does it matter if we can understand the proof? Right now a lot of human proofs assume the Poincare conjecture to be true. So from that perspective getting a proof would be helpful. But it would feel strange to rely on a proof that we can’t understand. We might take some comfort in it being machine verifiable, but of course there might have been something snuck in there along the way to make it work. Now we might pit machines against each other and ask them to discover flaws. And that might work but again we wouldn’t understand why or how. 

We might be willing to accept this in math, but of course this problem goes to the heart of the alignment question. If agents develop huge knowledge in language entirely disconnected from our own what hope can there be of achieving any kind of alignment? I used the domain of mathematics to illustrate this problem but it exists equally in any other domain. Now we might be able to ask models to help translate what they have found into language that is accessible to us but we wouldn’t really be able to tell if the translation is faithful. There are other failure modes here including machines saying that it simply cannot “dumb down” their knowledge to our level. Ultimately this is a world in which human knowledge is frozen and only that of the machines grows. It is a world of human dependence on machines.

Now we are already seeing plenty of signs of model specific languages emerging. Neuralese is the term that has emerged to describe language that models use internally or between each other that can share information in latent space without mapping it back to human legible language. On large language models we can currently use techniques such as those employed in the J Space work to map the latent space in terms of our language. But this technique may stop working if we pursue other approaches that take language models more in the direction of image models. Or it may become irrelevant as world models grow and machines invent their own language altogether.

So it is essential at this moment that we push forward on some models that are deeply rooted in existing human knowledge. That will allow these models to extend our knowledge in ways that will be accessible to us. There is also another sense in which this can help with alignment. It is more reasonable to expect models that stand on the shoulders of human knowledge to see responsibility back to humans.

So yes, Sutton’s approach might be the most elegant approach. It might even be the ultimately most effective approach. But it also feels clearly by far the most dangerous path to take towards AGI. Let us instead redouble our efforts to build AGI that’s deeply rooted in human knowledge. This will be our best bet at building systems that will be aligned with human thriving.

Thanks to Oliver Friedmann for helpful comments on an earlier draft of this post.