00:00
I love you devoting so much time showing just how wrong I am.

00:05
That's exactly

00:16
I think it's fine for resting bitch face to be the first words in any given podcast.

00:21
It's certainly a herald of good things, if though if those are the first words.

00:25
Welcome to Increments.

00:26
We're thrilled to be joined today by Robert Wright, who has written uh a veritable multitude of books, and uh importantly is the the host

00:38
of the non-zero podcast, but perhaps most importantly is I think the father of the video podcast.

00:46
Can we can we say that?

00:47
Is that true?

00:48
You can say it.

00:49
Uh it's it was it was uh Bloggyheads TV, which I co-founded with uh Mickey Kouse and Greg Dingle was the the first certainly the first split screen video thing uh uh discussion.

01:03
program about like politics, public affairs, and so on.

01:07
There was a couple of months earlier, Leo Laporte had started his his uh tech podcast the this week in tech

01:14
uh or the weakened tech uh or something but uh but he used a much more sophisticated technology uh and and so as a result he needed like uh broadband

01:26
uh connection to make it work.

01:28
And we had something that was more primitive but by the same token more versatile.

01:32
I I guess this

01:33
It's not mainly I'll stop there because I don't think that's the main subject of this podcast.

01:37
But we were we were early.

01:39
We were I mean we were the platform that allowed people to see all these bloggers

01:45
for the first time.

01:46
Because like they weren't on cable TV.

01:49
They weren't, you know, cable TV news did not deem them worthy of appearance.

01:54
But they were some of them were getting pretty well known.

01:56
I mean, you know, Matt Iglesias, Ezra Klein, Michelle Goldberg

01:59
Uh and uh so yeah, it was it was fun.

02:04
I've never heard of them, but I'm sure they're yeah.

02:07
Well they're not up there uh you know at our level of prominence, but um

02:12
You're doing your best.

02:13
Okay, well, there are lots of things we could discuss with you, but you have very kindly supplied us with an advanced copy of your newest book, which comes out uh

02:23
in maybe roughly a month or so from now, the God test.

02:27
Um, and we're gonna repay you with uh for that kindness by by perhaps criticizing

02:33
some of some of the book and uh ask asking you questions about it.

02:37
Um so I hope that when you inevitably get annoyed with us, you remember to take it out on Vaden because I am a non-zero paying subscriber.

02:45
So I hope that buys me.

02:47
I am, I am.

02:48
God bless you.

02:56
Um I mean the one that's the one that's responsible for me not getting like six dollars a month from you is the one I'm talking about.

03:03
Yeah, well we'll see how the conversation goes and then maybe you'll tip it.

03:07
Okay, I'll try to behave myself then

03:10
Um okay, so a recurring theme in a lot of your writing, this book but also previous books, uh Non-Zero especially, is sort of the concept of

03:19
shall we say, directionality in human affairs and evolution.

03:25
Um and perhaps just evolution more generally, and perhaps not even just human evolution.

03:29
And uh and maybe even a notion of some sort of climax.

03:33
to to evolution.

03:34
And so you seem to adopt the view that evolution is is pointed, yeah, in some d direction of sort of increasing complexity.

03:40
I'm I'm wondering basically how this relates to

03:43
teleology in your view.

03:45
So it seems like most of the time you don't want to say that this climax is predetermined.

03:51
in any way, um, and that we have that it's inevitable and that we have no say over how the climax goes.

03:56
But then some of your writing sort of maybe veers more into that side of things.

04:01
Um and I guess

04:03
The the main question is to what extent is this direction uh foretold?

04:08
To what extent is this direction inevitable?

04:11
So uh first of all I'd say yeah, I do believe that both biological evolution

04:17
And cultural evolution, which includes technological evolution, you know, cultural evolution blo broadly speaking is

04:25
Just the kind of the evolution uh of all kind of uh non non-genetic information that that humans produce in a sense, you know, science, religion, everything, memes has

04:36
Richard Dawkins used the term, not not the way the term necessarily is used now.

04:41
So yeah, and I think uh I mean first of all

04:45
uh it it is a fact that uh more and more complex forms of life have resulted from evolution and that when there have been setbacks because of, you know, mass extinctions or whatever

04:58
the system seems to kind of reboot and continue to persist in that direction.

05:04
Uh doesn't mean every single lineage is always getting more complex.

05:08
Doesn't mean none ever get less complex.

05:10
It does mean that on balance

05:12
The most complex organism there is at a given moment.

05:16
In other words, the outer envelope of complexity tends to get higher over time

05:22
Um and then with human kind of cultural evolution, uh if you look at the kind of political uh the social organization that that uh

05:33
has changed in concert with cultural evolution.

05:38
It's generally gotten bigger and more complex, you know, 20,000 years ago, hunter-gather villages

05:43
uh now it's a globalized society.

05:46
M uh uh which is not always a cohesive globalized society.

05:49
And one of the you know, big claims in my book is that we're if we're gonna survive the AI revolution in good shape, we're gonna have to approach it as a real global community.

05:57
But

05:58
In any event, you get, you know, chiefdoms, city state, ancient city states, nation states, you get the internet and so on.

06:06
So

06:07
Broadly speaking, you could say that human cultural evolution seems to sustain the direction of biological evolution.

06:14
Now, purpose or teleology is

06:18
Uh a separate but related question.

06:20
I I would say directionality is suggestive of purpose, but but uh far from enough uh

06:28
you know, to let you conclude with great confidence that a system is purposive.

06:34
Um and I I should say right away that uh you know a system can have a purpose

06:42
But on the one hand, it can be because some intelligent being designed it, you know, like uh if somebody designs a a watch, uh pocket watch is a is a famous example in the in the

06:54
Annals of intellectual history.

06:56
Um that that's uh that that's uh purpose, but a lot of biologists and I would join them are c are comfortable saying

07:06
organisms have a purpose, but it's instilled not by like an intelligent designer with a brain.

07:11
It's it's instilled by natural selection.

07:13
Uh b uh in the sense that

07:16
uh organisms are kind of quote designed by natural selection to be good at getting their genes into the next generation.

07:24
So the you you could say that genetic proliferation is a

07:28
is a purpose maybe in quotes of organisms, um and that they are designed in quotes, you know, the quotation marks in both cases being uh there to distinguish

07:40
organisms from things that are designed by a conscious intelligence with a purpose in in mind.

07:47
Um and then the big question the question you're asking about teleology is, well, is it possible the whole system

07:53
Is a product of design.

07:55
You know, the whole unfolding system of evolution and then cultural evolution.

07:59
In other words, uh

08:01
Was it whether whether it's an intelligent designer like a god that set natural selection in motion or maybe some kind of meta-natural selection?

08:08
And you can imagine such things.

08:11
uh would would would have designed it in quotes, you know.

08:15
Um but does it have the hallmarks of a purposive

08:19
system and I think it I think it does have some hallmarks of a purpose of system even beyond directionality itself

08:27
But I'm not sure we want to get very deeply into that right away if you want to keep people listening because that gets pretty arcane.

08:35
But anyway, that that is that is the connection between directionality and purpose, I would say, is

08:40
is, you know, i you're more inclined to suspect purpose if there is directionality and if there are patterns maybe in various senses and structure in various senses.

08:50
But uh

08:52
you know, um in any event I d I do think it's a fact that we're heading we're moving towards some kind of climax

09:00
in the uh in in in the cultural evolution of human beings um as we approach globalization and in my view either uh

09:10
Get better at working with one another across national bounds, form a true global community

09:18
And, you know, make kind of the psychological adaptations that I think that requires, which is to get just better at looking things from others' point of view.

09:26
Um, you know, in in in a sense, maybe getting closer to enlightenment.

09:30
I I think we don't do those things, uh, we're in trouble.

09:34
So yeah, it it's a climax that I think is

09:36
worthy of a novel.

09:37
And the book is mainly about artificial intelligence, but my argument is that although this is not the first technology that that kind of

09:46
uh encourages us to cooperate internationally to avoid its downside.

09:51
It is uh bigger by long shot than any other single technology in that regard.

09:57
Yeah, so um

09:58
I think we we're definitely going to get to the um AI component of the book, but I do want to linger on the evolution argument for for maybe a a few more minutes because

10:06
My reading of the book is is that um your views of evolution and the the directionality of evolution strongly inform your views on superintelligence and what's going to come.

10:17
Um and near the end of the book you had a throwaway line, which is that something I'm gonna paraphrase, but nothing uh brings you greater joy than revisiting old grievances.

10:28
That was a joke.

10:29
That was a joke.

10:31
Well it was a it was a joke with a grain of truth in it, but uh yeah.

10:35
So I'd love to revisit one of the old grievances.

10:37
One of yours or mine?

10:39
One of yours.

10:40
Oh thank you.

10:41
Thank you.

10:41
Thank you.

10:44
So the latter uh latter chapters of the book you you spend a significant amount of time talking about your uh disagreement with Steven Pinker.

10:50
Um and Stephen Pinker and uh Jerry Coyne the the blogger.

10:54
Um I actually don't mention Jerry Coin.

10:56
That took a lot of self-restraint, but you you seem to have known who I was talking about.

11:00
And I read the blog uh blog post as well.

11:02
Um

11:03
And uh and so just put my cards on the table.

11:05
I'm probably more on the Pinker side of things.

11:07
Um uh just to orient um orient myself on the the landscape here.

11:11
But my understanding of the disagreement is that Pinker

11:15
sides quite strongly with uh Richard Dawkins' view of like the selfish gene.

11:19
So evolution is about uh replicators and the those replicators which are more able to replicate themselves with high fidelity uh will do so until they hit resource limitations.

11:30
And then when you get um uh random variation, then that's where you start to get uh evolution and and higher order uh emerging.

11:36
And this is of course a purely mechanical argument, um one that doesn't uh And I buy all that by the way.

11:43
We're we're on the same page there.

11:44
Yeah, so I guess my question is, what do you find lacking in Dawkins' view that necessitates adding purpose into the system?

11:54
I don't f I don't find anything lacking.

11:56
I think it's a sufficient mechanical explanation of how evolution works.

12:00
And it's like

12:02
You know, if if uh you know the uh the pocket watch is the famous example that Dawkins himself kind of revived uh uh with with design, so I use that because there was this treatise

12:16
in I guess in the 18th century, uh by a theologian who was arguing that God must exist because organisms are kind of like pocket watches.

12:25
He said, look, if if you

12:27
You know, if you're walking along in a field and you see a stone, there's no particular reason to think it was designed, it has a purpose.

12:33
But if you see a pocket watch, you look at it, it's intricately organized, apparently to do certain things, then you're right to infer that it's uh

12:41
Why that that it is uh designed to do something has a purpose.

12:45
And then he said, and look, if you see an animal, which is it more like a rock or a pocket watch?

12:49
Well it's more like a pocket watch, obviously

12:52
And uh, you know, Dawkins said he wasn't wrong about that.

12:57
Paley, this theologian, was not wrong to make that distinction and say that an organism demands a special kind of explanation.

13:06
But Dawkins said where he was wrong was thinking that it was a hands-on design by a god.

13:11
It was designed by natural selection.

13:12
I totally agree with Dawkins about all that.

13:15
Okay.

13:16
And and if you

13:18
Look at the pocket watch and expect its mechanics.

13:22
You can get a entirely sufficient explanation of how it works on that basis alone.

13:31
But you you uh but the question of its purpose goes beyond that.

13:36
And what Dawkins would say is: look, you can have a system whose functioning is entirely explained in physical terms.

13:45
Uh yet it has a purpose.

13:47
And I think Dawkins would would certainly Dan Dennett would have been comfortable using this language to go further and say

13:53
Uh and so with an animal, you know, you have a complete you you you can explain the way it it functions by just inspecting it and doing that kind of science.

14:04
If you want to know what its purpose is and how it uh came to have one, you need to know about natural selection, okay?

14:14
So uh, you know, he he he's not saying, you know, if you had said to him, well, what's lacking in the in the biochemical explanation of of how an organism works

14:25
He would say nothing, nothing.

14:27
That's not my point.

14:28
My point is there's there's a whole second question of how it came to be like this, and that's natural selection.

14:34
Now I want to I want to stop and say

14:36
The question we're addressing, the one that I have been addressing, not for the first time in this book, is not how organisms came to be.

14:44
Pinker and Dawkins and I are on 100% the same page.

14:48
I cannot overemphasize that, okay?

14:51
A purposive system does not have to have anything spooky in it.

14:55
There don't have to be any spooky forces in the organism.

14:58
There don't have to be any spooky forces driving evolution.

15:01
It can be strict natural selection.

15:03
So far as I know it is.

15:04
The question is, was uh the fur the seed of life, whatever that was, a strand of RNA or something uh you know, some more primitive form of self-replicating information?

15:18
Was that, for example, put here by space aliens so that it would evolve into intelligent life?

15:23
In that case, then then yeah, the whole system has a purpose.

15:26
Was it put here by a god, then it has a purpose

15:28
Or is it the product of some meta-natural selection that involves replication among universes?

15:34
You can imagine such things.

15:35
Then it would have a purpose.

15:37
Okay, so

15:38
I I I just uh I can't overemphasize that I do not take issue uh with Dawkins or or Pinker on uh the way uh

15:50
You know, the way evolution works or or uh how how you explain it or anything like that.

15:55
I mean there are quibbles with Dawkins over the extent of emphasizing group selection versus individual selection, but that's not of of relevance to this discussion.

16:03
So if Pinker was here, what would his uh criticisms of your view be then?

16:08
Um is it just how the whole thing got started?

16:11
Well he he he would say

16:13
He would say uh I he doesn't think in inspecting the history of evolution, life on Earth, he doesn't think there's good evidence that the whole thing has a purpose.

16:24
He would agree it's it's possible, it's in principle possible.

16:27
He doesn't think there's much evidence for it.

16:31
And uh, you know, he he would he would probably agree that there's in a certain sense a kind of directionality

16:38
Dan Dennett agreed with that.

16:39
And by the way, I would encourage people to go to YouTube and look at my long conversation with Daniel Dennett, much of which is about this, where some of the issues get

16:50
um illuminated, I think, i in in uh you know he seemed at times to be agreeing to me he seemed to be agreeing with some important parts of my argument but

17:00
uh I'll let people judge for themselves.

17:02
The um anyway, the the the as far as the God test goes, the appendix

17:09
of the book lays out I try to lay out the argument and the issues clearly for the purpose question.

17:15
It doesn't play a big role in the in the book

17:18
as a plank in my argument.

17:20
I mean I I I clearly you know I I I I do uh I do think that uh if you look back at how we got here

17:30
It's not crazy to suspect that there's a purpose.

17:32
And and there are other, you know, uh things I can bring to bear on this that I consider kind of evidence for, but none of them involve departing from a Darwinian view of evolution

17:42
Okay.

17:43
Um Darwin by the way, Darwin himself said now you may say he was trying to please the crowd, but uh i it's funny, the last paragraph of natural selection, I think he changed this because he felt guilty about it, but

17:55
Uh in the first version of it, he says uh there's a line uh in the first edition of uh the uh the origin of uh species

18:06
The last paragraph, i i there's this famous thing about how, you know, when life started in some pond or something, but there's something uh there's a line about

18:14
uh breathe life was breathed into it or something.

18:17
Anyway, it says like by our Creator.

18:21
Okay?

18:21
Now I don't think Darwin, I think he was

18:23
I don't think he believed that there was a god that had started the thing off, and I think he later took that out because he was actually just trying to appease the Victorian sensibility

18:34
by saying that but the point is he understood that his theory was intellectually compatible with evolution having a purpose and he was right

18:45
Or it's the case that the argument that stuff has a purpose is intellectually compatible with everything, too, right?

18:51
Because you can always

18:53
say, why did life start at the Big Bang or the universe started the Big Bang?

18:56
Well because there was a God that put a purpose for for that.

18:59
Like you can always backstop stuff with purposes that you can always do that.

19:03
But the point of Darkens is

19:05
uh uh book the blind watchmaker, the point of the first part is he agrees with William Paley.

19:11
It is possible to inspect a system

19:15
and look for evidence that can be adduced in favor of the hypothesis that it is purposive and was in some sense a product of design.

19:24
So Dawkins agrees with me on this point.

19:27
You may not.

19:28
You may think

19:29
It is no point in thinking about it because you can't bring any evidence to bear that's worthy of the name.

19:34
It's not an argument.

19:35
But but Dawkins is on my side of this argument if if you're taking issue with it

19:39
So yeah, so maybe let's start bridging over to the superintelligence um and I'll just do that by um asking how these two subjects relate in in your mind.

19:48
Um in the appendix, which uh which I recommend the readers consult for a more uh fleshed out version of your argument, um you give a brief analogy um where you say

19:57
Case number one, DNA is in a fertilized squirrel egg, then the egg matures, and then you get a squirrel.

20:04
And then that is, I believe, um, in your view, analogous to the first self-replicating material on planet Earth several billion years ago.

20:11
then biocultural evolution, and then the bio-newosphere.

20:16
Um and so in the same way that the DNA contains all of the ingredients to lead to a squirrel

20:23
I believe the argument that you're making is that evolution itself contains all of the necessary ingredients to lead to

20:31
a superintelligence.

20:33
Um again you say the bionusphere and so maybe I'll pause here and let you clean clean that up a little bit.

20:37
I I mean l let me uh add one concept which which is just

20:42
The idea of like a global brain of some kind.

20:45
Okay, so looking at uh organized uh human intelligence, leave aside AI.

20:50
You know, a figure I bring into the book is Pierre T.

20:53
R.

20:53
Deschardin is

20:54
paleontologist and theologian.

20:56
Now he did think evolution involves spooky forces, but leave that aside.

21:01
The uh he invented the term Noah sphere from the Greek word for mind, Noah N O S

21:07
Uh because even he was writing, you know, this is like a hundred years ago, he coined it, a little more than that, but he could see even then

21:15
that, you know, human this increasingly globalized intellectual collaboration among human beings and kind of organizational collaboration

21:26
Looks more and more like a giant global brain.

21:28
There's a global mind.

21:29
That's what the Noah Sphere was.

21:30
And he worked it into a theology and everything.

21:33
But the point is

21:35
He he saw it as a logical kind of extension of the evolutionary trajectory.

21:41
So you start out with like, uh, you know

21:44
uh he was this was before we knew what DNA was, but you you start off with um whatever gives rise to single-celled life, then you get multi-celled life

21:54
Then you get societies of multi-celled life.

21:57
So organization keeps going to higher and higher levels, and then you get uh an intelligence that at our level sufficient to start a whole new kind of evolution

22:06
And that carries uh the organization of that species up ultimately to the global level, he would say.

22:13
And so so uh

22:16
Yeah, that that is so to get back to the question you're raising, the question is, could you say that uh

22:23
the global brain that we've wound up with was uh implicit in or inherent in or whatever you want to use, whatever term you want to use.

22:34
Yeah well I wouldn't use inevitable, no, because it's not inevitable that a squirrel a squirrel is gonna mature.

22:39
Squirrels die.

22:40
They die early.

22:41
They abort.

22:42
They all kinds of bad shit can happen

22:44
Okay.

22:45
The question is, is it is it likely assuming the environment remains generally benign

22:55
Uh, and you know, the squirrel isn't overrun by toxic chemicals and the planet isn't hit by an asteroid that wipes out all life in the case of evolution, but

23:05
Is it highly probable that given long enough you'll get the outcome we're talking about, a mature squirrel or giant global brain?

23:13
So, you know, then you get to questions like, well, how how likely was the evolution of some form of life intelligent enough to launch?

23:21
uh a cultural evolution, a socially inclined species that did that, it wouldn't it didn't, you know, wouldn't have to look exactly like us, wouldn't wouldn't have to have five fingers or anything, but

23:32
some form of life.

23:33
How likely was that?

23:34
How likely was uh multicellular life to form?

23:38
And you know, you can you can adduce evidence.

23:41
For example, multicellular life has evolved independently a number of times.

23:46
Okay?

23:46
So that suggests that there was a a a a strong evolutionary impetus behind crossing that threshold.

23:52
In fact, I think we have a pretty good idea what it is.

23:53
I won't bore you with it, but uh you know, winged flight has evolved independently numerous times.

23:59
So

24:00
But interestingly, intelligence and communication hasn't, right?

24:03
Intelligence has only evolved once, and I believe that was one of Jared Cohn's points, which is that

24:07
Elephant trunks have evolved once, but we wouldn't say that evolution is leading towards elephant trunks.

24:12
Um and so if we're counting the number of times things have

24:15
Evolved independently, it seems like intelligence is that's only happened.

24:19
No, but i if you look at the timescale that evolution operates on

24:23
Like multi- the different forms of multicellular life, I don't know when exactly they arrived, but I doubt they arrived within like 10,000 years of each other, right?

24:33
I mean, it's a very slow process.

24:37
And the thing about intelligent life is that once you get there, um you start asking the kinds of questions we're asking, right?

24:47
Like where are the other forms of intelligent life?

24:50
Well, I mean you know, we were the first, okay?

24:55
My premise is that if we if we just all committed mass suicide, you would eventually

25:01
see another one, but it was inevitable that whichever was first was going to look around and not see a second because evolution is just on such a slow timescale.

25:10
Also, they tend to wipe out their rivals, right?

25:13
There aren't any Neanderthals around anymore.

25:16
I mean it's there's some question of about which groups I guess wiped out which we have some Neanderthal DNA

25:23
But so no that that I don't think that's uh that's a good argument.

25:27
I think it has that in common with a lot of Jerry Coyne's arguments, but we needn't go there.

25:31
Yeah, fair.

25:33
Um okay, so I'm actually gonna read a quote from Non-Zero at the beginning of Non Zero, uh, which was published in nineteen ninety-nine, I'll remind readers.

25:41
So you say, quote

25:43
If directionality is built into life, if life naturally moves towards a particular end, then this movement legitimately invites speculation about what did the building.

25:51
And the invitation is especially strong, I'll argue, in light of the phase of human history that seems to lie immediately ahead a social, political, and even moral culmination of sorts.

26:02
Um, okay, so I want to focus on this idea of a climax or a culmination to this process.

26:09
And, you know, I take it part of the point

26:12
of this book, your most recent book, The God Test, is to say that superintelligence in some ways could be this culmination.

26:21
But presumably in nineteen ninety nine, when I asked you, you would have said, uh

26:26
We don't whatever whatever big thing was happening at that point was also likely to be the culmination.

26:31
So I guess I'm s I'm slightly worried that

26:35
there's a whiff of perhaps something like unfalsifiability to this thesis, or that at every moment in history we are likely to think we're somewhere near the end of history, because things always seem rather chaotic.

26:49
given where we are.

26:50
Um alternatively you could also just say that maybe you predicted superintelligence um and and the AI revolution that's currently underway in 1999.

26:59
But um I'll let you speak to that.

27:01
I don't think I quite predict superintelligence anywhere, including uh in this book.

27:05
I do consider it quite plausible and and

27:08
I think I think it's likely we'll get to to that.

27:10
But let me uh let me let me answer your question.

27:12
I I think actually fundamentally the two arguments I'm making are are the same argument.

27:19
So in non-zero, and this is now twenty-six years ago, um I uh I said that uh you know technological evolution, I mean the the the book's premise is that

27:33
Uh the growth of social organization, growth in the in in the scope and depth of social complexity from hunter-gatherer village globalization has a lot to do

27:43
driven largely by technological evolution.

27:47
And what tends to happen is that new technologies either facilitate or otherwise encourage

27:53
uh playing larger non-zero sum games, you know, more complex, over greater distance.

27:59
So like roads, riding, all those things facilitate longer distance, non-zero sum games, like trade.

28:06
Um and

28:08
And then and meanwhile, uh technologies that are put to uh unfortunate uses like killing large numbers of people, they they encourage

28:19
uh non-zero summits in another way, which is that they encourage societies to in their defense, you know, cooperate uh more intensely.

28:28
You know how societies get during wartime.

28:31
Um so anyway, the the point is

28:33
The the argument was that uh the this this this feature of technological evolution had uh driven us to to uh

28:45
to this point of of larger and larger realms of more and more complex human collaboration.

28:52
And I said, uh you can already see

28:56
technological evolution starting to encourage us to make uh the final leap to to something that worthy of the name global governance

29:05
and something worthy of the name a global community because, you know, we had started to see technologies that it was in our common interest

29:13
to collaborate to control or constrain nuclear weapons, obviously, uh climate change, the possibility of global pandemics.

29:22
And I said, I'm sure in that book, look

29:24
Bioweapons genetic engineering uh gives us more reason to collaborate.

29:29
And and I would just stop and say that

29:32
Whether or not you think COVID started at a in a lab in China, I think any reasonable person would would have to accept it.

29:39
You know, could've.

29:40
That's a real possibility

29:42
And this is a good example of why it would have been in everyone's interest to already have in place uh uh uh a more effective uh regime of internat the international governance and monitoring of labs, maybe.

29:55
Okay, so that's

29:56
The kind of thing I was I was uh talking about that these technologies for our own kind of mutual defense, the mutual defense of humanity

30:07
encourages us to collaborate and like put aside these stupid wars and focus on the fact that more and more we are in the same

30:17
boat.

30:18
Uh there's also a p a positive side.

30:20
I mean more and more we can benefit materially from collaborating over long distance and create, you know

30:26
uh more prosperity and so on in principle.

30:29
But in any event the technologies do uh pose the threats.

30:33
And I'm just saying artificial intelligence is another example of that.

30:37
Okay?

30:37
So uh it is a technology that

30:41
We cannot uh if we want to govern it uh in a way that serves the interest of humankind, we're going to have to get better at international collaboration and cooperation.

30:55
That's the main argument.

30:56
It just plugs exactly directly into the non-zero argument.

31:00
I don't have to revise a thing except to say, oh, here's another technology, AI.

31:04
Now I had written about AI before, as you know

31:07
Uh the book starts out with my conversation in the 1980s with Jeffrey Hinton back when I was writing a piece about AI.

31:14
But

31:14
I didn't I didn't emphasize AI in uh non-zero, but because one thing in in two in the year 2000 didn't seem like it was uh

31:23
Heading anywhere in particular.

31:24
Now, um I I also mentioned let me uh and then I'll I know I've talked a long time.

31:29
Let me just just finish and say

31:30
I did mention the idea of a a a global brain in non zero, and I I I even mentioned Terre de Chardin in passing, and uh

31:40
And you know, I think if superintelligence happens, it's basically gonna be the global brain I was talking about, except that uh

31:51
The neurons are silicon, um, and not just human, right?

31:55
Uh and so yeah, I just see this

31:59
Uh I I'm not I don't I don't see myself as as as doing a bait and switch.

32:03
Oh I was talking about these technologies.

32:05
Oh suddenly you're talking about that technology.

32:07
It's the same damn thing

32:09
And in fact, artificial intelligence amplifies some of the specific threats I talked about, like bioweapons, and it will hasten the advent of a kind of genetic engineering that can be used to ill and and and so far.

32:23
So

32:23
It it it it's all one thing.

32:26
Interesting.

32:26
So I wasn't gonna ask this question, but just your comments made me think about this.

32:30
Um

32:31
So on this idea of of the one of the theses in Non-Zero, uh which full disclaimer, I haven't read Non-Zero, um, so my apologies there.

32:38
You know, um a lot of people are in the same boat.

32:41
Uh I think they deserve our pity, but there it is.

32:45
Um

32:46
But it strikes me that uh we are way farther away from global governance today in the era of Trump

32:54
than we were in nineteen ninety, um when the Soviet Union had just collapsed and NATO was maybe stronger than it was.

33:03
And so I guess I'm just curious before we move to the superintelligence conversation, um

33:08
how the Trump phenomenon has uh updated your views about the prospects of global governance.

33:15
Um the thinking about the

33:16
gridlock at the UN, I'm thinking about the likely collapse of NATO um in these kinds of subjects.

33:22
How how is the the fall of the American Empire if that's not too grandiose?

33:27
I can think of bigger tragedies than the demise of NATO and the fall of the American Empire.

33:31
I mean uh the American Empire has You're never a big fan, for sure.

33:34
No.

33:35
Um but but uh but uh uh I I

33:40
I would say it hasn't necessarily changed my views in the sense that I have never felt super optimistic that humankind is going to respond to the challenge.

33:49
So the fact that we seem to be failing doesn't call for me to revise, you know, uh I mean I may I may have sounded more optimistic in parts of non you know it's funny uh

34:02
When when the hardcover came out and people said, Oh, you're so optimistic, I was like, uh I don't know, I think you're getting me mixed up with somebody else.

34:10
And I actually revised in between hardcover and paperback, I revised the introduction to emphasize, like

34:16
I'm not saying we're gonna pull this out of the fire.

34:19
I'm not gonna say I'm not saying it's gonna work out.

34:21
I'm just saying that if we do not get better at global governance and do not make a kind of moral progress

34:28
uh that that I think that facilitates uh global governance.

34:33
Um we're in deep trouble.

34:35
That's still my view.

34:36
Now you're right that from my point of view, Trump is almost, you know, the secular equivalent of the Antichrist.

34:41
He's just he's just

34:42
Uh I mean if he had at least stuck with his line about regime change wars, I would have but apparently but apparently he was just kidding about that.

34:50
So I I I can't think of a really a single thing I I approve of

34:55
about him.

34:55
Um, firmly in the same cat about that, just FYI.

34:59
Yeah.

35:00
Well, it's uh but now that said, you know, human history uh all trends, you know, of significance

35:09
Have their ups and downs.

35:11
I mean, for example, on balance, global prosperity has grown, but there have been depressions.

35:15
And you know, in the past, if you look at uh big leaps in

35:22
uh international governance.

35:24
I mean you would think right away of like, well, League of Nations was it was a very ambitious effort

35:30
United Nations, it was a very ambitious effort.

35:32
Well both happened in the wake of epic catastrophes.

35:35
You know, that focused you know the world wars that focuses people's minds and

35:41
You know, the way I feel about AI is is the way a number of other people I've talked to who believe that it needs uh international governance feel, which is that I just hope

35:52
Uh it's not a super costly catastrophe, you know, I that gets our attention.

35:58
I hope it's like a near-miss.

36:00
I mean, in a way, this mythos thing had some of the hallmarks of that.

36:04
It's like

36:06
You know, do you see what like I mean, of course there's controversy over to what extent, if any, anthropic is exaggerating the powers of mythos.

36:15
But anyway, if you accept the premise that

36:18
Uh, if they just released it, maybe some bad actors would have gotten it and done bad things.

36:24
You know, that's the kind of thing that

36:27
No, I may and maybe not a near missing the most dramatic sense, but it's the kind of thing you would hope would at least get people's attention.

36:33
Um so anyway, I'm not you know, I'm not predicting success.

36:38
I mean, uh i it we're a long way.

36:42
from where we need to be, not just in terms of of governance, um, but in terms of the psychology

36:52
that that w uh the kind of psychology that would help us get over these ridiculous, pointless wars, which I think is a prerequisite for good good global governance

37:02
I'd also say I think AI is just a huge challenge to govern, even then.

37:07
I mean it's like i it it it's very challenging.

37:12
Okay, well let's let's just move into that territory and and focus the conversation on AI and

37:17
and super intelligence.

37:18
So in the book you sort of discuss these two camps, you discuss the accelerationists and the doomers.

37:25
And

37:25
You often say that it's hard to outright dismiss the case of the Doomers and, you know, parts of you, um, even if you're not wholly convinced, you find yourself sympathetic.

37:35
to many of their arguments.

37:36
I'm wondering if you could just flesh out to what extent you are concerned about the prospect of sort of rogue

37:45
AI or rogue superintelligence.

37:48
Um, because I feel like the you're slightly non-committal in the in the book.

37:52
If if I can just echo that a little bit, like there is this um

37:56
Maybe I don't say rhetorical move, but a a device you use which is to talk about other people's views um and kind of withhold

38:05
staking a claim on your own views necessarily.

38:07
And even in the preceding discussion today, it's talking a lot about what Dennett thinks and what Dawkins thinks and and stuff, but it's sometimes hard to pin down what you personally

38:16
Think.

38:16
Um and so would love to uh to to know if you consider yourself a doomer or uh um uh or or what your what your views are specific.

38:23
Um

38:24
On the Dennett Doggins thing, I've tried to be clear, I think there is considerably stronger evidence that the overall system is in some sense purposive than people like like uh

38:33
Dennett and Dawkins believe.

38:35
Gotcha.

38:35
Gotcha.

38:36
And that I also think that's all you can say unless you have some kind of special divine revelation, which

38:41
I haven't had yet and time's running out, I gotta say.

38:44
Uh gotta run over to Hira the Mountain Cave and hope for the best.

38:49
Maybe.

38:50
Uh

38:51
But on the uh so yeah, and and i I don't think you're crazy to use the word rhetorical move.

38:56
I mean I'm trying to

38:59
Uh I'm trying to frame the argument in a way that uh

39:07
you know, is accessible to people who come in as skeptics of the strong doomer claims

39:15
And by Doomer, you know, I use the term kind of sci-fi AI Doomers.

39:20
There's that, Elias Yudkowski, AI's gonna take over and subjugate us or kill us

39:25
And that's the thing I say, you know, I I think it's just hard to dismiss.

39:29
I don't know if it's gonna, I don't have nearly his confidence that it will happen if we don't stop AI progress soon, but

39:36
Uh it turns out to be surprisingly hard to dismiss the argument, and that's genuinely the way I feel.

39:41
It's a little like purpose.

39:43
It's like I don't think it it's an open and shut case either way

39:46
And I think it's a lot more open than a lot of people say.

39:48
I I I would as a tangent right now say, you know, even aside from sci-fi doomerism

39:56
I feel pretty damn confident that AI, if not governed wisely, is just going to be profoundly destabilizing.

40:05
You know, because there's so many dimensions right ranging from employment to what bad actors could do with it to like

40:13
the way it changes uh just the nature being human and people freaking out about that and and so on, there's so many just kind of destabilizing things about it that

40:25
That alone warrants us to uh warrants our, you know, taking a very good look at it and and

40:33
for starters, uh I think concluding that it wouldn't be a super bad thing if the rate of uh progress slowed down, you know?

40:41
Uh just

40:42
Just I I just wish we could get the one argument against regulation off the table, which is that, well that would slow down the pace of innovation, you know?

40:49
Uh and of course

40:51
If you say, well, what's so bad about that?

40:53
Then they say, well, China.

40:54
China's so bad about that, which which gets us back to the fact that I think we need to mature a little psychologically and get better at working things out with other nations.

41:02
But um

41:04
So anyway, the the you I I think you're you're right that m the voice is kind of as like uh this this kind of naive

41:14
uh, you know, tourist of AI sci-fi doomerism, to the extent the book is about that part of the the doom.

41:23
Uh and uh and

41:26
You know, what do I encounter?

41:27
But that actually does recapitulate my journey.

41:29
I mean, I like I had a you know I had a conversation on my podcast with Elie Ezra 15 years ago or 16

41:36
I I watched it recently actually.

41:37
Yeah, and he was in he was actually in mid transition between Singularity Enthusiast and Doomer.

41:45
But um

41:47
And at that point, and up until you know right before ChatGPT, even after ChatGPT came out, my line was always like, wait a second, I don't think you understand

41:58
Human beings have a drive for power because of their specific evolutionary history.

42:04
It is not a property of intelligent systems in general.

42:09
Uh that was my genuine view.

42:11
And I'm not like making that up.

42:14
But I mean it's on the record.

42:16
I kept saying it.

42:17
And I've since come to realize why that objection alone is just not reassuring enough.

42:23
And um I can get into that if you want, but but I just want to say that I think on the one hand, you're right.

42:29
I'm

42:29
I I I'm I'm uh I don't want to come off as like Eliezer, like, you know, he's got he's got his sympathetic audience, and then he's got a lot of people who are alienated by how extreme he sounds, right?

42:44
And I, first of all, I don't feel as extreme as he does, but also I don't want to alienate people in the same way he does.

42:51
I want people

42:52
you know, who have not thought about this much and are reasonable people to at least take seriously the possibility uh that that could happen.

43:01
There could be

43:02
There could be rogue AIs getting out of control.

43:05
That could be super bad.

43:06
There could there could be an actual takeover.

43:09
You know, and a surprising number of actual leading AI researchers, Jeffrey Hinton himself.

43:15
Sometimes called the Godfather of intelligence as it may turn out, you know, the good n I forget, I quote him in the book.

43:21
Something like the good news may be that, you know

43:24
Intelligence has a long future.

43:25
The bad news may be that our particular form of intelligence was a transitional phase.

43:32
Uh you know, Yashua Binggio, uh these are the these are the the founders of deep learning who who um uh which is what gave us kind of more as much as any other school of AI

43:44
thought the the revolution we're in the middle of.

43:48
And uh, you know, the the the three people who kind of got the uh

43:53
the touring award for that together in the same year.

43:56
Those two and uh Yan Lacoon, two of the three are are

44:00
Pretty dumorous, you know.

44:02
So I think it's worth taking seriously.

44:05
Yeah, well so just on that, so um

44:08
I guess I don't want us to be or I'll speak for myself, but Ben and I um share a lot of the views here, um, as being perceived as being on the accelerationist camp.

44:18
Because I do think that there's a a third axis or a third position which wasn't um maybe fully explored in in that book.

44:26
Um and that's a position that AI is extremely powerful.

44:29
Uh it's gonna have a lot of transformational effects in our society.

44:33
Not all of them are going to be good.

44:35
I agree with your view of

44:37
uh calling it destabilizing, I'd maybe uh analogize it to the destabilizing force of say the printing press, which caused Revolutionary Wars and the Catholic Reformation and and all this.

44:48
Um but

44:51
The AI as tool people, um, who I would I put myself in that camp, um, excise the superintelligence risk.

44:59
from the conversation.

45:00
And um and maybe just uh one way I've been thinking about this recently is um you have your underlying technology, which is the large language model, the LLM.

45:12
And then you have metaphors that you can layer on top of that technology to understand that technology.

45:17
So one metaphor is of course

45:20
superintelligence or some um god on the mountain who is giving us wisdom.

45:25
Um another metaphor I would say is is one that you draw on a lot, which is um evolution

45:31
um LLM's uh the training as as being um analogous or metaphorically related to again you don't use the metaphoric language, I'm I'm introducing that of course.

45:40
Um

45:41
uh to the process of evolution.

45:43
Um another one which I've been thinking about the last couple of weeks is um maybe a bit more playful, but uh you remember those like

45:51
Cars that you could put on the ground and you could um wind them back and they'd let go and they'd shoot across the the room.

45:57
Yes.

45:59
Yeah, so like AI as like a slingshot.

46:02
It's a slingshot for your ideas.

46:04
It can take your ideas and it can shoot them.

46:06
much further than you alone would be able to to do.

46:11
And so when I use these tools, they allow me to explore all these ideas much more quickly than I would be able to.

46:20
Um and if I have misaligned the direction of my slingshot, I'm gonna be shot into a very wrong area of idea space and I could start going into um all sorts of of uh uh blind alleys and I could hit them more quickly.

46:33
Um

46:34
But if I align it in the right direction, then I can make a lot more progress more more quickly as well.

46:38
And so I guess um the distinction here uh is that I don't think these tools are ever going to um not be operated by a human being.

46:48
That I think would be the main um difference in in our views, uh where I fully agree with all of the the the real world concerns of

46:56
Um how is it going to affect um our ability to discern truth when you just have AI-generated images everywhere?

47:03
Or how what's the effect on the education system?

47:05
Or what's the effect in the arms race?

47:07
Or what's the effect

47:08
of self um uh directed weapons.

47:11
These are all extremely real concerns and and I part ways with like a Mark Hendriesson type

47:16
who would just say it's all fine, it'll just work itself up.

47:18
No, I don't think it's fine.

47:19
I think it's seriously problematic and we need to absolutely take that uh concern um to heart and work on it.

47:25
But

47:26
I don't see any possible way that these tools are going to just take on a life of their own and start operating without a human being

47:37
putting in the prompt and stuff.

47:38
And so I guess I would just um ask you uh what you think about that view and and like what what do you hear from the Doomers

47:47
that leads you to believe that these things will start working without human beings as operators.

47:53
Because that to me is where I get off the the the ride.

47:55
And that's what um we had brought up in our Scott Aronson

47:58
uh conversation um uh about is AI just a tool or or is it something more?

48:03
Well yeah I'd say a a couple of things.

48:05
I th I think it's a tool right now.

48:08
Um you know I do uh um I do address

48:13
this question in the sense uh in the book in the sense of I don't embrace that view that that that it will always be a tool and only a tool, but

48:22
You know, I I I raise such questions as, um, you know, if there's let let's imagine that uh there's a CEO who is uh

48:37
You know, getting some strategic guidance from an AI.

48:42
And uh

48:45
And, you know, okay, so it's a tool, it's a tool.

48:49
But if it gets to the point where the AI

48:54
The the CEO can't even really fathom uh the strategic the logic behind the strategic moves, but the AI has always been right in the past.

49:02
And so the CEO says, I guess we'll go with it

49:06
Hmm, who's in charge at that point, I'd say.

49:08
And I'd say um then if you imagine uh two corporations that are fiercely competitive.

49:16
And both of them have these AIs that have been getting giving them guidance.

49:20
And they just they just start thinking, I can't afford to wait, stick around, wait, you know, wait until I I grasp this, assuming I'm even capable

49:27
Just do what it says.

49:29
Now, so so one question is, at that point, it has some agency been meaningfully transferred to an AI?

49:37
I mean, I I I'm just raising that question, but but there's a whole nother um

49:42
issue with uh you know highly agentic AIs, which that really isn't, because uh the the nature of the relationship between the CEO and the AI is that the

49:52
You know, nominally the CEO's the agent.

49:54
The the CEO pulls the trigger.

49:56
And I'm just saying at some point in that case, if you

50:00
If the CEO's philosophy is, I do whatever it says, well, you're not really meaningfully pulling the trigger.

50:05
But but that aside, you know, if you if you look at agentic AIs, I mean

50:13
I I I I I I guess I'd ask you.

50:16
Um a year and a half ago, like if you look at what is now happening

50:21
with like open claw and some of the reports you're hearing about it like, oh it shit, it went off and did that.

50:28
I didn't expect it to go out and do that.

50:30
Now it it like it erased my file.

50:32
It did this, it did that.

50:34
Um

50:35
Uh given those reports and and also just the sheer power of the things, which I'm seeing, right?

50:42
It's like

50:43
Uh for example, just like software is becoming like make a wish, right?

50:46
Like, oh, I'd like software that does this.

50:48
And you don't have to know how to code.

50:50
You can use Claude code to create it.

50:53
I I'm wondering

50:54
A year and a half ago did you expect these did you expect us to be here in a year and a half?

50:59
Yeah, I did.

51:00
I did actually.

51:01
Um and the reason I did is because I think about these

51:05
tools, these engines as being entirely governed by the amount of fuel we can put in them.

51:13
And the fuel is the training data.

51:15
Um and so there was a a paper that got very little recognition three, four years ago that just estimated the amount of high-quality feed stock

51:26
that was available left to train on.

51:28
So feedstock here would be stuff like uh GitHub, Wikipedia, Project Gutenberg, um text that has been um s

51:36
qu uh uh refined over huge amounts of time with a lot of human labor.

51:42
Um low quality feedstock would be

51:44
stuff like um YouTube subtitles or transcripts of of um the um reality TV, stuff that doesn't

51:54
have a lot of information about the height of the Eiffel Tower in it, say.

51:58
And so you can estimate uh when does the available feedstock uh when's that going to run out?

52:03
Um and then you can use that to kind of estimate the amount of available progress left uh to make.

52:08
And so this paper came out uh about three years ago and it estimated that we would run out of

52:14
high quality training data around 2027, at which point the models would start to plateau.

52:19
Um and so I'm we were on the record saying that um yeah we expect

52:23
that these tools are going to get better and we're gonna expect them to get better in particular places where there's high quality training data.

52:31
So if there's a lot of um GitHub left, I don't think there actually is too much GitHub left to train on

52:36
Um then we'll see progress up until we've used it all up.

52:40
So I would expect, for example, a lot more progress uh left in video and image generation because we haven't tapped out YouTube

52:47
Um and now I'm expecting that in the next couple um five, six years, we're going to start seeing progress on like the second layer of AI, which is um so you talk about um agents.

52:59
But another term is harness.

53:01
So the harness is is all the stuff that wraps around the uh the model calls.

53:07
And so I expect there's to be a lot of progress uh on that front.

53:11
But um but if we just look at the raw um capabilities of these systems when they don't have tools and they don't have the ability to call the internet

53:21
um and they just are relying on their own um strengths.

53:24
Uh I expect that to plateau.

53:26
Um on your point about the CEOs um kind of thoughtlessly just doing whatever the uh Claude uh says

53:35
Again, that's to me just like thoughtlessly shooting your slingshot in a bunch of different directions and not knowing that no, it's up to you to challenge the thing that it says.

53:44
Um you can't just naively trust it to be this fount of perfect wisdom that will give you perfect answers, but it will accelerate your questions.

53:52
Um and so

53:54
I believe later on in the book you talk a lot about cognitive sovereignty, which is a term I like quite quite a lot.

54:03
Uh when you talk about cognitive sovereignty, uh looking through the quote here, um you say uh so paraphrasing a little bit, you say um

54:12
When people talk about cognitive sovereignty, they would want an AI companion that would give them honest feedback even when they didn't f when it didn't feel good, and would help them see the perspectives of people whose perspectives they've otherwise inclined to misperceive.

54:24
and this companion would foster various other forms of acuity that can emerge when cognitive bias is weakened.

54:30
It would do for the mind what a good personal trainer does for the body, make it stronger and more resilient even as it force forces

54:38
even as forces designed to compromise it proliferate.

54:41
So if you view AI as tool, then all the stuff you've just described is achievable tomorrow.

54:47
You can just put it into your prompt.

54:49
If you view AI as superintelligence,

54:52
then you're kind of just like waiting for the companies to do it for you.

54:56
And so I guess I I think that a lot of the issues that you bring up in the book

55:01
can be addressed immediately by reframing what the technology is as a force multiplier.

55:08
And it's up to you to guide the force in in in a way that's most conducive for your own

55:12
um wellbeing and and and um mental health.

55:14
And so that's how I would address the point you raised about the CEOs, who I think would maybe do that because they've been listening to too much doom debates and because they just are listening to too many people say that these things are just

55:26
getting smarter and smarter and you can trust them less and less.

55:28
Whereas I'd rather say as the technology improves, you need to damn well learn how to use it better because you can absolutely be slingshotting yourself into the wrong, wrong space.

55:37
So that would be a merge sponsor

55:39
Yeah, I mean on the uh for just as an asterisk, I I uh uh uh three years ago, we don't need to get into this now, um that was before the reasoning models.

55:53
came out.

55:54
My view is that that really added something beyond what mere more data would have added and and that then that that's actually very relevant.

56:03
to the ability of agents to engage in very complex planning and execute plans, right?

56:10
So

56:11
Uh I'd say that on on the CEO thing, and I think we can expect more advances.

56:17
You know, the the reasoning probably doesn't deserve to be up there with transformers

56:21
But it's not nothing.

56:22
And uh there are other there have been other significant advances and I I suspect there will be more.

56:27
On the CEO, you said, well

56:30
You know, CEO who's relying in that way on the AI kind of like that's what that's what they get for for not using the the AI in a more judicious way.

56:39
But what I'm the the scenario I'm positing is where

56:43
If they want to remain a CEO, they just don't have any choice because they're competing with CEOs that are using AIs that come up with effective strategies faster than human beings.

56:53
So they don't really ha i it's like saying, well, if you're if you're so foolish as to use autonomous weapons, you shouldn't be out in the battlefield.

57:00
Well, but if you're not using them, you're gonna die, okay, if the other guy's using them.

57:04
I mean, that's just the way it is

57:06
So and so I I would say that that at least is my premise.

57:10
Now you may disagree, you may think we'll never get there, uh, but it seems to me, honestly

57:16
We're not that far, you know, now.

57:19
I mean these things are very fast and they come up with ideas very fast and and some of the ideas um

57:26
are are actually very good.

57:29
Um on the on the cognitive sovereignty thing.

57:32
So you were kind of saying, I guess, well that CEO wasn't using cognitive sovereignty.

57:37
uh you know wasn't right, wasn't insisting on remaining in charge.

57:41
I I I'm saying you know, and in general a lot of my book is is it's very evolutionary at a number of levels in its perspective

57:50
And one of, you know, uh like all at various levels, you've got like systems competing with one another.

57:59
And in some cases, I w I think we should purposely put an end to that.

58:04
Like if the US and China are going to compete in some kind of death struggle.

58:08
That's going to greatly compromise our ability to g to develop these technologies wisely and not get into trouble.

58:15
So I'm not embracing a world, you know, the way a social Darwinist would.

58:20
I'm not embracing a world

58:22
in which uh competition leaves to the f leads to the fittest competitor.

58:26
But at the same time, I think we have to be realistic.

58:30
You know, the economy is a is a competitive place.

58:33
You have these companies, they're competing with each other.

58:36
Uh you have AI models competing with one another, and that's uh what I think drives uh will drive uh a certain amount of uh

58:48
Well, the development of AIs that maybe are regrettable and the use of of AIs in ways um that are regrettable.

58:55
So uh y you know, it it's kind of like

59:00
And and I'd say the same thing about but about AI is going rogue.

59:03
Like like if you imagine um

59:08
AIs that once they're out of control are super smart uh and they can replicate themselves and stuff, and they know how to exploit various parts of our infrastructure

59:19
I'd kind of rather none of them ever escape our control.

59:23
And and uh but I'd ask like like how easy is it to keep a single one from escaping from our control, right?

59:30
And all I'm saying, I'm not predicting this.

59:32
I'm not predicting this.

59:33
I'm just saying

59:34
If we right now are not considering these kinds of things serious possibilities, I don't think we're doing the odds right.

59:41
And and I think

59:43
we should we should act accordingly in terms of governance and other things.

59:48
In a few minutes we should move to the um

59:51
LM training as evolution, but I just want to make one more point on on this question of like what is the role of the human being in the development of these systems.

01:00:00
Um, because there was a few points in the book that I thought were verging slightly on the misleading in the way that um

01:00:08
you had described how these systems came about in the first place.

01:00:11
Um and so just to give an example, um so you say uh referring to um

01:00:17
how computers have uh learned to generate coherent language.

01:00:21
I believe that was that was a subject um earlier in the first few chapters.

01:00:25
So you say, in a way this seems miraculous.

01:00:27
Think about it.

01:00:28
You feed a machine strings of symbols that, though meaningful to humans, are from the machine's perspective machine's point of view pure gibberish.

01:00:37
And you give the machines no information about what these strings mean to humans.

01:00:42
And in an important sense, the machine proceeds to figure out their meal.

01:00:46
So this is a very uh autonomous way to describe the development of LLMs.

01:00:54
Um but I think it's misleading and maybe one way to describe that is um just in researching uh in preparing for this episode.

01:01:01
Um

01:01:01
There's all sorts of uh undeciphered writing systems that uh paleontologists and linguistics uh linguists are trying to to figure out

01:01:10
So one of them is called linear A, and it's a writing system that was used by the Minoans of Crete from 1800 BC to 14

01:01:19
50 BC.

01:01:20
Uh and this is a writing system for which we don't have a Rosetta stone.

01:01:25
Uh so there's no uh translation, so we don't know what the meaning of of these symbols are

01:01:30
So I mean according to the way you described it, we could just feed these symbols to an LLM.

01:01:36
These are symbols that

01:01:38
Um we've given the machine no information about what these strings mean and the machine can just spit out meaning.

01:01:44
But of course

01:01:45
We give these systems, when we train them on English, huge amounts of information.

01:01:49
So the information comes in via the curation of the dataset in the first place.

01:01:54
So you have to select what counts as symbols that are are valid.

01:01:59
Um it comes through the uh the post-training, um both from the like um supervised fine-tuning, um where you're giving it to human beings to go up vote, down vote, uh RLHF.

01:02:11
This is where the meaning is leaking from the human beings into the systems.

01:02:16
This is how we are putting our knowledge of these writing systems into the LLMs.

01:02:22
You mean it's not happening in the so-called pre-training where they're predicting the next token or or it is?

01:02:27
That's happening as well.

01:02:28
That's happening as well.

01:02:28
Okay.

01:02:29
Um yeah, so that's predict the next token.

01:02:31
But it's not

01:02:32
predict the next token of the linear A writing system because we have no idea how that writing system works.

01:02:38
Right.

01:02:38
And so it would just be producing meaningless gibberish.

01:02:41
When we make it predict the next token

01:02:43
We're the ones who are designing the loss functions.

01:02:45
We're saying, okay, we want you to predict the token in the following sense.

01:02:49
We want you to predict the token in the middle of the sentence rather than at the end.

01:02:52
because we know that language, at least English, um, is bi-directional and and a word in the middle of the sentence can have its meaning affected by words at the end or at the beginning.

01:03:02
And so this is how we are teaching these systems what these symbols mean.

01:03:08
And if we try to do this with a writing system for which we don't actually understand it.

01:03:12
um like linear A, that would be a pure case of what you're describing.

01:03:16
And it wouldn't work.

01:03:17
It wouldn't work at all.

01:03:18
And so I guess just there seems to be a bit of a rosy view of the way we wouldn't systems or I mean

01:03:25
We could i i assuming I don't know anything about linear A, I mean assuming that it's sequences of symbols, you probably could uh make uh the machine

01:03:36
Better at predicting what symbols follow others, um, you you couldn't give it uh a sterner test in the sense of, okay, now generate a sequence of symbols that you haven't exactly seen but makes sense.

01:03:49
We can't give it that test because we don't we don't understand.

01:03:52
So we don't we don't we don't know.

01:03:53
I mean could give it that test by choosing the training data and choosing an ancient language

01:03:58
For which we already have the Rosetta.

01:03:59
No, but I mean you can't if you don't understand the language yes, you can train it, but my point is, then if you want to give it the acid test

01:04:07
And say, okay, now here's the first few few uh symbols in the sentence.

01:04:14
Now continue it in a way that, on the one hand, isn't exactly like anything in a training text.

01:04:21
But quote, makes sense.

01:04:23
That's a test we can give it when it's English.

01:04:25
We can't give it that test with linear A, because we don't know what makes sense

01:04:28
If you show us a novel sequence of symbols in linear A that don't appear exactly in that way in linear A, we don't know if it makes sense.

01:04:36
Yeah, but we could do it with

01:04:37
We could do it with ancient Babylonian where we do have a Rosetta Stone and we just hide the Rosetta Stone and we see if it produces the Rosetta Stone.

01:04:46
Um I haven't seen that run, but I am quite convinced it wouldn't because the mechanisms by which it's a little bit more than a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a

01:04:52
There's a way to fly down to this

01:04:54
Yeah, totally.

01:04:54
Yeah.

01:04:54
Falsifiable tests for sure.

01:04:56
Falsifiable tests.

01:05:02
myriad of different ways that humans have massaged the data, corrected the output, said upvote, down vote.

01:05:03
Ye

01:05:07
That is how these systems figure out their meaning.

01:05:11
It is not the case that you give the machine no information about what these strings mean to humans.

01:05:15
You give it a huge amount of information, just in these ways that are different from the deductive rule-based systems that you're contrasting it against.

01:05:22
So just because it's an inductive bottom-up system rather than a rule-based

01:05:26
top-down system does not mean that no information is being moved from human to machine.

01:05:32
Well, yeah, information has to be moved.

01:05:34
There you have to give it uh information that has patterns into it because it is making in effect inferences on the basis of that.

01:05:44
Uh the the um

01:05:47
Uh now on the meaning thing, you know, here's what I meant.

01:05:52
We now I mean we need to get a little technical here.

01:05:56
So

01:05:56
When it's technical, so yeah.

01:05:58
Okay, good.

01:05:59
Uh maybe more technical than me.

01:06:01
Uh that may that may work to my disadvantage.

01:06:04
But uh when um

01:06:07
You know, words, I mean strictly speaking tokens, but to oversimplify, let's think of it as words, are are represented uh via you know long sequences of numbers

01:06:18
Known as vectors.

01:06:20
In principle, you can locate them in vector space, although the these are have so many numbers that I personally can't conceive of vector space

01:06:29
beyond three dimensions.

01:06:30
You may not be able to conceive it, but the fact is you can you can measure the distance between words in vector space if you have the vectors.

01:06:38
Now, what we say to the machines kind of at the outset, before any training at all, uh is

01:06:47
And and even before there are word embeddings, you know, which uh uh the word embedding is just that long sequence of numbers that represents the words, but you know, we we we do what humans did invent.

01:07:00
Is the idea that we're gonna use these things to represent words, long sequences of numbers.

01:07:07
But we don't tell the machines what the number should be.

01:07:10
And basically, in effect, metaphorically, what we're saying to the machines is

01:07:14
We want you this gonna be this long period of trial and error, okay?

01:07:18
We're gonna you you can you you know you're gonna try to predict the next word and we're gonna tell you how successful you were or weren't

01:07:26
You're going to make adjustments, you can change the numbers, and in the end, what we want is a sequence of numbers corresponding to each word.

01:07:35
Let's say we're, you know, it's a it's a it's a run that determines the word embedding.

01:07:39
What we want is a uh a sequence of of numbers that that

01:07:47
helps you guess correctly what the next word will be or what the missing worded will be or if it's translation, what the French word will be, or whatever.

01:07:55
And that's kind of all we say.

01:07:57
We don't say, you know, to succeed at this, you're probably gonna have to represent the meaning of words.

01:08:03
Okay?

01:08:04
We don't say that.

01:08:05
And I mean, I don't even know how you would say it.

01:08:07
We just say

01:08:08
We want you to find numbers for each word, and then in a large language model, there's more than that that's getting found, notably the weights and so on.

01:08:17
Those are also numbers.

01:08:18
We get in that, but the point is um we uh

01:08:22
The the machine we now know we have discovered that the the uh those word embeddings the sequences of numbers that the machine comes up with to represent each word

01:08:34
Do amount to a way to actually represent the meaning of words, okay?

01:08:39
And and in a way, this is a slight oversimplification and

01:08:44
I I I put the simplified version in the book and then the the the less simplified version back in a in a thing called a I think it's called a note on terminology or something.

01:08:55
But uh

01:08:56
It's like i if you started out by representing animals on a two-dimensional graph and so tigers, you know, one one one dimension is lethality, one is speed, so tigers are high on both, rattlesnakes kinda high on one, lower on another.

01:09:11
and then snails low on both, whatever, uh those are characteristics of animals.

01:09:17
And and and if you added enough characteristics

01:09:20
uh enough dimensions, you would be kind of yeah, coming up with a way to represent what the name of an animal means to a human being, because those are in fact the connotations.

01:09:30
I think of tiger, I think lethal, fast, it has fur, it has this, it has that.

01:09:34
Well that that is the system that the machine, in effect, comes up with.

01:09:38
We did say you should use these sequences of numbers.

01:09:42
We didn't say they have to map onto the meaning.

01:09:44
We didn't bring up meaning

01:09:46
And yet the machine found it.

01:09:48
That's what I'm saying.

01:09:49
And I think that's true.

01:09:51
And I think it's amazing.

01:09:53
And it's because the machines do this in various ways.

01:10:00
uh that uh I believe uh th that they're going to get much more powerful because the

01:10:10
You know, and you you mentioned evolution.

01:10:12
Okay, I do emphasize much more than some people that I think the way to think about a lot of what

01:10:20
Going on in training is not learning in the conventional sense of a human being learning over the course of a lifespan, but actually evolving like uh natural selection

01:10:30
Because I think uh, for example, uh the learning uh the system of representing meaning, I think is a pretty good bet that that's a product of natural selection.

01:10:39
I just do not buy a pure tabula rasa view of

01:10:42
human uh the human brain.

01:10:44
And in fact, uh psychologists, one theory about how we represent meaning had long been this theory

01:10:50
That in effect it's a it's vectors.

01:10:53
It's it's dimensions of meaning corresponding to attributes uh of semantic attributes of the thing being represented.

01:11:02
Um

01:11:03
And and, you know, when you think of it this way, that like you you just give it, you you let it uh

01:11:11
Th you know, if given enough data that is the product of the human brain, or often like input and output data, like in the case of driving, it's like, well, here's the environment that the the that goes into the eyeballs.

01:11:26
Here's what happens with the steering wheel as the output data.

01:11:29
You know, if indeed uh through through you know you can feed almost any kind of data generated by humans

01:11:38
into the machine and it will, you know, kind of reverse engineer in effect the human brain.

01:11:44
And I don't mean it's gonna do everything the way the human brain does, but

01:11:48
I do think there are other examples besides representing meaning, and I mentioned a couple in the book where, yeah, it is it is reinvented things that human evolution invent.

01:11:58
And uh I don't see any reason to think that that's not what fundamentally is going on, although in some cases is it is more like learning than evolution, like like for example, uh if it becomes conversant in English.

01:12:11
You know, a specific language is something we learn.

01:12:14
So that that that's kind of analogous to uh to human learning.

01:12:19
Uh but yeah, I think I think in a lot of ways it's uh

01:12:24
It's reinventing the human brain.

01:12:26
And that and that's why I think we haven't seen nearly the end of it.

01:12:31
I mean, you know, you probably heard recently like Facebook, Mark, Mark uh Zuck Zuckerberg in his uh

01:12:37
infinite compassion for his fellow human beings, um, informed his workers like like the same day he laid off 8,000 or something that from now on their keystrokes will be recorded.

01:12:49
Well, why is that?

01:12:51
Because

01:12:51
If you if you have all the input data and all the output data of your workers, you can in effect reconstruct

01:12:59
The functional equivalent, it's not exactly like the brain, but the functional equivalent of the part of their brain that's doing the job you're paying them to do.

01:13:06
And then you can fire the rest of them.

01:13:08
Ethics of him firing people aside, it does indicate that the systems can't figure out the meaning themselves, and that's why he needs to farm the meaning from his 8,000

01:13:20
or his staff of forty thousand uh uh workers.

01:13:23
Well what I would say is the meaning is implicit in the structure of the language and and and in the struc patterns in the words

01:13:31
And the machine figures that out through the trial and error.

01:13:34
That's what the neural network does.

01:13:36
I would rephrase that, but this is maybe just two different views of the same thing, is that

01:13:41
human beings have figured out a way to produce um vectors out of words such that the orientation of these vectors um uh has semantic uh meaning.

01:13:54
But

01:13:54
That's because we came up with that system.

01:13:57
And maybe as an example, um you can go back and you can read Claude Shannon's like 1955 paper where he came up with the concept of Shannon information

01:14:07
Um and in there he came up also with a way to represent uh the meaning of words and the next token distribution using a very simple Markov um Markov chain.

01:14:15
And you can see whether when he makes the system a little bit more powerful, that it starts to generate

01:14:20
sentences that strongly resemble human sentences.

01:14:24
Um and then you could say that uh if you give the system the cats at on the and then you have a distribution over the next tokens.

01:14:31
the fact that it puts a lot of probability on the word mat, uh the system has figured out the meaning.

01:14:37
The f system has figured out that there is some

01:14:40
meaning associated with the word mat when it precedes or follows the pat sound.

01:14:45
We have stronger evidence than that that they're representing meaning.

01:14:49
It isn't just that they successfully predict the next token or that they sound smart, right?

01:14:53
Like

01:14:54
You're familiar with the word to Vec paper and so on.

01:14:56
Yeah, yeah, yeah, of course.

01:14:57
I I mean yeah, so let's talk about evolution in uh next, but I would maybe just leave this as a thought for the the listener that

01:15:03
If these things could figure out meaning independent of human beings, if human beings were not really necessary to be in the system, then we would have solved the linear A.

01:15:14
cipher problem because we would just ask say, hey, figure out the meaning, please.

01:15:17
Tell us what that meaning is.

01:15:18
Um I claim that that's not possible because the meaning actually comes from us and we're imposing it into the system.

01:15:25
Um but the linear A or uh you could set up this experiment tomorrow if any listener wants to do it, where you take the Babylonian writing system or subsystem for which we have a Rosetta Stone.

01:15:35
and see if you can train it to repr reproduce that Rosetta Stone without the data in that.

01:15:39
I mean you need a lot of data and I don't know how much there is, but my answer to that

01:15:43
You know, remains.

01:15:44
Yeah, before we get to evolution, it it might just be worth saying, at least for my part, I'm not sure about how Vaden feels exactly, but I don't want to claim that I'm a wonderful person, you agree with everything I said.

01:15:53
Otherwise I wouldn't be giving you

01:15:55
Six dollars a month.

01:15:56
Are you kidding me?

01:15:57
That's uh Yeah, well you know that was before we started recording.

01:16:00
We should tell people that you're actually a paid subscriber to the non-zero newsletter, which is

01:16:04
Which is what's putting that six dollars in my pocket.

01:16:06
Although there are di discounts are available, you know, annual.

01:16:09
Sounds like if it's really six dollars, that means you're not an annual

01:16:12
You didn't you didn't really make the big one year commitment because it would only be five dollars.

01:16:18
But anyway, I hate.

01:16:19
I'm not complaining.

01:16:20
Who's counting?

01:16:20
Who's counting?

01:16:21
I could have taken that New York penthouse.

01:16:23
Now I'm just stuck in Pittsburgh in this old apartment.

01:16:26
Yikes.

01:16:27
Um but for my part, yeah.

01:16:29
I mean I don't want to make the claim that I haven't been surprised by any aspect of these systems or that there's no emergent behavior that I've been impressed by.

01:16:37
um or disconcerted by, for instance, I think some of the deception papers coming out of Anthropic um have been disconcerting.

01:16:44
I remain slightly agnostic as to exactly how serious it is, because there is some

01:16:48
publicity that is natural for them to to want to attract.

01:16:51
Um but the the question for me always remains, are these concerns in the realm of

01:16:58
engineering concerns about thinking about how to engineer systems to work as we want them to work?

01:17:04
Or are these concerns verging into autonomous agent superintelligence territory where they're gonna break free of their training regimes?

01:17:13
and launch into the stratosphere as as these as these machines that that we absolutely can't control.

01:17:19
And yeah, I'm firmly of the belief that all the

01:17:23
um behavior and the problems we've seen so far lie firmly in the first camp.

01:17:29
So that's not to say there won't be substantial problems, so I agree with you.

01:17:33
Uh Claude Mythos was worrying, insofar as the claims they were making there were were correct.

01:17:39
Um and that in the hands of the wrong people is worrying.

01:17:43
But that that's still a far cry.

01:17:45
from these systems sort of waking up one day and try and trying to or being able to sort of subjugate the the human race.

01:17:54
Like at b at bottom there's still these are statistical

01:17:57
prediction methods, right?

01:17:59
And so if if they were evolving towards some sort of super in super intelligence, I would expect to see signs of certain things.

01:18:09
So for instance, I would expect to see

01:18:11
uh less and less reliance on post training and fine tuning.

01:18:16
In fact we see more, because as you said,

01:18:19
We start to saturate what we can get with next token prediction.

01:18:23
So we have to rely on various post-training techniques, which I agree with you have been wildly successful, but we have to rely on them more and more.

01:18:31
Um

01:18:32
Uh whereas if superintelligence was around the corner, um, you'd you'd expect to wanna rely on them less and less, right?

01:18:38
You wouldn't have companies popp popping up that are

01:18:40
um doing sort of bespoke fine-tuning on various tasks uh because you would expect that these systems were just getting generally better at

01:18:49
Absolutely everything, um, and not having to be taught, you know, how to write good poetry sort of on on on the side.

01:18:56
And

01:18:57
Um I mean yeah, maybe this is getting slightly too technical now, but you can run experiments where you, for instance, give it a bunch of data, ask it to predict things, and then sort of extr try and extract a world model.

01:19:08
meaning like what it thinks the actual underlying reality is of the thing it's predicting.

01:19:13
And uh there are many examples of people doing this now.

01:19:17
One particular one that comes to mind is you can do this with uh astronomical data, right?

01:19:21
And so you ask it to predict the uh the rotation of of the planets and the stars, etc.

01:19:26
Uh we have lots of such data.

01:19:28
Um and it can get very accurate answers.

01:19:30
Uh but then you can ask it, what what do you think the laws are that are governing the underlying motion of these things?

01:19:36
And it's it's like absolutely gibberish, right?

01:19:39
Um and so this is all to say that statistical learning is

01:19:43
perhaps part of how the human brain work is working, but is is definitely not the whole story.

01:19:48
And I think where they deny depart the the the doomers is that they think

01:19:55
Well, it's a bit unclear what they think sometimes, because sometimes they'll they'll claim that LLMs might be able to get us the whole way.

01:20:00
Um but then in the next sentence they'll often say maybe LLMs can't do it and we should be concerned about some future technology.

01:20:05
But those are very different claims in my end on uh uh uh in my mind.

01:20:09
Those are um that's a very important distinction.

01:20:11
But insofar as we stay in the realm of statistical learning, and it you know, it it seems that that's the the the wave of the future at the moment, um I uh yeah, I think I want to sort of

01:20:22
nip any superintelligent fears in the bud.

01:20:24
I just want to add another uh uh point to to strengthen um Ben's claims here and and really

01:20:30
Um emphasize the two-on-one dynamic of this conversation.

01:20:36
Yeah, it's extremely important that we don't let that be forgotten that it is two-on-one.

01:20:39
Yeah, no, thank you.

01:20:40
As you know, junior podcasters.

01:20:42
Exactly.

01:20:45
Yeah.

01:20:45
In in the book you um uh give lots of examples from Anthropic's papers about it deceiving the the um uh the user and and these

01:20:57
Examples are so interesting to me because they only exist in academic literature and

01:21:05
each next generation of Claude seems to have been doing less of that.

01:21:09
Like where are the news stories of I was using Claude and then all of a sudden it uh asked me to empty my bank account and send it to

01:21:16
Dario.

01:21:17
Like you don't see these examples.

01:21:18
Um so what does that mean?

01:21:20
That means maybe that the AI safety problem is easier than we think because

01:21:25
All these scary examples that are coming out from 2003, 23, 2024, there's they don't manifest in the actual product that is being delivered to to people.

01:21:34
Um and so it's just I just want to point out how notable it is that all the examples

01:21:39
come from publications by the people who are incentivized to tell the world that they're making superintelligence, but they are not coming from just the common person who uses it.

01:21:50
Of course we can talk about

01:21:51
cases where where uh a person basically has like a mental health episode into the chatbot and then I think you gave one horrifying example of a person who killed themselves based on what the chatbot was uh was telling them.

01:22:03
That's of course appalling.

01:22:05
Um but again I

01:22:06
frame that as um shooting your slingshot in the wrong direction and not realizing that these things are just gonna echo and amplify what you put into it.

01:22:14
Um and unless you get it to be critical of you, it's not going to do that necessarily on its own.

01:22:19
Um and so there's just more examples uh of stuff that I wouldn't have expected to see if we're on the verge of superintelligence.

01:22:26
Okay, there's a lot there between uh the two of them.

01:22:30
Sorry.

01:22:30
Yeah we have a tendency to do that.

01:22:35
I mean maybe I'll kind of work backwards.

01:22:37
On the safety stuff

01:22:39
You know, I do think uh certainly the most dramatically alarming examples come from the lab, you know, where they were trying to come up with dramatically alarming examples, which is

01:22:49
Doesn't mean it's it's bad work.

01:22:51
It's important to understand like what will happen, you know, if you um

01:22:58
i i if you give a machine you know if you tell it that the only way for it to survive uh is to uh in some sense violate protocol and among the options is uh

01:23:10
you know, blackmailing some guy at work and and yeah, it's true they have to kind of keep narrowing the options to get it to do that, you know

01:23:19
You know, no, it's not a good thing.

01:23:22
Right.

01:23:23
Well, maybe.

01:23:23
But although I will say on that front, I mean first of all, I'd say there are examples of uh

01:23:31
Well, actually mythos I I haven't looked into this closely, but the Anthropic was touting this example where supposedly, you know, the one where the guy is sitting there eating a sandwich the myth the Anthropic guy is eating a sandwich in the park and suddenly he gets an email?

01:23:46
From the L L M and he it wasn't like it wasn't supposed to have access to email.

01:23:50
I couldn't tell to what extent they were overdoing that for the sake of drama, but that was

01:23:55
I gather the idea was it had succeeded in escaping some kind of confine.

01:24:01
But yeah, I I I I grant that so far real-world examples are

01:24:07
f are are relatively few and far between.

01:24:10
I mean there are people saying shit it wiped out all the data in this file and so on.

01:24:13
You're seeing this especially

01:24:15
uh with people who aren't as careful as maybe they should be using open claw or something.

01:24:20
But in a way the it takes us back to the first uh if we if we look at the series of questions um

01:24:28
uh encompassed by uh the two of you in the course of that uh that last phase of gang up on um the first was about autonomy and and this kind of gets back to

01:24:39
My point about uh the what the evolutionary environment is favoring.

01:24:46
The thing about autonomy is no, they're not yet autonomous enough to take over the world and kill us all.

01:24:51
At the same time

01:24:53
Autonomy is the holy grail.

01:24:55
Okay?

01:24:56
It's what companies want.

01:24:58
It's the kind of AI they'll pay for.

01:25:01
They want

01:25:03
the most valuable worker possible.

01:25:05
They want they want the AI that's like a good worker.

01:25:08
You just give it the goal, you turn it loose

01:25:12
it it performs, you know, thousands of steps, it runs into obstacles, it gets around them, and it comes back and says

01:25:21
Uh mission accomplished.

01:25:23
Okay?

01:25:24
And and and in fact, you're I'm sure you're both familiar with the meter graph.

01:25:28
You may have some criticisms of it, but you gotta admit

01:25:32
They're getting more autonomous.

01:25:33
The Mediagraph, M E T R is the organization, and of course they they they try to measure how long would it take a human to do

01:25:41
a task that like half of the a you know, an an agent can now perform with 50% success or 80% success, depending on how they do the study

01:25:50
And what they find is the you know human labor length that the latest and best LLM can do

01:25:59
Has been doubling.

01:26:00
Originally said seven months.

01:26:01
I think then it was six months.

01:26:03
I think now it's lower than that.

01:26:04
But uh in any event, you've seen the graph, and what that reflects is some degree of growing autonomy

01:26:10
So the research is leading to that.

01:26:13
And then I would just say it's what the market is encouraging.

01:26:17
More and more autonomy.

01:26:20
And uh that's why I can imagine someday something, you know, uh very surprising happening.

01:26:26
Uh uh with an agent that has access to things that in retrospect you wish it didn't have access to.

01:26:32
Yeah, okay.

01:26:33
So I'm I'm aware that we are verging on time and I don't wanna I don't wanna disrespect that.

01:26:37
Um I actually do I do uh speaking of meter, I do have complaints about their uh about their methodology, uh including

01:26:45
Small sample sizes.

01:26:46
Um, but also that the they've they've admitted themselves as you read their methodology that the cla the tasks that they focus on are significantly more cleaned up.

01:26:54
And well-defined than like sort of messy real-world tasks that you'd come across as like a real software engineer.

01:26:59
Um so I think there are issues there.

01:27:01
But let's just at the end here pivot to

01:27:06
the analogy between pre-training, LLM pre-training, and evolution.

01:27:10
I would say, oh, oh, I wanted to actually get it.

01:27:13
Can I uh well I'll I'll I'll remember it, but

01:27:16
I w I would I would analogize post-training to it as well.

01:27:19
I mean it's the same Okay.

01:27:21
Uh and you know I I think uh maybe Vaden you were suggesting that

01:27:29
Some of the post-training is this kind of massaging that indicates that pre-training wasn't enough.

01:27:36
Well, I agree, pre-training wasn't enough.

01:27:38
I mean there's more to there's more to intelligence than than being able to construct a sentence.

01:27:43
There's like responding to questions.

01:27:47
There's knowledge.

01:27:48
There's being able to do logic.

01:27:50
There's a lot of things.

01:27:52
And I think a certain amount of post-training uh is is an attempt to cultivate those things.

01:28:00
But I would say in that case as well

01:28:03
What I think is happening in some cases at least, is that you know, through the trial and error, as the machine gets better and better at doing these things

01:28:13
Whether it's uh answering questions about uh physics or s answering questions generally or doing various other things that they get it to do in post-training, or just be nice and polite.

01:28:25
Um that

01:28:27
In effect, you know, we have to remember we don't really know exactly what's going on inside the neural networks.

01:28:32
The weights are shifting.

01:28:34
We know that.

01:28:36
But we don't know to what ex you know, which kind of structures are in a sense being created.

01:28:43
And and I would posit that

01:28:45
In post-training, you're seeing what I think you see in pre-training, which is that uh the machine has to build structures of information processing

01:28:55
that are at least functionally analogous to some structures in in the human brain, like some of our reasoning equipment, for example.

01:29:03
Um so uh I I um

01:29:07
I don't know if that that may not answer what you're about to continue asking me about, but uh I I would em and I wish I'd emphasize that more in the book, actually.

01:29:18
I mean I I you know I talk about

01:29:20
I mean this field is changing so fast.

01:29:22
You know, I talk about post-training, how important it's become, but I don't really drive home that I think that like there, as in pre-training, I think

01:29:32
uh you know, you the machine is building information processing structures

01:29:40
uh that are functionally comparable either to some that are kind of in our genes, so to speak, like some of our basic linguistic uh equipment, or you know, things that are more in our in our learning.

01:29:53
It's funny, the uh the original reinforcement learning people, um, what's the name of the famous guy, the bitter lesson guy?

01:30:02
Rich Sutton.

01:30:03
Did you listen this is a tangent.

01:30:05
Did you listen to his Dwarcash podcast?

01:30:07
I did, yeah.

01:30:07
I thought he was a curmudgeonly old man making all the right points.

01:30:16
He's like B.

01:30:17
F.

01:30:18
Skinner, completely unreconstructed by reading my book, The Moral Animal, which of course

01:30:25
I would have encouraged B.

01:30:26
F.

01:30:26
Skinner to do if he were still alive.

01:30:27
The point is, you know, Moral Animal, I I don't mean, you know, my book, these aren't my ideas, but it's a book about evolutionary psychology

01:30:34
And there's another place I agree with Stephen Pinker is that uh I think there's a certain amount of stuff built in by evolution.

01:30:42
And what I realized

01:30:43
is that Sutton is just like, he's a blank slate guy.

01:30:47
You know, he he thinks that uh you should just

01:30:52
And this gets back to AGI.

01:30:53
You talked about artificial general intelligence.

01:30:55
I don't really I'm not on board.

01:30:57
You know, I mean, I I I think a number of these people

01:31:02
have believed more than I do in something called general intelligence.

01:31:07
I think the human mind evolved

01:31:10
you know, uh in a pretty kludgy way.

01:31:13
Like, okay, this thing evolved to solve this this evolutionary problem.

01:31:17
This thing evolved for that.

01:31:18
Then this emotion got added and and and it, you know, and and so on.

01:31:23
And

01:31:24
So I'm not I'm not really you know AGI if you want to define it pragmatically like OpenAI at one point said well AGI is when X percent of desk jobs can be done by okay that's a good practical thing

01:31:37
Fine.

01:31:38
But I still wouldn't think that well, general intelligence is in one sense the right term.

01:31:44
But uh I guess I'm also saying why explaining why I'm not surprised

01:31:50
That training has so many facets.

01:31:53
Okay?

01:31:54
You teach it to kind of a basic facility with constructing sentences

01:31:58
But you still have to add this and you still have to add that and even got to gotta deal with the visual stuff separately if you want it to be able to label, you know, objects

01:32:07
Because I think that's more what the human mind is like.

01:32:10
A lot of stuff mushed up together.

01:32:12
But now finally, Ben, I'm gonna let you actually ask the question that I think I interrupted or did I?

01:32:20
No, it's impossible for you to interrupt me.

01:32:23
We've been interrupting you the whole two hours.

01:32:25
So yeah, no, I'm not saying I'm e I'm not saying I've gotten even.

01:32:28
I didn't say that.

01:32:30
I'm I'm making progress.

01:32:31
It's uh it's an interesting point though.

01:32:33
So yeah, I guess your your claim would be if we were to get to something like general intelligence, whatever that may mean from these systems, um, at some point there'll just be sort of a jump to universality when we've added the right next module.

01:32:45
Um and uh

01:32:46
That's the thinking.

01:32:47
Yeah.

01:32:48
Right.

01:32:48
That's the thinking.

01:32:49
Yeah, I mean uh uh yeah, I don't want to just repeat what we said so far, but I would just say, yeah, insofar as all these modules are still based on statistical learning, then then I think we have principled reasons to to to reject that view, but um

01:33:00
But I won't belabor the point.

01:33:02
I mean you could probably anticipate at this point to be honest our objections to the analogy between uh human evolution

01:33:10
um or b uh evolution by natural selection in in animals versus training.

01:33:16
Um and the distinction just comes down

01:33:18
to the difference between like mathematical optimization and just more general optimization in life.

01:33:24
I think there's this there's this tendency to just conflate the two because they sort of it we view them both as these optimization processes.

01:33:31
But mathematical op optimization

01:33:33
um which is what m uh training, uh both both pre and post-training in uh in LLMs is is based on.

01:33:40
Um and it's it's it's a very structured environment.

01:33:43
Right.

01:33:43
There's only so many things that can happen.

01:33:45
We're parceling the observation space in very s in very specific, strict, structured ways.

01:33:52
Um, as Vaden was saying earlier, not everything can happen.

01:33:55
The algorithms are very specific.

01:33:57
Exactly how we update the weights is very specific.

01:33:59
That's not to say we can

01:34:01
uh predict the end of the process or the exactly all the emergent behaviors, but it is a very constrained environment.

01:34:08
Um whereas evolution doesn't have any of that, right?

01:34:11
It's truly

01:34:11
Uh it's truly blind.

01:34:13
Um it's very unstructured.

01:34:15
Uh there's no top-down designer deciding what exactly counts as a token, what the token space is, what the architecture

01:34:24
of the system, whatever that would even mean in natural evolution, um, how many layers the thing has, right, et cetera, et cetera.

01:34:31
Um, and so

01:34:33
Uh this is all just to say that I think the analogy starts to break down um between between the two of them once you take the specifics of mathematical optimization into account.

01:34:42
Yeah, but I would say can I can I just quickly say I would say

01:34:46
There are constraints in evolution, like namely the whole history of the construction of the thing that is now being revised.

01:34:54
And in a way

01:34:56
So like it's like, you know, when when you get to a point where uh say, you know, language, you know, uh

01:35:06
uh y you're gonna start like the at the beginning of the evolution of linguistic equipment, you know, it's like uh I guess uh initially it might have been like

01:35:15
you know, warning calls issued to kin who share some of your genes or something, who knows?

01:35:20
But whenever it starts, it's like

01:35:24
You have to build it on top of whatever's there.

01:35:32
Nobody steps back and says, oh wait, if I had known language was gonna come, here's the way I would have designed the human brain up till now.

01:35:38
No, sorry, you gotta work with what you gotta work with.

01:35:41
And the and and I think that happens again and again.

01:35:44
And that's it's it's a sense in which evolution doesn't

01:35:47
can't optimize.

01:35:52
With a lot of inherited equipment and then kind of make the best of that, given how likely various kind of kinds and magnitudes

01:36:02
of mutation, of of of innovation are to happen.

01:36:05
And there's kind of an analogy with training, but kind of a different uh of an LN, but but kind of different, which is to say

01:36:13
So they start out with next token prediction.

01:36:16
And then they say, let's build in reasoning.

01:36:19
And at that point, when they're building in reasoning, look, you're stuck with an LLM that is good at max, you know, predicting tokens.

01:36:25
Okay.

01:36:25
That's what we're working with.

01:36:27
Maybe that he maybe you would have designed reasoning differently if we had started in a different way, but this is the order in which we're doing things.

01:36:33
And the interesting thing to me is

01:36:35
It's not the same order in which hum the human if you accept my premise that in some ways pre and post-training mirrors the evolution of the human mind, it doesn't happen in the same order.

01:36:47
And so

01:36:49
That may lead to all kinds of weirdness, who knows?

01:36:51
But uh anyway, that's just a point.

01:36:53
Yeah, so I mean um I wanna hammer the the what I view to be the disanalogy between evolution and the training of LLMs.

01:37:03
fairly hard because I think it's it's um it's a common thing that you hear out in in the wild and and I think it it breaks for a number of reasons.

01:37:10
Um

01:37:11
So evolution is blind variation and selective retention, right?

01:37:16
Um I know that when you talk about evolution, you're more referring to like the stages

01:37:22
uh uh that stuff came online in the grand history of us from little amoebas all the way to now.

01:37:29
So first we got language, then we got reasoning and and stuff.

01:37:32
But evolution at its core is is about the mechanism of blind variation and selective retention.

01:37:37
Um so there's a professor in machine learning called Ryan Adams, who uh is at Harvard and he used to have a podcast, which I used to listen to a long time ago.

01:37:45
And he was asked this question because uh before there was gradient descent, there were a whole class of algorithms called evolutionary algorithms.

01:37:55
These evolutionary algorithms were specifically set up to copy this mechanism.

01:38:00
And so how an evolutionary algorithm

01:38:03
would work if we were to train an LM is we would duplicate the copies of the weights multiple times and we would randomly vary the architecture, so the number of layers, the number of neurons, um, and then we would

01:38:16
make a prediction and then we would uh s choose some environmental mechanism to let certain weight in architectures die off

01:38:26
and then spawn a new um variation of of the the the ones that succeeded.

01:38:31
Um but these mechanisms were all tried and none of them worked and what worked was gradient descent.

01:38:36
And gradient descent is definitionally not

01:38:39
Blind.

01:38:40
That's what the gradient is telling you.

01:38:41
The gradient is saying go this direction.

01:38:44
Um and so there's no blindness, there's no variation because you're keeping the architecture the same.

01:38:49
You're you're adjusting the weights

01:38:51
in a direction that is known and is known because there's a designer.

01:38:56
There is there is God.

01:38:57
We are the gods.

01:38:58
And we're the ones saying this is the right answer.

01:39:00
Well I mean I wouldn't say that.

01:39:02
I mean so

01:39:04
You know, I talk a little, you know, I mean Hinton uh uh is appears repeat Jeff Jeff Hinton re appears repeatedly in my book, partly because I happened to interview him in nineteen eighty, so it's a good narrative device, but also he's done a lot of

01:39:16
Important things.

01:39:17
He was a co-author of the big uh backpropagation paper, although even he doesn't I don't think he he even he says the idea originated then and there.

01:39:26
It had been kind of evolving.

01:39:27
But the point is

01:39:28
Uh i i it it helped make uh the mutations uh not blind, right?

01:39:35
It's a way of

01:39:37
of uh and and so I agree uh and so that makes the evolution I'd make the comparison in the book with Lamarckian evolution which uh uh and

01:39:47
So yeah, so in that sense it works, you might say, faster or more efficiently than than natural selection to the extent of the mutations um

01:39:57
are not random.

01:39:58
Of course there is a whole revisionist view But there's not even mutations though.

01:40:03
Look what is the analogy to the mutation?

01:40:05
No, there are mutations.

01:40:06
The weights uh change after every trial

01:40:08
Though those are the mutations, the mutations in the weights, in the strengths of the connections between the neurons.

01:40:15
Change yes.

01:40:17
Mutate, I think of random.

01:40:21
Well, yeah, right.

01:40:22
No, and it's not random.

01:40:23
It was random before backpropagation.

01:40:25
Now it's not.

01:40:26
But but my point is if evolution were Lamarkian, instead of purely random, it would still work.

01:40:32
It would work faster.

01:40:33
It would work better.

01:40:34
It would still be evolution.

01:40:35
But it but you'd have to fix the objective beforehand, right?

01:40:38
So there's this big difference where like we we sort of we know what we're trying to get with the LLM.

01:40:43
So we're actually not letting them

01:40:45
explore the entire evolutionary landscape, um, even within this fixed training architecture, which is already very limiting.

01:40:52
So we're like, we're not letting it evolve

01:40:54
um physically at all, but we're also fixing like what the objective is, right?

01:40:59
We're like, okay, you're gonna go out, you're gonna reduce binary cross-entropy as much as much as possible.

01:41:03
Here's what you're gonna do.

01:41:04
And if you have like a parabola, like a like think high school calculus, and you have a point on that parabola, and then you want that point to get to the basin

01:41:13
Um it's gonna change, but it seems like it's strange terminology to say that it mutates because you're guiding it.

01:41:20
I don't know why.

01:41:20
I mean the analogy seems pretty side in fact it is literally this

01:41:24
Strength of connections among neurons, which is another is is something that in our brain was subject to natural selection.

01:41:31
So I I and it changes.

01:41:33
The strength changes

01:41:35
And I don't know why you would not call that a mutation.

01:41:39
And they are selectively and they are selectively preserved depending on whether the mutation led to

01:41:46
Success as defined.

01:41:47
Now in evolution, it's defined as genetic proliferation.

01:41:51
In a large language model, it's defined as

01:41:53
uh predicting the next uh token in the first in pre-training uh and then it's defined as other things later.

01:41:59
Yeah not to get too second terminology and and again we should wind down because we're we're pushing pushing um

01:42:04
uh your time and so I just wanna uh not argue about terminology but just maybe indicate to the listeners the difference in the way that we're thinking about it, which is

01:42:14
Um when I think about LMs, I think of them as giant matrixes.

01:42:19
Um where a parabola is a good metaphor because a parabola would just be a matrix of a single.

01:42:25
um entry.

01:42:26
And then when you adjust the weights of your matrixes, you're just trying to get lower on these these basins.

01:42:32
Another metaphor or another way to think about it is as neuronal connections.

01:42:36
Right.

01:42:37
Um and and and that's I think how you're thinking about it.

01:42:39
So when you think in terms of optimizing, you think in terms of the neurons of the LLM.

01:42:44
forming new connections and and and all and all this.

01:42:47
Um and I would just leave it to the to the listener to recognize the intuitions that we are both coming here with and that's why

01:42:54
Perhaps you got stuck in it.

01:42:55
Yeah, and the neurons don't physically exist, of course.

01:42:57
They're they're they're represented in the in the mathematical abstractions you're talking about.

01:43:02
And I don't have the background to even think about it the way you do.

01:43:05
The good news for you is you can think about it either way.

01:43:08
Uh I can only think about it one way.

01:43:10
Yeah.

01:43:12
Awesome.

01:43:13
Well we spent uh the majority of the time uh trying to disagree with you.

01:43:17
But we should say yeah, we I think we both

01:43:23
This was Vaden on his best behavior, so you should be you shouldn't really be happy.

01:43:29
I guess we should set up an actual cage match that doesn't involve a physical cage or physical combat.

01:43:39
A ver virtual cage match, yeah.

01:43:41
Yeah, we specialize in those

01:43:42
No, I appreciate you guys uh reading the book and uh and carrying your skepticism into it and I I

01:43:50
Encourage everyone to everyone who's made it this far to pre-order it.

01:43:54
Yes.

01:43:54
Any plugs?

01:43:55
Plug plugs.

01:43:57
Oh, it's called it's called the the God test.

01:43:59
Uh the subtitle is Artificial Intelligence and Our Coming.

01:44:04
Cosmic reckoning.

01:44:06
I I mean I guess it's you know you know it's good for.

01:44:09
It sounds like maybe it's not that great for well no, I I like to think there are things

01:44:14
It was very thought broken.

01:44:15
Your audience would be interested in.

01:44:17
But also I think it's for like like their relatives, you know, who like like what is this stuff?

01:44:24
You know, I I try to make

01:44:26
it very accessible to a lay person since that's what I am and also that's what I've done for a certain amount of my writing career.

01:44:34
Uh and um

01:44:36
So I I do like to think it's a i i if somebody, you know, for people who are just now going, what is this?

01:44:44
Like it's sounding people are talking about this AI thing more and more.

01:44:48
What is it all about?

01:44:49
I I think whether you buy my arguments or not, it's a not bad way to be introduced to like

01:44:58
Basically the way it works, why some people like me think it's gonna get more powerful, and what the argument for being worried is and why some of us think

01:45:10
uh we we we really need to confront it as a global community and quit saying, but what about China?

01:45:16
What about evil China?

01:45:18
Which is, among other things, just a way for AI companies

01:45:22
to uh discourage regulation.

01:45:24
Although in the case of Dario, I think it's actually ideologically.

01:45:28
It's ideological.

01:45:29
He he's uh he's a he's a true China.

01:45:31
And then where can people sign up to your newsletter and listen to your podcast?

01:45:38
org.

01:45:39
You can Google nonzero and substack.

01:45:41
And the podcast is also called Nonzero.

01:45:43
It was a non-zero YouTube channel and a non-zero podcast feed.

01:45:46
And paid subscribers like Ben.

01:45:48
I mean, Vaden, you've probably been wondering what you're missing out on by not being a paid subscriber in non-zero.

01:45:56
Every episode I'm like I'm gonna say the phrase blue balled when the paywall comes down because that's when all the good stuff

01:46:02
Oh, so you actually you know about the paywall thing.

01:46:07
I mean who who could forget the Emily Bender episode?

01:46:11
That is true.

01:46:14
That was something.

01:46:15
Holy smokes.

01:46:16
Oh no, we were we were very much on your side in that conversation.

01:46:19
God bless you.

01:46:20
Good.

01:46:20
Now I know what your limit is.

01:46:27
The Buddhism really came out in that conversation.

01:46:30
I was amazingly impressed with you.

01:46:31
I would have lost my mind, to be honest.

01:46:33
Yeah, that was incre that was very impressive.

01:46:37
Uh uh that I was just kind of to get as perturbed as a person might, but I I don't know.

01:46:43
I I have no explanation for that because it's completely out of keeping with my character to not get perturbed

01:46:49
Well thank you so much Bob.

01:46:50
This is an absolute thing.

01:46:59
I hope we can be one of those old grievances that gives you pleasure to revisit.

01:47:03
Someday I will get pleasure out of revisiting this.