Note: The following is a rough transcript that has not been hand-revised by High Signal or the guest. Please check with us before using any quotations from this transcript.
===

steve: [00:00:00] I think that becomes a game-changing input into this partnership between the human and the machine. And if that's true, that means that we're complements, we're not substitutes. People who are lacking in those skills When they use these machines, what they're gonna produce is gonna pale compared to what people with deeper skills, more knowledge, critical thinking ability are gonna produce.

And at the end, I think it's going to be the exact opposite of an equalizer 

hugo: That was Steve Tadelis, professor of economics at UC Berkeley, on why AI may act as a skills-based unequalizer rather than a leveler. Steve has spent his career at the intersection of tech operations and economic theory, having served as a senior economist at both eBay and Amazon.

In this episode, we discuss the often invisible friction of bringing economic rigor to data science. Steve shares stories from the [00:01:00] early days of eBay, where misaligned incentives and organizational inertia left millions on the table, and contrasts that with Amazon's relentlessly scientific culture. A through line across both: trust is the actual currency for data scientists who wanna move from cost centers to value drivers, and a big part of earning it is making sure your stakeholders don't look like fools for the things they missed before you arrived.

We dive into why surface-level metrics like conversion rates can mask underlying rot, and why performance marketing often fails to measure true incrementality. We also explore Steve's perspective on the AI era. Steve argues that because LLMs complement critical thinking, they'll widen the productivity gap between elite talent and those producing generic slop.

Finally, we look at the barbell future of firms, where AI favors both hyper-lean startups and massive corporations over the middle market. We close on what Steve sees as one of the most important shifts of the AI era. [00:02:00] Technology is rapidly collapsing the time it takes to figure out how to do things, but it has barely touched the question of what's worth doing in the first place.

The human moat is curiosity and the critical thinking required to set the right goals. As Steve puts it in the closing minutes, curiosity is the thing children start with, and the question of how we keep it alive into adulthood is, in his telling, the real game. If you enjoy these conversations, please leave us a review.

Give us five stars, subscribe to the newsletter, and share it with your friends. Links are in the show notes. I'm Hugo Bowne-Anderson, and welcome to High Signal, brought to you by Delfina, the AI agent that supercharges your analytics. Hey there, Steve, and welcome to the show. 

steve: Hi, Hugo. Hi, Duncan. 

hugo: So great to have you here.

I'm so excited to talk a lot about what, what you've done and thinking through how to bring more economics seriously into data science, machine learning, AI, among many other things. And you've been doing this for n- for [00:03:00] some time, and I'm really interested in hearing what it was like like to do economics seriously 15 years ago at a company like eBay.

steve: It was on one hand extremely exciting and sometimes extremely frustrating because you imagine, and maybe this is one of the failings of coming from academia, you imagine that you know your stuff and you're gonna explain it to people and they're gonna see the light and everything's gonna just click.

And that's not how it always works. For me, there were projects where there was clear alignment between there being a champion for that project, for there being a real interest in the economic insights and then turning that into either an enhancement or development of a product or of something that the company was doing.

And then there were other [00:04:00] times where you walk around the corridors, you learn about what the company's doing and you suddenly see $100 bills lying on the floor and, and then you learn the hard way something that I think this quote goes to Upton Sinclair many years ago, when someone is paid not to understand something, they will not understand it, 

hugo: right?

steve: And, um, you know, identifying when that happens, how to approach it, how to get people on board oftentimes was more challenging than I would've thought. 

hugo: Something in there that I love that Upton Sin- Sinclair quote, and as an economist, I'm sure you, you'll appreciate this. Something embedded in there is incentives, right?

So I'm wondering what the incentives were at, at tech company like eBay and what incentives are like in tech companies. What's the variance with respect to being able to pick up those $100 bills? 

steve: That's a great question, Hugo, and my knee-jerk reaction answer is that all over the place. Mm-hmm. Incentives were just all [00:05:00] over the place.

And I think that it came from a combination of what people were evaluated on, obviously. Second, their own personal ambitions and where they wanna go. And, and third, the broader leadership, right? Let me give you an example of a short conversation I had with the then CEO of eBay after eBay launched what they called streamlined or something purchasing.

So this is taking you guys back to a period that I think you may remember. Anyone under the age of 30 is gonna have trouble remembering this. Amazon came up with a brilliant shopping experience that they called the one-click purchase. They actually had a patent on that, believe it or not, right? And eBay bought PayPal, I think it was around 2002, '3.[00:06:00] 

And I think it was 2013 or so that they launched this streamlined shopping experience that replaced A with B. So let me start with A. So when you bought something on eBay, anytime in the late 2000s and definitely up to 2012, the only method of payment was pretty much PayPal. So you would say buy it now.

Okay. Then there would be a question, do you wanna pay now? Yes. Do you wanna pay with PayPal? Yes. That's the only game in town. Do you wanna log into PayPal? Yes. And then you went to the PayPal login. You would log in, ID, password, then you would be in PayPal. It would ask you to confirm that you wanna make the purchase.

You'd have a click, and then after that, you'd go back to the eBay site for final confirmation. So Amazon had the one-click shopping experience, and eBay had the 13-and-a-half click shopping experience. Now, like I said, the companies were together for 10 [00:07:00] years before they streamlined it to you, you created this...

You basically accepted this new version, and then it would be something like three clicks, and you were done, which is a huge improvement, right? You don't have to remember your Pay- PayPal ID and password and all that. And I remember going to the CEO who announced this in great fanfare, and I was like, "John, I'm a little surprised.

Why did it take this long to adopt something that the pain point was pretty, pretty clear? And what's the kind of barrier? Where are the frictions?" And his answer was, "Well- Steve, you live in a world in academia, and in academia, if your dean comes to you and says, "Hey, Steve, I want you to do this other thing instead of the research you're working on," your answer would be, "Fuck you."

Excuse my French. Those were his words. And in the military, we know it's very different. It's [00:08:00] all about coordination. So someone says, "Everybody, you're gonna do this now." So in the military, it's all about coordination. In academia, it's all about innovation, and we're a lot closer to academia than to military, so things just need to happen more organically, and people have to be ready.

That didn't convince me. I get the idea that people need more flexibility, more freedom, but when you see such a blatant pain point, it's kinda like, "Hey, guys, put everything down. Spend three months, fix this shit, let's move on." The culture at eBay was very much kumbaya, let things happen, and therefore, there weren't incentives to try to fix big problems, try to identify them and get those rewards because it was very kumbaya.

You need to get a lot of buy-in. You need everybody to agree. There weren't clear single thread owners, and that was an incentive problem. Why it was there? It's a mystery to me. [00:09:00] So that's just one example where when the incentives are not clearly laid out, issues of short-termism, people are not gonna give up short-term profits for long-term learning is another example.

They're just ubiquitous, and as an economist, I don't have the full visibility into why it's so hard to make cultural changes that set incentives in the right direction. 

hugo: I just... That's such a beautiful example, the one-click pay, and there are three, maybe three plus one really important parts of it, and one you mentioned is it was patented, right?

Which is very important. Another is that it took a handful of lines of code, right? To implement, so it wasn't technically that groundbreaking. What-- The third thing is the really groundbreaking part was the mental model shift, which I think is part of the challenge here, right? The part of the genius of Jeff Bezos, people were still thinking about you're pushing shopping trolleys through grocery stores, right?

And he stepped back and said, "No, it's [00:10:00] actually a totally different paradigm." So it is a paradigm shift in some ways. And the plus one is it's one that they're now trying to bring into IRL shops, like some Amazon shops, you can one-click pay when you're there at the item. Right. Is how we can use these types of ideas behind paradigm shifts and cultural change to think how we can adapt to AI now as well.

duncan: And how much of that is-- comes back to kind of measurement challenges as well? And if you actually are able to instrument how many people are converting and you're relentlessly focused on conversion, then naturally you're gonna do that kind of stuff, right? 

steve: Right. 

duncan: And probably twenty years ago, eBay, that wasn't the case.

Curious, do you think we're getting better at that kind of stuff now? Do you think the world's evolved? 

steve: So it's definitely the case, Duncan, that we're getting better at measurement. Storing data is a lot cheaper, right? So there's more things that we could measure. I think it still goes down to whether we're measuring the right things.

And let me use another example from a project [00:11:00] that I worked on at eBay that ended up turning into a, a kind of fun research paper that was all about how to play with the search ranking. So eBay had a search science team, really smart people. Pretty much everyone, PhD in computer science, stats, physics.

Those are the folks I really enjoyed spending a lot of time with. And they built a search engine that was pretty much trained on conversion. And one of the things that is true at eBay is that you have variation in the engagement with the seller because eBay is a marketplace with third-party sellers, and some sellers are better at describing their item and sending an item that's gonna make you happy, packaging it well, shipping it well, et cetera.

So we know there's variance in the quality of the transaction that people experience at eBay. Now, when you look at eBay circa [00:12:00] 2011, I don't think it's changed that much. You basically saw that pretty much every seller had a 99%-plus rating, right? And as an economist, reputation systems, that's what people use to screen.

But when it turns out that the 10th percentile, uh, seller is, I think it was a 99.1, that was a 10th percentile seller, then you're living in one of two worlds. Either eBay is Shangri-La and everyone is awesome, or you have a measurement problem and these reputations are not really reflecting what's going on.

So one question that we asked ourselves, and this is something I did with Chris Mosco, who at the time was a postdoc at eBay, and now he's the central chief economist at Amazon. We're like, "How could we measure something that will tell us how happy people are at eBay?" And then we came up with this very simple idea, which was the following.

We [00:13:00] took a cohort of people who joined eBay In, I think it was 2009 or '10, there were about 10 million people who joined that year. So we took 10% of that so we could do some data analysis quickly. And what we did is we tracked them over three years, and we basically binned them as follows. What percentage of these people bought once?

Because the starting point was you joined eBay, say, in 2009, you bought an item. Okay, now we're gonna track you for three years. How many people only bought once? How many people bought twice? How many people bought three times? That's it And we created a histogram, and you look at that data, and it's in a publicly available paper, about, if I remember the numbers, about forty-four, forty-five percent of people bought once and never bought another item over a three-year period.

Roughly another fourteen percent bought twice, which [00:14:00] kind of means that almost two-thirds of the people who joined you bought one or two items and never bought from you again. Now, at the time, I didn't have data from Amazon, but I knew that numbers cannot look like that. And our hypothesis was that people are buying an item on eBay, and one of two things are happening most likely.

Either you have people with the following mindset: "I don't like the idea of buying stuff from other people, but my son just turned sixteen. He loves the guitar. I wanna buy a vintage 1980s Gibson Les Paul electric guitar. I'm not gonna find that on Amazon, probably. I'll go to eBay, but I don't plan on coming back."

So that's the selection story. But then you also have the treatment story: "I came to eBay curious. I bought something, didn't have a great experience. I'm not coming back." And how do we try to measure those things? But the first thing was to get people interested in letting us have some runway with that.

And I took [00:15:00] that very simple histogram, and I presented it to the team that reported it to the chief technology officer, uh, at the time, the CTO, Mark Harjes, and everybody's jaw just dropped more along the lines of, "Why did we never look at this?" Now, it's one of those things that hindsight is twenty/twenty, but when everybody is focused on conversion, right, then you're not asking different questions that go to the foundation of customer satisfaction, right?

We have another published paper that we did based on the following idea. Like I said, less than one percent of transactions get negative feedback, but it turns out that almost ten percent of transactions had messages sent from the buyer to the seller after the transaction using the eBay messaging platform.

Now, one story again is, you know what? A lot of people just get their shit. They're so happy they have to write an [00:16:00] email to the seller saying, "Thank you so much. This has been awesome. You made my day." Or maybe not. So we used a little bit of baby natural language processing, and we realized that close to half of those messages were, "Hey, where's my item?"

"Hey, this is broken." "Hey, this isn't what you described." And there was a lot more information there that we could rank sellers and then going back to the motivation using the search, demote them, promote them, et cetera. So it was just asking different questions about what should we measure rather than what is everybody measuring anyway.

duncan: Something really interesting there, Steve, around Thinking more deeply about the value drivers of the business and kind of thinking outside the box in creative ways. I think I'm on a bit of a personal mission in life to help data scientists and data analysts feel more like value drivers and actually be more value drivers in businesses, because today they're, they're far too often cost [00:17:00] centers.

Curious to get your take on why that is and how we help this entire discipline of data become seen as value drivers. 

steve: That's a great question, and I think a lot of that has to do with ownership and who has a seat at the table. And it ties back to the earlier exchange that Hugo raised around incentives If you are able to think outside the box and notice something that other people haven't noticed, then there are at least two huge social barriers to turning that into value.

One is, I believe sociologists calls it-- call it the not invented here syndrome, right? It's like someone owns it, you're an outsider, you tell them, "Yeah, yeah, you don't understand the business. You know, go back to [00:18:00] playing with your algorithms and stuff." Uh, the second is, if you suddenly found some hundred dollar bills on the floor that the business owner didn't notice were there, who looks like an idiot?

And then people have to basically justify why that thing that looks like a hundred dollar bill is not a hundred dollar bill. And it's really an illusion if you really understood how things work, right? I'll give you a little example of how I just noticed that, and I really think in many ways it was more of a kind of natural human reaction.

One of the things that we did at eBay was with Chris and Tom Blake, who is also at Amazon. He's a director I think in some of the search science area there, was our famous or infamous paid search experiment that had a lot of media exposure and was on Freakonomics Radio and it's a paper in Econometrica, which is a leading econ nerdy journal where basically we showed that [00:19:00] eBay was blowing away many, many millions of dollars on paid search.

And one of the first things we looked at was it was just inspired by the fact that eBay was spending twenty million dollars a year on keyword eBay. That's when someone types eBay into the Google search bar and there's an ad, right? And the consulting company they hired said, "This is your highest ROI because people click on this like crazy."

Again, conversions, right? And then we're going to these people and we're saying, "Folks, I typed eBay. Where the fuck do you think I'm trying to go?" You're just putting an ad above the organic link, right? And they're like, "No, you don't understand. This is branding, this is that." And then, you know, there was this accidental shift that they did on the MSN network.

The results were exactly what we said they would be. Then they let us run an experiment. We showed, yes, this is all wasted money. Obvious. A former undergrad student of mine when I [00:20:00] still taught at Stanford was a director or senior director at PayPal She heard about that work before it became public.

She's like, "Oh, you should come over and talk to us about this because we're also spending money on paid search." And unlike eBay, where they spent a lot of money on paid search because you could buy anything on eBay, a big line item in PayPal's expenditures was keyword PayPal. So I get in front of the CMO and the head of digital marketing and a few other people.

I present our work, and then the head of digital marketing looks at me and says, "Look, this is really interesting and very impressive, but PayPal is in the finance sector, and eBay is in the consumer product sector, and these are different." So I looked at him with a lot of patience and said, "Look, here is my theory of the world.

When someone goes and types a company name in their search engine, [00:21:00] especially if it's a rather well-defined company name, unlike, say, Mercury, maybe I'm looking for mercury the substance as opposed to Mercury Insurance. But when I type PayPal, my theory is I want to go to PayPal. And if someone else might even advertise in between, Square, I'll probably just ignore that and go for the organic link because I'm using it navigationally.

That's my theory. The eBay experiment confirms that theory. I'm curious, what's your alternative theory?" And I kid you not, he just stared at me for the better part of 10 seconds and said, "It's finance. It's just different." So, you know It's hard for me to think of a recipe to empower economists, data scientists, you know, other more science-driven, think-outside-the-box folks on how to have more of an impact on the bottom line because so much of it is understanding the [00:22:00] psychology and the incentives and the organizational structure of the people you're trying to influence.

And let's face it, us nerds, that's not our strong suit. So as much as I wish I could say there's a formula, the way I saw this work at Amazon, it's because at Amazon it came from the leadership. The leadership basically said, "You need to listen to these people. You need to embed them in your org. You need to have them be participants in writing the documents that lead to decision-making."

And that was a cultural thing. 

hugo: Amazing. And I love that response, it's finance is different. Actually reminds me of a tweet I saw some time ago, which was, and I don't-- clearly don't quite believe this, but, "Wall Street is just astrology for men," is what the tweet was. I also love that you mentioned sponsored results for Google.

I did this the other day, and I just did it again to make sure. I just searched Gemini, and the first sponsored result is for [00:23:00] Google Gemini. There are no other sponsored results, and then the first actual result is Gemini. And I've always wondered who pays who when I click on the sponsored result Gemini.

The other thing that's interesting here is that the original paper, The Anatomy of a Large-Scale Hypertextual Web Search Engine, Brin and Page in 1998, I've just brought it up, and they actually have an appendix, Appendix A, which is titled Advertising and Mixed Motives, and it has to do with incentives, right?

They state explicitly, "The goals of the advertising business model do not always correspond to pri- providing quality search to users. We expect that advertising-funding search engines will be inherently biased towards the advertisers and away from the needs of consumers." Right? Um- 

steve: Wow. 

hugo: So they were- 

steve: But they 

hugo: want...

Yeah. It's in an appendix in the original version, and it's an appendix that's been redacted from subsequent versions as well, for what that's worth. You couldn't make that up. 

steve: Incentives. 

hugo: Exactly. And so I am really interested in diving into, we've been talking around this, the incentives and ability to [00:24:00] do experimentation.

When we discussed incentives earlier, you, you, you mentioned the difference between shor- short-term, medium-term, long-term i- incentives, and I'm interested in how you think about getting buy-in to have perhaps even short-term losses, to have an experimental culture. But before that, I'm wondering if we could just have an example of running an, an experiment and the types of challenges you encountered at eBay.

steve: You know, with the paid search stuff, once we had our big win, or basically we sold our theory to the finance folks, and then we had this accidental change that gave us the data we needed to show them we were right, and then they basically opened the floodgates. It's like, "Okay, start experimenting here, start experimenting there."

There was a... You know, we had that proof of concept, and we got the credibility. And by the way, just going back to the question that Duncan asked, another thing, a word that I didn't throw in there is trust A lot of things happen based on trust. And building trust with business [00:25:00] owners, with stakeholders is another key way in which data scientists, economists could become more influential.

Because once you build that trust and you've shown that you know what you're talking about, you could deliver results, people are more likely to open the door for you to try things out and play in their sandbox and explore other things. And part of building that trust is making sure that they don't look bad, even if they did make mistakes in the past, right?

But going back to your question directly, Hugo, there's one project that, again, Chris Nosko and I worked on at eBay that was a lot of fun, and I really regret that we didn't turn this into a outward-facing paper. Shortly after we arrived, Marc Cargille, the CTO, wanted me to give a presentation about some research I did before coming to eBay that I did with a good friend and at the time colleague when we did the work at Berkeley, Florian Zettelmeyer.

He's now at Northwestern Marketing and also in a high [00:26:00] leadership position in the Amazon advertising machinery, where we worked with this company called Manheim Auto Auctions and did some research that showed how releasing a certain type of information improved the sales of their cars and made everybody better off.

So it was a very nice applied research in a business setting. So I presented that at eBay so people could see what economists could do with business teams. And right after the presentation, a director from eBay Motors came up to me and said, "We've been toying with the idea of having a subscription model for dealers who list cars on eBay."

I didn't even know people list cars on eBay. Who would buy a car on eBay? Turns out a lot of cars are listed on eBay, and they had a very different business model. The seller didn't pay a percentage of the final transaction fee because, not surprisingly, most people who want to buy a car want to see the car.

And then once they're physically with the seller, that's easy to [00:27:00] circumvent a final value fee. So that model was a listing fee, and they charged 50 bucks a listing no matter how many listings you had. And then they were thinking, "Oh, we know there are big dealers, small dealers. Probably big dealers should get a discount.

How much?" And this and that. And for any economist, they know exactly where this falls under market segmentation or the nerdier and less sexy term, second price-- second-degree price discrimination. And we're like, "Yeah, we totally would love to help you with that." Now, I've taught this stuff. I taught this stuff at the PhD level.

I taught this stuff at the MBA level, but I've only taught it conceptually, right? So I know the distribution of demand, right? I know the probabilities of every different demand type, right? I could do a nice mechanism design constraint optimization problem. I have to figure out demand. I don't know how to do that because historically, for every dealer, we only had one data point every month.[00:28:00] 

How much cars are they willing to list at $50 a listing? So I need to create some variation to try to estimate some demand curves for different dealers. So we first said, "Well, could we, you know, play with the listing price? You know, offer a discount or offer a surcharge?" And, uh, like, no, that's kind of... You know, we don't wanna confuse people.

Okay. Then I had the following idea. Let's imagine, Hugo, you're a small dealer, and you have 10 cars on your lot, and Duncan's a big dealer, and he has 100 cars on his lot, and you're listing three cars a month now, and maybe he's listing five, even though he has many more on his lot How about we pick a subset of dealers and tell them, "Hey, you're getting a special deal.

For the next three months, just list as many cars as you want for free. Naturally, each one of you is gonna list all the cars on your lot or close to that 'cause it's free, and I'll find out what the intercept of your demand [00:29:00] function is with the X-axis, with the quantity axis. And now I have two points, and I can start doing something with that."

And then they're like, "Yeah, we can't do that because then for all these dealers, they're not paying us, we're gonna not make our quarterly earnings," right? And I'm like, "Damn, h-how do I do this?" And that's when we came up with... I was very proud of this idea. We looked back and we saw that dealers were pretty consistent.

You know, let's say you, Hugo, were listing two, three, four a month. Duncan's listing seven, eight, nine a month. Let's go back three months, see the average number of cars each of you listed, and now offer you the following deal. Hugo, you were listing three cars a month. That's 150 a month. How about you pay us 150 a month and list as many cars as you want?

And Duncan, you were listing nine cars a month, 450 a month. Pay us 450, list as many cars as you want. Effectively, it's like a two-part tariff where the entry fee is for each dealer what they paid on [00:30:00] average, and the marginal fee of listing a car is zero. So I got that data that I wanted, but it was like pulling teeth because I need to figure out a way to create an experiment without jeopardizing their short-term monetary goals as a team.

Now, if I had some way to give some guidance on, look, if we get this right, it's gonna be a windfall of $25 million a year, which it actually ended up being at the time, and we're gonna need to spend about half a million dollars on lost revenue in the short term to get there. I would imagine if I could convince people, then they would say, "Sure, you know, we're not gonna hold this against the team because we're learning."

But again, you need the culture to say, "Yes, we're gonna spend money to learn in order to buy a lottery ticket with a higher expected return." 

hugo: That's an amazing story. And I'm, I'm just wondering what-- throughout every way w-where you've worked, how do you think [00:31:00] about getting buy-in to run experiments like that when it does give up short-term revenue?

steve: I think first of all, if you really recognize the culture of the business to be driven by we need to satisfy the Wall Street analysts, you know, very short-term oriented, the likelihood of you succeeding would be very low because it's just part of the culture. And again, just to use Amazon as a different example, sure, you folks remember the-- all the letters that Jeff Bezos would write to shareholders, you know, I don't know if it's quarterly or annually.

And, I mean, it said like, "We're not trying to satisfy Wall Street here," right? "We're playing the long game," and that was the culture. My understanding is that Google was very much like that, too. You know, back when they did the IPO, they did this out-of-the-box auction that no-nobody else did. I really believe that the culture comes from the top, and in the right culture, it's gonna be easier to convince people.[00:32:00] 

When the culture isn't there... And of course, I'm just mindful as an economist, I'm using the word culture so much. I'm sure that half my econ colleagues are gonna stop talking to me after they hear this podcast. But, uh, I mean, that's the truth, right? It's just if the culture is flexible enough to let that happen, you're more likely to be able to do that.

And if it's not, it's gonna be pulling teeth. You're gonna have to have some, you know, occasional big win and generate that credibility where you could go to an open-minded, high-level decision maker and say, "Remember I told you A? And here's A. And remember I told you B? Well, here's B. Well, I'm telling you C now.

Could you let me play around a little bit to prove that? Because here's how I'm thinking about it." And maybe one way to do that in a more productive [00:33:00] and more accountable way would go back to the, "Here's my theory of the world. Here's why I think this is gonna work. Here are things I could measure along the way.

Give me some leeway to get part of the way there, so that if the early things I tell you will happen, you're gonna let me go further." Breaking that down that way, I think, is a way to gain the credibility where you're holding yourself accountable for the way you're thinking about the world. 

duncan: How do you figure that out from the outside, Steve?

So we have lots of listeners who are early mid-career thinking about their next thing or their long-term gig, and obviously you went to eBay and maybe some teeth were pulled, some dental work was done there. You went to Amazon later, maybe it felt a little bit different. But how do you try to assess that from the outside and land somewhere where the culture actually is right to, to enable those kinds of deep experimentation and relentlessly re-reinventing yourself and the business?

steve: You know, I'm, I'm, I'm [00:34:00] laughing because I'm very happily married and with my wife for six years, and when you start dating someone, you're kind of trying to figure out if there's a match, right? And there's always that risk that when someone is, quote-unquote, more desirable in the grand scheme of things, the other side is going to basically, "Oh, you love hi- I love hiking," right?

And I'm just thinking of a friend of mine who was dating someone, and hiking was very important for him, and yeah, she hiked with him all the way to the marriage, and suddenly hiking stopped becoming a priority. 

duncan: And 

steve: she didn't. So... No, she didn't. And I think it's really hard to know the insides of an organization that you've never been part of, and this is where having the inside information is really critical.

And here I'll- Tell you, I'm trying to rem-- No, I think Pat Byrer left Stanford before [00:35:00] you were there, right? Yeah. Yeah. Pat and I were colleagues at Stanford and very good friends and very sad loss. It's a year li-- this month, it was a year since he passed away. He called me to convince me to interview for a position at Amazon.

And then, you know, because we were buddies, he knew about a lot of my experiences at eBay. And he said, "Well, Steve, before I tell you anything else, let me tell you about a conversation I had with my boss," who's basically the CEO of Amazon Consumer, and that was Jeff Wilke. Said, "I have this idea," and I wanted to pitch it to him of something we should explore.

And five minutes into my explanation, he stops me and says, "Pat, are you trying to explain Bayes' rule to me? Because I know Bayes' rule." And I said, "You see, Steve, that's my boss. That's the CEO. Do you want to work here?" Of course, the answer is of course, right? Because... And it's that same Jeff Wilke who, when I was there, sent an email to a bunch of VPs saying, "We have this problem.

We need to think about this [00:36:00] like scientists. We need to form hypotheses that are testable and then see what we can learn." If you're on the outside, if you don't have a trusted confidant on the inside, I think the likelihood of you learning that is very low. Former employees. I think that's the due diligence that people need to do, and it's clear to me that that's a really hard exercise.

I don't think there are any easy fixes to that. 

hugo: I'd like to shift gears slightly now. It's very related to e-everything we've been talking about and we've, we've hinted towards it. But I'm interested in your thoughts on and your experience with generative AI in, in the past few years. And last time we spoke, you mentioned originally you thought LLMs would be an equalizer of sorts.

But now you're starting to think the opposite. So I'm interested if you can tell us a bit about that flip in that story. 

steve: Yeah. It's interesting. I first have to start with the caveat that given my history in tech and so on, [00:37:00] I am so far behind on where people would expect me to be on my knowledge of LLMs and AI and all that.

And it's not so much because I'm getting old and my brain doesn't work as well, which is true. I'm getting old and my brain doesn't work as well. It's more because my priorities have shifted and I've become, uh, much more interested in spending quality time with people who I care about. I s- care a lot more about mentoring younger people from different ways of life.

And rather than going after that next big research idea that with probability epsilon changes the world and with probability one minus epsilon is just another printed piece of paper that nobody cares about. I'm getting a lot more joy and fulfillment from, you know, touching people's lives. Um, with that caveat, I have played around obviously with this technology and it's true when [00:38:00] it When it started, I'm like, "Okay, this thing could do research sort of, could write pretty well, and that means that people who are good at it are not gonna get that much out of it, and people who are not that good at it could leverage to be better, and it would be an equalizer."

And then I started playing around with it, and I noticed that, one, the quality of the interaction changes dramatically based on how much thought I put into the prompts. Prompt engineering, I think people refer to this. Second, yes, it generates a lot of really cool stuff, but doesn't always all fit together.

And even now, when hallucinations are significantly fewer, still there are cracks and gaps. And what I see is that My ability to employ critical thinking to the output, then to see how I [00:39:00] either need to challenge it or think about different forks in the road that I could go to from here, turn those into thoughtful prompts, go back and forth with this process, it requires really deep critical thinking skills, the kind that people in academia, people trained in, in higher degrees in all sorts of disciplines have.

I think that becomes a game-changing input into this partnership between the human and the machine. And if that's true, which at the moment I'm convinced it is true, that means that we're complements, we're not substitutes. And if we're complements, it means that people who are lacking in those skills, in a reasonable knowledge base and critical thinking skills, et cetera, when they use these machines, what they're gonna produce is gonna pale compared to what people with deeper skills, more knowledge, critical thinking [00:40:00] ability are gonna produce, and the results are gonna be noticeably different.

And at the end, I think it's going to be the exact opposite of an equalizer. It will exacerbate the differences in education, raw talent, gifts that you got from your genes and not from something you worked necessarily that hard at. And I think there needs to be some awareness around that, and at a minimum, more research into really figuring out the extent to which this is there and where it's more likely to persist versus where it might be less of a problem.

Do 

duncan: you have a take, Steve, on how AI will affect kind of sizes of firms? Like within the literature, of course, of the theory of the firm and frictions across firms, like what does AI do to the optimal size of companies? And do we still need huge [00:41:00] corporations? Do we now have these billion-dollar one-person companies?

Like how does this unfold in your mind? 

steve: Let's start with the obvious answer. I don't know. And now let's go a little deeper. If I put on my organizational economics hat, which was my main area of research roughly between the years 2000 to 2012, I'll borrow from one of the areas there that I've been a fan of, I've contributed to, and that's transaction cost economics.

A field that was pretty much created by my late friend and colleague, Oliver Williamson. And transaction cost economics basically looks at that question of where is the boundary of the firm? What will be done inside the firm versus what will be done in an arm's-length market transaction? And it basically answers it with, well, doing things inside [00:42:00] the firm has costs.

Doing things outside the firm has costs. Wherever it's cheaper, that's going to determine what is done inside the firm and what's done outside the firm, which almost sounds like tautology. Now the question is how do you oper-operationalize that? And that's where Oliver Williamson brought these ideas of relation specificity and incomplete contracts and the complexity of transactions, and then I've done some work along those lines as well.

And I think that AI doesn't really push that boundary in one direction or another. I think what it does, it really gives us a different way to think about how these internal versus external costs are going to change. So if coordination costs, which are key to internal organization of firms, is gonna fall thanks to AI, then it's gonna favor bigger firms.

Then 'cause it favors scale, and that's gonna [00:43:00] favor firms. If on the other hand, the transaction costs in markets, right? So for example, what is it? Is there something like LegalZoom legal? Am I getting that right? Yeah, LegalZoom. I'm trying to remember there's some company that... LegalZoom. Thank you. Right.

Like you're a small company, you need maybe a part-time lawyer and now, no, I don't need a part-time lawyer, right? So I could do everything with, with LegalZoom. And when we start having these auxiliary functions that are being better done through technology, it means three people could put together a company and a lot of the things that are not core to business will be done through all this outsourcing and technology, then you're gonna have many more smaller firms.

So I, I think at the end of the day, it depends on where the bottlenecks are and where the value requires this kind of tighter integration, say big data execution, versus a lot of these outsourcing of activities that are not core to business. [00:44:00] So if I had to guess, I think we're gonna see more activity at the tails, many more smaller businesses, and then the big businesses are just gonna get bigger and bigger where that scale really matters.

That would be my prediction based on how I think about this. 

hugo: Fascinating. A barbell. 

steve: True. 

hugo: I, um, we, we had Noah Smith, Noahpinion on the podcast recently, and he had a take on market transaction costs that I found interesting. There may be a barbell there as well in terms of once you can verify and trust a partner or a collaborator, then transaction costs go down a lot.

But because of the generation of AI slop generally, there may be huge transaction costs in some parts of-- b-between firms in the economy as well. 

steve: That's, that's an interesting observation. I think that here the focus seems to be on what is being generated by [00:45:00] the technology. So if the technology that allows for outsourcing is going to be maybe less cutting edge, more cookie cutter, like again, take the example of a lot of legal contracting where you have like probably fifteen types of boilerplate contracts and twelve different areas where you could tweak things a little bit, and it's not rocket science once you have that down, then I think that would really allow for the smaller, leaner firms that rely on these services because this issue of trust is less critical.

If we're talking about things that require more trust, meaning that there has to be room for more discernment, more expertise, more ability to evaluate whether it's, you know, doing it right or not It almost seems to me that really [00:46:00] pushes things inside the firm because you need to have those expertise to make sure that the shit doesn't hit the fan.

I don't think that takes away from that transaction cost lens that I suggested. I think it's quite consistent with it because you could think of this inability to trust very similar to those ideas that Williamson created from the failure of the market, and that's why it needs to be managed through hierarchy, fiat, et cetera.

hugo: Yeah, that makes a lot of sense. And I wonder whether the other thing to consider is we're not starting in a vacuum, right? It's initial conditions and like large firms currently at least have a strong sense of how to deal with all the legislation that occupies the data product space, machine learning space, GDPR in Europe, CCPA in California, and- Yeah

large firms already have first mover advantage for lack of a better term 

steve: there. Mm-hmm. And, and what you're suggesting is [00:47:00] not only kind of, you know, makes sense true, this has been studied. There's been work done by a bunch of scholars. I remember Garrett Johnson at BU was one of the folks. I believe there's a survey by JP Dubé that, that looks at the GDPR.

They've done a bunch of papers where GDPR was very well absorbed in large corporations, and it basically killed small, medium sized corporations, the whole compliance, the figuring things out, and that, you know, I'm-- I don't wanna come across as, you know- no regulation, business is good, et cetera, because we know that regulation is often necessary.

But in Europe, I think people took that to an extreme, and those regulations really killed innovation, smaller businesses in Europe that were facing the GDPR, and it was the big companies that were able to do that. And if anything, it created more [00:48:00] concentration. Uh, and I'm pretty sure that was an unintended consequence.

So, so that's a really good point. And yes, the data backs that up. 

hugo: Super interesting. One thing I loved about this co-conversation, Steve, is a lot of the way we framed it was around data, economics, the firm, really large companies, and you've always brought it back to humans, which, which isn't an easy, e-easy task to, to do, and it's very thoughtful of you.

And with the technologies that are emerging now, we've talked about this before, but there, there's a difference, right, between engineering and science and figuring out how to do things versus figuring out what to do and, and where, where to go. I'm wondering, and this is speculative, but where that gap stands right now between humans and the technology we're building and using.

steve: Yeah, we chatted about this a little bit when we first met, and again, instinctively, it seems pretty clear to me that the machinery that we're developing, uh, we, you know, [00:49:00] humans, I'm not developing jack shit anymore, but is really, at the moment for sure, showing great signs of how to get things done better, right?

Namely that engineering aspect Rather than what should we be doing and what should we be focusing on? What should our goals be? Where at the moment, I don't find that these machines are going to do any better than regurgitate philosophy books and, you know, tell us what we already knew one to three thousand years ago, right?

If I take the example of a lot of the younger recently graduated PhDs who are working in tech, and they tell me that right now with either Claude Code or ChatGPT, what would take them two or three days, they're doing in a couple of hours. And I think we're very close to... I know there's a [00:50:00] term for this, where you just code in natural language, and then it does everything for you.

So again, I see that as getting things done that we already know where we're going. And I think the real bottleneck, again, thinking about that term, is setting the right goals. You know, choosing priorities, being accountable for the outcomes. And I can't imagine that someday maybe the machines are gonna be a little better at this, but at the moment, I think this is where the human angle is really critical and at the risk of really diving into the deep end of losing a lot of credibility for many of my colleagues.

You know, when you look at the Western modern view of more, better, faster GDP per capita, and you compare that to the kind of values of more, you know, indigenous, they hardly exist anywhere [00:51:00] anymore, of community, of being there for each other. I was totally bought into the bigger, better, faster Steve Austin six billion dollar man or six million dollar, whatever it was at the time.

And I'm starting to feel that we may be a little, a little too extreme in where we've gone with that, and that worries me as, as a father, hopefully, no pressure on my kids, but maybe in a few years, grandfather That worries me. 

duncan: When you talk to younger folks and help them think through what they should do next and how they should build their skill set in this crazy fast-changing world, how has your advice changed there, and how are you thinking about what folks should be doing next to become-- make sure that they are able to figure out what to do next and not only the how to do it?

steve: So first, I, I circle back to there will never be a substitute to developing critical thinking skills. And the irony is [00:52:00] that I don't think I know how to teach that in a pedagogical way. I think I know how to teach that in mentorship. I think that's a lot of that relationship between advisor and advisee, uh, at a PhD level.

But let's face it, Duncan, people like you, Hugo, myself, we entered grad school, we already had a lot of those skills, and I think they need to be developed at a very young age, and I don't know that our educational system is doing that a lot of justice. I think Jim Heckman, for the past decade or so, has been working a lot on, on young childhood development, but I'm not sure how much of that has been in this direction.

I know a lot has been on some of the softer side of nurturing. So definitely learning how to ask questions, learning how to critically evaluate things. The second thing is I see from my own lived experience and a lot of the [00:53:00] people I interact with in these academia, tech, et cetera- We all have really powerful inner critics, many of us, not everyone, but many of us, right?

That little voice on your shoulder that says, "Oh, you fuck up. You, you know, they published six papers in the last two years and you only published three. What's wrong with you," right? And, "Oh, they already are senior director and you're still a senior manager." And right. And I remember the first time I took this wonderful mindful self-compassion class where it started with an exercise where the person who taught the class said, "Imagine a friend of yours that's coming to you with, you know, they, they just got laid off from their job and it was downsizing and they're miserable.

What do you tell them?" And you're like, "Of course I'm gonna tell them, 'Hey, you're awesome and this happens and don't be so hard on yourself and, and you know, I'm sure you're gonna get another job soon.'" And just how could I be of help, right? [00:54:00] Say, okay, now imagine you're the person who lost a job. What is your inner thinking?

"Oh, you idiot. If you worked harder, they would've fired someone else," right? And one of the things that I really wish more people were able to absorb is just being kind with themselves 'cause we all get beaten up by life. And again, I don't know if this is more of a Western modern world thing, but we're really hard on ourselves.

So critical thinking and just be kinder to yourself. And I think with those two at a minimum, you're gonna live a pretty good life even if you don't end up being the next unicorn founder and or VP of data science and analytics. 

hugo: I really appreciate that, Steve. And I have half joked in the past that TDD doesn't stand for test driven development.

It actually stands for trauma driven development. As a, because it seems to occur a lot and I think whether it's pu- it's Western or [00:55:00] not, I, I think there's definitely a correlation if not a causal relation between two things you've mentioned, which is self-critic and an urge to up and to the right as, as well.

I am also interested in, you mentioned critical thinking, and I know it's time to wrap up in a minute, but I, a, a question popped into my hu- mind something I'm curious about and my question is, could critical thinking be somewhat downstream of cur- curiosity? 

steve: Huh, that's a great question. Is it downstream or is it upstream?

hugo: Well, I wonder if you start to develop these skills when you find something you're curious about and if part of the answer is to- Yeah ... give people breathing space to find something that they really wanna figure out. 

steve: Yes. And you know, um- For some reason, I'll tell you where my, my, my head is going. When you just use the word curiosity and right, I, I, if I ask the question, who's the epitome of curiosity?

Right. Curious, what would your [00:56:00] answer be, Hugo? 

hugo: Two people came to mind, actually. 

steve: Okay. 

hugo: Albert Einstein and Charlie Chaplin. 

steve: Okay. It's interesting that you focused on people. I'm thinking of a large group of people that are pretty easy to identify based on a certain characteristic. 

hugo: Oh, scientists? 

steve: Nope.

hugo: Explorers? 

steve: Nope. It's true that they're all... I'm thinking of a much larger group, like really huge. 

hugo: Humans. 

steve: You're looking at the wrong demographic. 

hugo: Plants. 

steve: Think age. 

hugo: Ah, children. 

steve: Children, right? Children are the most curious humans, right? And the interesting thing is that for children, when they have that curiosity and like, "Oh, how does that work?

Why is that like this?" Often there are two very different kinds of answers that will satisfy them. One is the kind of science of it, right? "Well, let me explain to you," and of course, age-appropriate, and the other is magic [00:57:00] or religion. They both often equally will satisfy the four-year-old, right? And what turns scientists into great scientists, like you said, Albert Einstein, is keeping that curiosity together with, "Okay, let me focus on the science."

And I believe it was even Albert Einstein who said, "There's always gonna be mystery," uh, right? There are always gonna be things that we just... You know? If you think about in some way, right, that's one of the, I think, the great contributions of Gödel, right? On, you know, that there are certain things that we won't be able to prove or disprove, and that's part of the mystery.

So to, to your question or point, Hugo, yeah, I think that curiosity and critical thinking, when they come together, that's just-- that's the explosion. That's the beauty. The curiosity is there for all the kids, and then the question is, how do you augment that with critical thinking? And I don't know how to answer that question.

I just don't. 

hugo: Well, thank you for [00:58:00] such a wonderful, wide-ranging, and thoughtful conversation, Steve. 

steve: Thank you. This has been a treat. Thank you, Steve. 

duncan: Thanks so much for listening to High 

hugo: Signal, brought to you by Delfina. If you enjoyed this episode, don't forget to sign up for our newsletter, follow us on YouTube, and share the podcast with your friends and colleagues.

Like and subscribe on YouTube and give us five stars and a review on iTunes and Spotify. This will help us bring you more of the conversations you love. All the links are in the show notes. We'll catch you next time.