The following is a rough transcript which has not been revised by High Singal or the guest. Please check with us before using any quotations from this transcript. Thank you.
===

roberto: [00:00:00] The pinnacle of your insight. It's never data. Only it's data plus some assumptions, plus your judgment. That's usually based in your domain expertise. Only with that combination, you can get your recommendation that we should stop doing X, we should do Y, or we should do Z in a different way. It's never only about data.

People who interpret the data science jobs. It's purely counting things in a smart way, or even modeling things in a smart way. Ultimately, I think find a ceiling. During that was Roberto EDRi, 

hugo: VP of Data Science at Instagram. I. Talking about the limits of data and analysis and why real impact comes from judgment and decisions, not just data and models.

In this episode of High Signal, I have the great fortune of speaking with Roberto about the incentive problem in AI product development and how to change it. We dig into why most experiments don't actually matter what teams get wrong about shipping, and how reels on Instagram went from a struggling feature to a [00:01:00] global success.

If the conversation about product strategy, data science is a decision function and why crushing your ego may the most underrated skill in tech. If you enjoy high signal, leave us a review. Give us five stars and share it with a friend. Links are in the show notes. Before we jump in, I'd love to check in with Duncan Gilchrist from Delphina.

Hey there, Duncan. Hey Hugo, how are you? I'm well, thanks. So I thought you could just tell us a bit about Delphina and why we're making high signal. 

duncan: At Delphia, we're building AI agents for data science. Through the nature of our work, we speak with the very best in the field, and so with the podcast we're sharing that high signal.

hugo: So Duncan, I know you really love this episode with Roberto. I'm wondering if you could tell us a bit about what resonated with you. 

duncan: Roberto and I go back more than a decade. He's been a friend, colleague, and mentor since my first job in tech. We worked together at Wealthfront when it was still crammed into a two room former dry cleaning shop by the Caltrain in Palo Alto.

[00:02:00] In the episode, Roberto offers an incisive point of view on what it takes to perform at the highest levels of data science. You don't get there by equivocating on this and that, but by staying close to the data, forming sharp opinions and being right, it's an inspiring conversation because he makes both the challenge and the opportunity crystal clear.

Let's get into it. 

hugo: Hey there, Roberto, and welcome to the show. 

roberto: Hey, you go. Thank you for having me. 

hugo: It's such a pleasure. I'm so excited to hear about everything you're up to at Instagram and, and meta at the moment, particularly in, in your role as VP of Data Science, but with all the product thinking that you do.

But as you know, a lot of people in our space there are like, there aren't many traditional paths C career paths. We're figuring out what that looks like and your history from being at Wharton and your MBA through to Instacart and Fair is such an interesting journey. So I'm wondering if you could just set the scene for us with respect to your journey and how you ended up at Meta.

roberto: Sure. Absolutely. I'm Italian. I was born in Italy, a small town, uh, [00:03:00] called Lake Como famous because it's beautiful and Georgetown is there. And I was born from a very blue collar family. So I'm the first person to graduate high school from my family, which is sets you like on an interesting path by the time you go to college.

I absolutely didn't have like in mind to move to the United States or work in technology at the time in Italy. And most continental Europe, if you're smart and ambitious, there's really two paths. You can go into consulting, you can go into investment banking. I went into consulting at a firm called BA Company where a lot of the job can be automated.

And at some point I was very proud of myself because I automated part of my job and my reward was I almost got fired. And the reason is that these, these firms, and you can put some banks there, some law firms, like they're in the business selling time. They would sell my time for a big multiple of what they would pay me for that time, that arbitrage.

[00:04:00] It's a lot of the services job, so if you're automated things and it'll take you less time to do the same thing, it's actually not a good thing for the firm. So good thing for you, and I think you're creating value in the system, but not a good thing for the firm. They overside me and I realized that that job was not gonna be the right long-term fit for me.

That's why I started looking at. Okay, like what are jobs that exist to, if you automate things, that's a good thing and you can get like that leverage. And they really introduce me to product-based organizations, like things where like you're trying to build something which then costs very little to replicate and it's very hard to do.

It's much harder than consulting or like being a lawyer. But if you do it, then it can be much more scalable and much more profitable and fun. And I quickly realized that that industry didn't really already exist in Europe and we're talking almost 20 years ago. And that's when I moved to the United States.

I went to to graduate school because that was a way a, to get a visa for two years [00:05:00] in which you could convince a US employer to go for a tremendous pain of sponsoring you for a longer visa and then eventually a green card and two. To try to make myself relevant for product-based companies because as many people in Italy I grew up studying, lacking an ancient Greek that already prepared me for career in technology or data science, except from knowledge of Greek lettuce.

So I went to work and I took almost no business class. You go to work or you can take any pen class, which is a major advantage. I took a bunch of. CS and starts classes, which was a very humbling experience for me at the time. And then I graduated in 2011. I took advantage of the fact that data science was a very NAS field, and so you could be a very mediocre data scientist to find a job.

And that's what I did. And I became the first scientist, a company called Etsy. Which at the time was basically a [00:06:00] warehouse in Dumbo, Brooklyn with people, great people who have missed year and, and dogs, and now it's a big public company, but wasn't at the time. And it was messy enough and growing enough.

And they said enough that if you had some drive, you could become valuable to the company and really learn on the job. And that's what I did there for a few years. And then also because of family reasons. I moved West Coast. I was the first data scientist for another company called Wealthfront, which is an asset manager except we use a software based set of people in going to the automation team.

And I was there for almost four years. And then something happened to me, which I think happens to a lot of data scientists who are somehow good at communication. At some point, the founder or the CEO will ask you if you wanna become a product manager. 'cause they know that you know what you're talking about, that you can also get through to very senior audiences with short attention span.

It's a useful combination. And I said [00:07:00] yes, and I was a product manager and a product manager leader at Wealthfront, and then at another company called Instacart. She's a marketplace for grocery delivery. Now, also a public company. And that was at a product at a company called Fair, which is another type of marketplace for a B2B retail independent goods.

And then I joined Instagram going back to a data science job title, looking after what you could describe as basically the consumer experience of Instagram as opposed to, say the creation experience or the advertiser experience. I joined that company five years ago to the date. Today is my fifth. Year anniversary at Instagram when I joined, that was clearly the first and only big company I'd ever been a part of over under among my friends was that I would last about six months, six to eight months before getting myself fired for not being a big company [00:08:00] person.

But I think Meta is not like your usual big company. And so actually is a very good fit. And now it's the job that I think I've had the longest in my life. 

hugo: Amazing. Firstly, congratulations on your work adversary. A as we say. And that, that's incredible. And what a journey. And I, there's something I, I love in there is that you went through several, you worked at several organizations which were at the forefront of data sciencey stuff at the time, like what Etsy was doing at the time.

We were all really EE excited about, right? There were a lot of challenging business problems where we're all trying to think about, and Etsy was doing really interesting stuff as was Instacart. Instacart with technology and of course Instagram and, and Meta now and something. I've never worked at Meta, but a lot of my friends who work there and have worked there in certain parts have talked about how wonderful it is, at least in their orgs, that it does feel like almost a mid stage startup, their small org within a larger organization.

So they have all the support of a larger org, but they don't have the bureaucracy of a larger org. Now that isn't all the stories I hear, but that's a lot of them. I'm so [00:09:00] interested in the intersection of. Data and product and business. And your role really involves using data science to frame business questions and influence strategic decisions rather than just deploying models.

So I'm wondering if you could unpack what this actually looks like in practice. 

roberto: Yeah, so I think data science teams can mean vastly different things at different organizations. I've always gravitated towards companies where data is closest. To the product management function, mostly because I absolutely don't have an engineering background.

I had like my first laptop, I was maybe like 21. I worked my first line of code. I was probably like 24 or 25. It's very hard to catch up in, in that way. I learned data science very, in a very utilitarian way to try and be helpful to these organizations that I was. Admiring and very much wanting to be a part of, [00:10:00] and most of the stuff that I learned on the job from some of the people at those organizations that have made those organizations really revolutionary at the time in this field.

Then it turned out to be for the first part of my career in tech, very much a source of imposter syndrome, but in the second part of my career, definitely a source of competitive advantage in the sense I. That I'm probably attuned to try and find the value and the practical implication in the form of what else we should do or stop doing or doing differently based on data and for sure the scientific method, more so than trying to do smart things for their own sake.

And that tends to resonate with people, especially senior stakeholders, who ultimately. Wanna win. They're not like interested in applying framework [00:11:00] A or analysis technique B for their own sake. They're interested in doing the ugliest, fastest, more direct thing that will result in a decision that is better than the random.

And I think that both at Etsy and Instacart. And fair. And now Instagram, we've been by far large able to do that. So sometimes people ask me, oh, you're a data scientist and you're a product manager and product manager leader, and now a data scientist. Again, like what's the difference? How do you interpret these two things differently?

The reality is I think my job is very similar across all of these job functions. I think it's about building a better product. Doesn't matter. The color of your shirt or what like your job title says. Probably the most impactful thing that I have done at Instacart, that has been something that I've done by going into a metaphorical cave and interrogating a [00:12:00] database for a few straight weeks as an ic, and that gave us like an equation that.

Could support our growth, geographical and merchant growth for a few months. Some of the things that are the most impactful that I've done at Instagram are things that are more like into the product management, real. It really doesn't matter, and that's like a philosophy and a standpoint that I try to encourage.

Also in the teams that I hire and support, and the people that join my organization, especially when they're early in their career. So that they don't feel limited by their job title or by a functional organization, and the walls are paper, and actually for data scientists, I think it's important to understand that like the pinnacle of your insight, it's never data.

Only it's data plus some assumptions, plus your judgment. That's usually based in your domain expertise. Only with that combination, you can get your [00:13:00] recommendation that we should stop doing X, we should do Y, or we should do z. In a different way. It's never only about data. People who interpret the data science job as purely counting things in a smart way or even modeling things in a smart way, ultimately, I think find a ceiling in their back.

hugo: I couldn't agree more. And one way I've thought about it historically is you have business questions or other types of questions and you're looking for a business answer, and you can factor that through a data question and a data answer. But in the end, you wanna come back with a business answer or a business solution, right?

And so much so, in fact, we were talking about consulting earlier, but I remember several years ago McKinsey even tried to push a new role called Data Translator. And it was, and I totally disagree with this role because I think it separates decisions from data more than bring, like it inserts middle management into the data process essentially.

But the idea I think is important that having a function or connective tissue between that brings business as close as possible to data and having those translation cap [00:14:00] capabilities. I, I love that you also. Talked about the, the need to, you hinted towards the need to do experiments and figure out what.

The quote unquote best course of action will be with error barss. And there is a paradox of experimentation. I can frame it in a different way. First, a lot of people say 90% of models don't make it into production. How do we fix this? And my response is always, oh, if 90% of models don't get into production, maybe that's a great thing because you're doing such wonderful experiments, right?

And so if the 10% are fantastic, of course my background's in scientific research where negative results, although we don't have a journal of negative results like we should, are incredibly important. And this is something that's important culturally, right? So at Instagram, my understanding is you all run tens of thousands of experiments like every half year or something like that.

Yet only a small fraction actually yield. Significant impact. So I'm wondering how you create a culture in which, 'cause this is the right thing to do, but how do you foster a culture in which people understand this and [00:15:00] have the liberty to have this freedom? 

roberto: Yeah, I think like not only at Instagram, like throughout my career, even like in companies or like maybe smaller, any which you could run as many experiments as we can run here.

The reality is I think 90% if not more of certainly what I did, but I think that applies to more than myself, was at best a know all, a zero at worst, probably like slightly negative for the organization. And then the remaining five to 10% that's to pay for everything else and more hopefully, and that's very hard for getting the human brain and sensibility to.

Understand and accept. I think somehow, like our brain is still wired for the days in which you go out to in the fields for three hours and you get 30 lids of or pounds of corn, and you go for five hours, you get 50. And that's like that linear relationship that's like. What [00:16:00] do you want? But in, in knowledge work, that's not true.

It's only a small amount of things, and you need to accept that everything else doesn't really matter. And if you can accept it, it can actually be liberating. It doesn't mean that you cannot be smart in trying to increase that percentage, but I think that it's helpful and healthy to accept it and understand.

It's hard to predict our priority. What's gonna be like part of the 10% as like an outsized return? One thing that's very arithmetic based that I wish people did more is just like impact accounting of what they're trying to do before they do it. One of the most important people in my career is engineer and data scientists and general renaissance man, Dan McKinley, who, um, I met at Etsy.

And he has a bunch of decks that encapsulates some of the lessons that we [00:17:00] learned at Etsy. And one is this. You cannot predict what's gonna work, but you can predict most of the time whether something is absolutely no chance of moving the needle. And so it's worth doing that very rough and quick arithmetic before you start working on something.

Even if the instinct is just, oh, I have this idea. This is cool. I should get working on it right now. This is something, a failure that we're all subject to. So like this sort of like the impact accounting, this ending a mental map, for example, of the platform that you're working on. It's important because it's a little bit like.

A city. There are certain streets that are prime real estate and important. Everybody's there all of the time. And there are others where almost no longer, and if you are working on building the city and you look at the city through a map, it's easy not to have that pulse because you're not like in the streets.

So I think it's important to train yourself on a mental map of what are like the few places [00:18:00] of real estate, so to speak, that really matter and. Trying to make them a little bit better, trying to make something that happens all of the time a little bit better. It's much more impactful than trying to build a new neighborhood or trying to get people to go into the not great neighborhood that no one wants to go to anyway, and which is just maybe a recommission.

This is some of the. Mistakes or lack of diligence, lack of questioning oneself that I see I've seen happen throughout my career. I see that in my own work, even if I'm aware of these pitfalls, and one of the things that can get the percentage of what you do that ultimately moves the needle from 10% to maybe 13%, which would be like a massive differential in productivity and a massive differential in terms of the outcomes.

That both you and the organizations you're proud of would eventually drive. 

hugo: Amazing. And I love this idea of impact accounting and you know, [00:19:00] I've worked in startup land for many years now, and there's the old trope, like every day get up and say, what's the most important thing I can do right now? And I think that's a naive version of impact accounting in, in a lot of ways.

Like you, you wanna rank order the things that are most important and figure out what will drive. The most impact. And funnily, I have, I went freelance about a year ago and I've always done some form of impact accounting, but definitely as a solo entrepreneur, quote unquote, doing impact accounting is incredibly important.

'cause otherwise one would burn out. There's, I, there's a question I wanted to chat on later, but it's really relevant now. I think particularly in terms of deciding what to focus on. And you were discussing incentives in consulting earlier and the kind of the pathological incentive of. Not the desire or the incentive to not automate and almost getting fired because of that.

In a lot of tech companies I've, I've worked at and, and, and know of teams often overemphasize shipping new product as milestones and people are incentivized organizationally and in [00:20:00] terms of their career as well to ship new product as opposed to. Maintain existing products and continually improve e existing products.

So I'm wondering how you think about it as a leader, how leaders can adjust organizational incentives or culture to, to just avoid, like launch, launch, launch, launch, launch as opposed to maintain continuously improve in, in the life cycle. 

roberto: Yeah. Yeah. I feel like very similarly, I think it, it goes back to the way that like our braids are both wired and also like by.

A lot of the media and the narratives of the media, the stories that we are exposed to when we grow up, particularly like you think about all the movies, the brain is gullible to the fascination of these discrete moments in which things change. Like anytime you watch like a Hollywood flick where there's some permutation of the hero's journey, there's always like this moment.

The hero [00:21:00] finally overcomes things and everything turns, and there's always like this sort of discreet milestone. And you grow up with this thing, it remains with you. And so like I, I think it's important to accept and to realize that we're all like fascinating by this like moment of like shipping, there was nothing.

And then like at some point there is something is out there in the world. Like there's Steve Jobs in 2007, like launching the iPhone and that's a thing. And there's like a before and after again like. I think another of the, of the war stories that, that, that achie at Etsy put out at this like fantastic slide, which encapsulated that pattern, Etsy, where people work like on a product and then they're close to shipping it, and Etsy is a very social place.

So you would like book a room or like a venue from a ship party. And then ship it and then have a party and have schwag and really [00:22:00] remember that day. And it was like a success having shipped. And then, you know, 18 months later, the feature would be quietly sunset from the website for complete lack of usage.

And I think like the important thing is one, like to realize that this happens. And some of the most fun and more meaningful moments in my career are part of that pattern. And that's okay because again, like the fact that 90% of what you do doesn't matter, it's ultimately okay. The second thing I think, is to try to rewire your brain and if you're a leader, the brains of your team into impact.

And I think that the meta does this. Better than any other organization that that I've been a part of. If you ship something, you have just flipped the switch, which is entirely in your control to flip. If you have made something that people use and that people want, then you have created value in the [00:23:00] world.

So flipping the switch is not creating value. Creating value is like flipping a switch and then like people are continually using this thing and achieving better results against their objectives because of that. And it just so happens that if you, the platform that has put that into the world, you're able to capture a part of the value for your own business.

So I always go back to the ship launch party 18 months later, like sunset. Also to try and the flip side of that is like fear of failure. A lot of the people that work in these successful companies, especially the people who grew up in the United States, like they come from a life of perfect choices.

That's how you get a Etsy, you get an Instacart, you get a meta. Like these are like, these are sterling resumes. I was lucky being born in Italy and being afforded to make many mistakes and still end up in just sort of like. Eum of, of the word. The problem when you make [00:24:00] a life of perfect choices is you become addicted to the lack of failure.

And one surefire way to avoid failing is never try to do something that might fail. And so it is okay to shift something and then it doesn't work out and you sunset it. The thing that's not okay is not doing things or keeping like things in a little of months or a year testing because they're not quite ready.

I. To launch, but you're also not quite ready to stop working on them because you have developed an attachment. 'cause you're worried about your own standing in your place of work. So it goes back to your psychology. I think you want to shift things pretty fast or sunset things pretty fast. The real mistake is keeping things in this limbo.

For way too long, and this is true of products, this is true of personal choices that sometimes you make or you don't make as a leader. Very [00:25:00] rarely you get people thinking, oh, I wish my sunset that product later. I wish I gave it like another few months to thrive. I wish I didn't ask that person to leave the company this early.

You often hear the opposite, and so I think it's a matter of understanding how our human brain is wired. And its limitations as it pertains to the game that you play in a world of nearly infinite leverage, then try to do slightly better at that because of this knowledge, while recognizing that in the end we're feeling massive meat and we're gonna make suboptimal decisions.

hugo: Absolutely, and I love that you framed it and I paraphrased, but around like the spectacle of the launch. And in fact, I don't know if you've seen Danny Boyle's, Steve Jobs movie, it was written by Aaron Sorkin, but it's in three sections and it is actually, it's framed around three different launch speeches in the history of jobs leaving Apple.

So it's a movie which is around this spectacle, and it ties into that, that moment where you open. Your new [00:26:00] MacBook Pro and you hear the angels of technology sing to you, right? And that smell comes up and it is almost like our modern religious e experience, our secular religious experience. I, I, in so many ways, I, I also wonder how we can, and maybe we can use something in Instagram as a way to.

Look into this. Perhaps not, bud. I'm interested in how we can reframe failure as learning as as well. And I think that's something we're talking around is if you're doing experiments and you're trying things in personal life, business life, whatever it is. And something doesn't work. What you learn from that, and I have a kind of a naive example, but I was recently helping out some friends with some marketing and we did a campaign that totally fi, like literally no, there was nothing good about the results of this campaign.

And then we looked into why and we learned certain things about what not to do in future, and we didn't put a lot of spend into it, but enough to learn something. So I'm wondering in your mind. How we can culturally reframe failure is learning. 

roberto: Yeah. I think one of the advantages of working, especially in software, if you have a [00:27:00] hard worker bone and then there's some like hard constraints, but a lot of working in software and working on platforms that are evidently hosted is not everything, but almost everything is the proverbial two-way door.

You can take it back. The only. Come to that is a slight feeling of embarrassment, but the outside world doesn't care. Like that's that foster world quote. You'd be surprised and you'd make like better decisions if you knew how rarely the world thinks about you. This is true even like a of big companies.

So keeping in mind that almost everything is actually irreversible. And having a very high threshold for embarrassment, I think are two of the best things that you can have. The in experimentation culture, one of the biggest pitfalls that I see is you launch tests with variants that are not different enough, [00:28:00] and if they're not different enough, then the likelihood increases.

You're not gonna learn everything from the test. You mentioned like we should keep drawing outs of negative results. I very much agree with that. I. But people are embarrassed and we make a lot of embarrassment driven choices in all of the organizations they've been a part of. And one way to avoid that is like launch something, a test where most likely you're not gonna learn anything in the positive or in the negative.

And you're gonna tweet and you're gonna pretend that infra is 2020 eyesight when clearly it's not. And then there's gonna be like a lot of. Practices around that. The best thing to do is to launch things fast, incur the risk of embarrassment, be very quick in changing your mind, be very quick in sun setting and changing things.

When that happens, if you know that most of the value will come from the occasional home run and then [00:29:00] you're gonna hit, then the most rational thing to do is to try and feed as many at bats as possible in the allotted. Amount of time. This, again, it's a thing where like media and also schooling doesn't prepare us well for that because schooling, especially for people who are like these high achievers, it's you start from a hundred and then you get like doc point for every mistake that you make.

This is the opposite of how the word of product works. The word of product. You start from zero and then try to make as many points as possible. In a constraint of time, you learn like constrain of like how many tries you get. You get like an unlimited number of fucks that you can make. The important things is that is like how many good things you've made, not your percentage, and that requires a slightly different mindset, and I think it's a mindset that founders has because if you're like, you like a solo [00:30:00] entrepreneur, then you don't have that problem.

Incent are alike, but as a company. Increases in size. And that also means by definition that that company has had a measure of success beforehand and have been a part of a lot of companies that were rapidly growing because they were having the level of success. Then your mind gets into this and tries to avoid embarrassment, tries to avoid surprises, tries to avoid changing one's mind, and the problem with that is that also learning.

Take takes a nose dive. But if you're able to embody that sort of low ego, low sensitivity to embarrassment mindset within a big company, you're gonna be in the minority and you're gonna have an outsize result. Exactly. Because people around you are gonna be more insecure, more risk averse than than you are.

hugo: Absolutely. And I'm half joking now, but for people who do get embarrassed around this type of thing, maybe. We need [00:31:00] embarrassment, my milestones or like embarrassment driven development or something like that. I've talked about anxiety driven development, which I think I've practiced and a lot of us have in the past, but not being fearful of embarrassment.

I'd love to jump in and talk about some of these ideas with respect to a particular product that I presume many people know called reels on Instagram, where initial user metrics were negative, yet it eventually became, and I understate it, a major success. So I'm wondering if you can tell us about. Uh, what this journey looked like, and in particular, talking through how you balance data-driven decisions against the need to support strategic bets that initially show negative results.

roberto: Yeah. Yeah. I think the big advantage in my mind of meta is, and then that's probably like the. Big company at this level of size that's still like founder led. That's an amazing advantage in terms of the ability to change course and make bold decisions [00:32:00] when needed and overcome. Innovator's dilemma.

Clearly you launch something new. Part of it is gonna cannibalize your own existing business. The difference between a successful company or a company like. IBM of yesteryear or Intel most recently is realizing that cannibalizing yourself is better than like someone else cannibalizing you. In our case, we have a very big wake up call with TikTok.

You. 2020 and 2021. And we also had this advantage of being extremely impact driven internally, which allowed us to bet big on reels beyond what you could see like in the experimental timeframe. The biggest weakness, I think, in general of AB tests is practically speaking, you can only run them for so much.

In a company environment, [00:33:00] many of the most interesting effects that you're gonna see are gonna happen outside of that timeframe. So by definition, you cannot use a vanilla AV test to evaluate what to do next, especially when it comes to things that get better very rapidly. So how good you were is something last month.

It's not predictive. Where you good you are today or next month. For real, specifically, I think GPU Driven modeling and the ability to recommend things off of a potential inventory of hundreds of millions or billions of piece of media is something that's become much better, not only for meta, but in the industry in a relatively short amount of time, pretty much at the tail end of COVID and 20 22, 20 23.

Because we wanted to bet on this thing and we [00:34:00] knew that was what people wanted. We were able to write that way and come out with a, with a product that I think is now so much better than the early incarnation of reels and the ability to recommend, to use something that you will like. And I think you do something like that, that people want and uh, and the results follow.

There's a former employee of Meta, his name is Nikita. Beer is some sort of, of a growth ledger because he came to meta through an acquisition and he has this reputation and track record of having developed a few viral consumer apps, which, which is not easy. And when you, you talk about consumer, and he's a very snarky guy.

I think in the early years of reels he was making fun reels versus TikTok for. The inferior quality of our recommendations at the time. And then a few years ago, someone was saying, Hey guys, you know what, like this was on Twitter. It was like, you know what, like I've been giving a [00:35:00] realtor another chance and now like it's actually, it's showing me things that, that I like, which is not an easy problem in, in recommendations.

And Nikita replied and replied, yeah, reals is actually good now. And that to me, like from such a very practical and very snarky guy at that. It was like the ultimate achievement. So much so that now the LinkedIn, you can put not only your picture, but also like a background picture, like in other social networks.

And the screenshot of that, not even a a tweet, a tweet reply, is my personal cover screenshot LinkedIn because it's one of the things that I played only a small part on, but I'm very proud of. 

hugo: Fantastic. That's such a great story and I love your framing. Of when self cannibalization can be good, and one, we don't have the counterfactual, of course.

One example is if Blockbuster had tried to cannibalize its own brick and mortar stores by doing streaming stuff early on, perhaps we would've seen a different story with respect to that, those shifting curves of [00:36:00] Blockbuster and Netflix. I am. Interested. We're working in such a rapidly moving complex environment technologically and socially at the moment, and I'm just wondering what you can share about your approach to evaluating and integrating emerging technologies like generative AI or large language models into products that Instagram, and what criteria guide decisions and adoption to help scale these types of innovations.

Yeah, 

roberto: I think it's to me like not dissimilar to reels and the advancements in recommendation modelers that have happened spurred also by having figured out the GPUs could be used not only for gaming, but also for matrix multiplication in a very interesting way. Meaning trying to incorporate as much as possible the rate of change or something that's getting better very fast.

That is very hard to do, and that is something where I think. Even being a founder, very involved. It's, it's a major [00:37:00] advantage if you think about it. Like even, I don't know, two years ago, one year ago, a few months ago, in certain respects, even today, you look at the state of the art generative AI models, some of their outputs are still embarrassing, right?

They're not like ready for prime time. I was looking earlier today at somebody who prompted. I think GBT four oh make me a map of Europe and they'll make like a map of Europe that looks like one of those maps from like middle age and all the labels are wrong. And that's like an attached to what human brain seems like.

So simple that the result is underwhelming. There's still like a lot of these like underwhelming things, but I think that the very definition of disruption, it's a product that initially it's inferior to the competition. On almost every aspect, but whose cost structure, it's even smaller. So if [00:38:00] you normalize by the cost structure, it's actually a very interesting product.

And then eventually that product gets better rapidly and it's able to eat the incumbents launch. And being able to anticipate that, I think it's important. So right now you look at a lot of AI generated content. I. Maybe not ready for prime time yet, but certainly much more so with a few use cases which we saw, which we see like in terms of like ways or memes we see like the Ghibli for example, portraits much more viable today than they were even like six months ago.

So like being able to extrapolate from that and make a bold bet within your product. I think what distinguishes, like companies are gonna be able to ride multiple waves. Of platforms from companies that remain overly attached to the wave that made them originally successful. The way that we [00:39:00] encode this in product development at Meta is by establishing clear betting budgets.

You think like about in product, we always talk about bets, big bets, strategic bets, but what is a bet. A, that is a wager. You do? Yes. In a band. There's a point in which like you have some value, you have typically some money and you part with it. You put it somewhere. The part in poker or these games, you might, it might not come back to you or it might come back to you multiply it.

So there's an element of risk embedded in the word bet. Oftentimes we use bet, but it's not really a bet. It's just a way of saying somebody wants to do this. And we're doing it. There was actually no upside in some of these things. I think there's a lot of upside, and we definitely wanna continue to have that impact focused culture, that way of being aggressive betting in [00:40:00] parting with some short term value in search for a higher, longer term.

Optima, anything that, for some of the reasons we already discussed, only for our size, we're one of the best organizations in the world at that. 

hugo: Fantastic. And I love how you framed the I the idea of a bet and of course for a lot of organizations, and it may not be the case where you are all the time now, but the cost in doing such a bet is not often even monetary.

Now it's, it is in terms of headcount, but it's really in terms of the time of the people involved. I also loved how you made clear something a lot of us know now is. How LMS, for example, can be wonderful at so many things, but a certain percentage of the time and over 10% A, a lot of them, they like serious stupid fa failure modes, like what you would consider a human doing something stupid.

And I do a lot of work in helping people build and ship LM powered powered products. And some of the most successful stuff I've seen, and it depends what your appetite for risk is, what type of product you're building, but like having them [00:41:00] embedded in like serious business logic and significant guardrails because.

Depending on the use case, am I gonna let a random LLM response serve that to a user? And a lot of the products, like for example, certain generated AI products on LinkedIn, they still state this is very experimental. Please give feedback. And you know that you even give plus one minus one give feedback 'cause we're still figuring this out.

roberto: Yeah, yeah, yeah. No that's true. And I think like, I think Ben Thompson over Strat like this interesting framing in which. With this Nissan technology, oftentimes like you wanna do things even like simpler things, but very predictably versus some technology like Siri is not being a success to date, where like they tried to go for the home run way before the technology was ready.

And so it is a very inconsistent results and very frustrating results. And you born through the asset, which is the only non replenishable asset, which [00:42:00] is the trust of people. I think like very few people trust Siri with trust Siri with less than elementary tasks and even those right now. Whereas if you're able to show progressively more ambitious tasks but perform reliably on those, you can create a lot of trust.

You can create a lot of consumer habit on that, and that is good. The same grace is that there are so many. Parts of product that are baseline bad and so many tasks which are actually complex for humans that LLMs can excel at. For example, LLMs can do a very good job at telling you what a piece of media is about, right?

A picture or a video or something like that. Even when you don't have, uh, metadata that can power a lot of use cases. Search being one of them. Like one of the reasons that like Google search [00:43:00] is good, it's been good for 25 years, is that it's an easier problem when it's about words, but when it's about videos or pictures, that's like a much harder problem.

You might need to use like a lot of humans to annotate and produce metadata about something, but that is of course very expensive and. Gives a sort of labor cost advantage to countries that are not the United States, but then we build this technology and this technology is great. Even like the Apple Intelligence notification summaries are pretty good at telling you what your notifications are about.

That's something that's very easy for another to do at 70, 80, 90% of the level of a human four. Less than 1% of the cost that is disruption, right? It's something that is worse, but it is much, much more economical and [00:44:00] convenient than it is worse than the incumbent solution. So that's, I think, the exciting thing, finding those things where the cost is a tiny fraction and the value is.

In the same order magnitude. 

hugo: Totally. And I love that you mentioned apple intelligence summarization. 'cause I do, I say this to a, a lot of people I work with, who a lot of people I work with wanna build like multi-agent architectures, multi turn con conversations. And so sometimes I say to them, look, we're still figuring out summarization in generative ai.

It's actually that early. Sure, let's go and build this stuff. But there'll be significant failure modes. And I compare it to, I was watching something about. The guy in the late nineties who he built the first website, which had a video playing where people could chat while watching the same video. And that seems like an ancient technology to me now, but when he did this, that was like, wow, that's incredible.

And we really are at that stage of like very nascent technology, right? So a super [00:45:00] exciting space. 

roberto: People, I think over always overestimate how much it takes to create a lot of value, but I think an intelligent question is, what is the minimal thing that I can do that will create a lot of value and then do it over and over again?

Many times, like what you mentioned, there was a moment in the development of, I think what was called Web 2.0. We're talking about like mid to late August. At the time I was like. At that point it was trying to get to the United States. It was reading everything about technology is Ajax a synchronous, JavaScript and XML, which was this way to be the first sort of reactive webpages and reactive chats where you need to refresh to the server every time.

And you could like actually from such a simple technology, you could actually build things like real time chat. And there were like people building them like. 37 signals built [00:46:00] campfire out of that, and that was pretty incredible. But people are like, oh yeah, this is just like a web page and then like Slack with somebody at the end who's a product genius that could understand how much leverage you could get out of this very simple dynamic was able to be on the big business out of that.

Mm. Look, a lot of big businesses, they're built on a deep, but usually tiny. Observation, uh, novel, novel sort of observation. The whole Cash app business is built for a very particular way that you can make an a ledger writing in US financial assistance, it wasn't possible before. Builds like a very big thing on, on top of that.

So the, I think, product instinct is being able to find these pockets that change from one day to the other and imagining. How big of a business you can build on top of that. People think that you need some major [00:47:00] breakthrough of a wide surface area to do things that are of a scale. But that's not true.

Like that's not true about page rank or the auction system that powers a lot of Google. That's not true about Slack and Ajax. And if anything, like I think on with LMS, there's a lot of these things that are happening and a lot of value will accrue to the people are able to build the product layer on top of these models.

hugo: Absolutely. And many aspects of the product layer, you actually spoke to a key point, which is annotating data and needing to hand label stuff. And of course several platforms including, if I recall correctly, Facebook back in the day. Realize that you can add value to users by allowing them to tag themselves in photos and then have that to be used as training data so that eventually you can tag them and ask them for confirmation, essentially.

So figuring out what, and once, we haven't said this explicitly, but all the products we're talking about are [00:48:00] seriously data powered products, figuring out how to establish. Those data flywheels in product with users in the loop can be a fascinating avenue for this type of product development as 

roberto: well, right?

Yeah. Yeah. I think that's true, but there's also like a change that's happening over the last 15 years. I wanna say a lot, even like of the social products that we use now, we're built maybe 15 years ago, Instagram must be more than 10 years ago. Twitter now XI think started in 2006, like at that time.

People would give the time to these products to build their own graph, build their own lists. Many of DOG Twitter users have lists for their various interests. You tag people, you'd add people and create your graph on Facebook and on Instagram, and then what happens around 2020 with the rise of things like TikTok, but also I think with a, another behemoth of consumer entertainment, which is [00:49:00] YouTube is.

Just getting accustomed to being served the best stuff for you without the need to add, curate a graph, subscribe to a YouTube channel, search for anything, right? You just, somehow the algorithm has figured out what do you like, and it's there in front of you, and that changes the propensity of people to even annotate these data.

Access that propensity, I think is today. By far large part of that is just like a generational, makeshift lower than, than it was. And if your product was built on the strength and banking on people being able to annotate the data and make your job easier by giving you free signals, then it became much harder overnight.

And that's I think, a lot of what made us uniquely suited to overcoming in terms of challenges. And now we have these other. [00:50:00] Technology, a large language models, which are basically like free people, free manary, people that can go and annotate stuff for you at scale. A scale that's much larger and not limited by the constraints of humankind for a very convenient cost and don't think that anybody has yet figured out.

How to use this to its full potential. Absolutely. So there's a lot of areas just waiting to be unlocked 

hugo: and some of the most exciting things I see even before product launch with LLM powered products, you can do a product spec and actually generate some synthetic user queries, hand label some, and then build an L LM as a judge.

And then some of the exciting things are then figuring out how to align your own judgments with your LLM as a judge. But one of the things I've found fascinating is having to go through this process makes me be, makes me have to be so explicit. About what I think is good and what I think isn't so it isn't just intuition anymore, it's actually writing criteria, heuristics for these types of things.

I [00:51:00] sadly we have to wrap up in, in a couple of minutes 'cause I feel like I could chat with you for hours. Roberto, I am interested, there's so much meat in here. I'm interested what practical lessons from your experience at Meta. And Instagram. Do you think smaller organizations or data teams could learn to a, apply their own strategic decision making processes?

roberto: Yeah, I think if I were to encapsulate a couple of tenets, which resonate with me more and more as I get older and as I, my sample size of people that, that I support grows larger. The two things would be as follows. One is All Can by Stephen COVID, which I think is still like very current. Try to begin with the end in mind.

Don't just do the first thing that comes in URL. Start from the outcome that you want. Work backwards from there. And the second is at some point there is a very clear [00:52:00] fracture between playing to win. Playing for the outcome that you want versus playing to be smart. If you want to play to win, you really have to crush your ego in the process.

Most people are not an animal to doing that. They're rather loose than do this. One of the thinkers, which you say it's the most influential to me that talks about this, especially his subscription XA lot, is a product leader formerly of Stripe. Once you have that mental model, you see that everywhere, including in yourself, right?

One dynamic we discuss is embarrassment. Embarrassment is just ego, right? If you cannot be embarrassed, you're much more likely to win. Mm-hmm. If you cannot be embarrassed, you're gonna check your assumptions. You're gonna end up like starting with the end in mind. You're gonna be more likely to win any given game.

Over a large enough sample size of gains, and fortunately, every day it's [00:53:00] a new at bat and this abundance driven line of work that we're in, you're gonna end up with a completely different and better outcome. So a lot of this is just crush your ego. Try to do things to win and to learn versus to fit it.

hugo: Amazing. Well, I think that's wonderful advice, Roberto. I've enjoyed this conversation so much. Thank you for such a wonderful conversation and bringing all of your e expertise and history here as well. 

roberto: Thank you very much for having me 

hugo: here. Thanks so much for listening to High Signal, brought to you by Delphina.

If you enjoyed this episode, don't forget to sign up for our newsletter, follow us on YouTube and share the podcast with your friends and colleagues like and subscribe on YouTube and give us five stars and a review on iTunes and Spotify. This will help us bring you more of the conversations you love.

All the links are in the show notes. We'll catch you next time.