The following is a rough transcript which has not been revised by Vanishing Gradients or the guest. Please check with us before using any quotations from this transcript. Thank you.
===

hugo: [00:00:00] I'm so excited for this chat, as you know, Ravin, for a number of reasons. 

but, You're now at Google labs doing a lot of research into, foundation models and the products,that we can build for them and I, from them. And I want to get to all of this, but, you've had such a fascinating journey to where you are now and it ain't over yet, but from Tesla, SpaceX to sweetgreen to probabilistic programming with PyMC, so I'm just wondering if you can give us a Kind of a brief introduction to what you've done historically and how they've led you to your current position at Google.

ravin: Yeah, I think as you pointed out, I think I'm been lucky to have worked at, in a number of teams that are pushing the frontier for various things, whether that be rockets or in this case now, frontier AI. just for me, going way back, as a kid, it just fascinated me so much how you could use mathematics to, just describe the things around you.

I remember being in high school and learning about Newtonian mechanics, and it just, explained how things fall, and how [00:01:00] momentum worked, and I was like, oh wow, you can actually describe all these phenomena with, And if you really scale it up, you can then describe how planes fly, how bridges are built, how buildings don't collapse, like so many things.

so I got my degree in mechanical engineering and that led me into SpaceX and, the hard mechanics industry where we're building rockets. Actually, just like the Starship launch that happened two, two days ago,for me, early days on that project in 2017. But when I was at SpaceX, I realized I also wanted to describe the things that were more intangible.

I was in the department at SpaceX that was figuring out how to build the rocket, the supply chain department. So not necessarily the one that was figuring out how the rocket flew, but the one that was like, how do all these pieces come together? when you see something like Starship, there are literally tens of millions of parts that have to come together to get that rocket to fly that one time.

It's like little bolts, little washers, just sealant O rings come, they come into these bigger structures and bigger structures, the bigger structures. and it, you can describe this whole thing as a [00:02:00] computer, as a graph in computer technology. And you can describe the P the way pieces get to each other as edges.

So now you have this sort of bill of materials, just like a Lego. and I realized, what would be really cool is generative modeling would be neat to explain the variation in how these things get built. That's how I got into PyMC.

hugo: Just quickly, would you disambiguate the term generative modeling?

Because we think of generative as being, the term's been stolen, just like the term inference, Ravin. 

ravin: That, that's a fair thing. I feel like, I like to say, I used to work on, the original generative modeling or generative modeling before it was cool. so just think about when you work, when you're working at SpaceX, you have this really big piece of a rocket.

It's called the first stage, right? It's the part you see that comes off right as it gets halfway through its flight. Okay. A big question is how long is it going to take to build the first stage? It was a very common question because you got to get this thing built so you can go actually launch it into space.

now typically what, let's say a junior data analyst like I would do is you go into your database and you say, the first one took two days. The second one took four days. The fifth one took [00:03:00] six days and maybe the other one, five days. So you got four data points and you go, let me just take the average and you get some number like 4.

12. Um,that's a number, and you can go tell people it's 4. 12. The problem is it's never actually 4. 12, right? And you have this variation, and I was struggling to explain this variation in,the next thing that a junior data analyst did, which, which was me, was you just, you say, let me just take the standard deviation because, that's what I've been taught in school.

It just doesn't work for rockets though, because things aren't normally distributed. things can't go below zero. That's impossible. You have all these constraints. And I was like, wow, what am I going to do? I didn't, I don't feel like I learned. Enough of the mathematics to describe this. but I got really lucky and I got into the thing called Bayesian modeling, which Hugo, I know you're all about as well.

You, you have great, great tutorials. So pitch for Hugo. If what I'm saying doesn't make sense, go watch his tutorials. But this Bayesian framework lets you do all sorts of wonderful things. You can specify that your distribution can't be below zero. You can say that maybe it maxes out at 30. And once you set all [00:04:00] this, you can use, some technology called like MCMC or other estimators.

To come up with this distribution that you can now sample from, you could say, I've seen these five versions in the past. What is every possible version of the future that I can see given my constraints? And now I had this thing where I could just pull numbers out and say, I think we're going to see four a lot, but we might actually see five a bunch too, and six a bunch too.

We're probably not going to see 12, 12 days, but it might actually happen. But now we can start planning for, okay, we probably think it's going to be between six and seven days, but in case it's 15 days, here's the things we can do, because it's not very likely, but there is a chance. That sort of.

Generative thinking of generating all these possibilities and the relative probability that each one would happen Unlocked a whole new set of, capabilities in what's called decision theory For me to go in and have conversations with people within SpaceX about the various, possibilities. To allude to Hugo's point, now the term generative now talks about these large language models And we're getting all [00:05:00] these chat GPT things and tokens and images and stuff like that.

But if you scale it all the way down to what I call a one or two parameter LLM, that essentially is the core of generative modeling. 

hugo: Yeah, and it actually, in the Bayesian sense, and I, we, as we've joked before, we're here to talk about generative AI, but we'd love to do another podcast on, on, on statistics, whether you it Bayesian or not.

I, it does force you to write down your assumptions and, it forces you to write down what you think the generative, the data generating process is. What is actually happening in the real world to generate the data you're seeing, and then it helps you make decisions under uncertainty. The other thing that You mentioned, but elided over slightly that I want to drill down into your background was in mechanical engineering, which makes sense of how you went to SpaceX, but you were there as a data analyst as well.

So it isn't so I'm very interested in career paths because none of them, most of them are non obvious. How do you go from mechanical engineer to being a data scientist to, to working on generative AI at Google? [00:06:00] 

ravin: Yeah, so I think one thing that is, I'll say has a lot of noise in it as I was navigating through my career and even still as I'm navigating through my career is, job titles.

I, they're just broad labels that different organizations apply to skill sets. and I think people get really fixated on the term data scientist or things like that. So for me, you're right. I've actually, I have a really weird career because none of my titles actually match. I went from mechanical engineer to industrial engineer.

To supply chain engineer to sweet green actually literally just called me engineer because they didn't know what category to put me That's awesome So literally my title was senior engineer and now i'm at google. i'm a senior research. Data scientist so they put a lot they put a lot of my old titles to together into one.

If you read the titles, it becomes hard to understand how that word soup works. But what's been a constant in all this is it started with just mathematics. and we'll, and even actually mathematics are still being used. I had to learn linear algebra, differential equations, some amount of probability theory, and my mechanical degree.

I leveled that up when I was at SpaceX, working on learning, Bayesian [00:07:00] statistics, just as you said. And statistics in general, in, under a supply chain engineer title. but, that actually got me into, what I think is my most fun org, or one of my most fun orgs, is, PyMC and Open Source. So by working with those folks That gave me a lot of the skills and stuff I needed to get, Google as a data scientist.

the truth of that is just learn the fundamentals, in my opinion. really get down, learn the fundamentals, and you'll get the job title that matches the thing you want to do, which is the core mathematics that you're actually working on. 

hugo: Yeah. And I love the variance,the spread of the types of companies you've worked at.

So working on mechanical engineering stuff at SpaceX rocket ships, through to sweet green when I lived in New York city was like my go to lunch and I love salad. So you were working for. What seems to be a salad making company, but is a very, in some ways, sweet green is actually the first salad making company.

That's also a tech company. So maybe you can tell us a bit about kind of the data driven approach there and why [00:08:00] they even wanted a data science team. 

ravin: Yeah. what compelled me to sweet green? If we talk about the company itself, at the time, they wanted to do two things. One is, if you really squint at a squeakereen, like you just squint your eyes really hard, it actually is a manufacturing plant, right?

They get, they, just like SpaceX, they get material in. SpaceX get metal, but sweetgreen gets kale. And then, there are people that are skilled at putting those things together. and then it goes out the door in an assembly to a customer. at Sweetgreen we had to have the right stuff at the right time, and had to get people that could build it, and that looked very much like manufacturing.

And then at Sweetgreen in particular, like you said, is very tech forward,it was a really great time in my life, where I got to use, the AWS stack and a lot of different technologies in the cloud, to, To affect the real world. it was just, it was right at that intersection. Very similar to SpaceX plus getting a lot of free salads at sweet green.

that was a really great perk. 

hugo: Yo guacamole greens for the win. If anybody hasn't been to sweet green, check it out. And also like the logistics and supply [00:09:00] chain stuff was clearly gnarly. But I've said to you before, when I was. Eating a lot of sweet green. this is relevant. I'm not just, state building and I was going to the sweet green on Broadway and like 27th in Manhattan or whatever.

And you needed to use the app. There were hundreds of, It was like a music festival there, dude, right? Like you'd like, they really needed to figure out how to get as many customers in and get them their lunch as possible. serious concerns about if people wait too long, there'll be churn. So these types of things, in addition to having a good mobile app, I think allowed them to scale incredibly in Manhattan.

ravin: Yeah, it's just New York was the, Sweden talks about markets where there's like LA, Chicago, New York. New York was just a whole nother level. I remember going to New York and waiting in line. It was like an hour and a half just to get through. People waited the whole time through. It was amazing.

So, 

hugo: what's up at Google, dude? What, what are you working on? what teams do you work with? And what's your, what's your general mandate and purview? Thank you. 

ravin: Yeah. so one is a [00:10:00] lot has happened in Google, even more than I know. the company is, has always been full on steam ahead with, with generator technology or AI, as Sundar likes to say.

but me in particular, I work right in the intersection of a Google deep mind and research and then Google labs where we're deploying products to, to, folks. So if you folks want to see some of these and ask questions, there's, labs. google. com has all of the public experiments, but it's just exactly what we're saying.

a little bit at the beginning. It's there's so much technology that's being developed. That's just so fascinating. what is the right way to get it into people's hands? So it's not just some mathematics running on some server in the cloud, but really something that,improving your day to day experience, organizing information for you, all the wonderful things that people have come to love Google for.

hugo: Very cool. So I've linked to labs. google. com in the chat and I'll do so in the show notes. once again, just reminding people here, if you have any questions, please do put them in the chat and feel free to introduce yourself and let us know,what your interests are. Um,I love, and this is one of the reasons I wanted to chat with you about all [00:11:00] this stuff that you're working at Labs, but at the intersection of Deep Mind, as well, and also working on research, the wonderful triangle of like research applied AI and product.

So I'm just wondering what excites you the most about working at this intersection? 

ravin: I think,I saw a post from you about AI applications and I remember a lot of it resonated with me, but it's,the. The key themes are, again, common echoes from my past. The math is changing every day. It's just new equations or formulas or methods that are just pushing the boundary one step at a time.

But the second part is these steps are occurring quite rapidly. It's not like one new innovation today and another one in a year. It's every step of the way. even half day at this point, sometimes every day, somebody's come up with something interesting and new. And we're just moving quickly in a lot of different directions.

And it's the interesting thing is it's not just that you just [00:12:00] stick on a path and you're just getting, incrementally better at the same thing. Let's just say, let's just say text generation, but we're noticing all these new things that are just unlocking. that wasn't possible even six months ago.

you just, you're like, Oh wow, this now we can now suddenly do this new thing and it might be useful to these new people. in this new way, how can we put it together in a way that they can actually get their hands on it. That rapid development, is just fascinating to me. 

hugo: And who is the end user or the goal?

Who are you building products 

ravin: for? So at Google, we're building products for a lot of people. enterprise. There's a lot of things that go to folks that are running businesses. And I think we're going to talk about a couple of those things later, but some of them are going just to individuals like you, your mother, Hugo, she could use, like a Gemini app or notebook or things like that.

So it's just for whoever it makes. sense to some of it is developers. So while I know you're a big data science guy, and probably folks in the chat as well, some of these are very data science specific. So there's features that go straight into CoLab or, experimental data science agents that are meant [00:13:00] strictly, not strictly, I shouldn't say that, but targeted towards, data minded folks or folks that wanna break into data.

you just have to look at each one case by case. And if you want, we can go through specific examples. But, for whoever, it'll just make the most sense for. 

hugo: I would love that actually. Because, as you mentioned, we, we've got Gemini, we've got Gemma, we've got Notebook LM, we have,multi modal, we've got vision models, we've got experimental data scientists in Colab notebooks and this type of stuff.

So maybe, yeah, maybe we could do some principal component analysis or talk about like the top five or something like that, that the team is working on and you're putting out there. 

ravin: Okay. So going through the ones that I've personally worked on, so then I can talk about it, With a bit more depth and clarity.

like a big developer focused one that we worked on, in labs was AI studio and, the Gemini API. So in that case, we were really thinking about how could, the Gemini model be exposed to developers in a way that they could build their own applications. It doesn't have to have Google branding on it.

It doesn't have to say it's Google, but just like powering whatever they needed to get done. 

hugo: And [00:14:00] just to be clear, Gemma is the open weight model, but Gemini is. The analog of,GPT 4. 0 or something like that. 

ravin: Exactly. Gemini is the massive model. it's one that has multimodal understanding and all sorts of, cool features like that.

and it is a big model and you access that through the API. Gemma is an open weights model that you can download the weights and use, use yourself on your own local device. I actually use both, day to day. I use Gemini at work a lot,when I'm using the clusters and things like that and I actually use Gemma on my personal device, when I'm in an airplane and I don't have access to the internet and I still need a code completion agent, for instance, we have a code Gemma variant.

you're right, in that space, we got a lot of things that are targeted for people that are more technical, hands on, want to get their hands dirty with the weights or code, they want to go and build their own thing. And just to be clear as well, Correct me if I'm wrong, Gemini has a free tier?

Gemini does have a free tier. You can go to aistudio. google. com and it's something like three clicks to get an API key. That's so cool. And look, 

hugo: just to be clear, I'm not paid by Google. I have no [00:15:00] affiliation with Google. My friend Ravin works there, but go and check out Gemini's free tier. I'm actually incredibly surprised that, and I can, I will name names.

They're huge. Like Anthropic doesn't have a free tier. Mistral now has a free tier. OpenAI doesn't have a free tier. The reason I'm interested in this is as an educator, I go to conferences and teach stuff. And the fact that to use open AI, I need to get learners to put in a credit card is ridiculous.

So props to you and Mistral for having free tiers. it's stupid. I'm I'll be on the record of saying that give me like 60 cents of free credits for students or something. so also I,I don't know if you've used, I don't know if you've used MLCChat. I have not. This is an app which, you can see, like recently when Llama 3.

2, I just put Llama 3. 2 3B on my phone the day it was released and was able to chat with it on my phone. Before that I had Gemma two, two B, right? so I'm sorry, I've interrupted the ability to be able, this is amazing, man, to be able to take open weight models and put them on my cell.

so now we're talking about [00:16:00] the actual models. Do you want to maybe give us a rundown of the language models and vision models and even how many parameters they are and that type of stuff, get a bit technical. 

ravin: There's some, Oh man, if we're going to be sitting here all day for talking about models, let me, double check.

I have my latest understanding, but there, let's see, sorry, this may 

hugo: be going a bit too deep. So I do want some point and hear about the other things you're working on, of course. 

ravin: No, this is great. Let's do it. There's, I would, there's one, there's maybe one fine point I could put here, which is great is There are a lot of everything's called large language model now, or there are many large language models now, but the way I tend to segment it into my head, are there ones that are so big, they have to run on the cloud. there's no possible way you could run them on your own, even, bigger than that.

So you got to go to an API and you got to get them. But these models tend to have, these days have audio understanding, vision understanding. they're really good at reasoning. if I need something that requires all of that, like heavy horsepower, I think like semi truck level type of stuff, then, I got, I'm going to go to the cloud and I'm going to use the latest and greatest, not only just in models, but you have to remember there's an entire stack under those [00:17:00] models.

there isn't, there's a heavily optimized, server cluster that's running, custom software that really just makes it a smooth, and easy experience, even though you're doing crazy amounts of crazy computation. But, every now and then I don't need all of that horsepower. and in that case, I use a smaller model, like a Gemma And that lets me run just locally on, on device.

I don't have to worry about the internet connections or HTTP requests. I can, sometimes I like just going to the park and doing my work there, but I don't have an internet connection. Maybe you're on your phone and you need it there. these, smaller models are really good for those things.

Now, they don't quite have the same reasoning capabilities. Not all of them have the multi modal capability quite yet. But for text to text tasks, or especially for coding, because, coding is a text to text, highly constrained space. Then I'll use this, one of these, smaller models instead.

That's my personal philosophy. 

hugo: So we've touched on Gemini on Gemma. If you want to say anything else about those, please do. But otherwise we can talk about some of the other products you've worked on and are working on. 

ravin: yeah, let's switch to that because it's, while we were [00:18:00] talking about how people use things, these developer products are really good for people that know code or can take text, LM outputs and put them into your own thing.

But not everyone is, is like that, there's most of the world are not developers. Another part of my work at Google is packaging these things into, user interfaces and things that make it easy For folks to access and they don't have to worry about the underlying code You want to jump into those hugo?

I'd love to Okay, so I think hugo dropped the link to labs. google. com. You'll see a bunch of them there but the one that you know I'm most i'm frankly the most excited about personally right now because and I have a bias because i'm working on it but it's notebook LM audio overviews. So if folks have seen it or tried it when you go to Notebook LM, you don't see anything that screams, AI or LLM or what not to you.

it's actually meant, the interface much more is a, as the name implies, like a notebook. And you put your sources in, and you put your stuff in, and your ideas and your thoughts. But now, instead of just having them, in a, let's say a physical notebook, we use Gemini to help you chat with your notes.

That [00:19:00] was, the first, Core feature of NotebookLM and the second part now is there's a little button you can hit and it'll actually just generate, a podcast for you, in about a couple minutes, using Gemini and other technology to take all your sources and, create an audio format. And the reason I love it so much is it, I think it really highlights this transformative element of the new generative,capabilities that we have.

Our computers can now take things that are in text or video or images and turn them into text, audio, text or audio right now. Or, and it, as a human, it just, I've, sometimes I have a really hard time reading research papers, for instance, just pouring through the details. But when I put it in a notebook and I hit the audio reviews, I get this sort of, explain like I'm five.

Summary, and that just helps me understand the concepts more. So when I need to go back and actually read the paper, I'm just that many more steps ahead than I used to be. To me, it feels magical, even though I know how the whole thing works. 

hugo: Absolutely. And I do think, to your point, products like Notebook [00:20:00] LM to be able to make these types of models more accessible is fantastic.

We can drill deeper into notebook LM or talk about a few more other products and then pivot back. Why don't you tell us about a few other things. 

ravin: Okay. the challenge I'm having, I'm going to need to think through, because some of these products are gonna be coming out soon, so I can't talk about them just yet, but on our next podcast, we can come back to them.

Another product that I'm really excited about, especially from labs. I'm trying to think about the ones that are public, but, this is actually one that isn't as, there's one called breadboard and I'm going to, I'm going to look, find the GitHub link for you. Great. I'm typing away at my second screen to get, to make sure I'm giving you the latest and greatest. Um,there's an app called Breadboard, or not an app, it's a visual programming language that, that another team within my, within Labs is trying, and it's a way for folks that are not It's a way for folks to put together LLM applications, in a visual block sort of coding style, rather than just straight code.

And the reason I like it is, I've had a lot of, I've had a lot of parents actually, because, now that I'm 30 or 35 and whatnot, and [00:21:00] they have kids, they're asking, how can their kids get into generative AI? And how can they learn, and they see it as just the next wave of technology, and they want their children to, be able to use it for their own work, be able to get work, use it in their learning.

and it's a bit much to say, oh, you gotta go learn Python to like really understand what's going on. spin up a conda environment and get an API key. still it's too much for an 8 year old or maybe 9 year old. but Breadboard is this in between where it allows Kids to or any adults as well, but to put together LLM applications using blocks So they don't have to understand all the underlying code details or whatnot But they can get a block that says generate text and then they get text And they make it put another block that does another thing and that gets them Into LLM applications while having to dive full bore into Computer science background with LLM understanding and tokens and all this sort of stuff.

And I also, being an open source person, just appreciate the open source nature of the whole thing. 

hugo: Yeah, absolutely. And I do think, a lot of the time, the mental model of Lego blocks and building blocks is actually incredibly important across the board for the type of [00:22:00] work we do, it always has been in data science and machine learning, and data engineering, 

but even more so. With generative AI, because we're really coming back. What becomes very important with generative AI is composability. Again, it always has been important, but figuring out how to build these systems, essentially, which consists of a variety of generative quote unquote microservices or something like that, and connecting them up to build a more robust system that can do a lot of multimodal things and so on.

ravin: Yeah, I just fully agree with you. There's just, it's another block in the, in this whole toolbox that we've had, so I do want 

hugo: to continue speaking about products and UIs for non technical people. But before that, we've got a great question in the chat from Noble. Noble Ackerson, great question from a great name.

Um,I love this question cause I struggle with this. And I don't think there's much science to it, to be honest yet, but the question is, what is the rubric or framework to determine which model to choose for a given product objective? I'm surprised decisions are often hand [00:23:00] wavy, vibe based at this point in Gen AI maturity.

Now, I'm not going to answer that. I want your thoughts on that. what I will say is that, This isn't new machine learning has been like that. Like we're getting to more scientific machine learning in a lot of ways, but machine learning and data science have often felt like. incantations of some sort as well, right?

ravin: Yeah, I fully agree with you. One is I say, I think you, I've seen that you're going to have an LLM, applications course coming out pretty soon. Exactly. Yeah. On 

hugo: Maven. 

ravin: I, yeah, I, I'm going to give a pitch. I've always enjoyed, I haven't seen, okay. While we talk about caveats, I've not seen Hugo's course or any of the course material, but I've seen a lot of Hugo's prior material, with, Statistics and all that and I know his course is going to be good.

I have a suspicion you're going to answer this question in your course and the folks in your chat should go watch that. I'm also, to your point, I get this question so much that I wrote pieces of it into my, into this guidebook I think we're going to talk about. I'm going to write more into the guidebook for literally people that just need guidance like this.

Um,the blue, the answer I'll give you the quick one is, I [00:24:00] think people jump way too fast to trying to pick a model. When they come, they're like, I want to do this thing. I want to build this new application that does all these things. Which model do I need? do I need a GPT? Do I need a Gemini? Do I need a Llama?

they get really hung up on that. I really think what people need to do, actually advise people to do, is you should go, what are the evaluations I need? to check my thing actually works. forget the model, forget the code, just what are the inputs that are going to go into my system?

Is it text, image, what type, what's the formatting, are there typos, who's providing it, and what are the outputs that I need? And come up with 10 examples. actually I'm gonna say come up with 20 examples. 10 will be like the good case where everything works perfectly. And then 10 will be like what, malformed inputs, things that are wrong, things that could come in but not be quite correct.

And once you have that, then you can go to the models and say, Let me try a Gemini and see if it does the thing I need it to do. Does it? Okay, awesome. I know this is at least possible with the latest frontier tech. it's going to be expensive. I'll have to use the internet potentially. Or, I don't want to say it's going to be expensive, but it's going to [00:25:00] require a big model.

Okay, but at least we know it's possible with Gen AI today. But now let me actually creep down the ladder and go to a Gemma 9b. does it still work? Is my thing still working? If it works in Gemma 9b, then awesome. Now you've got two models you can pick from. And you've got all the choices of how you want to do things.

But it's that, I think that's the principled way to work with Gen AI. I'm product building right now. Build your evaluation set first and think about the model second. But Hugo, actually, I want to kick it back to you because I haven't heard your thoughts on this and I want to get a preview. what do you think?

what would your answer be? Yeah, so I 

hugo: totally agree. I think You want to think about translating whatever business challenge you're trying to solve, into an evaluation framework. on, but on the other side of it, you want to think about the data you have as well. And that may involve user conversations, user interactions.

Let's say it's customer sort pre customer support, previous interactions with customer support. Let's say it's email outreach. Then you want to make sure, exactly what data you have there. And once you have an eval framework [00:26:00] and knowledge of the data you'll be using, and then use that to generate synthetic data, which I think is going to be increasingly more and more important, then figure out which, which model you're going to use.

And to be honest, I eval framework, incredibly important. data, I honestly think data over methods, even more so than it used to be with ML, right? Like you may be able to switch out a model to get better performance, but if you have higher fidelity, more high signal data, either fine tuning on it or using in context learning or using it to do what some people call prompt engineering, what I call prompt alchemy.

I think, Is incredibly important, but that's always been the case with ML as well. Like when people would say, should I use a neural network to do this? xgboost, right? Yeah, exactly. Or logistic regression. It's like, why don't you start with a baseline model? And actually I like that.

You suggest try the state of the art model first. To see what performance you get here. Cause in ML, I'd always say [00:27:00] do the opposite. If you're doing binary classification, I'd say don't start with machine learning, start with a majority classifier, and then see what lift machine learning gives you. Like you'll never actually deploy a majority classifier, right?

Yeah. It's a baseline that allows you then to see what lift you get from different machine learning models. and then you need to take into account, it's more obvious with generative AI, The cost we're talking about when we're, pinging vendor APIs at scale, but in machine learning, if you're using sophisticated models and you, that maybe aren't interpretable, these types of things, all of these things are costs in a variety of ways.

So maybe, it was a decade ago that we're all building logistic regression models that really help businesses, do the predictions they needed. Some people called it AI then, right? Because they, they had to, they're 

ravin: going to market. Yeah. 

hugo: Yeah, but I don't think the details have changed, right?

And we can get to that, but having robust evals. and knowing what your data is and knowing what your [00:28:00] prompts are. And this is, our mutual friend, Hamel has a blog post called, fuck you show me the prompt and I'm quoting him explicitly and I approve of his cussing there cause we need it.

Cause at that point, some, and still so many frameworks you're trying to build an information retrieval system or something, and the framework doesn't even allow you to introspect to what the prompt is so you can barely look at your data. So solving all of these things is important. to the question about evaluations as well, though, I do want to.

Split think about a graded approach to evaluation. So actually we did a lightning talk, which I'll link to in the show notes, for our, Maven course on building, LLM powered applications for software developers and data scientists. And we went through an example where let's say you're a recruiter.

and you're consuming LinkedIn profiles and you want to automate emails to, to potential good candidates, right? now at a basic LLM level, you can see, wait, did it extract the correct information in a structured form that allows me to populate the, these [00:29:00] emails?

That's cool. you need that, right? But that's an LLM micro eval. then you can sit down. With the person who used to write the emails and they should be part of this conversation. Maybe you should put the emails in a spreadsheet in a Google sheet, which allows them to plus one minus one, like thumbs up, thumbs down them.

and we see that in a lot of ChatGPT for example, you can plus one minus one, those types of things that allows them to label whether these emails are actually appropriate or not. I think another point there is that, there's usually a human doing these things beforehand, right?

Most of the things we build ML and AI for are things that humans do that don't scale. So we want to involve computation to allow them to scale. So we've got, the first one is, Did the LLM extract the right information? The second level is, do these emails look like what the human would approve of? But this still, all of these things are important.

The real question though, is do you hire good candidates? Okay. So do you get responses to emails? Do you [00:30:00] get people in the interview flow? This is a long term goal. and a lagging indicator in a lot of ways, but it is your actual goal. So you want to tie the eval. Back to that final value of, do you actually get good candidates?

And that, I think that really, what that speaks to is systems thinking, like thinking through the entire, we're not just building models or using models. We're building software systems that automate certain things. and then do they deliver the actual business value? So we want that macro eval, and we want to think about perhaps all these levels of evaluation as well in that example, but I'm interested if that resonates with the way you think, 

ravin: yeah, I was going to say, I just actually love the way that you you put that, I think, I won't put you in this camp, but maybe I'll put myself in.

It's someone that's interested in the math, right? It gets really easy to be myopic on what's the latest model or what's the, parameter size and things like that. But especially when you're thinking about applications, like you just mentioned, what actually, none of that actually matters. Truly, what matters is, did we get the, did we get the candidate at the end?

and just the way you unfolded that as well,this [00:31:00] application is acting in a human system where there's signals coming in from all over the place, right? It's not just the text and texting out to the model, but there's, the responses, there's the people that wrote the emails, there's the actual end goal of whether the, you got the right person or not.

And these things all support,The janitor of model or LLM or whatever you put in the middle. it's not that the model should outshine, frankly, any of those other factors. I 

hugo: also think this is a great way to manage expectations with business lead. I'm look executives, business leaders have so many incoming pressures to use AI do like they've got Shareholders, board members, competitive landscape, what shareholder board member, what's our AI strategy?

We need it. And so they come to you and they're like, what's our AI strategy. And it's our job in a lot of ways to understand their incentives and look for the meaning under the words, right? Because They want to keep generating business value, right? and they have an obligation to shareholders to do that as well for better or for worse.

a legal [00:32:00] obligation. What they're really saying is that they want to keep up with modern technology, but they don't want to start. Messing stuff up as well. So if you can show them that a logistic regression actually delivers more business value, and you can brand that as some form of AI as well, and you can maybe add a bit of generative flavor at the end to certain things.

That's what they want. They're not necessarily saying we need to use AI across, across the board. So if you can speak to their business needs as much as possible and understand their real needs, I think that very much, and I honestly think I tell that to a lot of people because everyone's freaked right.

About needing to use AI at the moment. So I think managing those expectations within an org, understanding the external pressures, is a way to reset expectations and get the work done. 

ravin: Yeah, I think that's, I think you hit the head on the nail right there. a lot of people are trying to figure out how they can put this in.

I would say that the counter risk, which you're pointing a good way, is a lot of people feel like they have [00:33:00] to force generative AI into their company or business just ham handedly somewhere in a non principled way. But, the framework you just suggested is a great way to think about, where is it actually going to make sense in the context of the thing that I'm already, the thing I'm doing or the company that I'm working at or the org that you're, Happening to lead.

hugo: Yeah. And also let's say you have a customer service agent first deploy it internally to your customer service people before deploying it externally, please. because otherwise you'll see all the nonsense that we've seen. Secondly, you don't necessarily need to have one big LLM call and stuff the prompt and YOLO and hope it, you know, because then you end up with stuff which, your former company would probably like whereby someone goes to.

Ford's website or whatever, and asks, what's the best car. And it says a Tesla, right? so you can avoid those types of things. You, if you're building these systems, get them to respect your business logic. Don't let them do things outside your business logic. And then perhaps add a bit [00:34:00] of generative flavor at the end as well.

Like you can literally build an, a pre LLM style chat bot that,maybe uses, LLMs to understand the intent of the user as opposed to old school natural language understanding, then embed it in business logic, then get the response, then add some generative flavor. Now that's a lot more work, but it has a lot less failure mode.

So I think now we're seeing more ways to embed LLMs in production application without letting them go absolutely wild. so I do want to jump into firstly, I always forget the difference between jealousy and envy, but I've, I experienced one of those towards you, I, and I'm so happy for you that you get, no, but that you get to work across, you get to work on foundation models and all the nitty gritty of training and fine tuning and all of these things.

RLHF or, DPO, all of these things all the way through to building user,facing products. And I think that's [00:35:00] probably quite rare these days, but what type of skill set do you need? if someone wanted to get into this line of work, like it seems like you span the gamut, right? 

ravin: Yeah, I think.

I got lucky or maybe I became this generalist by accident, but I think maybe there's a I was You don't need all these skills, but I'd say maybe Segment yourself into where you would like to be and then get the skills closer to that But one is if you want to be closer to the application builder that hugo is talking about.

Then you're gonna have to be The titles there would be like program manager or product manager. And a lot of those skills, frankly, are not technical. they're human understanding skills. Just as Hugo mentioned, it's understanding businesses, being able to empathize with the executive and the customer service rep and figure out how to, tie their processes together because with that empathy, then you'll understand how the technical solutions can solve their pain.

that's a, I'd say that's the application developer,spiel that Hugo just gave for the [00:36:00] last 10 minutes. the other one, which, which is a little bit more concrete, and people tend to go to is if you want to be an AI researcher, then you're going to have to get a lot of nitty gritty math skills, right?

You're going to have to understand how JAX or PyTorch works, put together models. train a couple on your own, understand gradient descent and derivatives. and how gradient, how those two come to come together, read float values from disk and understand what quantization means. And that's a key tool in the,AI developers, toolbox these days.

I think you literally, you don't necessarily don't have to do one or the other, like these things are now blending together. But I would tend to think which one you'd like to trend towards, trend towards first and then, get a strong foothold in one of those and then use, and then bridge the gap if you'd like to then bridge the gap.

So are we also at a 

hugo: different point where we can.

We can build systems and use models without training them now, which is incredible. Now, this isn't quite new though. I think people are like, wow, vendor APIs. Now people forget all the amazing work hugging face did over a long period of time to [00:37:00] allow us to do that with inference endpoints as well. and not just hugging face, right?

It's a entire community. but so I've encountered people who can build generative AI applications. Who don't even know about neural networks, can't fine tune that type of stuff. They ping APIs, they understand the data mindset, which as we spoke to before, evaluations and looking at your data, incredibly important.

do you see for people building applications that, knowing about how to train, knowing about the math behind neural networks, is that still needed? Or can we get can we import software engineers into this entire landscape now? 

ravin: Yeah, I think so. There's a, you're right. There's this third category that's in between that I should have put a finer point on.

I'm glad you actually came back to this point. I'll just be frank with you. I don't entirely get how pieces of the internet work these days, but I can import requests from Python and get my, get my, stuff off the internet in, into code. And a similar thing I think is happening, is definitely happening in the, this modeling landscape.

I think, yeah. I would say a big shift from when you and I were doing data science in the [00:38:00] 2010s to now is something you just hit on a whole bunch, is that, at that time you had to download, a scikit learn or something and specify a model and, and at least see some math, even if you were just an application developer, but now in the LLM landscape in the 2020s, like you said, Hugging Face and others, Google and OpenAI and whatnot, They have these really nice APIs where you don't, A, you don't have to train the model yourself.

And actually for a lot of reasons we could talk about separately. I think, for large models people are never going to train these models themselves. It's too expensive and whatnot. You're just going to take a pre trained foundational model and use it. that is the rational thing to do now.

Rather than what we used to do, Hugo, which is, initialize the Bayesian model and then train it or xgboost model and train it from scratch. but yeah, these people, if you want to be, let's call an AI developer, You don't have to know what's going on under the hood, in a great amount of detail.

You can just use an API, and some prompt alchemy as you, so funnily put, and understand, get, I think for now you're going to get a pretty good initial version of the application. I do think that where I'm seeing people get stuck though is if you can't get there [00:39:00] with just natural language instruction, then if you don't understand what top P sampling is or top K or temperature, or if you don't super understand how tokenization works, you.

These layers of abstraction start breaking around the edges and you'll get into, you could get into trouble. So there's a lot of things now I think you could build without having to understand any of these details. but if you're really going to get into it, I would still advise that you, you understand the basics just so when that layer of abstraction breaks, you know where to go and how to, So yeah, not necessary, perhaps, but recommended is my advice.

We might move to a place real quick where we don't need this at all anymore, maybe in two or three years, but right at this moment. Just understand a little bit. 

hugo: So I'll say something and then I'll ask something. I don't know if I've ever said this publicly. I called this podcast vanishing gradients for numerous reasons.

One, and it's actually not the, not the most important reason, but one reason is I think you can work with deep learning systems. This was pre these types of [00:40:00] API calls as well. I think you can build deep learning systems. Without knowing multivariate calculus, but once you get a vanishing gradient problem, maybe you need to know about multivariate calculus.

So I was thinking about the abstraction layers, necessary there. My question is though, I love that. You said we may get to a point where there's things aren't necessary. Let's use the car analogy, right? I don't need to. When I drive a car, I don't need to understand the internal combustion engine.

If I'm a formula one race car driver, I need to know a few more things, right? So what will it take to get us to a point where people can use and build these systems without knowing the calculus or multivariate calculus and which involves linear algebra for those. 

ravin: I, this is such a good question. Cause I, I'm going to, I'm going to pull some threads from an earlier discussion that we had,so being a mechanical engineer, I know exactly how an internal combustion engine works, but I don't need that skills day to day.

My mom doesn't know how an internal combustion works, but she still drives the same car I do. So I think if I take my original analogy and then actually switch it back to yours [00:41:00] is, yeah, she doesn't need to know how a car works. but when there's an issue, she has someone she can go to. So there is a function, which is a mechanic, and there's an entire ecosystem built that when she runs into an issue.

She can have someone debug what's going on. I don't, in LLMs right now, I don't necessarily know that there's there's a go, and every organization has that go to person that's like their IT person for. Actually, maybe I'll take computers. A lot of people use computers and have no idea how they work at their day to day jobs.

When they run into an issue, they run to an IT person, and then the IT person solves it for them. I don't think we have that for, with every technology, I think we're always going to run into issues, whether it's cars or computers. But other technologies have had that job class of people that figure it out for them.

We just don't have, I don't think we have that class of people yet. So I'd say that's one thing that needs to be built. Just a restaurant in 1950 did not have an IT department. they straight up did not have one. But Sweetgreen now had an IT department because we used computers, even though it was a restaurant company.

Similarly, I feel like a lot of businesses probably are going to end [00:42:00] up having consultancies, or, AI mechanics as a job function. I think the second one is, if we go back to cars again, the first car, the early versions of cars had hand cranks and you had to, you learned that an engine had to rotate because you had to do it yourself and they didn't have electric starters.

But once they got electric starters, a lot of people don't even understand how a car starts these days. They just turn, push a button, frankly, for a mechanical car. Dude, there are no keys anymore. There's no, it's not even keys. Yeah, it's not even a thing. So I think, the car manufacturers got better at building layers of abstraction over, their engine.

The engine didn't change, but they added a new component that made it easier. I think the same thing is happening also with, with the generative technologies, right? you. I don't know what it is because if I knew what it is, I would have a startup maybe or something.but there's going to be additional technologies built like,electric starters that make it that abstract the way the LM in a way that make it easier for people to use and not have to worry about how a crankshaft turns over on a cold [00:43:00] start.

and there are a lot of people trying a lot of different abstraction layers right now to figure it out. But I think it feels like in the next couple of years we're going to end up with two or three other key pieces of technology that just hide away these things. when you run into an issue on an LLM, it can self correct itself rather than you having to debug it yourself.

You were just talking about this earlier, alluding to you build your application and you use your initial test cases to figure out the inputs and the outputs. But, There could be a version of LMs, and we're starting to see with agentic frameworks, where the agent itself figures out that it did something wrong, and it fixes it even before you know what's going on.

stuff like that is just incredible to me.

hugo: you had to say the word agent, didn't you? 

ravin: I did. I, I just, it's sorry. 

hugo: this isn't, this is not a LinkedIn post, bro. I know. no,I'm like all jokes aside, I'm very excited about agents and agentic frameworks. maybe you could tell us a bit about how you feel about agents, what's happening, whatever you can say at Google Labs with respect to agents and so on.

ravin: yeah, to Hugo's point, I think agents are a little bit overhyped right now, so I don't want to [00:44:00] steer the whole podcast towards this thing. But it's just this idea that, instead of just making one call with an LM, it's able to act on its own outputs. in a regular, let's say, Gen AI call, you pass in some text, tell me a recipe about eggs, and it gives you, let's say, some text back.

In an agentic framework, the LM can say, actually, you know what, let me actually revise this, because I could make it better. When I say, I'm anthropomorphic. Morphizing a bit, but it, the thing can act on its own outputs and then it can go do a bunch of stuff. And now we're seeing this in all sorts of areas.

You're seeing it in code, you're seeing in text generation, you're seeing it in,in a lot of different products. it's a really neat, it's just a really neat capability that some of these systems have because as we were just talking about earlier, instead of you having to write a bunch of code yourself to put things together, this thing is smart enough to do it, yeah.

The analogy I like using is maybe LLMs right now are like a two year old toddler. You have to specify every step to build a peanut butter sandwich. And you have to be like, Oh, put the peanut butter on. Oh, the knife fell. Maybe you should go pick it up. And, you talk to a child. You have to instruct a lot to get it, the child, to be able to [00:45:00] complete a task.

if I was an adult and I'm dropping, I'm making a peanut butter sandwich. You just expect me to get it done and fix all the mistakes myself. And that, is, a bit of what's happening in, let's say, an agentic project, if I have a rough analogy, but yeah, super fast. We should, either, either, we can be, I can be back as a guest in a year or you should get a guest in a year that can talk about this.

Cause it is, a lot of innovation that's happening all altogether. 

hugo: Absolutely. And the power of agentic approaches to ping APIs and gather information and take actions as well. I think there's some pretty weird, like YOLO danger vibes happening where, agentic approaches with Go and buy me something, right?

Here's my credit card information. And you're like, okay, we need to, just imagine a hallucination in the middle of that. but I am, increasingly interested in the types of products you're building and interfaces you're interested in. last time we spoke, you mentioned, I just want to keep returning to real world value as well though.

and you mentioned, a bakery automation project. And I thought that was a fascinating example of using AI to streamline work [00:46:00] flows. So maybe you could tell us a bit about that. 

ravin: Yeah, this is a project I worked on with a Gemma model in, in particular. So actually it's very analogous to things you were talking, we were talking about at sweet green or, The email assistant that you had just brought up for recruiting.

the mindset I had here and I'd say, I don't know how to say this without, spurring more jealousy or envy in you, but,I'm really lucky. and I think a lot of AI builders are really lucky that we get access to insane amounts of compute,at Google, or even at home

I have a 4090 RTX. So I got a lot of VRAM. But, there's a lot of people, like you said, that are not living in AI world day to day. they're not in this day to day. They're literally at just a bakery. they run a bakery or, some sort of other small business. but they're just getting a lot of emails and, when, can I get this kind of cake?

Can you come at this time? what products do you offer? They're getting all this information in through text. And for them to go in and answer a lot of these questions, they have to take time away from doing the thing that. People pay for . People don't pay a bakery to answer emails, right? They pay, they usually pay a bakery for the baked goods.

and I don't assume a lot [00:47:00] of people open a bakery to say, You know what I love getting? I love getting random emails for customers. that's, I'm going to start a bakery to get as much random email as possible. They usually start, I assume they start a bakery because they have some love of baking and want to produce things.

So now, you know, but you have this bakery started. You're getting all this free text, email coming in for what your products are, what you can offer and things like that. You're not working at a Google where youv've got gazillion hours of compute. And you're not, you don't have an AI engineer on staff. How can you use this technology to improve and make your life easier for what you're doing, which is baking goods.

and in that, so in that I made this tutorial that it's like, if I was someone who was working in a bakery, how would I access these systems? What kind of compute would I have? And what would I need? And in that framework, it was, okay, let's build a model that uses, uses Gemma because that's a reasonably sized model that's, cheaper to use than a large, a humongous model.

let's use a Colab interface because while it's not, it's an easier place to start for most folks to start programming and you've seen this in tutorials starting with a [00:48:00] Colab abstracts away a lot of the A lot of the environment details and whatnot. And let's just start with the emails that they're already getting.

Is it possible to make a bot that can transform your emails? And it turns out now that yeah, it is. With a Gemma model, it's easy to go in and do this sort of thing. for a, for somebody who has an afternoon at a small business to read, a tutorial, spin up a collab that, I've already created for them, take some of their emails.

I'm not even, I'm not even talking about hundreds of emails, thousands of emails, like big scale, take 10 or 20 of the emails that they just got this day and, and automatically transform that into, into an ordering application. and to that point about business value as well, this thing connects to a flask front end, which makes it easy to get these in a format that isn't just JSON, but something that, one, a bakery staff could read.

Do you have resources for this online now that we can share? I'll put it in the show notes. But if you type it in. There's a whole tutorial on YouTube where you can watch the video. It's 11 minutes long. but the Colab and everything else has been open sourced and whatnot. so it's one click for somebody now to go to a Colab and try it out on whatever they [00:49:00] happen to be doing.

But it's really, Hugo, like you said, it's that idea of business value. what is it that these businesses need? what is it that the skill sets they need? What skill sets would they have available to them to actually make this thing? Because, it's cool. Yeah, cool. LLMs can do your bakery stuff, but it requires you to have an AI engineer full time spending six months on a research project.

Probably a little bit out of reach for a bakery, right? can it be done? Yes. Is it feasible for somebody who's baking goods to take a six month sabbatical to learn AI? No. but we're just at this magical moment where there's enough, free technology out there. There's enough infrastructure like Colabs and, GPUs and the cloud in nice wrapped up interfaces that somebody who works, is a little bit technically minded or has a teenager at home that is learning Python, for them to spend six hours in, in this case, zero to 20 dollars and really get something that they can actually use in their business to make their lives easier.

Yeah. 

hugo: And to your point, the ability to hire AI engineers very much ties into business value as well. My hot take is that we go [00:50:00] back to data science. My hot take is that 2010, the 2010s was the decade of data science. And in fact, I'm extremely bullish on data powered software and data powered decision making in the medium to longterm.

I think that much is obvious. I think people probably think I'm bullish on it in the short term. I'm actually not, I actually think. Data science hasn't delivered its promise that it had in the 2010s. A lot of people hired a lot of headcount, significantly high salaries, and didn't see the return on investment.

I think FAANG companies did, and there's a short tail of other companies, but non digital native companies, definitively didn't. And I think unless as AI developers, We show to businesses how valuable it can be. It's going to be the same thing. There's an AI winter coming anyway. but I think it will be exacerbated if we don't figure out, especially with the kind of significantly high prices of headcount for AI engineers, you hire A team of a few [00:51:00] AI engineers, and we're already talking over a million dollars purely in head counter.

That's before insurance and all of these associated costs. Right? So that all I'm saying is that's another reason to be very, motivated about demonstrating business value to your business leaders as well. I do want to now move to thinking more about UX. Okay. And, interface design being critical for adoptions.

So And also products been critical for adoption. So yeah, I'll give two examples. the first one is close to your heart. Notebook LM has been around for some time, right? But it was someone chanced upon this podcast functionality and it blew up, in, in, in the cultural consciousness. Similarly, our chat GPT moment wasn't due to new transformer architectures and new models.

It was. Due to a thin wrapper around existing models, In order to try to get more users. I don't even think open AI expected it to blow up in the way it did. But my [00:52:00] point is that the reason we had our chat GPT moment at that point wasn't because of a new model. It was because of a product choice and a user experience.

So what is the role of. of UX,and products in, 

ravin: general adoption. I, so I will push back on that. I agree with you mostly, but I want to, there's a bit of a nuance I want to put on that last chat GPT thing, which is, you certainly the UX and putting it out there was a key part of it, but there was a conscious decision.

Somebody. And I haven't ever worked at OpenAI, so I don't know for sure. But there was a conscious decision somebody made that it's let's take these models and chat tune them in a way that, that would be good for the general consumer. I know within Google, I can tell you for sure this does happen.

I worked on,the first APIs that we released called TextBison. And I was part of the deliberate effort where we said, what is it, we've got this model, what is it, how could we tweak it just a little bit, like not a lot, but how could we salt it and tweak it just a little bit? So it speaks in a way that people really want To use.

So 

hugo: it takes both. I do agree. Sorry, my point wasn't that it wasn't. Somewhat intentional. My point is that there wasn't [00:53:00] the expectation that this would create such a huge moment where everyone started to scramble for generative AI. 

ravin: God, no, that's a good point. and I think you're right on that one. clearly the Chat GPT was the moment where, we realized that this was A lot of people realize, including me, frankly, that this isn't just the technology that's useful in a large company, but within a, but to the general public.

And I think that maybe that cuts to the crux of your question about products, Hugo. is it that,what makes it that a model is useful for researchers doing some, very esoteric tasks versus the general public, Who maybe your point isn't necessarily like an AI nerd, but just needs to do the day-to-day thing that they want to do.

Is that the Yeah. Okay. Absolutely. So one is, I'll say,I'll put it in two ways. the, I'm not a UX designer first off. so I'm gonna be speaking for a discipline that I've not worked in, even though I've had a thousand random titles. but I do work with a lot of great UX designers within Google, directly on my team for Notebook and other.

other products. And so there's, people that are formerly trained in UX design and [00:54:00] PhDs and things like that, that are thinking really hard about how is it that we can make, expose this technology in a way that really just works for people. so I'm going to be paraphrasing the things that I've learned from them, but again, not, I'm not the core person on the team that does this.

so I, alluding to what you said is we've got all this cool tech and you can write Python and whatnot to get to it. But in the context of the value that you're intending to derive from the system, what is the best interface to get you to the thing that you need?so with like with notebook LM, for instance, I specifically worked on the audio overview feature of that product.

and we just, we, when I was working on audio reviews, you realize something that You know, we, there's another team within labs that's already working on this thing called notebook LM. People put a lot of sources in there already. they've already invested some time into building their own chat notebooks.

we're at a point where we think you can have these podcast hosts talk to you about your stuff. So the interface there isn't even just like a, something on the screen, but it's like the interface is just, it's literally just audio. Like how cool would it be [00:55:00] to just listen to your content? and so the, the first thing there is like, Okay, we're getting away from the screen interface.

I think there's a lot of focus right now on screens, which I'm not saying is a bad thing, but things you have to look at on a phone or on a computer. But,the audio interface has been there for a bit with a lot of audio things, but we're, again, like I said earlier, there's this inflection point where, oh, there's a different type of audio experience we can get that we're now seeing from the new technology coming from the Gemini team.

Or the GDM team. So we built some early prototypes internally. quick evaluations like we talked about earlier. And we're like, oh, and some collabs. And we're like, oh, actually, this is pretty promising. you can, with some wiring and collabs, you can create audio that sounds really compelling.

But who are the type of people that are going to use this? And realize it could be anybody. It could be students, it could be researchers, it could be people planning Thanksgiving, if you're in the US or Canada, it could be anybody. okay, the clear interface for this is not some technical, thing at the moment, at least on this user group.

Maybe [00:56:00] it's just adding a button in notebook where you can just click one button and you get your audio out and like Honestly, if you go look at the interface for it for an audio overviews, there's really not much to it you literally click one button and you get what you need and honest That's what we settled on and we had all this powerful technology behind the back end that made that one button Really powerful and now you've seen a lot of people adopt it because I mean they weren't compelled by the button They were compelled by the audio The experience they got once they got that, that audio overview, and maybe to underscore the last thing to belie this point is, it doesn't require the super complicated UX to make these magical moments happening.

it was just, in this case, refining the technology to where this one button click made a lot of magic. happen.and so I think I don't, I think, maybe the other way I can put it is I think if you remember web design, I feel like there was this explosion where everyone realized they could put buttons on your website and you suddenly had these websites with a ton of buttons like all over the place, and they had all these links and everything like that.

but over time, I think we've seen website interfaces simplified to make them easier to [00:57:00] use. And I feel like we're in a similar trend with these AI systems. It's every, I think every application is trying to put all these buttons in or controls or things like that. but what we're willing to learning the new ways to simplify back to something that a lot of people can use or make sense, make sense with the technology that we're starting to get.

I just, it's hard to say, but there's so many cool things happening in this space that I want to have a podcast with you in six months again, or maybe a year where we follow up. Absolutely. Because I really think there's a lot of interface changes that are coming, in ways that we just areI'd love that question.

Yeah. I just, yeah, I just having a tough time even explaining how you've, seeing why. Yeah, 

hugo: and we were talking about the web recently, but, I do think

we're all very bullish on the web in, in, in the, late nineties, we had dot com bust. And then we had a huge amount of value created and captured through Google, YouTube, social media. I do think two tools that we forget a lot about are Blogger. I think [00:58:00] Blogger and then WordPress, then Squarespace, this trajectory of tools that allowed non technical people.

Sure, sometimes you got to do a code injection. That can be annoying. but allowing non technical people To create content on the website on, on, on the internet. and the other tool I love historically is optimizely, creating a front end user experience for people to run online experiments without having all the technical know how was absolutely incredible.

And I'm excited. I think notebook LM is a step in, in that direction, which I'm very excited about. I, I do want to have a slightly nuanced conversation now. I want to give a few pieces of evidence. Maybe tell a couple of stories. One is, in the early days of Twitter, my understanding is that the handle you could have was limited to a certain number of characters.

Maybe it was 12 or 15 characters or something like that. or that, or what you could put in your name or something like that. and that was eminently appropriate for the founding team of Twitter who were on average Anglo [00:59:00] Saxon and had relatively short names. when Twitter became popular in South Asia, there literally wasn't.

enough characters for people to spell their names, right? So this was a bias that then they had to reflect on and change, change in the end. the reason I bring this up is because we still have a lot of products in technology that are developed by a small amount of people, many of whom are in Silicon Valley.

I do think chat GPT is an interesting example of this where,In Australia, we have like American vibes in some ways, but ChatGPT is polite in a peculiarly American way, which doesn't resonate so much with Australians. I've been in Germany recently, as you know, I was there when we chatted a few weeks ago and my German friends are like, I'm can it just tell me I'm wrong and say, shut up and just get on with it and that type of stuff.

They're like, this American politeness is just getting to my head. Now, why this is relevant is, I think, I had a similar experience with Notebook LM when I generated a podcast, about [01:00:00] a friend's LinkedIn profile. it mentioned that she had done, like, all her college stuff in the same city she went to school in and how that seemed odd, but, whatever.

and that brought an American perspective that in America it's very common to, It's almost a ritual to travel somewhere else to go to college, right? and the fact she didn't do that was picked up on and that would the very American accents, made it strangely like non Australian and she's from Australia.

So it seemed perhaps, I know I'm doing a random walk here. but it seemed perhaps we could have. More appropriate local models for people. So I actually released a podcast. I do another podcast called high signal with, a professor from Stanford, the business school who he's Chilean and he's gone back to Chile.

And he's there are organizations there that perhaps they should have. Chilean large language models that they work with. So I'm wondering about your thoughts on the ability for people to build their own language models or having local region specific, business specific large language models, [01:01:00] or are we going to have Sam Altman's view of one model to rule the world?

ravin: you're giving a good point. Like people should be able to take some models and build them themselves. So an example of this is, on the Gemma side, we just released a version of Gemma that's fine tuned for Japanese with, with a bunch of Japanese folks and released it there. And then there's a, actually there's an active Kaggle competition right now to, take a Gemma model and fine tune it into your own, into whatever, I think, whatever language you happen to be, speaking or interested in or the community that you're trying to support.

I'll, there's a, there's another one where somebody took a Gemma model in India and they fine tuned it to be, If I need it specifically to one of the Indian languages, being Indian, there's a number of languages that are in that country. And I think it's, you hit a point in the head that it's all about enabling folks to have the tools to be able to create the experience that they need, for themselves.

And Gemma is like one example where that's already happened in a bunch of different ways, but it, this, sorry, this technology, should be accessible. And I [01:02:00] think it already is accessible to lots of folks who want to use it, themselves. Now, I think the second part, which is something you and I have always faced in the data science community in general, is just because we have a lot of open source tools, doesn't necessarily mean that everyone understands how to use them.

and it's like the knowledge is immediately accessible. So I think there's a, still a need for folks like you and I to go in and perform outreach in other countries and things like that, teaching them how to use even like a scikit learn, let alone an LLM. And that continues to be, need to be done.

It's not enough just to like, publish the technology and stuff out there. But yeah, I definitely agree. humanity is quite diverse and, we got to get models out there for everybody. 

hugo: Yeah, absolutely. And to your point, even like historically, I'm a huge fan of scikit learn, clearly.

But historically wrangling like the holy trinity of scikit learn numpy and pandas together hasn't been, it's just Oh, I've got my file of utility for utility functions and all of that. I, I do want to jump into a demo in a minute, but I've shared in the [01:03:00] chat, and we'll be doing a, a demo of, notebook LM for those that's currently watching.

I've shared a link in the chat. To your generative AI guidebook, which I love so much. I do. Want to talk about the page on personalized plans where you have, plans. so resources to study and read for the app developer and fine tuner, those who want the fundamentals society and,security specialists, research scientists, and then the executive.

so I'm just wondering if you could tell us a bit about these personas, what they need to know about AI and why you chose to put there. What you did. 

ravin: yeah, I think you. A lot of folks are now interested in this technology, and like you said, different folks have different questions or considerations.

The executive is going to be thinking about things a lot differently than, the, the bachelor's level student that's looking to the future of their career. and there's so much happening all at once, so as I was getting questions from many folks of many walks of life of how they could get into AI, [01:04:00] I realized the same answer doesn't work for everybody.

I'm not going to tell the executive to go start PyTorch and install Cuda Kernel. but I'm not going to go tell the student who's a second year sophomore to learn about organizational management and how AI can improve the efficiency of their business. it just doesn't make sense. On a side note, I wish 

hugo: I didn't have to tell anyone to install a CUDA 

ravin: Kernel.

I wish I didn't have to 

hugo: install 

A 

ravin: CUDA Kernel. Exactly. But, you and I have to in the line of work that we've chosen. so yeah, the idea behind those personas was, I think the other challenge is now there's so much happening with Gen AI all at once that there's, if you even just do a quick Google search or read the headlines of a newspaper, it's hard to understand which information you need to know.

Yep. depending on who you are. So the point of the guidebook as the name implies is to guide people to the right resources that are available. out in the internet. You can think of it like a sort of tour map, a tour map or something like that. Where I want to get to a certain place because I'm an executive.

I want to get to a certain place of understanding because [01:05:00] I'm a student or an AI developer. what are the set of resources? And I'm just trying to give him a quick, quick, highlight as to the key places on usually the internet, but also, sometimes off the internet and actual books, of what to read or what to look at, if you're this type of person, 

hugo: I like it.

I am interested. You've made different design choices to what. I'd make, and I love that. for example, in the executive section, you've told them to watch Andrej Karpathy's talk, actually two talks, but one is him building GPT from scratch in code spelled out. and I would not have decided that. So what was the design choice behind getting executives to watch building GPT in code from scratch?

ravin: To be fair, I need to, I need to go update this thing. I wrote this particular one about eight months ago, and it's, in this case, it's already just ancient stuff. at the time, the questions I was getting were from folks that were really curious about how these things work. And so I figured if you're an executive and you want to know exactly what's going on behind the scenes, this is the one video.

that really spells it [01:06:00] out. but to connect this back to an earlier conversation, I actually don't think it's worthwhile to, I don't think an executive now needs to understand how a transformer works, how attention works anymore. So I likely actually, I'm going to now be updating the sections that you called me out to removing it to a much more, higher level guide.

I don't, it's not important anymore to understand, how the code works under, under the hood. And my intention wasn't to call you out. It was no. But it's, it brings up something you and I've just been talking about the whole time, right? It's these. The space is moving fast. What you needed a year ago, isn't necessarily what you need to know.

Now we can talk 

hugo: about this after, but I'd love to chat with you about this and maybe even collaborate on your gen AI guidebook, give it like with respect to what exactly you need to know and that type of stuff. That'd be super 

ravin: fun, man. It would be really, actually, I totally invite that for, I'm going to pitch you again.

Hugo is a really good author. I'm actually excited to, To work with you directly on writing. This would be fantastic. 

hugo: I appreciate that, man. awesome. Well, look, far out. We've been talking for like 80 minutes, man. It's felt like three. but look, we're going to wrap up [01:07:00] in 10 or so. So why don't we jump into a demo?

I'd love to see, a tour of notebook LM from, from the man himself and, yeah, see what's 

ravin: up. Yeah. yeah, let's do it. I would be happy to, let me figure out how to share my screen. 

hugo: Yep, you should have permissions. 

ravin: Alright, are we seeing are we seeing Hugo, your beautiful face? Oh yeah, woah, oh. So what we're doing, let's say I want to learn more about this wonderful guy that I've, I just met. Yo, can I also 

hugo: just say this photo in the bottom row, I don't know why I'm drawing, you see the one of me in the bottom.

Yeah. That one I've been trying to get, I thought. I don't live in Europe, but the right to deletion from Google, this was taken on a work offsite when I had food poisoning. And I hate that photo and I hate the fact that it exists on the internet. anyway, why did I just draw attention to it?

Who knows? let's go on. 

ravin: All right,I think it's a snapping image of yourself if you ask me, but I can see why you would dislike it now that I [01:08:00] understand the context. Let's say I want, I've just met Hugo, for instance, not knowing that he had food poisoning in Europe, but I want to learn more about him, as a person.

What we can do, oh, this,

this, Zoom bar is getting in the way of my, This is, 

hugo: I've experienced, I experience that all the time and it's annoying to say the least. If anyone from Zoom is watching, do better. 

ravin: I wish you would snap a 

hugo: I'm kidding, that was rude, but still do better. 

ravin: Alright, so what we can do with NotebookLM is we can go in and we can add in, let's say, your website here.

actually maybe I already have, I already added one of them over here, so we're just going to use that. Okay, great. One thing I'll notice is, nowhere on this interface are you really just, is it screaming that you need to know AI or anything like that. But you just have some buttons you can put some sources in and you can say, you know what?

I want to create a frequently asked questions about Hugo and I click this button We're gonna wait a little bit while some magic happens under the hood. [01:09:00] But here we go within about 15 seconds or so we get a lot about Hugo, independent data science, AI scientist, educator and writer, podcaster, which we obviously know is true given that we're on this podcast, his experience in Bayesian data scientists, and you can see a lot of this, actually is about, Hugo himself.

Those are questions I ask about myself every day, man. You might as well. How do you contribute? Okay. This is cool. We've got it an AI overview. I will say, we're very savvy people and we understand that if we're using an AI system, there may be hallucinations or things like that and can't get around that at the moment.

so I can just go in here and just say,what college did, Hugo go to? I actually have no idea what the answer is to this one. I'm not pre prepared. But we'll give it a second. Hugo did not specify. However, he does mention he spent time teaching to researchers at Yale university in New Haven.

hugo: I did my postdoc there and actually I'm going there for Thanksgiving next week to hang out with my old friends and colleagues. Amazing. okay. I also, I love that. What, so this is [01:10:00] actually really important. This be, and we don't want this here, but when using llama index, for example, it will do the same.

It will say the sources do not specify, which I think is very good as opposed to trying to make up something or draw from memory as well, because Gemini knows about me, Gemma knows about me. but the fact that. At least with Llama Index, it's difficult to find out what the actual prompt that was sent to the system is a challenge.

But I think we don't need that here. But the fact that it says the sources do not specify is a really nice proof of principle as well. 

ravin: You're exactly right. And I want to, let's, I'm going to bookmark that for a second. But I, let me talk about references and we'll go back to exactly what you said.

I think, since you're on this podcast, a lot of folks are getting a sense that Gemini is powering this whole thing, which is absolutely true. There's a Gemini model under the hood that's taking in all these sources. It could be more than one. It could be videos, audio, things like that. And it's, it's, I don't say capturing, but it's, ingesting all the information that's coming from the sources, themselves.

Now, we know, obviously, the generative models [01:11:00] also have, weights where they also get information from. But in the notebook case, we don't, not that we don't want the weights to do things for us, but we're really interested in our sources. And we're not, we want Gemini to help us understand our sources, not that we want information necessarily from Gemini's weights.

And in the UI design that you, we were talking about earlier, For those who are AI savvy, this essentially is a RAG system. You don't have to know the word RAG. We don't put it anywhere on the screen, but it essentially is a retrieval augmented generation. And on top of it, you get links to your sources.

So let's say you just want to double check that this is true. It shows me exactly where in Hugo's source, which was his website. It says that he, he's going to indeed be at Yale University, New Haven. And I can verify myself that this is true from a source. I don't have to trust that the LM got it right in its generation magic.

That for me is huge as a, as when I'm doing research, because I need to really understand if this is 100 percent factually true and exactly where this is coming from. And 

hugo: can I also ask, did it, Does, sorry, it gave the reference? 

ravin: Is [01:12:00] that what the one is? Exactly. So if I put, I'll let you folks do this yourself, but if I put multiple sources in, you'll get one, two, three, four, all the sources that you're seeing the same thing come from.

So something that I do actually, sometimes I do this even with my own medical research to be frank, is I'll put in a bunch of medical articles, and if I see that it's been cited in three or four different places, then I'm like, okay, this claim probably is true, because three or four different independent sources are Gemini has found that these three independent sources have all said the same thing.

just seeing the number of references here starts giving me, credibility, to things. 

hugo: it's Can I also just quickly ask,for those, most people will know what RAG is, I think, but for those who don't, it's a technical term for a version of, essentially, information retrieval. I personally, and I say this to too many people, Sure, build a RAG system, but perhaps, benchmark with a bag of words like bm25 or something and then use a generative response to build it as well,But, my question for you is, if I wanted to build some sort of information retrieval system or RAG system like this, [01:13:00] which gave me the specific reference, Do you have a suggestion for how to do that? would I use semantic search or I may even just use grep dude, but, or like regex matching or something, but do you have a sense of how to advise people on how to get references out when building rag, rag based systems?

ravin: There's,so the one is the, in the RAG is another like a, I hope, actually I hope you talk about this in your application courses, like common application design patterns. I have a section on it in my Gen AI guidebook. We had mentioned agents, for instance, as being one pattern that's getting more popular.

RAG, I would say, was the pattern that was in hype last year, still being used this year. So there's a lot of different solutions, just as you said. You could use traditional like TD IDF bag of words methods, as you just said. I, these days, tend to use embeddings a lot with my writing systems. a quick overview.

these GenAI models are really good at giving you next token text, but there's also versions of these that, tend to tell you what two things are, like, correlated with each other, or, in the same sort of space. you, in the intermediate output. [01:14:00] if you pass in a recipe, some,the, some GenAI, gen, I guess it's not generative in this sense, but some of these large language models in the embedding space will tell you that these recipes are more related to each other than these.

technical research articles versus these pieces of legislation from Australia. And it's able to put them into the subspace and then you can use that subspace to pull out the things that are most relevant to the thing that you're typing, typing. Amazing. 

hugo: And I do want to say, I'm glad you mentioned embeddings when we started doing data science at scale, people like neural networks, and forgot about.

Databases and forgot about counting and forgot about group bys and that type of stuff One of the things that's happens with generative ai is llms llms multimodal. We've forgotten about embeddings There's a culture and they're the most They're one of the most important things. They form the model of relationships in which you're embedding your entire world knowledge, essentially, right?

And if your embeddings are crap, your model's crap. 

ravin: Yeah, exactly. It's just like the latent space of, Bayesian models. We don't think about it, but it is like where all the power is happening. I 

hugo: know we're [01:15:00] jumping 

ravin: around 

hugo: a bit, but I am You, all of this stuff would be super interesting to chat about in my course in, in January.

I'm wondering if you'd be interested in coming and giving a short guest lecture on this type of thing. 

ravin: Yeah, Hugo, any opportunity I have to interact with you and the folks that you associate with, I'm all for. 

hugo: Dude, that's amazing. All right. I've got a guest lecturer, as well as collaborating on gen AI for executives for your guidebook.

Great. Okay. Let's jump back into the demo. Now we're, I love we're building. Collaboration in real time on a podcast also. Yeah, 

ravin: this is wonderful. So maybe, to connect to a point you're saying earlier, Hugo and I are talking about all these crazy concepts that are below the hood, but you don't have to know any of this in the UX.

you just go to this UX and all this magic between generative AI, embeddings, retrieval, all this is happening under the hood. And you don't, you're not exposed to any of it, which is actually, even as a technical person is nice, because I'm not trying to get into Python code, all the time.sometimes I just want to understand what's going on with Hugo's life.

this chat, let me do that. This is the, this is the chat interface of NotebookLM. The other bit, which has been, which has [01:16:00] been the viral component in the last, two months, is this audio overview. So in here, if I click this one button, generate, a whole slew of magic is happening under the hood.

And what's going to happen is, in about two minutes, you're going to get these two podcast hosts talking all about Hugo and his accomplishments and what he's doing. So.one thing is going to happen for sure. Hugo's going to get huge ego boost out of this most likely about all the wonderful things that he's done.

And you would get an overview of all the wonderful things Hugo has done. But I actually think a more powerful thing is happening. is that. When I need to read a lot of sources, it does take a lot of time to sit in front of a screen and read, and it's a very screen dependent experience, and I'm already spending so much of my life on a screen, a lot of days now, I just generate this podcast, I put on my headphones, and I just go walk in a park, and this has been a new way for me to keep up with so many things in my life, from research to trip planning.

Like I have a notebook on my trip to Hawaii pretty soon. it's just an easy way to then just be able to get away from a screen, but still have the information in a way that, I'm [01:17:00] able to listen to and ingest so I can later on go make decisions about Should I take this tour experience?

Should I take that tour experience? Should I read this paper? Or maybe I can just skip it. and so while the UX is like really simple, it's literally just one button. It just, this modern technology transforms information into another format that just makes it so much more useful for me and the way I want to consume it in my life.

And it's, to me, it's just so incredible that, computers are just at this point, man. I can't imagine in the 90s knowing that my computer could just read everything and create a podcast. it'd be insane. I, yeah. I dude in, in the nineties each, I was playing 

hugo: in the early nineties. I still had floppy discs, like playing like text games in the terminal.

I, I am interested. So all, and this is just one proof of principle, like you can create vision models and put them in here. Yeah. Or, image generation models. I do have a question in terms of these, as I mentioned before. So when I go to chat GPT, right?my desktop app, I can talk with it.

Then I can, sorry, I can type to it. I can talk with it. I can get it to generate [01:18:00] images and it's using GPT 4. 0. It's using Dalle three for image generation. It's using some version of whisper for the,speech to text and something else for text to speech. now if I want to start using more state of the art models, if I want to plug in.

new llama 3. 2 model or stable diffusion or whatever it is, I need to start building my own systems. And in fact, I taught a, a workshop at, PyData NYC two weeks ago about, Composability and building systems that incorporate essentially building your own versions of ChatGPT, where you can, incorporate all of your own different models and I'll link to the repository in the chat.

And hopefully the video will be live soon. Self promotion aside, I am interested.is there a future where something like notebook LM, I can plug in a different model or I can use F5 for example, which is a new voice cloning. Okay. Voice cloning technology is actually insane. And, I'm not necessarily supporting the use of it, but if we could generate a podcast, just cloning your and my voice, that would be cool.

[01:19:00] So is there a future in which we can have tools, UX tools like this, which are no code, but if I wanted to plug in new models, I could do that as well. 

ravin: Yeah, So I think there's a future where we're going to have both. I'm going to go back to the cars analogy, because I think it's, this is true of my, my past.

Um,you can go buy a car from a Ford or a Chevy or whoever. and they've already put all the components together for you and you just drive the car and you're done. it's very hard then in the future to then swap in a Ford motor into a Chevy car because they've been designed, so specifically to be good at what their task is.

and in that system that it's not really feasible or like practical that you'd want to ever do that. and you're gonna see applications that are like this from every, from everybody, even just even in notebook LM. I'd say. We've put a lot of work into fitting all the pieces together, that the experience is seamless.

And this works at a super high quality level. Just like if you bought a car off of the.but there is still a future, like you said, where you have home kit builders where you can buy a kit car and you can put in whatever engine you want, you can put in whatever, put in whatever, seats you want and build your own custom experience.

Like [01:20:00] that feature is also happening at the exact same time. That's like the Gemma style models and things like that. 

there's one thing though, which is making this very hard for the home builder. And this, I think people don't realize it because they can't see it, but it's if I wanted to build my own semi truck at home.

Yeah. There's you'd have to, there's no way I could build a semi truck in my garage. These, the trucks are so massive, they require so much machinery, the engine blocks for semi trucks are so huge, you need a crane to pick them up, a normal person does not have this sort of machinery in a regular suburban house.

And some of these models are so massively big, or require such big infrastructure to run, that it is unlikely, somebody at home is going to literally buy a data center, like a straight up data center and run these models, models themselves. And that part, when you see a huge airplane, like a Boeing 747, nobody reasonably goes I want to build a 747.

At home, like it's just, it's obvious from the size of it, it's ridiculous to, to do, you have to rely on a Boeing and an airport to, to even service this machinery. [01:21:00] Absolutely. So I think that, that's the other part that's quite hard. And I've, I faced this myself too. I'm like, can I run this myself? And I realized that actually, probably not because I cannot run a 3 million token context model on my desktop.

There's no possible way for my GPU to handle that. I do need a Google level infrastructure to get that sort of context window. Makes sense. In this one application. So that's the other part about this, whole thing we're talking about. It's just is it, what is the feasibility with the infrastructure and stuff that you have?

And sometimes you just gotta let, like a Google type of company or cloud vendor just do it for you. Totally agreed. So we've got an audio. We've got a podcast now, right? We do have a podcast. I'll try playing it. I don't know if the audio will come through, but I'll tell you if 

hugo: I can hear it.

ravin: Okay. we'll say this. If you can't hear it, I will download this Hugo and I will send you the wave file and you can put it in the podcast notes. That sounds great. But let's try it. Let's see if we get it.

hugo: I can't hear it. Oh, it's cause it's coming through your headphones. I think if you maybe take your [01:22:00] headphones out and 

ravin: off. I think if I, I think I'm going to break my audio thing if I do that. So,I will, I'll download it and I'll give it to you. I don't know how to pipe the audio through my audio interface.

But we'll, people can hear it. But of course, if you want to recreate this yourself, you don't even have to wait. Just take Hugo's website, go to notebook. lm, hit that button and you'll get your own audio overview in about five minutes. 

hugo: Amazing. thanks so much for that wonderful demo, Ravin and chatting through all the possibilities.

it's time to wrap up. I would,

I'd like to know for people who, data scientists, ML people, who want to break more into working with, AI models and building. LLM powered systems among other things. What's what's a takeaway you'd hope? Hope they'd have from this episode 

ravin: it's a I think it's a theme that I want to highlight that you've brought up a couple times in this episode I think the knee jerk reaction I give you is oh go learn some technical thing Go read the guidebook even go read the material Hugo's course But I actually want to say a different thing which is there's just a wonderful community of people around that Hugo is [01:23:00] one of them but You know, the PyData community, other open source communities online.

Even each company has community groups. Go meet the people. Go talk to people and capture their excitement and learn from them where you should go and do things. I think it's the people that are making this technology cool and fun and worthwhile and valuable. Not just the numbers and the math.

So my encouragement, get out, Join virtually, if you can any like just like you have today, but if not, if you can go to a meetup in a local space that you have where this AI stuff is happening in every country that I've, I've seen meet some folks, shake some hands,andget to know the folks.

and I think you're going to find so many more opportunities than you ever expected. I love 

hugo: that so much, Large context windows are important, sure. but versus, get there and chat with people about things. And actually, I can say something a bit personal. I, I was working at Yale University, a decade, far out a decade ago, and I was living in New York City.

don't worry, I didn't commute every day, but I did a [01:24:00] bit too much. But at that time, 2013 2014, I started going to meetups in New York City, and hackathons that on, on West Broadway, there was, It was an AWS loft, which Amazon just paid for. And people would hang out there and, talk shop and that type of stuff.

Jared Lander was hosting incredible meetups in the R space as well. And it was being on the ground and meeting a bunch of exciting, interested. Thoughtful practitioners that actually was the forcing function for me to leave academia in the end and enter this space. And finally, I just got a message on LinkedIn from someone who's a tenured professor at a good university.

Who's like, there's too much exciting stuff. Hugo, I need to leave academia and go and work in this space. what would you suggest? And one suggestion was, speak with as many people as possible. So I couldn't agree more. Ravin, okay, first, thank you everyone for joining and listening and, and the great questions.

Ravin, I'd like to thank you for your, going to the frontier, doing all of this work and bringing it back to share with all of [01:25:00] us, for your wisdom, for your expertise. But also for the several collaborations, which just decided we'll work on together as well. And I can't wait for all of that and can't wait to do another podcast.

I'm actually thinking a series of podcasts on, Bayesian thinking. I want to call it statistics. Actually, one of the best things. one of the best tutorials I've ever seen, which I still, plagiarize from, is by Chris Fonsbeck. Yeah. And it is a tutorial. I don't plagiarize. of course I referenced Chris.

but, it's, it was a tutorial he taught at PyCon, I think, like 2017, 2018. And it was called something like. Statistical thinking in Python or something along those lines didn't mention Bayesian at all. And I was like, Why did you not mention Bayesian? He's cause it's statistics, Hugo. And I was like, that's so great.

That's so great. So I do want to have a series of episodes on generative modeling, and generative thinking and, all the wonderful probabilistic programming stuff, which I hope takes off even more soon. but in closing, thank you once again, Ravin. I can't wait for our next [01:26:00] collaboration. 

ravin: Yeah.

Hugo, thank you so much again for all the work that you do putting, giving people like me an opportunity to come talk with all the others you have on your various podcasts and LinkedIn posts and things like that. You're also doing incredible work building. I really appreciate it, man.