The following is a rough transcript which has not been revised by Vanishing Gradients or the guests. Please check with us before using any quotations from this transcript. Thank you.
===

hugo: [00:00:00] While we're waiting for Hamel, Dan, perhaps you could tell us a bit about yourself. 

dan: Yeah, whenever I'm asked to give my intro, I never know where to start. I think of myself as being a hobbyist, machine learning person or AI person until about 2011. At that point, I got second place in a Kaggle competition at a 500, 000 prize for first place, but no prize for second place.

I was first loser, but, since then, I was an early employee at DataRobot and consulting for, I think, six companies in the Fortune 100, for Google, For a while, I think the thing that I was probably most well known for there was created an educational platform as part of Kaggle Learn that had 40, 000 monthly active users writing code on it.

And now I do independent LLM consulting I also, founded a company called Decision AI, which combined machine learning and simulation for something called decision optimization, helping you not only [00:01:00] make predictions, but make better decisions. In response to the things that machine learning model predicts.

hugo: And it was decision AI that took you back to data robot for some time as well. 

dan: That's right. Decision AI was acquired by data robot and, I spent a year or two there after that, leading the product team in charge of model building tools. 

hugo: Very cool. And it's no coincidence that we're here to talk about.

A course that turned into a conference that you just taught. Cause also at DataRobot, if I recall correctly, you did, a lot of internal education or you had an education platform there. You did that at Google and Kaggle. And when we first met, we built a DataCamp course as well. Which was an intro to deep learning, which I loved creating that with you for a number of reasons.

But one was we taught Keras in the second half of the course. But we decided, the first half to teach deep learning with NumPy. So people would get a sense of, how these things are built. And we're living in a very different educational world now in a lot of ways. 

dan: Yeah.

When we created that [00:02:00] course, it sounds a little hard to believe. I think people will be skeptical of this, but I believe that was the second broadly accessible online deep learning course. The first one was a Coursera course by Geoff Hinton. So we didn't quite reach up to that level, but even Andrew Ings Course on Coursera for machine learning.

Didn't have any deep learning part of it yet. this was still eight years ago or so we were still very early in deep learning and, glad to be able to be part of it. 

hugo: Absolutely. And I've never heard anything I've worked on mentioned in the same breadth of as Jeff Hinton.

So I'm getting goosebumps currently. Hamel. What is up, man? You returned from the jungle. 

hamel: Yeah. Thank you. Sorry about being late. 

hugo: Don't be silly. The suspense makes your presence even more enjoyable. Dan, just, we just talked a bit about Dan's background. I'm wondering, Dan, could you just tell us a bit about Hamel?

Could you give him a intro? 

dan: I should be better at the handle intro. I've given it before. I think [00:03:00] the thing that I find almost really revolutionary is if you were to say, what are the top four generative AI projects or products that are actually having an impact and are widely successful today.

So you've got chat GPT, Claude, I think somewhere in that top three or four. Yeah. Would be co pilot and Hamel, aside from doing many other things was really the person, the range who first inspired co pilot. When Hamel was at GitHub, he worked on something called code search, 

This was a deep learning model, our generative AI model that descriptions, of code, plain English descriptions, translated that into code. I think I and most of the people before they saw what Hamel did would have said that it was impossible for this to work tolerably well. And Hamel, [00:04:00] Through some combination of force of will and genius, was able to make it work and inspired what is today, one of the most impactful generative AI products.

He, is also an independent consultant where he has a huge impact for many companies primarily by telling them to look at their data. everyone who's worked with him loves him in spite of him having such a clear theme. he and I recently, taught a course to about 2000 people.

Again, Hamel was the force of nature that made that so successful. I think of him as really a thought leader, one of the top thought leaders in the generative AI world. 

hugo: Hamel, what do you have to say in your defense? All these brilliant accusations of being an AI revolutionary, 

hamel: really humbling type of introduction.



dan: The amazing thing is that I worry about the things that I left out. 

hugo: I remember when Hamel start, so Hamel also worked with you at [00:05:00] DataRobot, was at Airbnb. Remember when amel started at Airbnb and he was like, dude.

Someone showed me a spreadsheet of their results that they're putting in production and they're just like sitting around these Google sheets and stuff and it's really intense. So Hamel, to that point you've seen a lot of a lot of different ways of working. You've seen a lot of failure modes and through your original consulting work in, in, in data science, right?

When you were, going to Vegas and that type of thing. Seeing the early days there, I think the consulting work is something which has definitely fed into, everything you're doing now. have we left anything out that's relevant Hamel, or do you want to make any admissions of doing horrible things just to balance out all the awesome things we've said?

hamel: So that project was called code search net. I don't know how much credit I deserve for co pilot. It certainly inspired lots of co pilot. There's a really talented team of people that worked on co pilot. And I didn't do any of the work on co pilot itself.

I [00:06:00] worked on the precursors to co pilot. So those people definitely deserve credit. And then, yeah, I guess I feel like I'm still learning a lot. There's a lot of stuff I don't know. There's a lot of people I look up to as well and learn from. Yeah, I don't really I feel like I know too much compared to in some dimensions compared to other people, that I learned from, including Dan, like Dan, when I first joined DataRobot, I barely, I was just learning machine learning at that time.

and he was teaching it. I was like learning from him constantly and yeah, I was just, I've been learning it since then. I don't know if you want me to introduce Dan or not. Maybe 

hugo: Dan introduced himself before you turned up, but let's do it, man.

hamel: So Dan is the best teacher that I know of. Everything like data science, machine learning, he really thinks about it, thinks about the craft of teaching more than anyone I've ever met, instead of a [00:07:00] passion for it since a very early point in his career. when I met him, I learned a lot from Dan personally.

he's also this really clear thinker. I'm a little bit chaotic in my thinking. Dan is a really good teacher and able to create learning experiences that really make sense. it's someone I really respect because he also really understands things at a deep level and he's able to teach them.

And yeah, I just think he's one of the smartest people in the space. Thanks. And just the joy to work with one of the most like trustworthy smart hard working people but yeah, like about the teacher stuff hundreds and thousands of students, data science, both on the Kaggle platform, he, spearheaded these courses, Kaggle learn.

And he also, I think he may have done some data, has he done data camp courses? I don't know. 

hugo: Yeah. Just before you popped in, we talked about the deep learning course we taught together and a bit more context for you. I just mentioned to Dan and he knows this, what I loved at the time was second half teaching how [00:08:00] to actually.

Use deep learning using Keras, but the first half was an introduction to deep learning using NumPy where people built toy, neural networks themselves. And the type of intuition Dan helped now, I think maybe a million students worldwide, but it was definitely hundreds of thousands a couple of years ago who'd taken the course.

hamel: Really well. It was like, Oh, Dan wants to do a course. Okay. It was basically the best machine learning educator in the world.

I'm super excited to jump in. We've actually just got a comment from Chadi in the YouTube chat saying, I'm dropping this comment to let you guys know that you left an important mark on my early start in data science. So thank you for that comment, Chadi.

hugo: That is beautiful. I want to jump in the course that we're talking about. I've dropped a link to in the chat. It's now called mastering LLMs, a conference for developers and data scientists. Originally, it was a course. On fine tuning, with all the possible things you could have taught as the first like serious course you're teaching on generative AI, why did you choose fine tuning?

Because there are strong arguments that [00:09:00] fine tuning is not the first thing one should teach or what people need to learn.

hamel: I think the reason, okay, so it was basically like the idea for the course was, created like way before the course began, as these things happen and when, all this generative AI stuff, especially like large language models first came onto the scene, especially in the very early days.

There's not, the only thing like technical people could do aside from using these APIs is to like fine tune stuff. If you wanted, you could build products, but if you wanted to do some like interesting machine learning, like tinker with making the models better or seeing where fine tuning can take you, then I think people with our backgrounds, like Dan and I, we were naturally drawn towards fine tuning.

exploring fine tuning because it's like something that we like have a grasp on already, in like comfortable training models. And, basically our community is like a lot of machine learning people. They're, [00:10:00] training models and it's very exciting. So we thought, okay and we got to talking.

So one thing I left out is Dan is also a consultant. He has very, he, works, he's a, he works at this, place called strive. He's the chief journey of a ops officer there. And Dan was doing like, he was trying lots of different fine tuning techniques and trying a whole bunch of different things. And I was also doing fine tuning, like a whole bunch of fine tuning. I was doing fine. I was fine tuning open models. I was fine tuning open AI. And we thought we were like, keep trading stories about it.

We were like, okay, what's interesting. What's not interesting. What's working. And that, okay. This seems. really interesting, like maybe we should do a course. And I was like, okay, I'll do a course with you. know. I was like, whatever Dan wants to do a course, I'll do a course.

Let's just do a course. Cause he, you can't turn that down when Dan says he wants to do a course. 

hugo: There isn't always a lot of machine learning to generative AI. There's a whole realm of generative [00:11:00] AI where you do know machine learning at all.

hamel: So something I'm hearing is you had the machine learning itch and you're like, I want I'm, I do machine learning. So I want to work with models in a way that leverages those skills as well. One of the things that I did early on is, I was like, okay, let me instruction tune my own model.

And when I instruction tune my own model, you just tell 

hugo: tell people what 

hamel: that is. So like a large language model by itself, just base model. It's not trained to answer questions or chat with you like chat GPT. It's just going 

hugo: to token completion, 

hamel: just complete stuff. And so you have to, train it to you have to train it to talk to you in that way.

One way of doing that is taking a corpus of text and getting another large language model to reframe it as a bunch of questions and answers. That's one popular way of doing it. So what you can do is. fine tune a model to turn it into a kind of a chat bot, 



hamel: Chat interface. 

hugo: By [00:12:00] giving it question answer pairs, essentially.

hamel: And I think like that process, I realized that's a good way to understand large language models. Cause you have to understand what prompting is, you have to understand synthetic data generation, what that can do, like what kinds of, you can change the behavior of your model, or you can just.

Use it to generate data have to understand like about inference okay, how to set up prompt templates, those prompt templates have to be, The same as what the model is trained on. All kinds of other like nuances, like you have to go through evaluation. You have to run this, like a little bit of a gauntlet to get there of all these different things that can go wrong.

And then it's just a really nice way to learn a bit about large language models, even if you don't care about instruction tuning, like if you, even if you don't want to fine tune, I thought it was really good. It was a good like way to solidify that knowledge. Incredible. [00:13:00] And so that's, I thought also okay, it's like a good, it's a bit of a good background, that I find like really helpful.

Like whenever I get clients to do instruction, tune a model I've had, I've For some people to instruction to in a model. And then all of a sudden they understand what prompting is. They understand, like it all gets flattened into text. They're using an API, but they may have like function calling.

They might be doing whatever. They don't really have a mental model until they do some kind of exercise like that. They're like, Oh, it just becomes I'm using this. API, but it's actually just becoming text and it's getting assembled into a prompt without doing it once. Like people feel like it's a big mystery and they don't understand all how this stuff works.

Maybe get there without instruction tuning, but I think it's just like really exciting cause it's you can, it's also this magical feeling where you can take a model that does one thing and turn it into a very useful thing with a little bit of fine tuning.

using another language model. So it's just like you create this very useful [00:14:00] product almost. And you're like, wow, like that was very powerful. 

hugo: gives you a lot of intuition. That's awesome. And Dan, I'd love your thoughts on all of this. I do for those watching who may be interested in instruction tuning.

How would you suggest someone instruction tune their first model? 

dan: Yeah. One other thing I was gonna say is that. And I'll come back to, I'm going to wrap that, the answer to that in this, if we'd created a course on prompt engineering, we could have done that.

We have some things that we can tell people that would be useful to them, but no one feels blocked from even getting started on prompt engineering. If you can make API calls, then you can, if you know any language that OpenAI knows, which is. Most people, you can fiddle around and experiment, and even if you're not perfect at it, you can follow some gradient and get better at it.

If you wanted to do rag, so we could have put together a rag course is the first course that we did, but. For the [00:15:00] most part, if you want to do RAG and you, take one of the public notebooks on, for instance, the Llama Index documentation, you can get started. I think that not that many people are meaningfully blocked from getting started on RAG.

Fine tuning is actually not extremely hard, but the perception of it was that it was so hard that people who couldn't Who would have learned it were just totally blocked and they probably didn't even know what the right tools were to get started. Hamel and I both really like Axolotl, which is, a wrapper around some hugging face libraries, but people were going to use lower level libraries because that's what there's a lot of blogs about.

If you went into the Google search on how do I fine tune, you might finish that search a little bit further from being able to fine tune than you were when you started. And so I think fine tuning is unique. Among the things that we could teach in the lack of good public content that will steer people in the direction that gets them unblocked.

Once you've trained your first model, you can [00:16:00] iterate and you can experiment, but even being able to train and deploy your first model is, people are just going to get led astray. And that was the reason That we wanted to teach fine tuning. And actually, if you're going to teach fine tuning, to come back to your question, Hugo, it's not that hard.

I think starting with the Axolotl library is really, the best place to start by far. It, as a rapper has a nice combination of being Relatively easy to use if you do one thing and I'll see what that is. And then also at the same time being powerful enough that you could grow with it and use just that library.

For the next year and use it all the time and never feel like you were blocked by some of the limitations of this wrapper. The one thing that would be important for you to know when you get started with it is to start by just running. One of the examples. So axolotl, it takes configuration files that these YAML [00:17:00] files, maybe a handful of people, but maybe zero people in the world could sit down with a blank sheet and just write out a config file, but to take one of the config files that they provide you.

and run it following the instructions on the axolotl readme is relatively straightforward. Someone who knows almost nothing about fine tuning could at least get it to run in, I think, less than an hour. And then changing out the data sets to use a different data set is relatively simple. You're just changing one line in a YAML file.

And now we're back to you're experimenting, you're seeing what works, you're seeing what doesn't work, and now you. It's a follow some gradient. 

hugo: Very cool. what type of hardware do people need access to? What type of platforms would you encourage them to? 

dan: Great. So I'm glad you asked. For hardware, you need something with a keyboard.

And a screen. I think that if you bought, I think you could probably do it on an inexpensive tablet. [00:18:00] I don't use tablets enough for real computing that I'm confident. But the key is that you want to do it on somebody that has a GPU, your hundred dollar tablet or your computer, or even my actually quite nice Mac book is really not the right tool to do it.

instead you should use a cloud provider. I happen to use, I, I've used a few, RunPod. RunPod. RunPod. Thank you. So I happen to use RunPod. An instance that costs about 49 cents an hour will be able to complete your first fine tuning job in half an hour. So you're looking at about. 25 U. S. cents to probably spend another half hour, on editing your file and other things.

So maybe you're talking about. 50 50 US cents or half a US dollar to get your first model to run on run pod. The axolotl readme actually has quite good instructions for how to run this with run pod. If you chose not to use run [00:19:00] pod, There are a lot of other compute providers, lambda labs, and yeah, almost any of these will work just fine.

I like Lambda Labs, but RunPot is great too. Sometimes I want to fiddle with the things RunPot will use Docker. but sometimes I don't wanna use Docker for various reasons. And then I use Lambda.

But whatever, you can use anything, for anyone listening who hears like that uses Docker and feels oh, I don't know enough Docker, I'm gonna be blocked. You don't need to understand Docker to use Docker on Run Pod. You just. Click in the GUI saying you want to use a certain image and the axolotl readme tells you what image to use.

And that's the last time you'll need to think about Docker. 

hamel: to be fair, like sometimes I want to develop axolotl or do something more crazy so I need access to the host. Yeah, that's 

hugo: very cool. So I love that you mentioned perhaps people don't need as much help with prompt engineering and rag and these types of things.

Of course, something we'll come [00:20:00] back to is that now you're teaching a course on rag, Dan, which Hamel is helping with Jason Liu. But you've made a very nice design choice, which it isn't for people who just want to know about rag. So people who are already building rag, and then helping them with the challenges they have then.

So this sounds so far you taught a course on fine tuning. That is definitely not what happened. A lot more happened. I suppose the elevator pitch I would give. Is a lot more people got involved. You had talks from a lot more, very interesting people.

Sorry, not a lot more, very interesting. They're not more interesting than either of you. you had Sophia Yang from Mistral. You had Simon Willison. You had Jeremy Howard. you had Paige Bailey and then Emmanuel from Anthropic beforehand, before we started live streaming, I was just saying how, what a huge fan of, Haley Sholkop I am, who's at Alutha AI and she maintains the LM evaluation, harness, Jono Whittaker gave an incredible talk on fine tuning napkin map.

Then. So you had all these people doing interesting work, [00:21:00] then on the other side you had vendors come in and provide, it was a 500 course and we all got 2, 000 worth of credit. So it was turned into this, many sided marketplace of ideas and tools and learnings. So I'm just interested in how that evolved, how you all thought 

about it. I think at some point you were just like surfing, right? Like you're riding the wave. So any insight on the evolution of the course, how it turned into JJ a layer. Sorry. This is the real deal. how did it evolve?

hamel: One of the things that I try to pay attention to in this course is to follow the gradient of what people want, and I could see the data of how many people are signing up, the velocity at which they're signing up, and how they're getting excited, while we were designing the course, it occurred to me, That, Hey, like actually, I don't know everything about fine tuning.

Okay. So fine tuning doesn't live in a vacuum. to do fine tuning properly. You have to do evals, you have to curate [00:22:00] data. And if you're fine tuning your own model, you want to deploy that model. Now, Dan and I could talk about all those things. I knew a lot of people that are experts in various pieces of those and I thought wow, okay I'm teaching a course but it's a shame that I should bring some of those people in the course like why not like I know them I know the basically the people that I would consider maybe the foremost experts in the world bring those people in the course I know wing really well from axolotl.

Why would not? Why wouldn't he be in the course? And so what I did is, 

hugo: Wings talk, which was early on, or the workshop was absolutely beautiful. And I'll actually, I'm just going to link to, I haven't put the link to all the videos that you have up. And I presume wings talk is up as well.

hamel: Is that right? Yeah. He also has office hours, but really what I realized is so people, when we got these guest speakers, And they announced that, hey, we are, like, they're giving a talk in this course. It was [00:23:00] clear to me that people really liked this idea of having these multiple perspectives from experts.

They're like, wow, we get to hear from not only Hamel and Dan, but also from, all these other experts. Recognize experts in those areas. Experts on eval, experts on deployment, experts on fine tuning, whatever. And then I realized that and then I started to realize Hey, there's a lot of, this stuff doesn't live in a vacuum.

And then from the first lesson, I could tell, people just wanted to know about LLMs. They're like, this is a fine tuning course. They're like, okay, how does it work with rag? And we would say something like, okay, we're not covering rag in this course, but it'll still be a lot of discussion and comments about rag.

there would be a lot of questions about When should I use open AI or Anthropic or whatever? following the grain of people once I get people really like these different perspectives. And so let me just give the people what they want.

So I just went to my network [00:24:00] of people. And I invited all of them. I started inviting all different speakers that I know on different subjects. And then at the same time, like some of my clients. I'd say most of my clients to date are like tools and vendors in the LLM space. And so I started, so I went to, one of them is Replicate.

So I went to Replicate, I said, Hey, I have an idea, like we're doing this course. It's like really interesting, maybe you want people to try Replicate. Why not give them credits? You just tell people what 

hugo: Replicate is. 

hamel: Yeah, Replicate is a kind of GitHub for models. Replicate. In a sense they host models, you can try them, you can try to put you can push open models, open weights models, and they host it.

It's, really popular in the text to image space. And they're building a whole bunch of stuff for, large language models. And they have large 

hugo: language models. I like the idea of GitHub for models. I also think of it a bit as a bit like, hugging face in some way. So you can go there and [00:25:00] explore all the models out there and the serverless inference capabilities are do it like sometimes on Saturday mornings.

hamel: And I think I've said this to both of you, like when having my morning coffee, or just go to replicate and be like, what's up on the explore page and discover new models and start playing around with them Yeah, no, totally. one of my clients were Replicate and another client of mine was Modal.

when Replicate agreed, I went to Modal and said, okay, this seems like a good idea. They said, yes, maybe it's a good idea for other people. So I went to Modal and say, would you be interested? by the way, just nudged them and say, replicate, threw in 500, and they have a little friendly rivalry going on.

So they decided to throw in some, credits as well. And then when we announced that, that created an immense excitement as well. And so it's these combinatorial factors. You know how it goes like the interaction between the guest speakers and the credits just Created a lot of excitement that kind of [00:26:00] kept snowballing and it just kept going with it I 

hugo: just I love that.

I do want to say one thing about modal as well For anyone wanting to play around with fine tuning or a lot of other things modal and because they're examples page And by the way, i'm not sponsored by modal Or paid by modal or anything along those lines. I don't know if any of you are, but, the featured example page, which I think Charles Fry has contributed a bunch to also like Charles does an incredible job and I'll link in the show notes to a podcast we did with him.

He does such an amazing job being out there and letting everyone know all the capabilities of modal, but this featured example page on modal just allows you to get spun up with all types of cool examples immediately. Any other products out there. and vendors have these cool example pages that allow people to do this stuff because it's a real game changer.

Dan, I'm interested in your perspective. what happened for you when this course started taking on a life of its own and became a fully fledged conference? 

dan: Yeah, I think, the feedback loop that Hamel described [00:27:00] is really the, phenomena that just objectively happened where the more people we got, the easier it was to get speakers, the more speakers we got, the easier it was to get more students, the more students we got, the easier it was to bring in vendors who wanted to give everyone, access.

this grew. I think there was a point when I was surprised that we were going to get 200 people enrolled. And we ended up, I think a little over 2000. It was really quite a phenomenon. Discord, server for this was really widespread. And, there's just a great opportunity to see a lot of different people's, opinions and thoughts on a wide variety of topics.

hugo: I love that. You mentioned the discord as well. I want to triangulate a couple of things. I've linked to all the educational resources from the course, which includes so many of the videos, wonderful learning paths. And I want to get into how you thought about structuring this and those types of things as well.

If you 

hugo: all could, put out some of the chats from the [00:28:00] discord on a blog post or something that would be so powerful because there's, there is so much. incredible information, questioning, conversations, experts in their answering questions.

So more generally, look, I want to be clear. I probably learned the most from being involved in the discord, the talks were, or the most that I couldn't get elsewhere. So I'm wondering what the role of the discord was for you and how you thought the experience was for the students and everyone else.

dan: Yeah. Going back to your suggestion of putting on a blog post, it'd be interesting to see if we get an LLM to there's so many channels, I don't even probably 50 channels. And some of them have been quite active. So be interesting to see if we can get an LLM to extract out the interesting parts so that we don't have to read through, 30, 000 messages.

I'm, I've always. Not just a data scientist, but somewhere between a data scientist and a product person. And so I've always been curious, what will generative [00:29:00] AI get used for? And one of the things that was very striking to me was, and I think because of the way that it was marketed, we got a lot of people in this course who wanted to learn, but didn't yet know what the use cases were.

dan: And really they were trying to figure out what are the I want to have the skills now so that I can pitch them within my business. And when we spin up some use cases, I'll be ready to hit the ground running on them. so discord had a lot of abstract, technical questions. Questions like, Hey, I'm going to try and.

What model can I run on a given GPU? And I think people didn't even yet know what their use case was, but I think it was just a place for people to feel out the space. And it was interesting to me to see where the interest was and where the interest, like just to understand like what people are excited about.

hugo: How about you, Hamel, with respect to the discord and what people were interested in? 

hamel: Yeah, I thought it was really [00:30:00] interesting. Like anytime we had a lesson or a speaker, there'd be a lot of lively discussion throughout the entire. Lesson or talk with people bringing in lots of different perspectives and resources and links to things and like lots of discussion going or going on every single time sometimes hours after a talk was concluded.

So I thought that was really fun and engaging and there was a lot of people that I knew in the discord. So it was like fast AI. I would say half of it was fast AI. former fast AI students or fast AI people that basically came into the course. It was very familiar kind of reunion in a way.

hugo: You say a bit about fast AI for those who may. 

hamel: Yeah. Fast AI is, created by Jeremy Howard. Who's been on your pod, I think many times, but he's created like one of the most popular. [00:31:00] Courses teaching people deep learning and machine learning called Fast. ai. There's also, created software with by the same name.

That's like a library on top of 

hugo: PyTorch as well. One is Fast. ai and one is Fast. ai and I always mix up which is which. I may get into trouble occasionally. 

hamel: A lot of people. And I knew most of the time, I would say someone asked a question and knew who it was. I had familiarity with that.

So it was like hanging out with friends in a way. 

hugo: I want to go back to something Dan mentioned, and I feel like, perhaps you've understated it, Dan, your observation that many people weren't asking questions about building practical applications and how that perhaps even concerns you slightly.

dan: Yeah, it was actually in overall, we probably had 30 talks. the average talk might have had 30 questions. So we had, ballpark, a thousand questions. And it was in the [00:32:00] second to last talk that someone asked a question where they said, here's what I'm working on, here's how that relates to the topic that you're talking about, can you help me understand it so that I can't apply it?

And that made me realize how, I think that was the first time that I'd heard a question where someone really had a specific question Problem or use case that they were trying to get unblocked on. at the time we finished that course, it made me wonder if actually it's harder to find use cases for generative AI.

Then the world set on, and you've seen a little bit of, I think Goldman Sachs recently had some published report where they said they think the economic impact of, generative AI is not going to be that great in the next 10 years. And, Duran Osamoglu, who's one of the world's, probably 20 most prominent economists recently said something similar and none of those guys know for sure, but it made me, Wonder if actually maybe this tool, which we thought was so [00:33:00] incredibly versatile, maybe it'd actually be much harder to find use cases for it.

And I certainly felt that with predictive ML, the XGBoost style stuff we did five or 10 years ago, at one point we thought it's going to revolutionize every industry. And then as someone who saw a lot of, businesses adopted when I was at data robot, actually it was almost exclusively used, for, Big tech.

So Google and Facebook and so on. It was really like churn modeling and some marketing analytics, but it wasn't that broadly useful, or wasn't that broadly used. And this made me think maybe that's going to be the case with generative AI. And then in the last week, I've started this, course that you mentioned, with Jason Liu about, RAG.

And we've done a few things with the way that course is set up. We took applications and we were, we tried to get people whose jobs require them to do RAG. And now I'm actually seeing that my belief, which was the belief that I had a week or two ago, that it was going to be very hard for people to find use cases because I wasn't seeing them.

I think that's actually [00:34:00] flipped and. When we have questions in this rag course, everyone starts with. Oh, the thing you just taught, how will that apply to this problem that I'm trying to get unblocked on? And they can explain all of the real world details that come up when you have real projects and use cases they have are all things that, seem useful to solve, they're not octet or canned examples.

Yeah, it's quite interesting to see that. I think they're just. Sociologically, some circles you can get into, and it's all people just trying to learn. They actually don't yet know what the use cases will be. And other circles where everyone is already a little bit further along. I would really encourage AI engineers to try to think about product.

I know in the consulting I do, A lot of it is with people who are not with non tech companies. Fortune 500 companies that, are trying to do AI. The people who don't have a technical background really struggle with things that if you've been programming for a while, they might seem [00:35:00] second nature to you and they don't have good intuitions for where they're going.

Generative AI will be useful and where it won't be, if you are a technical person and you're willing to stray from early coding into doing some product thinking, I think there's actually such a need for that. And I think there actually probably are quite a lot of use cases, but you need to move past say, I just want to learn, about memory management on the GPU to say, step back and think, where is this useful?

I think there's just such a big reward, or as Jason would say so much alpha in doing that. 

hugo: I love that. And I do want to dig deeper into why RAG may be able to deliver more value, and why you've been led to that belief now. I also, want to mention that, I've discussed these types of things with Jason before on the podcast.

There's an episode wonderfully titled. By Jason, how to build terrible AI systems, where we use an inverted thinking approach, where we think about how, what would you do to build a terrible AI system, and then to flip it on its head and say, [00:36:00] hey, actually, maybe this can help us think about how to build good ones is the goal there.

But one thing Jason and I talk about is how a lot of the value of these systems, a lot more value can be generated By automated generation report generation and, decision automation and these types of things. This comes back to, your work in decision sciences as well, Dan, but rag systems seem like they're.

Able to, be powerful at report generation, for example, which will in the end deliver more economic value than someone just chatting with an interface. So I wonder if that plays into why you think rag may be more powerful for, economic value than, than other types. 

dan: Yeah, I've heard Jason say that I think there's some truth to that.

I don't buy it entirely. The amount of economic What 

hamel: are we comparing RAG to, by the way? More useful, you're saying RAG more useful than Sorry, say that again. It's what are we [00:37:00] comparing RAG to? Some of the questioning is about, okay, RAG is more useful or able to provide greater economic benefit, but what relative to what?

Yeah, I would actually, I've got an idea, but I'd flip that on both of you. 

dan: Okay. I'm going to compare it to, if I divide applications of LLMs to RAG and everything else, I want to push back on, so before I started this course with Jason, I had the view that both those two together.

Finding use cases for them would be difficult and people weren't going to find so, so many use cases, and I didn't have to hold that view strongly, but I thought maybe the economic benefit actually would be less than most people anticipated. I think the claim that. You go and is to Jason and maybe they would make together is [00:38:00] that rag is going to have disproportionately high ROI on the time spent on it compared to everything non rag.

And then I think the logic for that is the value of reports as they inform decisions. I don't entirely buy that. The fraction of economic value that is created by report generation. Is extremely small. So reports can be useful, but most things that we do in life provide value. Most of that is not report generation.

And a lot of that is not even informed by reports. And, yeah, I think you guys know, and I haven't talked about it a ton publicly. I'm working on a, application that it's basically L M, Based alternative to CAD software so that you can, if you've got here, a bike chain. So if you wanted to, build a 3d model of a bike chain, you could do [00:39:00] it in A couple of minutes.

So bad software that is designing physical objects is something that many people do provide a ton of value. I'm doing it without rag at the moment. And it certainly is not report generation. And I think that as. Technical people start to expand out from San Francisco and think about the things that more and more people do, that are not what software engineers do specifically, I think there's actually a lot of applications.

I now think there's a lot of applications to new fields. You just need to get exposed to the new fields and some of it will be report generation. Some of it will be designing physical objects. Some of it will be robotics and maybe that will be to improve house painting. But there's just so many things that.

People do and I think many of them can be improved with LMs. And one of the small things that people do is, that creates economic value is writing reports, 

hugo: That makes sense. And that's also why I framed in terms of automating decisions as well, 

It's [00:40:00] not clear whether generative AI has actually had a return on investment for any organization. And I think Copilot is probably an example where clearly has. But I am wondering if we're thinking, how can we even think about this? If we think about headcount, the cost that goes into generative AI, a lot of it is speculative, right?

Hamel, the companies that you've consulted, it has been not inexpensive for them, right? To build out. The systems that they've built. So I'm, I don't necessarily want you to speak about them of course, but is there a sense that we haven't necessarily seen, ROI for a lot of generative IAI use cases yet, 

hamel: you haven't seen like massive shifts yet, like we haven't had our internet moment where.

the Amazon has taken over this in the industry and 

hugo: we had the. com bus before we saw serious internet moments as well, right? 

hamel: so I think [00:41:00] like probably there's people are not, I think in the short term, it's going to feel like it's going slow in the long term.

It's going to feel like it's going really fast. That's what I've always said. I don't really know the exact timing, but I think we're pretty early right now to say, okay, like here definitive ROI. It's really hard to like calculate ROI in a lot of cases. Like when you start to it is like in things like marketing and, stuff like that.

But We're talking about making users lives better. There's a lot of like mix of AI plus non AI stuff going on. And to really parse out like what is attributable to AI, what's not, it's like pretty hard. And there's definitely like AI first driven companies, but I think yeah, I can't think of anything where it's like just slam dunk.

I think the bar there is a little bit too high. So let me give some examples. In my consulting work, [00:42:00] we have done a project. You've got a team of experts in various academic fields, and their full time job is to take images, That are in academic textbooks and write what's called alt text descriptions or alt text, which is what a blind person would read on a braille reader in place of that image.

dan: And, that's a line of work that we've had many people doing for many years. And then with, one of my clients, we have built a piece of software where, in this case. A fine tuned, vision to text model, or specifically Lava, sees the image, what's in the alt text, and then the person who otherwise would have written it from scratch just clicks either okay, or they edit it, and we've seen that it's made them much more efficient.

And the payoff on that was, I think we probably were, [00:43:00] A month into that project before it had a payoff that was greater than the investment. And then ever since then, it's just been fair wins. if you said I want existence proofs, like I've done several projects where we know the ROI is positive.

That isn't building the next Amazon, but I hope that. We can start thinking about success or high ROI projects without saying it only happens when we can make decacorns or more. there's clearly a bunch of those projects that have been successful and clearly, I think, more than them that have not been successful, broadly speaking.

if you said it in Agria, yeah, I suspect the ROI is very poor. So far, and then we're just at a point of trying to figure out is that going to turn around? Are all of these upfront investments going to pay off? And that's the thing that I think we're at a point of speculating about.

hugo: I love that. And I love the resetting of expectations there. I do want to probably be [00:44:00] slightly provocative, but with the goal of getting somewhere, I wonder how much of this we've seen in like classic machine learning as well. And there's a common conception that perhaps classic machine learning hasn't worked for a long time.

A lot of organizations, but it has for some of the really big fang companies. But having said that there are probably a lot of small organizations that have used it and just aren't as publicly, putting their ideas out there as well. So I wonder, we've seen in the past few years, a lot of organizations reduce their machine learning capabilities just because it hasn't had the ROI also due to the downward macroeconomic pressure we're all experiencing as well.

I suppose that the provocation is, has machine learning, not even LLMs. This is classic ML delivered the value we thought it would outside a handful of large tech companies. 

dan: Yeah, it's certainly my view that it clearly hasn't. And that's, I think we're in the same place of, there are some projects.[00:45:00] 

Small companies, some at large companies that have clearly been successful. my experience has been, by and large that, there's a lot of innovation budgets and people wanted to do something that seemed cool. They, Adopted, they started building machine learning models in order to use this innovation budget and try and get ahead of the curve.

I think the thing they discovered is that. Actually, there aren't that many places where predictive XGBoost style machine learning is that useful for a typical company. The example I gave when I started my company is, even if you're a big company making decisions at some scale, let's say you're Broker, one of the most obvious, there aren't that many places that it's obvious you should use machine learning, but I think one of the few obvious places is, sales forecasting.

But the sales forecasting. doesn't really tell you what to do. So if you say the given location is going to sell a thousand mangoes before the next shipment comes. Okay how many do you want to have on stock? If [00:46:00] you stock exactly a thousand, since predictions aren't perfect, there's roughly a 50 percent chance that demand exceeds the forecast.

And now you need to make a decision. You could build some software to do that, which is what I attempted to do. But people who have been, produce managers for many years. They can roughly do that. And once you've got them in the loop, they also have just as good an intuition about how to make predictions.

And Even in the places where it seemed like machine learning might be useful demand forecasting, it didn't have that great a payoff. You ended up requiring humans to make decisions and the human can make just as good a prediction, of demand as a machine learning model could do. And so in end, why even have a machine learning model for many of these seemingly good use cases.

hamel: There's also the inflatable tanks. So just to go back to Dan's point, I do think that in classical machine learning you had maybe some positive ROI, but lots of negative ROI, and it's a lot of noise in there. And then maybe with LLMs, it's probably a bit [00:47:00] more positive ROI, but still a lot of negative ROI right now.

So still a lot of noise, but maybe slightly less noise. But one characteristic of ML and maybe even AI is a lot of people love to talk about it. they like to talk about how they're using AI more than they are actually using it. And that was always the case.

That was the case with like machine learning. And so the reason I call it inflatable tanks is like, I love to say, Oh, like we have this market position because of AI or because of machine learning. But then when you look here behind the scenes, like that, there's no machine learning even deployed.

Or it was just a blog post and that was very common with, machine learning, even in Silicon Valley. And I found that out, just by, by being there and I would talk to a lot of people and I would find this out. And so that's the interesting thing. Maybe, it's like with classical machine learning, it didn't have like direct balance sheet ROI, but it had a lot of psychological ROI. Maybe potentially the [00:48:00] company has some therapeutic effect. That people are buying into. 

hugo: I love that. And also a lot of the efforts are speculative. And this has always been the case with technology, even going back to the railroads.

Or, that's one of many examples going back historically, but it's like, Hey, this is something we believe that can deliver value. And it may not have yet, but we're going to build this and see how far it takes us. 

hamel: Yeah, definitely. And people wanted to articulate visions that were, bigger than themselves or have some vision towards the future.

And they would sprinkle some machine learning or some like AI stuff in the mix to put dressing on that like red light and then I just had to be completely empty rhetoric. But it would be just like very superficial and that was very common. So that could maybe be going on here. You definitely see like at least the early kind of a lot of people making money [00:49:00] in the ML space, there's definitely a lot of vendors.

A lot of the people selling shovels and the people and consultants and stuff like that. I would say right now in AI, like those people are like, I'm definitely making good money, like the tools and the, people that help you dig.

So yeah, I think it's just very early, but this is an inflatable tank thing that we can't, I think people are too afraid to talk about it, but it's there. 

hugo: I love that. And this actually leads nicely into, I got in, with the goal of reducing the amount of work I have to do, I got you each to prepare a question for each other.

No, I'm just kidding. That isn't why I did it. I always find it interesting when guests prepare questions for each other. Dan, perhaps you could ask Hamel the question, that you prepared for him. 

dan: If you had a research assistant. Full time and super competent who was ready to run any experiments you want or research any topic you want, and they start tomorrow.

What are you going to have them start working on tomorrow? What experiments are [00:50:00] they going to run? 

hamel: Yeah, I would actually like, one thing that's been bothering me lately is like, there's a lot of open model, open weights model API providers, and they all have different costs. Their latencies are all very different and their quality is all very different because they're all doing different things like different kinds of quantizations And whatever that you don't know about and it's actually hard to reason about what to use And there's also a lot of bullshit going on in the space I would have them like do that research and compile that for everybody's benefit it's very hard for a vendor to do that 

They can have a benchmark They try to be neutral, but it doesn't work because they're they have a dog in the fight But yeah, some like neutral third party who doesn't care. There are some people like that, but I think that's it's super confusing right now to operate in the [00:51:00] space because you have to basically do all that work yourself.

Matt Turk puts out that mad landscape with that great name every year. 

Yeah. And I'm just thinking about API providers. So I'm like, okay, fireworks versus together versus, replicate versus. Whatever it is like i'm probably i'm like blanking on whatever they're gonna get mad at me later But you know what i'm saying?

Like maybe the some of the more popular ones And, doing analysis is like, why is your 405 billion parameter model almost the same speed as your 70 billion parameter model on this one provider? What the hell is going on there? Or why is the quality of the one model on this provider not, so it's just like really confusing if people are trying to use this stuff.

Just even on that one thing. so that you can zone in on a lot of different things like that. Okay, that's just API, let's just like open model API providers. I think this is the thing that's interesting because I want to see if the open model providers, if there's anybody on the Pareto [00:52:00] frontier compared to open AI and Anthropic.

Is it the case that we have an open model provider that's cheaper, it's faster, just as reliable, And just as good. I don't know. It's I haven't had the time to do that. Yeah. And for 

hugo: those that don't know, the Pareto frontier is what you get when you have a bunch of trade offs.

And it's where you can sit to balance all these trade offs to be as optimal as possible across them. I also, I love Dan's question, Hamel. And I am, I want to build on it by, if you had an army of people working with you, I'm not interested in what you'd get them to do. I'm interested in what it would free you up to do.

If you had people who could do all the stuff that you do that you enjoy, but isn't the number one thing you can work on, what would you be doing with your time?

hamel: Yeah, I mean I would probably not do any sales calls with clients. I would just be working on problems. Because I'm an independent consultant, So I probably wouldn't do [00:53:00] business development. Yeah, I would probably spend most of my time doing work or writing.

And that's pretty much it. 

hugo: So I got you to prepare a question for Dan 

hamel: My question is, what are, so okay. You've. Taught these courses the conference lm mastering lms and then the rag course and you're also doing a lot of work what is the most exciting to you?

Like when it comes to what to work on what do you want to work on? 

dan: Yeah I always See applications and like how is no one? Doing that and actually are probably a ton of startups doing most of these, but I would really like to, practice focus. I have to focus on one of them, but, there's several of them.

I'm like, I would like to just tinker on solving that problem and focus on that 1 problem. So I talked about, I'm also very interested in this idea of. Crossing between bits into atoms or physical space. The [00:54:00] two that I'm like very excited about and enjoy playing with are this, app to design physical objects, that I've been tinkering with.

And then the other one, is in robotics. I'm really, inspired by some of Chelsea Finn's work, I built this, robotic dog. It's got a raspberry pie. In there, and it's got a microphone, it can hear what you say, send that to a whisper API, and then send that to, other LM APIs and use function calling so that you can give it an instruction, based on the instruction, it can understand that and make plans it can do things like navigate around a room and look for, an object that's hidden.

So I'm like, The thing, the things that I want to work on are things that are real applications, anchor with something, but then something that crosses over into the physical world. Yeah, I remember hearing Sam Altman talk about this Moore's law for everything and how like soon [00:55:00] everything's just going to get so much better.

And I'm like, okay, but most things that I care about. Have a physical component. And he says that research will get better and I guess bricks will get really cheap and everything will get really cheap, but I don't think many people are working on that. And so how is housing or construction or anything else in the physical world going to actually start getting much better?

that's, I think the problem that I'm excited about and would like to really have more time to tinker with. 

hugo: I just with the stuff you were just talking about, it did make me think that classic ML, has a lot of applications in supply chain logistics, warehouse, optimization, space optimization, business processes, that type of stuff.

Of course, I want to make clear that this doesn't exist in a vacuum. It is a mutant child of Fordism and Taylorism and management theory, which means that. There is a huge dehumanizing aspect of it. There's a huge literature into Amazon workers conditions and that [00:56:00] type of stuff when it's driven by machine learning algorithms and all of these.

So we definitely, it isn't always for the good of everyone. I am interested, you mentioned with your robotic dog that you send. You ping a whisper API. I know nothing about running whisper locally on very small devices. would it at all be possible? Did you play around with having like whisper tiny, on the actual hardware?

dan: I thought about that. I've never actually tried it. Yeah, since it's a Raspberry Pi 4, which is not a very powerful device, my impression was always basically asking chat GPT various questions, I don't think that you can run, any whisper in real time on, a Raspberry Pi. I think you could run one of the small ones doesn't work as well on a laptop.

hugo: I can run medium pretty well on my laptop as well. 

dan: I wanted to eventually be able to, use it outside. [00:57:00] So then you're gonna have a lot of background noise. You can have all sorts of audibility issues. I'm probably less intrigued by local, LMS than most people are like, why not just Send it to the cloud.

hugo: Let's do a demo. I'd love to see, like the CAD stuff you're working on. I just want to say, we've got another comment in the chat from Jay saying, thank you, Hamel and Dan for open sourcing the course. I can't thank you both enough as well. And that's a sentiment we've seen over the past couple of weeks as well on, on social media and those types of things.

But as I've said, I'm including a link to all the resources in, in, in the show notes and would love to know what you all think of them. So are you able to share your screen, Dan? 

dan: Yeah, I will share a screen. This is, I think it'd be very transparent that this is a super early stage app.

You guys, people who are watching will be roughly people five through whatever. In, seeing this because, I've shown it to my wife, my kid, and one of my neighbors and, I think [00:58:00] Hamel, but I'll make an object. I'll tell you what, why don't we make a cup with the name Hugo spelled out on the bottom of it.

hugo: Amazing. And just for those listening who aren't watching, I'm going to narrate, Dan's got a local host running, and a little, UI in his web browser where he's generating a 3d model of a cup with the name Hugo spelled out on the bottom of it. 

dan: Yep. I'll need to start the backend.

in a moment, we're going to start seeing some models, most of them are going to be really bad. You'll be surprised at, how difficult spatial reasoning is for, LLMs right now. 

hugo: I remember even being able to have text in an LLM well has improved significantly. can you say much about what models you're using or without giving away any of your secret source.

dan: Okay. This, like all live demos, may not work. LLMs are quite bad at this, and as a [00:59:00] result, I, try pretty much every LLM. OpenAI 4. 0, CloudSonnet, LLAMA, 70 something B, which I wrote along Grok, a couple of others, so I try all of them. And then, there are also many. Basically CAD style languages.

So each of 

hugo: I just want to clarify one thing. They're all language models, but aren't you generating visuals here? So wouldn't you be using a diffusion model or something like that? 

dan: Diffusion models generate images. Those images don't have real 3D printable, they're just a bunch of pixels.

Yep. So they, there are a variety of languages that are basically CAD languages. And each of these is specifying a physical object with real dimensions and real corners to it. And, so I have Did you fine tune using CAD languages or something like that? I have not at this point. At this point, it's all using publicly available LLMs.

Amazing. I think my [01:00:00] backend is not up at the moment. This live demo may not happen. The name Hugo spilled on the bottom.

Okay. I can tell from my logs. We'll just set this aside for another time. For sure. Yep. Yeah, we try all of these different, this Cartesian product of LLMs and modeling languages, and then, we show them all to the user. Some will have good results. Some will have bad results. You'll pick one of the ones that has good results.

And then there's a whole iteration workflow that is built into the app so that you can take the one that's best and say, here's what I want to change and, create that object. And then. you can download the 3D printable file and print it out. And I do that all the time and find it extremely fun.

hugo: Incredible. And last time you showed it to me, I, you were able to, it would give you something and then you could annotate it or change things on it. 

dan: Yeah. That's still the case that you can, it's this iterative workflow and [01:01:00] you can highlight things in the image that you want to change.

You can say what you want to change, and then it runs through this, recurring workflow of iterative improvement, much like you and I do for almost everything you're writing code or if you use CAD software, you would draw something and say, actually, that's not right. And then you would make a change.

And so it's, that same workflow. 

hugo: Great. So I'm going to have you back and we're going to do a live demo and, but the demo gods have not shined on us since the start of today, but we're still going live, which is exciting. I just want to say, we've got another comment in the chat from John Beers saying, I'd also like to thank everyone for this course.

It's been the cornerstone of my career reinvention after a year recovering from burnout. The conference has been really inspiring and motivating John beers. That's such a beautiful sentiment. And I think, The three of us can definitely relate to aspects of burnout as well. Although Dan does seem the most stable and centered of us all.

And that isn't to cast shade on Hamel or myself, but we get pretty out there in a way that [01:02:00] Dan seems Zen. Anyway, is that a fair characterization from your perspective, Hamel? Dan's a Zen dude. 

hamel: Yeah, I think so. Yeah. I wouldn't necessarily call it like, Zen is like a little bit more mellow, but on the spectrum of the people in this video chat, for sure.

hugo: I like it. You got some Zen vibes as well. Look, I've got two more questions and we'll wrap up in 10 minutes or so. I mentioned before that on Saturday mornings, I'll chill with my morning coffee and play around look at Replicate and see. And I've actually got Whisper hooked up so sit here and drink my coffee.

And talk to my laptop in order to generate like play with models, which is super fun. the sense of play I'm feeling now and the sense of experimentation is just something I haven't really had before in this space, like even building or working with machine learning models back in the day, there's a lot of like hands on keyboards and stuff you got to do. Whereas now we can play around with things [01:03:00] immediately. And I think that speaks to something you saw in your course as well, but I'm wondering from both of you, what are your thoughts on just the role of this playful experimentation in driving innovation in the field?

dan: Yeah, I absolutely see it. In some ways I did have a sense if you go back to 2011 or so when Kaggle first came into being, People really didn't know there are people who thought linear regression models were the best models and someone else neural networks would be the best models and no one really knew.

Some people thought that SAS, the piece of software SAS was the best tool and other thought it was R and others thought it was Python. And, there was a sense of everyone trying to figure something out in experimentation over time. If you were to look at approaches today, like everyone, especially for tablet data, the path is so well paved that everyone's doing roughly the same thing.

And there isn't that sense of experimentation. We really have no idea what large language models will be good at and bad at. [01:04:00] Over any sense of time, we don't what are the best approaches for rag and there's experimentation happening there. And so I think just the amount that we don't know because of how quickly this is developed is at an all time high that lets us experiment and have a sense of play and do things and not know what the results going to be.

hugo: And I think the skill set, I'm now telling a lot of my friends learn a bit of bash scripting, learn a bit of Python, and then you like go to hugging for, and then you've got to hack things out and deal with, Environment variable issues, sure, and that type of stuff, right? But if you can deal with that stuff, anyone can play around with models today, which is a very different scenario, right?

hamel: It's not just even models, the barrier to learn anything or get started with any kind of technical thing is a lot lower. 

hugo: But 

hamel: you can Get into things like you can learn new programming languages a lot easier If you use these tools correctly i'm tinkering a lot with chrome plugins 

hugo: I 

hamel: I don't know too much [01:05:00] javascript, but I use it to learn really fast Like what is this doing?

Why is it like this? Is there another way to rewrite this? What if we did it this way? I ask tons of questions really fast And it's directed yeah. And so it's like really interesting. I say, okay, give me a challenge on something to change and then I use it as a kind of a very fast tutor in a way that is not possible.

It would just take me a lot longer, to read a book. 

hugo: Totally. And the fact that you said, and let me paraphrase, but you said something on the lines of, if you know how to use them or something along that. And I actually, I put a link to, A new, piece of research by the Upwork Research Institute, talking about the expectations of C suite executives, with respect to 96 percent of C suite leaders say they expect the use of AI tools to increase overall productivity levels.

Then however, this new technology has not fully delivered nearly [01:06:00] half employees say using, sorry, nearly half of employees using AI say they have no idea how to achieve the productivity gains their employers expect and 70 percent say these tools have actually decreased their productivity and added to their workload.

Now I want to say I use Claude and chat GBT on a daily basis. I'm actually having a lot of fun using them adversarially. to get really good. But there's all types of shit I have to do in these conversations that if I did not know about, it would mess up my work day and mess up my work life, I think.

So I am, like, even knowing how chatGBT can get stuck in local minima and when to start a new conversation, when to remind Claude to use artifacts, cause it won't all the time. These little things that I know, cause I'm getting to know these people. These softwares relatively well, but do you have a feeling that if you don't know how to use them?

There's almost like a public awareness crisis around how to use them. 

hamel: The bar is really high for the general public. I think as developers, like we this intuition that you have to wrestle with the technology a little [01:07:00] bit, because we're used to wrestling with technology. But if I go out into the general world, like public, when I'd say Hey, have you used chat GPT were used for, or whatever, ask these like general questions.

I'm like really surprised often about how many people just completely write this stuff off or not having good experiences with it and when I dig deeper they're not using it. They're not really using it. They're not setting up they're not prompting it in any sensible way Not using like they're not creating gpts Or they're not like putting context in there they're not, using memory or using like system messages They don't even know about them.

They're like, what the hell are you talking about? What is this a message? there's like just type in the most terse thing that a human wouldn't even be able to, and just hoping for magic to come out the other side. it's a really counterintuitive because it seems like magic.

So you're like, oh, it's like magic is going to do it, but learning that I think is, 

hugo: Also the fact that they're trained [01:08:00] to please as well and to convince you that they're pleasing you as well is quite challenging. And I remember that I was testing something out and I uploaded a PDF to chat GPT a while ago to get a summary out of it.

And it's an old essay about, sociology of the introduction of photography, particularly in, police, situations called the body and the archive. I think anyway, I uploaded this to chat GPT. It gave me a summary and. Then I was like, wait, did you actually look at the entire PDF? And it said, no, I saw that the first page, what the essay was, and I know that essay or something along those lines.

So I gave you the summary of that. And so I realized at that point if I'd given it a collection of things, it wouldn't have given me. It just so happened it coincided because the thing it had made up is actually what I was asking for, but I had to dig several levels down in order to figure this out. And I think those types of things will be incredibly challenging as well.

hamel: Yeah, I think so. That's interesting that you like asked it to introspect itself and tell you [01:09:00] I feel like I don't I have a lot more skepticism. Like whenever I feel skeptical, I'm like, okay, let me just see. Let me like try to figure out what happened. I don't want to ask it what happened. Like I have like more distrust, I would say.

Yeah, and of course, 

hugo: like chain of thought reasoning and this type is like getting it to explain it steps and all of these things. I also remember, I don't like it's happened several times. I think where I've sent you screenshots of my conversations with chat, GPT and you're where I'm getting really angry at it actually.

And you're like, why are you getting angry, man? And I was like, bro, you get angry at software as well. And you're like, yeah, but I don't tell it that I'm angry at it. And I'm like swearing at this bloody chat bot constantly. But Dan, I'm interested in your thoughts as well on just general awareness around how to use these things.

hamel: I'm still trying to figure that out. I think it's quite interesting when it says I only looked at the first page. It's got a string of tokens, clearly, like recursively, like it did touch every token. Does it look at what its attention was [01:10:00] attending to? Like I have no faith in it. I have Absolutely. wow, that's interesting. 

dan: I wonder how, I wonder what things it would do to try and figure out. What it had previously attended to.



hamel: ask that. 

dan: yeah, that's super interesting. 

I love that. I had a level of skepticism and your both levels of skepticism is one level lower or higher than my level of skepticism for 

hugo: sure. 

dan: I'm not sure that you're, yeah, I'm not sure that it can't know, I just, the mechanics of trying to figure out how it would know, like what it would.

Be aware of that. We tell it, 

hamel: I don't know if it does. It depends on, I want to see the context. Is there markers around the pages that says like page one, like in like XML tags and page two, then maybe I can see that happening and be like, okay, yeah, that could be, 

dan: I get all it has is a string of tokens. And for it to be able to say, here's the tokens that I looked at and the other ones, 

hamel: even 

[01:11:00] I just, but why would only look at page one? Like it made a conscious choice. I don't know. We should, I was like, this is very interesting. We should you should do a 

dan: Absolutely. 

hamel: If 

dan: you were running at your LLM locally, you could actually look at the attention. You could actually this is what the McIntyre people do. You could actually determine what was the data it actually used in formulating a response, both originally.

And when it said that it only looked at page one, what was. It actually, attending to what it made that claim. 

hugo: I might do that in several situations and do a live demo or something, I think that would be super fun. And another thing I did recently I've been helping out, a bunch of companies with their documentation, developer relations, like movements, that, that type of stuff.

And I just wanted to go and look at some. Some examples of documentation that I love, right? So I chose scikit learn, Keras, and spaCy to start. And I was like, I wrote down all the [01:12:00] things I liked about them. And then I got Claude to, no, sorry. It was chatGBT. I got chatGBT to analyze the pages and tell me what it thought was good about these landing pages.

Told me all this stuff about the Keras landing page that just wasn't there. And I told it to specifically only use the landing page. And I was like, None of this stuff is on that page. And it was like, Oh yeah, I thought you were asking more generally about what type of stuff could be on this page. That would make it cool.

And so even 

hamel: gaslighting. 

hugo: Yeah, exactly. But really positive gaslighting. Like it's not the classic gaslighting. It's you're great. You're great. Yeah, this is great. So I do think like we, we need to be so careful and I think rag systems among others, information retrieval is 

hamel: that will send me their chat.

Like I'll ask a question about something like, Hey, like, how does this thing work? It would be something not related to machine learning, like some legal thing or whatever. And they'll send GPT answer. [01:13:00] And I'm like, why in the hell did you send me that answer? That we don't know if it's correct I'm like, but you're a machine learning person.

Like, why would you do that? But then it like blows my mind how people they're just like, even machine learning people, sometimes it's just they have completely different mental models. It just blows my mind. I don't know. Absolutely. 

hugo: So we have one question in the chat and then I want to ask one final question, but Bruno has asked, 

What are your thoughts and teachings on rag Or W slash E. This may mean more to you, Dan. I'm not sure. E. g. natural language search to LLM interpreter to get a query. Postgres SQL query with vectors retrieval LLM grounded answer.

hamel: I just heard a bunch of words. is there a comment on something? 

hugo: Yeah, there, I'll share it with you. It's in the YouTube chat and I'll paste it in our, I've actually pasted it directly in the chat. Sometimes like my friends and family tune in on to these and they're like.

How many acronyms do I need to know to [01:14:00] understand what you do bro? Or like how much jargon is there in your space? I'm like, it's pretty much all jargon. 

hamel: Oh this garage thing Okay, i've heard this for the first time. Okay, there's a friend that we have Who is a search expert and he like jokingly says he wants to invert rag to gar congenerative augmented retrieval 

hugo: Yeah 

hamel: And I think that's riff on that I don't know if I understand LM interpreter to get a query, postgresql query, a lot of error, 

hugo: arrows.

More generally, just on these types of techniques.

dan: Yeah, so this is like a LM interpreter to get a query, so that'd be like a query expansion. the one takeaway that I have. Yeah, this is the guy, 

hamel: Doug is the guy I was referring to. 

dan: The, one takeaway that I have from 13 years of doing various versions of machine learning is that, you [01:15:00] should.

set up a way to do evaluation and then you should try things. it's a little bit like this, it feels a little bit to me, if someone said, what's your view on XGBoost first linear regression versus deep learning yeah, You should set up a way to tell which ones are working. And once you've set up evaluation, then everything else is basically an implementation detail.

LL interpreter to get a query. So I'm query expansion. These are all. Things that are worth trying and sometimes will work and sometimes will not, and then I think the other piece, and this is something we emphasize a lot in, the RAG course that Jason and I are doing, is that you should figure out what are the categories of queries that are working particularly well and working particularly poorly, there's some things that are across the board tend to work Rewriters are an example of that.

But a lot of times my experience is that when rag [01:16:00] is failing, it is because you don't have the right data. And then you can think about how do we either bring in the right data or reprocess the data we have in the right way before we've And then we can build our, database with it, and that can be very targeted, to different data types.

And then what's the right way to do that? It all depends on, what are the queries that are, that you want to do better on. Then you should experiment with different techniques in those queries. 

hugo: That makes sense. And I do, once again, love the framing of set up evaluation or what, how you're trying to measure what you're trying to build, and then jump into the tools and techniques that may be useful.

Okay. I've got one final question. I want to travel to the future with you both. And as we all work in machine learning, our predictive powers have no match. No, I'm kidding clearly. But if you were to teach a course in five years on AI, and we would have a conversation in five [01:17:00] years, what topics do you think would be focusing on in the world of AI and LLMs?

Would we still talking about fine tuning or what we'll be talking about? 

hamel: I think I would be excited to teach people how to build full stack applications. For themselves or like small ones that they can be productive with right away, like the whole stack, like all the way. We're not just talking about models.

We're like, how do you just build something?

dan: I anticipate that, you're going to have a lot more cases where LLMs are actually taking actions. So right now LM is basically show some text and then a person reads the text and then decides what to do. I anticipate that there will be a lot more cases where LLMs are taking actions and to take actions, both function calling, which is The way that it calls an API and takes some action on the world, but then there are other places along the way where it needs to, , look something up, which would be rag like, but it could also be running a SQL query, but it's going to look up some [01:18:00] information.

And as a result, I think there's probably a future where you have networks. of something comes in and you consult an LM and then it decides it's going to run certain functions. And then those functions determine that you need to interpret something. And so it's going to send some information to a certain LM.

And you're going to have this actually like directed acyclic graph, almost of different ways that you have functions being called and LM is being called. All of this ending up in some action being taken. And so I think it's going to be something around, just stitching together many different pieces.

Those different pieces being the very concrete or, logical, which are function calling and the more heuristic parts, which are LMS, into some sort of 

hugo: network. Awesome. I love that. I love Hamel's response of being at a point where we're teaching and thinking a lot more about building, building full stack generative AI applications.

Something I hear in what you're saying, Dan, once again, is bringing it into the real world in more ways. Both of you have [01:19:00] mentioned things which take me back to these ideas of composability in these systems that I think are just so incredibly, composability in, in machine learning has always been incredibly important, but now we're really seeing it come to the fore again.

The other thing I'm excited about is more low code and no code, and people being able to build full stack generative AI applications that interact with robotics or the real world or decision making, with minimal code possible. So it's a really exciting space to be living in and working in.

I'd like to thank everyone who joined the chat and all the lovely comments with respect to Hamel's and Dan's course and work more generally. You guys are awesome. We've all been friends for years and I've had you on the podcast more than anyone else.

And I always appreciate, we even joke about, you probably should be co host sometime. But I'd always appreciate your time and wisdom and Dan, I've been wanting to have a conversation with you publicly for years. So not only appreciate your work, the work you do, but the time you take to come and talk about it [01:20:00] to let everyone else know, from the frontier, what's happening.

So thank you both so much.