The following is a rough transcript which has not been revised by Vanishing Gradients or Hamel Husain. Please check with us before using any quotations from this transcript. Thank you.

hugo bowne-anderson
Hey there, Hamel, and welcome to the show. Yeah, I'm

hamel
really excited to be here.

hugo bowne-anderson
I'm so excited. And we've been chatting a lot for years, but a lot more recently, because for the first time, we've collaborated together before, but for the first time, we're working together for the same company. Yeah,

hamel
it's really exciting.

hugo bowne-anderson
So it's something I'm interested in. I don't know if anyone listening will be interested in but I'd love to know, first off what it's like, working with me.

hamel
Oh, it's great. I've known you for some time. I've also listened to your podcast from the OG podcasts that you did while back. I would say yeah, you're pretty consistent personality. So yeah, it's really nice to work with you. And like, I make jokes all the time.

hugo bowne-anderson
The jokes are some of the funniest stuff. And also, we'll share some links in the show notes. But you're a meme Lord, as well, who gives Elon Musk a run for his money as far as I'm concerned. And the data means just you and Mark Seraphim, who we may converge on in conversation later, because he has a lot of interesting things to say as well. Two of my favorite memes, there's in hashtag data, Twitter space, but dude, one of the things I'm really interested in is you're passionate, and seriously opinionated, and have a breadth of experience across a lot of different like data, ml stuff. So we work together out of balance, now building productivity tools for data scientists to allow them to focus on the top of the stack, like the feature engineering, model ops, that type of stuff. So they don't have to, like worry about configuration files and compute, like, they just get access to that. But you've worked at GitHub before this, Airbnb before that data robot before that, and then did a bunch of different stuff before that. So that was, I just gave like a rewind version, I'd love to know how you got into the data world and your trajectory first. So maybe we can play that forward. And you can tell us about how you got excited about it and kind of your trajectory.

hamel
Yeah, so I first got excited about data stuff. In college, really, I mean, I was, I was doing computer science, just because that's what everyone told me I should do and really know any better. And then my teachers were really bad. And I ended up not liking programming. Surprisingly, the teachers

hugo bowne-anderson
were bad. But also, in my experience, computer science curriculum can be like at various place to place but it isn't always practically focused, I suppose.

hamel
Yeah, it wasn't practically focused, it was very, like, low level stuff. And the teachers were not really that good. And so I ended up focusing, I kind of pivoted to an adjacent thing called industrial engineering. That is a field that's about kind of convex optimization, optimizing schedules, optimizing production lines, like thinking about that is super practical, because it's kind of a applied engineering field that is kind of born out of the Industrial Revolution and trying to apply kind of engineering to that. That was super interesting. I studied a lot of statistics and math and stuff like that, but also like some computer science, but it was all like any programming, it was practically focused. That's kind of what turned me on to this, like, doing practical stuff with data. And then I ended up my first job was at this bank called Washington Mutual, which is now defunct, as bought by this other bank chase in the US. When I was working. My first job out of college was record working on a credit risk models, the credit risk models, there was just like forecasting if someone will default on their loan. So I did that for a bit. 

hugo bowne-anderson
When was this and what were you programming 

hamel
that was in 2003. And I was programming in SPSS in a little bit of sass, but it's a statistical software called SPSS, maybe it's around still, I have no idea. And it was writing a lot of SQL queries. That's the first time I learned SQL. And so I was doing all that stuff. It was very, like basic classification models, like decision trees and logistic regression, basically. And from there, I got kind of bored of that. And I decided, oh, like, consulting sounds interesting. Like, I can go visit a bunch of different companies, and see like what they're doing. Maybe I won't get bored. Like, I'll just be at every different company for like, a month or so. And get to like many different types of projects. I joined this company called Accenture and their management consulting group. And I was focused on a lot of like data stuff there. So I did mostly telecom there like helping at&t And at the end up buying all these companies, but basically like a lot of like telecommunications, internet providers, things like that. Work there to help optimize their customer satisfaction and diagnose various problems in their business using they did that over four years. Then I decided

hugo bowne-anderson
Did you do other stuff outside telco? I remember you, you did some crocks machine learning at some point.

hamel
Yeah. So that's the second phase, which I'll get to so then, like, I got bored of doing that, and

hugo bowne-anderson
I'm noticing a trend. Yeah.

hamel
Because it's consulting and I just felt okay, you know, yeah. So at that point in my life, I never did anything I always did. what felt like mentioned computer science, industrial engineering, whatever, I always had this, like engineering stuff that I was doing, but I never like in my life explored anything else. And so I thought, hmm, maybe it I wasn't sure like, this is what I what I like because I didn't really experience something I didn't like. So I didn't know if this is what I like. So I decided to just do something kind of crazy. And then I went to law school decided, okay, maybe I'll be a lawyer. So I went to law school, quit the job then moved everything and moved myself to Michigan, which is an arbor college town. And I lived there for three years went to law school, I ended up hating it. That that was like one of the most depressing experiences of my life, but you finished it. I finished it. Yeah. I don't know why I finished it. I should have not finished it to be honest.

hugo bowne-anderson
Well, from all I know about, I mean, I can be an incredibly stubborn human. And I'm not projecting all of that onto you. But I get a sense of that as well.

hamel
Yeah, it could be some of that. It was more like I was trying to give it a chance of kind of thing. And so yeah, that makes sense. So yeah, I did that for three years end up hating it. I didn't like it because it was so opposite of tech. stuff. Like everything moves slowly. It's very stuffy. It's very status base. It's not necessarily about the work you do sort of it's very hierarchical. Things move slowly things. It's about precedent and blah, blah, so forth and so on. I just didn't enjoy it for those reasons. And then I thought, Okay, I have to get out of this somehow. So the lifeline available to me to get out of that was doing consulting again, I didn't really want to do consulting again. But I said, Okay, let me do consulting again, because at least I can extract myself from this law thing, and then I'll figure it out. And then so I went to this other consulting firm AlixPartners in there, I did like a really wide range of projects, all focus on data science stuff. So that's where I worked with like a lot of retail clients. So this is what I call like, uncool data science, which is I worked with companies like Crocs or to the restaurant Ruby Tuesday, I worked at all kinds of retailers like hot topic, Pacific Sunwear, I don't even know maybe some of these companies are bankrupt. I'm not really sure. I worked at MGM, that Casino in Las Vegas,

hugo bowne-anderson
Vegas data science. I like, I just spoke with Katie bower at a Twitter. And she told me she's doing some professional development stuff, where she's meeting a lot of data scientists. And she spoke to a dude who's a data scientist for Grand Theft Auto. And for the game, he does stuff like figuring out how to price cocaine in the game and stuff like that. That's fascinating. Yeah, that's really that's really fascinating. So I Vegas just reminded me of Grand Theft Auto for some reason, it's unclear why Yeah, I could

hamel
see how that would bring that up. So a bunch of companies like that, I don't even remember all the companies, or at least 30 or so different ones. But basically, I even did some consulting for a gun company. So this is like everywhere. It was crazy. Maybe some companies that didn't even agree with what they're doing, for example,

hugo bowne-anderson
ever kind of a pretty naive question or set of questions just around what these processes even look like. I presume you clearly weren't deploying models like on sage maker endpoints and stuff like that, right? Will you like giving them CSVs predictions? Or was it like keynote slides? Or like dashboards? Or like, how was it roped into the coupled with the decision function?

hamel
Oh, yeah. So it's really interesting question. So I'll give you an example. So I'll give an example of this restaurant experience. So like, this is how it would go, they would somebody will call some consulting partner and say, Hey, our business is not doing so well. Can you help us fix it? Can you help us figure out different strategies of fixing it? And then they'll say, okay, Hamill, Just come with us. So we would arrive there, I have no idea what we're gonna do. I just go, go, there you sit, you walk into the room and say, okay, like, here's, like promotional data, where we send in coupons to everybody and promotional information. Here's like a giant Excel file full of all this stuff. Can you find something interesting in there? How will define some of the interesting things like, okay, so I just open it and you start digging around? You say, Oh, yeah, did you know that like, only 10% of your campaigns have like 90% of your lift, or something like that? In things like that, you just start talking about that really fast? Like very quickly, I'll go in there and start doing stuff like that. And that will lead to another conversation like, Oh, that's interesting, okay, like, can you help optimize that? Or what other What else can you find easy to kind of like build trust by like, poking around and like asking questions. Other times, it's like, hey, or like customer satisfaction is like going down. And you spend a lot of time building context. First, you don't start with data, you start with, okay, like your customer satisfaction. Scores are going down, like,  What do you mean? Like a like, how are these scores calculated? Where are they? And sometimes I just go interview a lot of people. Let me go talk to people. So like in the telecom example, I would have meetings with all these call center agents and They like focus groups and say, Hey, what is going on, tell me about opportunities that you see, maybe basically, I will be gathering hypotheses that I could test from real people on the ground. And sometimes I would just sit with them, I would actually, like sit with them for an entire day. And I just listened to them interact with the customer, like, so I can really deeply understand, like, what is happening in this business, I would do the same thing. Like some of these retail places, I will go work in the restaurant or the retail store for a day, I'll get really deep into it. I will go like work in the factory for like a week or a day or something like that. Just to really understand the context. So like, that's, it's really interesting. Like the data stuff only came later, in every one of these projects. First, it was like, okay, understand the business and the context and build some hypotheses about like, whatever problem they have presented to me and tried to chase that down. It kind of feels like investigative journalism, to be honest, with the fact that I have for sure, data skills. And then sometimes they would end up in okay, like, here's a model that can help you. But only in rare occasions, it would end up there are like fairly rare. Okay, I will say that maybe 20% of the time, so maybe not that rare. But

hugo bowne-anderson
and the rest is more like analytics. Oh, yeah.

hamel
Like, I wouldn't even call it... Analytics can sometimes be generous. Like, a lot of times, I would discover really deep business problems, just by doing this investigative journalism. So for example, so you mentioned crocs, when I came into crocs early in, they said, okay, like we have a problem with their inventory is costing us a lot. We have all these shoe sizes, and we don't really, we don't know how to optimize that. So can you help us like, do some magic to help us optimize our inventory? So it's like, okay, like, I've always been trained not to just jump into the problem with data science. Okay, like, you want to optimize inventory. Great. Okay, so how's your inventory? Like? How do you sure you have this inventory data? But like, how is it collected? Like, what's the process for collecting this? Does it even make any sense? So the first thing I do when I get data, is I try to brainstorm ways I can verify it. So there's like some really interesting tricks that you can do. The people that are pretty underrated one is, take any dataset, you get, and try to think of other systems that have the same data, or like an aggregate version of that data, especially like financial systems, or payroll systems that are supposed to be accurate. So like, if you get a bunch of data about inventory, low level inventory and stuff, can you recalculate? Can you figure out like sales from that you should be able to, and then doesn't match, like financial reports, then something is wrong.

hugo bowne-anderson
We're talking about consistency, verifying consistency, and a lot of ways Yeah, yeah. Consistency.

hamel
Yeah. And you can think of very interesting ways of verifying, like thinking of all these different rules of verifying consistency. And it's a very good trick actually, to verify the quality of the data set. I think it's pretty underrated. Like people don't do that, where it's just an instinct that I've had over the years, because I've always run into these problems, almost, I would say, I always find these problems.

hugo bowne-anderson
And it's very related to I mean, it's a snapshot of issues you get with data drift and that type of stuff as well. Right. So yeah, it's like, we need to be constantly doing verification and validation, like that's a snapshot of it occurring at one point in time making sure that all of these things are correct. Working together. Yeah. But then when you're modeling, you want to make sure that essentially your training data is representative of what you're trying to

hamel
predict. Yeah, definitely. Like in this example, it was that okay, the beginning inventory, minus the sales plus the receipts did not equal the ending inventory. So I mean, where's it going? Like?

hugo bowne-anderson
And the funny thing is, people did this before data science, right? Yes, the bar owner who's employing a bunch of people will make sure that the cash in the till at the end of the night, kind of matches the amount of drinks sold minus what a few that like, you know, you give away a few freebies. But if it's $1,000 off, you're like, hey, who took the money out of the till?

hamel
Yeah, so it means like a systematic data problem. And like, usually, when I'm given data science problem, I find something like this. This is not by any means. Unique to uncool data science, I would say I found just as same or bigger things in Silicon Valley. So like in Airbnb, when I first one of the projects I was working on is optimizing LTV.

hugo bowne-anderson
Lifetime value of a customer or a property. Yeah, so predicting how much money you make from this relationship over your joint lifetime.

hamel
Yeah, exactly. With regards to like, the reason for predicting this LTV was to allocate marketing dollars to different campaigns. In sort of, when I looked at, okay, the marketing campaigns And there was a notion of like somebody being converted, for example, and I wanted to just like figure out, okay, the consistency between our internal notion of what is a conversion. And then what the marketing partner so in this case like Google, or somebody who were using for like AdWords or something like that marketing campaigns, what is the conversion numbers there. And so I noticed when I was given this project, I noticed Hey, like, conversion number that we're reporting to our marketing partners completely different, maybe like an order of magnitude different than what we have internally, in like the people who were saying, converting internally and externally are completely off. Like they're very, there is almost no intersection between them. And then if you understand how ml works, so the way these like marketing stuff works is like you take your marketing partners, they give you some code or something like some JavaScript code, you embed it in your site at a place where a conversion occurs, and it sends them a signal. And they use that signal to help you try to optimize future conversions. So you can imagine an ML model where it's trying to predict rare events. And you're feeding it very wrong labels, all the time, is going to totally destroy that model. It means if you think about it deeply. So for example, Airbnb, I said, okay, like you need to, we're not going to forecast LTV, like you have a much deeper problem that you are giving your partners incorrect signals, there's no way they can be able to optimize

hugo bowne-anderson
anything, you're measuring the wrong thing. Yeah. And it's

hamel
kind of interesting. Like how often if you dig deep enough into try to think holistically, when someone says, Hey, like, Hey, can you do some ml? Or do some may build a model? I would say, like, 70% of the time, I come back and say, I don't think you should be building a model. I think you should be doing something else. Because this or that, or whatever reason. And I think that's one of the biggest issues in data science is kind of facing that reality. Because I have ever faced this like many times, like where I work, where a lot of times the incentive is just to ship a model and celebrate the shipping of a model. I mean, we aren't we are incentivized as an industry to do that. Like, as a profession. There's a whole ritual of, hey, like, build a model, ship it, write a blog post about it, give some talks internally. But the problem is, is like the smell is when there is no shipping, a model is celebrated. But in like moving metrics, or achieving concrete business outcomes is not so much like on the radar seems like

hugo bowne-anderson
an analogue of in product and building features and that type of stuff, what they call her this time recently, resume driven development.

hamel
Exactly, yeah. And you can get quite far in data science as a career, just building models and celebrating those models and writing blog posts about those models. I mean, it's kind of

hugo bowne-anderson
so this is something I want to hopefully get to talk about cargo culting. And something I'm hearing here is there's even a concept like there's a lot of tool and model cargo coating. But there's also general machine learning cargo coating, which I think will be interesting to get into a bit later. But I'd like to get back to your trajectory after Oh, yes, yeah, I went on a tangent. All the no great tangent because it foreshadows some things that happened when you went to Airbnb, but in between your uncle ml vibes and Airbnb, there's a very rich story to be told there. So maybe you can tell us what happened in

hamel
Yeah, so like, I learned a lot from the uncool things like how people are doing stuff, how different industries are operating, like, whatever the real problems on the ground, it helped like really ground me in sort of what all kinds of people are doing with regards to in their needs, and their impressions of data science and all kinds of stuff like in kind of what the mismatch is and where data science is good match for people and when it's not. And then like, again, I got bored of doing that. And I thought, okay, I saw all this cool stuff being done in Silicon Valley. I was also reading these blog posts and stuff like that. And also, I felt okay, like, I really want to kind of not do these projects where I'm, I kind of make something and leave I want to, to actually stop being consultant and sort of go into industry be more invested. Right? Yeah. Yeah. And so at the time, I was living in Boston, and I actually had worked with this guy, Zach Mayer, who maybe have him on the podcast. He's very interesting character

hugo bowne-anderson
love to him. And he's Zacks one of the correct me if I'm wrong. Did he work on Caret with Max

hamel
with Max Kuhn? Yeah, yeah, he's one of the cofounders of carrot. He's one of the

hugo bowne-anderson
my few like, Python meets our Venn diagram heroes as well. Yes, he forgets the flame wars and just get shit done and builds in both. Yeah. And it seems really lovely about all of it, to be honest.

hamel
Yeah, no. So he was really brilliant. So he was working at the same consulting company. I was for some time and he raised He was quite an impressive person. And then he went to data robot, which is this company that had a bunch of Kaggle grandmasters and people like Zack, that were just super sharp. So I thought, okay, you know what I'm gonna leave, I'm just gonna drop what I'm doing. I'm just gonna go work there.

hugo bowne-anderson
And just quickly would you describe data robot is like democratizing like auto ml stuff that makes building a lot of models in parallel relative, I'm trying to Yeah, describe data robot, that's my sense, is building a lot of models in parallel with big like, red button, you press it builds all your models, and then it does all of that stuff. So you can focus on the interpretation and seeing what works best in this type of stuff.

hamel
Yeah. So like, around that time, 2014 or so like building models. If you think back to the time of carrot, the package, you mentioned our package you mentioned, like building models, as

hugo bowne-anderson
I just want to say for those who haven't necessarily heard of carrot carrot is something Max Kuhn and Zack build together. Max was at Pfizer, at the time, working pharmaceuticals, I think doing a lot of drug discovery. But I may have got that I'm pretty sure that's where it came from. I've got that slightly wrong. Current now, min max is now at Rstudio building tidy models on Hadley Wickham is team working with a lot of really cool people. And this is software to really follow to look out for easily productionizable. If you're one of those people who thinks R doesn't work in production. I'll include a few links in the show notes to dissuade you of that completely. Don't come at any of us on Twitter, come on the show if you want to start a flame war about it. So that's my PSA anyway,

hamel
no caret, it was sort of like back in that time. If you wanted to build models, he was pretty painful, like you had to. So there's so many models to choose from. And they all had their own unique quirks. And some of the esoteric quirks, like okay, like, how to tune them how to use them, like so on so forth. There's like a plethora of different

hugo bowne-anderson
 even what you're inputting to them, like even the format, like the data structure, you input into a model could differ between models, so you couldn't quickly switch them in and out,

hamel
right? Yeah, exactly. And so the first package that I remember, in open source that wrapped around a whole suite of different algorithms and said, like, Hey, you prepare your data in a certain way. And then he will help you try 30 different algorithms, all in one common interface. So you can just try all of these things at once. And that was really super important. It allowed you to focus on other things that matter like feature engineering, the problem understanding the domain, so on, not spend all your time on just like modeling.

hugo bowne-anderson
And the other thing I'll say about carrot is the documentation. And the vignettes in particular, meant if you wanted to find out how to use it, and that type of stuff, the vignettes were so beautiful, they talked you through in kind of narrative based forms, how to use carrot, I think that's one of the reasons scikit learn garnered such adoption around the same time, actually, is because they're incredibly well documented for humans.

hamel
Yeah, it was really good. And so like, kind of around that time, kind of the same mentality of hey, like, so people like Zack had all these recipes in their mind, and other people like him. Some of these people were Kaggle folks, some of these people weren't. But basically, it was, hey, can we take like a lot of the best practices that we have learned, working with, like, not only cargo problems, but like real world problems. Because granted, like, also, these cargo people, we're not just doing cargo, like there are many of them are actually working in the real world doing real things. And they're all super impressive. So they all kind of got together in this one company, data robot, and made this platform that basically, you know, you prepare your data, and you feed it to the platform, and it tries many different types of models, many different types of algorithms, tunes them all, actually, like one of the most magical things and does a lot of feature engineering stuff for you. So this is pre-deep learning. So like every algorithm, it will try, like many different types of feature feature engineering approaches. And so in my mind, like they really invented the true auto ml, like there's different spectrums of auto ml, like different tools on the spectrum. But this is kind of the first tool that I saw that and used that I thought, okay, like this is really auto ml. And you know, I can say why, but say why? Oh, because so if something is truly auto ml, you don't want to kind of have to draw a box from your data to the output and specify like what models you want to try and you specify like what feature engineering steps you want to try? Yeah, there's no code lightly. A lot of things are no code ml platforms are a lot of people automate one part of the modeling process like tuning a lot of people are automating hyper parameter tuning, or maybe some people are automating, maybe trying different models, but kind of gluing together all the modeling steps like the feature engineering part plus the modeling plus the tuning, and then giving you all these diagnostics. So the diagnostics are really helpful. To all kinds of diagnostics about your data about the models, different kinds of models, what's working and what's not working, any kind of diagnostic that you can potentially think of all this stuff is like automatically produced for you,

hugo bowne-anderson
by the way, we're not sponsored by data.

hamel
But okay, so less is more about auto ml like,

hugo bowne-anderson
Yeah, but I have a question around. I'm a contrarian in some of the best and some of the worst ways. Yeah, but something. My spidey sense, tingles. When I hear automated feature engineering, is there a concern? Shouldn't we be injecting very serious domain expertise into the feature engineering process, among other parts of the pipeline?

hamel
Yes. So it's not like that you connect data robot to your database, and it just joins all your tables together. And like just has some benevolent AI that goes in like does all your know, you have to bring your data to data robot, you have to, like collect the columns. In a tabular data, you have to collect the rows and columns, you have to provide the features.

hugo bowne-anderson
So you do all of that before you give it to auto and let's actually not talk about data robot in particular, like your ideal auto ml system, does a lot of feature engineering, but you're prepared with all your domain expertise beforehand to give to the system, yes, and then does all this stuff, and then tells you the output of lots of models with lots of parameters with lots of features, and helps you navigate and diagnose what's happening those?

hamel
Yes, autoML should make it such that you only have to do Domain Specific feature engineering, and not any model specific feature engineering, right. So what I mean by that is like, okay, you know, like some algorithms, you have to normalize your data, some algorithms, you have to do something with texts, you know, you may want to treat text in a different way, depending on what algorithm you're using. I'm not talking about just deep learning, I'm talking about all the different algorithms that there are out there. Absolutely. You know, you might want to try different kinds of tricks, you might want to have some kind of model ensemble for certain things you might want to stage models, or you might want to do, depending on the constraints you have, and the compute you have, and how simple you want it, whatever. And so like, that's the thing, that's what I mean by feature engineering is, there's a lot of these tricks for kind of, I would say, model specific feature engineering, like, hey, with this model, like these tricks work really well. And it's not just there's actually like quite a few of those. And so it's really helpful not to worry about all that stuff, and just say, Okay, I'm gonna get the features, I'm gonna try to collect the features that I think are important, and that is feature engineering. And then at least, there's some, the auto ml system will aggressively try many different feature engineering steps. There's not a magic wand. It's not a free lunch, though. I mean, it's not that oh, it will, is guaranteed to find the best feature engineering stuff that by any chance, as we

hugo bowne-anderson
know, Hamill, that. There's no free lunch. Yeah, there's no free lunch, right?

hamel
But I will say this, yeah, people get the wrong idea about auto ml. They think, okay, like, this is gonna automate my job automated modeling, like how sending an automated modeling, how are you going to compete with the human being? And the answer is like, it's not about competing at all. It's about augmenting you. So just like would you ever cybernetics? Yeah. Would you ever write a blog post without spell checking it? No, probably not, especially as a professional then you would use a spell checker. You know, it's a tool that's there to help you. So

hugo bowne-anderson
Funnily, I have to now this is the bane of my existence being Australian having lived in the US for so long. I'm an inconsistent speller. Now. So I need a spell check. And I need to set it to whatever region I'm writing for, essentially,

hamel
yeah, absolutely. And the same thing here, it's, Hey, if you're going to work on ml problem, it's really helpful if you have access to one, an auto ml system, just to get a really strong baseline. And just see, like, what kind of signals that you get that you have all the diagnostics, all the reports, maybe you don't end up using anything that you find in auto ml, maybe you find that the model that you have beats the autoML I can you're like, I don't care, whatever. I also

hugo bowne-anderson
love, I wonder if you feel the same way. I love the idea of always your first model not being a machine learning model, like you base it around slicing the data in a particular way, making predictions based on features, that type of stuff, do not do any machine learning to see what accuracy and other things you get when building that model, right? But never use machine learning as your first model.

hamel
That's one of the things I really liked about data robot, is it really baked in the best I would say is best practices around model evaluation and just general thinking in something like data robot, they have a leaderboard, basically, of all the different models as try but it always placed the dumb model on the leaderboard.

hugo bowne-anderson
Well, you did hire a lot of Kaggle people so I wouldn't, yeah, I'm messing with you.

hamel
They'll always have the dumb model. So they're always have just the average model or the max model or the mean ma Whatever it would always put that on the leaderboard, statistical. And you would see really fast like, Oh, something weird is happening because why is the average model on top like something went wrong, these types of things would really. And there's a lot of things like that. But what I'm getting at is, with auto ml, it's kind of augmenting you and allowing you to like, kind of also check a lot of your biases. So like when I was using auto ml at Airbnb, when I, for a couple of projects, it was super interesting, because people would say, hey, like random forest does not work on this problem. People be very convinced, you know, people develop these affectionate sort of attachments to their algorithms. Sometimes, when you hear a statement like that is pretty your spidey sense should go off. Totally. Because I mean, what do you mean doesn't like I don't know. It's not because of the data was like, something about the data?

hugo bowne-anderson
I always prefer the statement. It hasn't worked yet. Yeah, exactly. As opposed to it doesn't work. Like it's actually a very slight nuance, but it is. Yeah, you're right. We want to be more like pundits, we want to be more certain. Yes, we still don't deal with uncertainty Well, right. Like, we will actually 80% probable then they're like, Get out of here. What's your posterior?

hamel
Yeah, so there's just like, Okay, if you try the auto ml system, on a whole bunch of projects, even at a place like Airbnb, found that, okay, like it really defied people's expectations in kind of counter a lot of biases, but also did a lot of things like found data leakage and a lot of places and things like that, you know, things like, perhaps like you want tools to help you as a human being. I think that's kind of underrated.

hugo bowne-anderson
Absolutely. And I do think the idea of tools to augment what we do, and I'm gonna give, it's not clear to me yet in machine learning what should be automated and what shouldn't I think some of the things you're saying definitely resonate, but I'm gonna give an example that I don't think is as what I'm actually more certain about, and something that may come to it involves GitHub actions, actually, something I'll come back to later. So as you know, I haven't spoken about this publicly, I'm not even sure whether it should be chill i So I'm writing a book, as you know, with my good friend and colleague, Eric Ma, for Pearson on a Bayesian inference and probabilistic programming, right. So we're doing this in a GitHub repository. Eric has done this before, you've done similar things that we'll talk about later with NB Dev. But I was like, Eric, how you want to do this, he was like, Oh, I got these GitHub actions set up that essentially, I write a notebook and is set up some browser based version of VS code with GitHub or something like you'll know more about this than I do. I'm just doing the work, then I like to do pull requests from within VS code to GitHub. And what that does is it turns sets up another branch called Build and like, converts my notebook into LateX, and then converts it into a PDF. So we have a branch called build that just has PDFs of every chapter. We're writing, right? And so what has happened there is that I'm doing what I want to do. I'm writing code and what my time is best served doing right, I'm writing code and notebooks, and then all the other crap that I would hate to do. I hate that man. Right? Yeah, like the computer is the AI. No, I'm joking, with all that to say that, we're not allowed to say that. Us, uh, you and me, other people may and we have to correct them when the AI did this, but then GitHub actions or whatever does it and like that, that workflow augmenting me in that way is absolutely amazing. So figuring out, I think that's uncontestable. That's kind of a good workflow here. It's how we do that with ml. And I love the idea of automated model building hyper parameter tuning once you feed in domain expertise, and features and data and that type of stuff. So that that seems quite, quite attractive. I hope that analogy is helpful for listeners, it's definitely helpful for me.

hamel
Yeah, I mean, automating stuff is really helpful in general. And I think CI is actually kind of cool. CI systems, like GitHub actions are kind of cool for automating all kinds of stuff.

hugo bowne-anderson
So we'll get back to that CI is continuous integration. To me, it's not clear if it's the best paradigm for machine learning workflows. But I want to get back to that, because we're still this is great. I mean, some of this conversation we're planning on having later but we're getting kind of in the weeds with some very interesting stuff to me, based upon your journey. So perhaps, what happened after your data robot?

hamel
Yeah, so it's really interesting, like the little bit just to flesh out data robot a little bit more. I also kind of did some consulting when I was at data robot, like helping customers use data robot. And I saw, like, time and time again, like, people take very practical approaches to I saw time and time again, where, like, simple approaches beat complex approaches, in because we discovered that with the auto ml system, like hey, like, you know, logistic regression like works really well if you do it the right way. And you do the right way. are, you know, you can simplify your whole system by doing using this approach or whatever. That really was interesting. And to see that across hundreds of different companies totally. So then okay after that, so I was living in Boston, and Boston is not Silicon Valley. And you still

hugo bowne-anderson
do have one more question about datarobot. Does that mean, I recall, there are a lot of interesting positions at the time that you're they're called customer facing data scientists that interacted with so was that partially your, your role?

hamel
Yeah, I did that for about six months. And then after that, I kind of did this untitled role, which I just worked on special projects and stuff like that. But yeah, that's how I kind of got in. Is this customer facing data science, special

hugo bowne-anderson
projects? That sounds like some, like, intelligence agencies stuff, like special ops?

hamel
Yeah, it was just kind of all kinds of different things. Really. It was interesting.

hugo bowne-anderson
All right. So then you also moved to the west coast in this journey, right?

hamel
Yeah. So then I decided, okay, it would be really interesting to work at, like, what is this, like Silicon Valley stuff about? This seems very interesting. I mean, these companies seem like they're doing something very different than the rest of the world. Or they seem like there were very ahead. At that time, I thought, okay, like, let me what's a really interesting data science company to work at. And at that time is like, 2015, or 16, or something like that. You know, I thought Airbnb seemed pretty interesting. For that perspective. So I got a job there.

hugo bowne-anderson
Can I just tell people like you like, hey, yeah, Airbnb seems interesting. So I just went and told them and they gave me a job. Talking about how do you get a job at Airbnb when? I mean, both have lots of friends who are working there, and they optimized for true positives. Yes. And get a lot of false negatives in doing that in their hiring flow, right? Yeah. As do a lot of companies up the top. So how do you get a job there?

hamel
Before Airbnb, I got rejected from like 20 other companies?

hugo bowne-anderson
Awesome. The reason that's awesome is because people need to hear that like with careers like yours, people need to understand that the amount of times what I've been rejected as well, right, so

hamel
Oh, yeah, I've been rejected way more than I've been accepted it. But especially in tech interviews, I'd be like, Yeah, I in fact, I didn't even I had applied to Airbnb, so many times, they never even got back to me. It's only when I had a friend who referred me. And then like, I went to this interview. And then I kind of got, I felt like I kind of got lucky. In the interview, even though the problem was something that I kind of, I was able to do well on it. But it didn't have to go that way. They could have given me some other problem. They could have stumbled on it. That's how I feel like interviews are. And to be honest with you also, like I think Airbnb was kind of, it was definitely from the resume perspective, that was the time where I feel like it made a huge difference. Because for whatever reason, maybe for better or worse, like, sometimes having some brand, like that opens up a lot of doors. And so like that was kind of led me to get into like this, quote, Silicon Valley kind of ecosystem.

hugo bowne-anderson
You've also spoken to an interesting concern in the middle there from us from the supply side of the labor market is getting through HR, right? Yeah. So I actually I met a kid. The other night, I met him with a friend had dinner with him. He's from Baltimore, moves to Sydney, his backgrounds in aerospace engineering or something, come to Australia couldn't get clearances. So couldn't get the jobs couldn't get through HR things in tech companies. So this kid, this is a real American Hustle. I'm so impressed. And this is one of the reasons I love America man, to be totally honest, is this kid he recognized in downtown Sydney where all the tech places are, he was like, I'm gonna get a job in a bar in one of those places. And just start talking with executives and let them know my, my thing to avoid going through being rejected by HR, he got a job in a bar. When he starts, he started hearing people talk about tech stuff. He was like, Hey, I've worked with Python work with NASA do this era aerospace stuff. Here's my resume. I started getting offers based on that, man, what a great hustle. That's great, because of the barrier of people teams, right, and HR. And I've got a lot of love for a lot of people, teams and HR. I think the incentive systems are skewed. They have a lot of challenges themselves. I want to be clear about that. So this isn't just us to point out some of the paradoxes.

hamel
Oh, yeah, I would have never gotten into Airbnb. If I didn't have a friend that worked there that referred me, I'm pretty sure. And maybe I probably wouldn't have gotten the same jobs after Airbnb either. Sometimes it can be difficult to break into a system like that. It's funny to say that nowadays because like now, I don't feel like my resume even matters. But that's because, okay, like, you can build a brand through social media or something like that. But it's kind of interesting to think about the data science, industry, job market can be kind of difficult in this dimension.

hugo bowne-anderson
You're blowing my mind right now. And I think for another conversation is the idea of brand building and resumes and I think a lot of what we do is kind of live action role playing and kayfabe right where if you haven't heard of kayfabe this is something I'm fascinated by at the moment. It's the theater of north american wrestling essentially in the simulation aspect of it, but how it bleeds into reality. And once again, this is something Mark Seraphim is very interested in. I'm gonna chat with him on the podcast at some point. But what we are hired to do at Airbnb, and how did it start? And how did it end?

hamel
Yeah. So Airbnb, I was hired to work on growth marketing. So my first project I remember it very vividly.

hugo bowne-anderson
We weren't Robert Chang's team. Was he doing growth marketing? Or was he somewhere else?

hamel
No, he was on, I was on guest growth, and he was on host growth or something like that, like, Sister team. Cool. I don't know if his host growth it was, I think it may have been host growth. I don't remember any more like, but it was on a related team. So when I first came in, like first day, they're like, Hey, we have this LTV model, where we're doing some we have this model in production, blah, blah, can you just take a look at it? And maybe can you improve it or do something? I'm like, okay, so I go there, sit down. I'm like, okay, like, show me the model, like, how does it work? Like, can you just walk me through it? The guy sits down at his computer, he's like, Okay, I have this R script, I run this our script. And like, what do you what do you mean, like this, our script is somewhere else, like, no, it's on my laptop. So that when he runs his our script, the our script is some model that whatever, it prints out some coefficients to the terminal. Okay, like he's using our studio. So he copies and paste the coefficients in the terminal, can he opens an Excel spreadsheet? Yes, any copies these coefficients into a spreadsheet, and the spreadsheet has formulas that then make take these coefficients, embed them in a SQL query, that is put in Excel, then he takes the he copies and pastes this Excel, SQL query that is generated from Excel, he copies and paste that into airflow, which is the system that they were using, and basically this airflow, quote, running inference, you know, linear model that was kind of doing this. I looked at that model. So first, my mind was blown. I thought, okay, like I have gone on this pilgrimage to the Silicon Valley. Yeah. And I was expecting to find something very advanced in probably it was bad luck, or something, or maybe just my funny circumstances. But I saw wow, like, this is some hacky stuff that rivals, whatever crocs is doing, or whatever you might want to say. It really taught me that, okay, like, you know what, this is really all the same stuff, like there's nothing special about, it's not really that special. So we're basically starting from scratch here, like what is going on. And then at that time, there was a couple of ml systems floating around Airbnb. So there was actually an ML system called Air... it wasbased on Spark, I can't really remember what it's called. But it was some kind of, I have to look it up. But it's made by this really smart guy, Hector Yee, you probably have him on the podcast. Yeah, yeah. Yeah, it's really nice guy. But anyways, like the model had to there is a feature in that system where you could do feature crosses, which is basically making your equation polynomial, multiplying features with each other against each other, whatever, had resulted in something like this model having something like 3 million features. So I said, that's crazy. I wonder if there's overfitting going on because, and there was there was some overfitting going on this is a different model, like related to LTV is something else, like predicting growth. And when

hugo bowne-anderson
I hear that I always like you should always regularize, or most of the time as far as I'm concerned, but when I hear that, I'm like, please be regularizing Yeah,

hamel
even 3 million features, he was like 3 million features based on only

hugo bowne-anderson
like, like L infinity regularization or something like

hamel
150 features may see or something it was like, I don't know, exactly, but it was like some ridiculous ratio, but basically, in anyways, I found all these problems, and I decided, okay, it was hard to kind of put anything in production. I thought, okay, like, This is crazy. Like, why is this copy and paste thing, Excel spreadsheet, whatever. There was some data leakage and some stuff like that, then I found that's fine. There's every problem has that. And then I started like thinking, okay, you know what, like, we need better tools. Because how can we do this is copy paste stuff and all this stuff. So I kind of spend my time at Airbnb just building tools like are exploring different tools like how we could what is the best path in Airbnb to productionize ml. And I started working closely with the team, like this ml infrastructure team, that eventually they went forward and built this tool called Big Head, which is ml infrastructure tool that Airbnb made. I don't think it's open source, but it's something that they've talked about and kind of, I think they've used a little bit. But yeah, I just focused on tools. I realized, hey, tools are really important. There's no way anybody can do ml. Now there's other places in Airbnb that were doing really good ml. So there's other like teams, like there's a search ranking team, but they have their own infrastructure, like custom built for that problem. There wasn't something like you can just use this is kind of like a nascent. At that time when I joined Airbnb, like, it was a nascent kind of capability.

hugo bowne-anderson
And also I mean search, right is like a foundational aspect of the business model. And the value prop, like lifetime value was important for optimizing stuff clearly. But if you don't have search as Airbnb,

hamel
yeah, so that's a really good observation. So with the search ranking, that's something that you can see immediate value right away, you know, like, if that's doing well or not, you can see conversions change in a day. And you kind of have really clear attribution mechanisms, you can say, hey, like, No, this search ranking algorithm is making a difference. Versus like marketing is very fuzzy, because like marketing becomes like, into goes into this attribution problem, like, Okay, you showed these ads? Like, did it really give you a lift? Or were the people going to convert anyways? Or did they convert is a double counting from another channel or whatever? There's all kinds of complications?

hugo bowne-anderson
Well, there's the old joke, right? From the 20th century, that's still applicable 50% of marketing works. We just don't know which 50%

hamel
Yes, exactly. I discovered, actually, with this growth, marketing stuff. You know, Airbnb was falling victim to some aspects of the situation where you can't really measure stuff definitively. Or definitively enough? And so what I mean by that is, you know, I was asked to, for example, can we make LTV, like a bigger number, that it's not really big enough? Like, can you just expand the time horizon? Like, instead of like a warrior? Can't you

hugo bowne-anderson
just edit it in an Excel spreadsheet to make it bigger? Can't you just like plus?

hamel
Almost, yeah, almost like, you know, like, it's like, okay, LTV, over what time horizon two years, four years? 10 years? Yeah, I mean, yeah, we can keep making it, like, maybe 20 year LTV. I don't know, what's gonna happen in 20 years is a very noisy prediction. You know, I was asked to do things like that, that was a signal that maybe kind of wasn't going anywhere. But like the story I told you earlier, where you were. So I found that there was incorrect signals being given to marketing partners. Now, and that was very unnoticed. And so that's also indicative of sometimes, when we're asked as data scientists do ml and data science, if they're, like, really pay attention, if the thing that we're being asked to do, is it really driving a business outcome, one way to look at that is like, Hey, if you turned it off with anything different happen, and so like, basically, that's what was happening here. It turns out when like, when COVID-19 sort of hit, there was actually some talk like how, like the growth or the like, stop spending, almost cut their growth, marketing budget, Airbnb did, like significantly, something like 90%. And there were some comments, hey, like, we didn't see any difference, really. So you can imagine like, okay, at some point, that was example where maybe things went wrong. That is something like I think like data scientists get caught in they there's a performative aspect of a lot of times machine learning, and data science, where it's almost as if you can see if you can't measure the outcome, or there's no like direct business value. Oftentimes, it does become performative. So it's something to watch out for, and something that I learned. Anyways, so I was doing that for a while. I was there for about a year. And then I had to move to Portland. And my wife got a job in Portland. So I decided to look for remote companies. And then you watch the works in medicine. Is that right? Yeah, she works in medicine. Yeah. Cool. And yeah, GitHub was like a really cool company looking company at the time.

hugo bowne-anderson
This was pre acquisition by Microsoft.

hamel
Yeah. As a pre acquisition. So I was like, Okay, I'll give it a shot. GitHub might be cool.

hugo bowne-anderson
And just to be clear, I mean, I think most listeners will feel the same way. Whether we love GitHub now, or however we feel about it now. GitHub changed our life. Yeah. And our practice in in a very profound way. Yeah. Both in open source and in terms of building products. Yeah and building businesses, right. Yeah. I

hamel
thought it was cool. I thought, okay, like, this is a cool place. And I like their product. I'll give it a shot. Yeah. Okay. So one thing I want to say about Airbnb, it was like, the people were awesome. I think it's like one of my favorite places I've ever worked in terms of the friends I made. I mean, the people were like, super smart. And kind of just like the friendliest people I've ever met. There's something really special there.

hugo bowne-anderson
Nice, man. You're working. You went into the office? Yeah.

hamel
I went into the office. So like, Yeah, I'm in it was like super talented group of people, like working in data science, even though like I found various problems that I talked about this not any reflection upon, that's more reflection upon like the much larger system. There's not like a I don't think that is really has anything to do with data science, per se. I think data scientists, they're pretty sharp. probably the sharpest is I've ever worked with, to be honest. Most of it is like similarly, like GitHub. So like went to GitHub, and GitHub. It's pretty interesting, like so I started also kind of started out with building ml infrastructure at GitHub. And then at some point I started building tools in open source. So

hugo bowne-anderson
so what I'm hearing is you're doing less data science at this point and getting more excited about building productivity tools for data scientists.

hamel
Yeah, I was getting more excited about that. Just because you know, all the problems I encountered so far, and the things I shared about, you know, sort of whatever is happening, I thought, okay, maybe I can affect this with tools or help people with tools. And so GitHub, I started doing that. GitHub went through some tumultuous time where their data science, they didn't have any, like, data science leadership was like, completely gone for, you know, the reorg, the company, and it was just kind of an orphan group, sort of, so we kind of worked on whatever we wanted to how did that feel, man? Yeah, that was kill the challenging, but I kind of made the most of it in terms of I thought, okay, like, I'll just work on stuff that I think is interesting. So I just worked started working on open source stuff. So that's where I met it, a whole bunch of that was pretty unsuspecting, I wouldn't have predicted that. But I started working on open source and met people like Jeremy Howard. And just started like working with on projects in open source

hugo bowne-anderson
And that relationship with Jeremy turned out to be beautiful and so productive, right? Maybe tell us about that.

hamel
Yeah, that was really productive. So the way it came about, was, okay, so I've been a student of, I've taken fast AI classes and stuff like that. And I think Jeremy got to know me a little bit as a student of his class, you know, he'd always do these things like support students by sharing their work and stuff. So he did that with stuff I did, I actually created this project called code search net, based upon stuff I learned from Jeremy. So it's basically code search net is like, I think it's github's like first representation, learning with ml of code. So it's sort of so it's a dataset that we shared, the head is a parallel corpus of code in natural language. And then we created a bunch of reference models that build embeddings of code, so that you can like search code, with natural language, for example, for information retrieval,

hugo bowne-anderson
I mean, I'm trying to read between some of the lines here, you're able to then collaborate with Jeremy on some of this stuff, essentially, that will not essentially like the Creator, co creator of ul emfit, which turned into GP three,

hamel
no, so I didn't get to collaborate with him on this project. But it was interesting, because I took some of the stuff I learned. And then I got to collaborate with a bunch of other people like in Microsoft Research, and all kinds of cool people from different places. That was really big at GitHub, that was a cool project is more of a research project that did get people really interested. And like this idea of representation learning of code, and maybe you can do something with the corpus of GitHub data and deep learning, and some of those people working on a co pilot.

hugo bowne-anderson
That was when my mind just went actually, yeah, so

hamel
it really generated some of that interest. I showed that project to Jeremy and Jeremy got really excited about that. Because it was using a lot of stuff I learned in fast AI and some stuff from fast AI. And then also, he came to GitHub, even he visited gave a talk and get up and you know, started to get to know him a little bit better. And then GitHub actions came out. And you know, I didn't even know see, I didn't understand how CI like the systems worked. I've only seen

hugo bowne-anderson
 Can you tell us a bit more about that? Because I talked about how I've been using GitHub actions for some stuff recently. But can you just give us like a very brief elevator pitch on continuous C ICD in general and GitHub action? Maybe?

hamel
Yeah, I think continuous integration is like pretty confusing jargon. To me, Ci is basically a thing that allows you to run arbitrary code triggered by various events, you want to run some code, when you push to a GitHub repository. Great. You want to run some code 9am Every morning, great. You want to run some code, when you make an issue, comment, or you open an issue. That's great. You want to run some code when you open a PR. Okay, so what do I mean by running? So what kinds of things do people run? The most common thing is to run tests, so to say, okay, like, I'm going to open a PR, I'm going to take the code in the PRM and test it. And so it's integrated with GitHub, and it's usually used for testing. You can run anything on and you can basically use it to automate stuff.

hugo bowne-anderson
Yeah. And so the example I gave, of course, was I put something and it converted through a series of actions into a PDF that I can share with my editor. Yeah, exactly.

hamel
Yeah. I mean, if you have a workflow, that where you write code and you want to take that thing, and basically automate it through something with it, it can be good. So

hugo bowne-anderson
what's the CD part? Maybe you can tell us what CD is then yeah, CD

hamel
is stands for continuous deployment. So the idea that you can take your code, like let's say you're building a website, or an app of some kind, is like you can build it. When you merge your code into the main branch of the repository. You can kind of build your thing and then push it to wherever to serve it. You can hand it off, or you can do the steps required. to actually put it in production or do something, that's the CD counterpart. So I was always also, like really fascinated by these systems because I didn't really understand how they work. But I thought they're super useful. Like, I was like, Oh, wow, this is like really cool, open and PR, and you get these colors tell me like green, yellow, red, and like, seems really useful, but didn't really understand what was happening. And then github came out with this product, GitHub actions, which up until that time, you had to put third party stuff in your GitHub repo that said, Okay, this is cool. I'm gonna, like, learn this now. Because like, I work here, and then maybe I can, like justify my existence a little bit better if I like, get into this. And I thought, okay, maybe there's some interesting data science related things I could do. I don't know. Let me just check it out. Yeah. So I got to know it really well. And then I decided, okay, like, what's really interesting is one idea that came to my mind is, what if I could build something that takes notebooks and makes it into a blog post? That would be cool, because I always want to make a blog post but notebooks. So blog posts is like, you start writing and medium at that time is like, Okay, write some stuff and medium copy and paste code. Now, if you change your code, you have to copy and paste it again. That sucks. Copy and paste. I used to use

hugo bowne-anderson
something. I don't know if you ever use pelican. Yeah, like there were a few things floating. It wasn't automated per se. Like I had to do a bunch of Yeah, tech judo and essentially, like cable management, like jamming stuff together.

hamel
Yeah, like Jake Vanderplas had created though I think like a static site generator. I think he used something where he did this gluing, but

hugo bowne-anderson
I think Chris album did something similar for his early books as well, but I don't know.

hamel
Yeah, I was really fascinated by that. And I thought, What if I just made a cool project would be like, you drop a notebook into a folder in your repo and boom, is a blog post that can I do that? Then I saw like, Jeremy has a new that he used this project called MB Dev, which I was like super fascinated with didn't really know too well. I was like, okay, it kind of messes Jeremy one day, I said, hey, what if we do you think this is a good idea? Like, oh, before that Jeremy had this project called fast blog or something past pages? No, it was before fast pages. There's something else like, it's not even using GitHub actions, or it did not know books or anything. It was just you can use a Word document and becomes a blog post on or something like that. And then I was like, hey, like, what if we could do this with notebooks? And he loved that idea was like, Yeah, that's great. You should do that. And then I created this fast pages. So I created those fast pages. Then I said, Okay, Jeremy, let me set up CI CD for this. Get up actions on all your repos. Whatever repos you have, like, let me I think you should automate some stuff. He's like, Okay, that's great. And I started working with him. And like getting into all this stuff, setting up tests everywhere in the setup tests everywhere, really had to understand like, What the hell is going on. And then I started getting deeper into his projects, and more into just by the nature of what I was doing, I was getting into the development or his development workflow really deeply. And I thought, okay, is this development workflow, but no, Jeremy's development workflow is super unique. It's like, he built an extension to the Python programming language, which is fast core. And he has his very own development workflow system called env. Dev. And that was really fascinating. So I had to learn those pretty deeply. So the way I did that is I basically went on one month documentation sprint of fast core and documented everything I could about it. And then MB Dev, I started basically contributing the project and trying to improve it and taking some of the stuff I learned from fast pages and reorganizing MB Dev, and really getting into it and improving it and all that stuff.

hugo bowne-anderson
Can you give us the elevator pitch on NB Dev? I've we've talked around it a bit, what exactly does it do? So MB

hamel
Dev is a really exciting project. It's literate programming. So what is literate programming is this idea that you should be able to write your code documentation and tests in one context. So you shouldn't have to write code in one place, then put your tests in a different folder, and then put your documentation in a different folder. Because the reason is like, okay, nobody writes documentation. And very few people write tests nowadays, despite what we tell people to do. Like we kind of have to beat people over the head. Why? Because it's not natural. So they're programmed this environment has created is sort of you can do all that in one go kind of and it encourages writing lots of documentation and tests. And so we create a lot of videos on it. And something I really encourage anybody listening to this to go check out

hugo bowne-anderson
Yeah, and we will include links to the all the videos and docs in the show notes. And if you haven't heard my interview with Jeremy Howard, we actually talk about MB dev a lot as well. So definitely check that out.

hamel
Yeah, I think it's great actually blew my mind in the sense that I thought, wow, like this MB dev stuff is maybe more impactful than even deep learning, like, deep learning is pretty impactful. And the fast AI library but I thought wow, like this apply As to all of software, I can make high quality software that's documented, tested, and that other people can get into, or they can understand just by the nature of how it's built.

hugo bowne-anderson
And you only have to write it all in one place in your notebook. Right? So that goes through the documentation that goes into your docstrings. It goes everywhere. It goes to your help functions, right? Yeah, it's pretty robust.

hamel
And so then Jeremy, and I kind of did some really cool projects, too. We just things that are not data science related at all. So we ended up making a GitHub Python, a new Python client for the GitHub API. That was really cool. It's called G H API. Is that doing MB dev we created? So the CEO of GitHub at the time, Nat Friedman, he had this side project where he was taking the GitHub event stream and making it into like a news feed in the CLI in the terminal. Like where you just stream it all. It's actually not that much as it seems, you can kind of stream it in your terminal. And you know, he kind of built it out. And then Jeremy looked at that I looked at I was like, at that time I was really fascinated by this project called Rich is built by will think I don't know how to pronounce his last name. But you know, I'm talking about the Scottish got mcgoon McGoohan. Yeah, yeah, yeah.

hugo bowne-anderson
And so we'll include a link. I love rich. I love textual. He's doing cool stuff. He's actually hiring at the moment as well. But he's turning the terminal into all this like cool browser stuff as well. I mean, he can do the pitch way, way better. I should have him on the podcast at some point. Really nice stuff.

hamel
No, it was great. So the really cool thing about this story is, so we rebuilt that streaming project thing in rich, we took it and we said, okay, like we're going to stream all GitHub events. I'm going to show it in this nice interface in the CLI using rich, then we ended up sharing that and then we'll made this blog post that says, hey, you know, like, I was just this rich thing was just a side project. But this thing that Jeremy and Hamill built, it blows my mind. I'm gonna make this into a product. This is crazy. Like I'm gonna make texts. I'm gonna make a terminal based applications because this is I'm like, wow, okay, like inspired somebody to do something with their product. And it's really fun. Rich's like an amazing product. We it's so much fun to use.

hugo bowne-anderson
Yeah. And also bashed client is awesome. In general, like, bring back a lot a lot. I can remember one of the first times we met you introduced me to a CSV kit for like data cleaning in the terminal. That blew my mind, man. Oh, yeah.

hamel
Jerome Janssens. He writes his book in this book, data science at the command line. That's, that's an awesome book.

hugo bowne-anderson
It's awesome. There's a second edition as well. Okay, we're going slightly off topic now. So let's bring it back. I want to know about what then happened to GitHub, and then your journey towards us working together out of bounds. And then I think it'd be great to drill down into more about the tooling landscape, and how much you love the tragedy of the commons we're currently experiencing. And literally,

hamel
basically, I worked did a lot of work with Jeremy, it was a really cool experience. I got to learn so much by working with him closely. So then they kind of came up on like four year mark, or four and a half year mark of at GitHub, had a second child went on paternity leave

hugo bowne-anderson
congrats as well, by the way. Yeah, how old you your youngest now,

hamel
six months. So cute. Yeah. But then I've always been interested in tooling and various kinds. And so at GitHub, I was using all kinds of different tools. So we were using stuff on AWS, we were one time we built our own tool, we had used Argo, we did kubeflow, I also got really involved in coop flow when I was at GitHub, the coop flow community and working with that. So we like we use some of that stuff. And then eventually we got by Microsoft, we use like Azure ML and Azure ML is basically a white labeled version of ml flow.

hugo bowne-anderson
getting a headache already, man, just hearing. Like, you got a great blog post as well, about an end. I'm going to read this and end to end example of deploying a machine learning product using Jupiter Papermill. tacked on get ops and kubeflow. What is our life dude?

hamel
Yeah, it's pretty. There's too many tools out there. And it's not tenable in the sense that if you're working in a company like GitHub, in your data scientists, you can only engage with two to three tool vendors, max, doesn't matter how many problems you have, you have a fixed budget of time, because like every time you engage with the vendor, it actually takes a lot of time, you have to talk to lawyers, you have to like fill in some weird contract thing. And you have to get it through all kinds of processes and red tape. And it's like, it's not easy to buy stuff, usually. And so you can only do that a finite number of times. And if every tool is only covering 4% of the surface area. I don't know why I picked that number, but whatever.

hugo bowne-anderson
Like very small, I wish it was 4% I'd be happy with 4% I think it's like

hamel
one person and things like 2% 1% is a very small this is so many there's so much surface area. And there's like so many point solutions out there. I mean, you have tools for surveying, for labeling for experiment Tracking Model workflow systems here. Model monitoring. It just goes feature stores, feature stores, metrics, stores,

hugo bowne-anderson
metrics, stores, experiment tracking data cleaning data validation

hamel
stops startups. Yeah. Is that a term? You just make that?

hugo bowne-anderson
I don't know. It's pretty good. Yeah, no, well, I'm just doing like jazz riffing on regex, I suppose. Right? Yeah, I want to bring back regex into just regular conversation.

hamel
Yeah, you know, there's so much fragmentation, there's not really tenable to see composed your data stack with 30 different vendors, there's no way anybody can do that. There's, it's impossible, nobody has enough time to engage with 30. And 30 is a generous number. It might be slightly more, but 30 is like pretty realistic in terms of like the number of different vendors you might want to go to, if you want to truly like today, if you want to use this best of breed, thinking of like, I want the best tool for that thing, then yeah, you might need 30 different tools. There's certainly tools that are more monolithic in nature that cover like more surface area. That's interesting.

hugo bowne-anderson
My mind just went to open source, right. And we always say that this is almost a trope in our industry now and across Python in general. But Python isn't a best of breed tool, right? We've always called it a Swiss army knife, right? It's never the best it will it's very rarely the best tool for any given job. But you can do all the jobs with it and glue them together. Right? Yes. So I wonder how that apply. But that's open source. I wonder how that applies in a productized space when the incentive systems are different. I also want to add, as you know, the past couple of weeks I've been in a kind term would be dependency purgatory. I've been in all forms of API hell, to be honest, across spread across many different products and many different services. I think I sent you a series of incredibly frustrated messages about... to step back. My background is in research science. And part of my mission, and part of my job is to help I want to help 10s of millions of scientists do industrial science, essentially, right. And they can not be expected to do what I did the past few weeks in terms of like, after dinner for five hours dealing with permissions and roles Amazon sage maker, right. That's very concerning. That's what people need to do currently, also concerning that, that's what companies pay PhDs in astronomy to do, right? Like there's totally impedance mismatch there as well. And I was ranting about this to our colleague, friend co Ville Tuulos, right, because he's such a patient and generous individual. He was like, I understand your concerns you go and it's totally valid. But they said to me, dude, what you've just done is you've trained like this wild LSTM and set it up on an endpoint that's accessible globally, man. He was like, What are your expectations? He pulled out his phone and said, These things only came out in 2008. Like satellites only went up x years ago. He said, I totally understand your concern. But this is magic. Also, like this is absolutely incredible. What's happening here? And I was like, wow, yeah. It's kind of like, you know, that joke that if you can't get the internet in a plane, now you're like, Oh, my God, where's the internet? I can't send my emails when you're flying in a, like a metal container going around the globe? Like, yeah, when did your expectations get so so I feel like Ville kind of set me straight. That doesn't mean it's not the most frustrating shit, right? Yeah. But the transformation of expectations, I think, and recognizing what you're achieving, as well, that was kind of a slightly incoherent rant. Does some of that resonate. Now, that resonates a lot. We're able to do incredible things.

hamel
Okay, like the fragmentation the tool space, you have millions of different vendors, whatnot. And I think, you know, in some sense, it is a giant human grid search on what are some good tools, it will probably, you know, it will result probably in better tools, I'm pretty confident that it will. And that's a good thing. Now, it's kind of feels crazy being in it at the moment to say, Okay, there's billions of dollars, perhaps trillions of dollars being invested in this space at large. It's pretty noisy. But I think people also recognize this super valuable, and that there's lots of opportunities, and that's why they're different folks are trying to build good tools. I think the current state is, I think, if you take the really long time horizon view, yes. It seems like we are going to end up in a better place with better tools. But you know, it's a shorter time horizon. Okay, when you opened your podcast, I think it was like a five minute 15 minute thing and term data science was coined and, I don't know 2014 Or something like that.

hugo bowne-anderson
In 2011, it was actually coined in 2008. I think Jeff hammerbacher coined it then but then popularized with that HBR piece by I actually looked that up because I was like, was it coined or was it popularized so that's why I

hamel
see okay, but yeah, you open with to say, hey, it's been popularized such and such a day. And then now we're here and nothing. Things seem like they're still frayed in terms of retooling. Like, what is happening?

hugo bowne-anderson
Yeah, I think I actually said something like, we're here? And then I was like, Wait, where are we? And that's what we're doing, man. I don't even I'm trying to find a compass. Yo,

hamel
yeah, it's very interesting, like what has happened, you might have to take a really long time horizon, like, as Ville is alluding to, like, I remember when I first started with this stuff, I've seen a lot of tools come and go. And a lot of movements die out. Like for the one most notable one is this Hadoop, big data kind of movement, where everything was like MapReduce, and you build algorithms. And I remember there's this framework called MHout on Hadoop where you did ml in Java, it was really painful. Like no one liked it. There was a lot of uncertain moment in time you thought, okay, like relational databases are kind of not cool anymore, not going to be used for data science, there was like, certainly that kind of at least marketing. And like, we reverted all the way back really towards something that looks like more traditional, like for a lot of things, more SQL interfaces. I think that's really good. We had to do all of that exploration, it seems, seems like okay, sometimes you take one step back to go two steps forward or whatnot.

hugo bowne-anderson
What are the forces at play? And I think talking towards cargo coding, at some point would be useful. But also, perhaps we can bring it back to giving listeners unsolicited advice about how to find tools that will help them in a landscape, which seems to be difficult to navigate.

hamel
Yeah, well, I think one thing that trips people up is how to think about open source tools. One thing that I think is useful to understand is that open source tools often are not meant to be something that for you to use, or that's not the only goal. open source tools have so many goals can they can have so many goals. So one is recruiting. One is marketing of some kind. Another one is, hey, like, sometimes at companies, career ladders and things like that, open sourcing your work is a way to kind of push yourself further along on some kind of ladder. Also, like some people just, I've even seen things like some people just want access to their work later on their own work later on. Say, Okay, like me open source stuff. So it's not so much about, hey, here's a product that you can use it to understand, like different products have different goals. And then people can kind of get carried away and say, Hey, like, Oh, why don't you just use such and such tool like Google is using it? Facebook is using Netflix, using it, whatever. And you have to really think like, Hey, is that a product that you the expectation is not? No, you just use it off the shelf? It's going to work? Because SO SO and SO uses it, it's you have to really think about like, what is the goal of that tool? Like is it really meant for you just pick it up off the shelf and use it? I think people forget about that get lost in that.

hugo bowne-anderson
I agree. I think there's a lot this covers a lot of ground could be grounded in an example. What example comes to mind?

hamel
Yeah, one example comes to mind is TF x. So TF X, I gave a talk about this in chip humans, Stanford sis ml class, which is a great class, by the way, she's actually does excellent work around ml systems, and tooling and stuff like that. Teaching people, TF X is pretty difficult to use. For most people. It's not well documented. There's a lot of places in documentation where literally, the documentation says, this is a function, am I making that up is actually exist. It's pretty high complexity. In some cases, it's not like really gentle sort of onboarding are sort of disclosure of complexity, like you jump right into, like, very high complexity. API's can be really confusing. It's super fragmented, so on and so forth.

hugo bowne-anderson
And what's the intention of TF X? What does it claim to help you solve?

hamel
Yeah, TF X is like, help you productionize your deep learning models that are built with TensorFlow.

hugo bowne-anderson
And then what are the other goals that the people developing? Do you think are trying to achieve? If it isn't? Only to help people do that?

hamel
Yeah, some bit of it is marketing to say, hey, like, these are our thoughts around production using ml. Look at us, we are the leaders of that you should use TensorFlow because we give you tools for that. But it's also like, Hey, you should think about using Google Cloud because we are the people that we create. We create good ml stuff and or we know something about ml. So you should use our products, you should use Google Cloud. I mean, it's not really you could really see it like in the sort of documentation and some of the tutorials, a lot of this stuff like nudges you towards the Google Cloud, like some of this stuff is easy on Google Cloud or whatnot, that may not be not so easy, outside Google Cloud. And a lot of it is kind of like a it is a marketing sort of effort. And then like some of the biggest customers of TF X, like Twitter, Spotify, for example. They are using Google Google Cloud for this very reason. Because you know, that's probably where if you are going to use this A platform for it to be tractable, you probably have to use Google Cloud. And plus, that's probably the way that you're gonna get support for this stuff. So it can be tempting to just cargo code this stuff and say, Hey, like, this is the best tool like Google is using it, they're smart, you should do this. And I think cargo coating is something that you have to be it's not just ml me, you really have to be careful of it. In general, AI can cause a lot more harm than causes than you expect. With regards, you know, a lot of things that are meant for Google or someone else are probably not appropriate for your company at all. And in fact, that will cause you a lot more technical debt, and a lot more problems than they're worth. So when you mentioned Kubernetes, yeah, I mean, Kubernetes does help a lot of people put a lot of problems, but it's also super complex. There's definitely a lot of situations and Tech where people probably overreach for Kubernetes when they don't need to be overreaching for Kubernetes. And so I think it kind of applies to ml two, I think, like,

hugo bowne-anderson
it's tough not to use these things when they're on the billboards, you're right. I mean, there are many different analogies we can use for the landscape. But it does feel like Times Square sometimes, right? And Google, for example, along with a handful of other companies has a huge amount of purchasing power in this landscape. So we see these flashing lights, right? So how do our listeners? How do our friends how do our colleagues, how does our community how do the next generation of data scientists and analysts navigate the tooling space? When we've got these giant billboards in front of us? How do we find signal on what we can use them?

hamel
One thing I would say is, and this is what I learned from working with Jeremy, for example, is learn to trust yourself. If something is not intuitive to use, do not buy into this idea that you are stupid, or that these engineers that are much better than you are doing in a better way and use have to strive harder and learn something, you have to like graduate out of your ignorance. Now, of course, like you should always be learning, but I think don't throw away all of your knowledge and your perspective about what tools fit your workflow. And if something is not well documented, it's very clunky to use, the API's don't make any sense. Or they're super complex. And it's not really clear why if you have to really ask yourself, like, why am I using this tool? Is it because Google is using it? And often the answer to that question is yes, that's the only thing you can come up with. You know, you really have to introspect and trust yourself and say, Hey, like, do I like this? Or not? Because we are constantly being bombarded. Like you said, in Times Square, and saying, You don't know what you're doing? Only if you could use our tool, we'll show you how we did it at Uber at Netflix, or Facebook, or whatever. And then it's tempting to say, I mean, because human nature is okay, like, yeah, of course, maybe I'm missing something. And maybe I can learn from something, which is the case, but there's a limit to that in to say, hey, like, actually, no, it's not okay, for things not to be documented. It's not okay. For API's not to make sense. It's not okay, for example, for something to just jump directly into distributed computing, when you don't need it. Do you really need this complexity? You have to always ask yourself?

hugo bowne-anderson
Absolutely. That's a beautiful point. And I do think something we think about a lot. I mean, a huge part of our job now is to manage and time complexity, right? And yet, we always seem to be reaching for complexity, not only with tools, but with models, as well as like, as we were saying earlier, it's like, Hey, do we use these very complex models when we have a baseline? That's kind of chill? So how do we navigate this paradox of managing complexity without the force is constantly pushing us there? Yeah. And

hamel
I think there's a lot of incentives to push people towards complexity. And this has reared its head. And this is actually somewhat exposed by fast AI. So there is something called the dawn bench challenge, which was how fast can you train ImageNet to convergence? And there was a leaderboard of this for some time. And, you know, it was like the people on top of the leader board were basically using lots of compute. And, but fasciatus said, okay, you know, what, like, let's take a different approach, like, can we think more deeply about what we're doing? Do we really need that much compute? Can we use hyperconvergence techniques? Can we change the way we're training, let's think really deeply about it. And they blew away all the benchmarks using something like 10x, less compute. And this is going against Google, Stanford, people, all kinds of open AI, all kinds of people.

hugo bowne-anderson
Yeah, I mean, I was actually like, I know, Jeremy is fantastic in so many ways, of course, but when like I read, fast AI, like started to outperform, like TensorFlow, and Pytorch and all that. I was like, wow, I mean, seriously, man.

hamel
So really just like paying attention to a lot of times complexity, it's sold to us as something that is desirable, in some sense, to learn into kind of skills to acquire, let's say of like, how to deal with like, very scalable systems and things like like that, and then we can get too biased in those things towards those things. To be frank, a lot of career ladders in tech are geared towards awarding people with building complex systems.

hugo bowne-anderson
Yeah, I totally agree. I am also interested in thinking about like, we're trying to think about navigating the tooling landscape. I think the idea of following your intuition and being honest about your skills, and what tools help you with those is really important. What would your advice be for things to people to learn when starting out, like going from junior to maybe something a bit more senior or Okay, let me reframe it. Let's think about the Machine Learning stack more generally. And as we discussed is all this like data validation, feature monitoring, feature engineering, experiment, tracking, metrics, stores, feature stores, all of this stuff, let's say someone can use the PI data stack, or the tidy verse to like, do pretty well in a Kaggle competition, like they know all the tools, but they want to start really getting in the weeds of learning the ops staff and this type of stuff and taking things to deployment. It's so overwhelming, where can people start? And what progression? Can they take? I suppose,

hamel
yeah, my answer to this will be to get better at asking why. So I think kind of to circle back to sort of where I began, this data science journey is to really ask, I think it's good to learn tools, like what you described, but I think you should always when you go through this journey, you know, it's not linear. And I think it is worth after learning some tools to kind of go back and really think about like, why are we doing Why are you doing this? Like, why are you doing this data science stuff? Are you doing ml? Like, why does it matter? Why does it matter to the business be very skeptical? Like, is the data Correct? Where's the data telling me? Let me learn the domain interview people that are working in that domain, go work in the domain yourself, be really skeptical of everything, like why you're doing it? And if it actually provides value? And I think that's not really a satisfying answer of like, tools to learn. But it's

hugo bowne-anderson
no, no, I love it. I love it. Because it provides not content, but a heuristic a rule of thumb, a methodology. It's not, like I say, teach them how to fish. And you're like, hey, I'll teach you how to think about where to find food dude, like, I can teach you to fish. But if the river dries up, you're done. So let's think more foundationally about it. So for example, using that methodology, I can tell you, several years ago, I would build machine learning models, and I'd have a spreadsheet open putting down scores and accuracy and all of this stuff and like writing, dude, I pretty much built a spreadsheet on my forearm at one point writing down like model diagnostic stuff. So painful. When experiment tracking became a bit more evolved. It was like, oh, that's something which will will help me right here. So start using experiment trackers because of the pain and the suffering. Now, as you well know, I don't think in my humble, very humble, whoever says in my humble experience, I mean, is there anything humble about me having an opinionated take on this, but my hot take? Is that experiment trackers for the fantastic, but they solve something symptomatic downstream, very useful. Don't get me wrong. But I think the real question is upstream, and has to do with versioning, and artifacts of model runs that are in production and experiments. And as you know, that's one of the reasons I think metaflow solves that really well. That's one of the reasons I'm really excited to be working on metaphor with you. Well, that's for another conversation. But I think your heuristic of thinking about what where your pain is where the paper cuts are, I actually don't think we want to solve all the paper cuts there are people say, 1000 paper cuts, like what's a few orders of magnitude between friends, right, as my postdoc advisor used to say, but it's millions of paper cuts, right? And we don't necessarily want to solve all of them. But the ones that keep coming back and keep recurring, finding the tools, which help us to solve that and building community around these types of questions as well, I think and conversations like this, hopefully, I think one of the challenges of building tools and finding which tools to use, it has to do with pattern recognition, right. So we know in analytics, for the most part, kind of the types of things we need to be doing, right? Like there are errorbars on that statement about far more than we do in machine learning, like machine learning at crocs versus machine learning at Airbnb, versus machine learning on Antarctic research stations versus machine learning in real estate versus machine learning in biotech. It's not clear what the actual patterns are yet. So once we start to as a community, develop a shared reality around the collective patterns and figure out which tools are useful for those things, I think we'll start to make a lot more progress.

hamel
Yeah. And it kind of like what I was getting at is just being better at sort of connecting with the problems more. I mean, it's hard to do that at that too much at the outset. Because like when you're learning to your question when you're learning progressing through data science and kind of building up your skills, it's hard to it because if you try to just do this stuff I describe asking why learning about the domain interviewing people too much at the beginning. It can kind of feel like a lot of pontification. But if you kind of get some of the skills and you are able to like looking at data, you can then kind of go back and have a different perspective. Totally. And, you know, this leads to a lot of things like being able to smell problems of various kinds with data, knowing where to get data, what questions to ask, and really the ultimate question in life is like what I should be working on. Exactly. And this, you know, I think that's also really important in data science, should you even be working on choosing the problem to work on or what you should be working on is really important. And I think that's probably the most important thing. And the way to build that skill is to kind of go back and say, really introspect. Like all the problems that you're working on.

hugo bowne-anderson
 This actually comes full circle very nicely to your work at Accenture, where you professed to go in there and actually, like, look at the data, figure out what problems sit down with people and think through what they're actually doing business wise, as opposed to going in and being like, Hey, we're gonna build this deep neural network and deploy it on Mars, for example, which clearly wouldn't happen. Yeah. So we've been talking about tools a lot and ways to navigate tools. There's a paradox here as well. And you and I hate conversations about tools as well. But we know we have to have them. And I think I hope that this is instructive. And I'd love any feedback from people listening to let us know. But my question for you, my dear friend, is, with all the current focus on tools. What important conversations are we missing? Having? Yeah, I

hamel
mean, I think it's a lot of those things around setting the right expectation of ML, like, what is ml? When should you be doing it, when you should not be doing it? When is it a good fit, making sure people have the right sort of understanding like, of that, you could see this happening in tech a lot. For example, some young companies who are growing, they might want to build some ml capabilities for something they might think they need ml. A lot of times, they'll just hire a bunch of data scientists, when in fact, that's probably the wrong way to go about things, probably want to hire some data engineers and some infrastructure people and maybe like, maybe one ml person, maybe

hugo bowne-anderson
yeah, I'm very excited about the rise of data engineering. Now, it's been far too long coming. And it's arisen out of pain as well. But the fact that there's such an incredible focus on it that's still growing, I think is a really positive firm for the field.

hamel
Yeah. So I think continuing to have that conversation is important. And just kind of having more and more domain expertise, as well is like, you know, I think that's really helpful and not doing data science in a vacuum. It's really hard to do that, though. How do you do that you have all you have to learn all these algorithms, if you learn programming, or infrastructure, all of these things. I mean,

hugo bowne-anderson
it's funny, this actually comes back to Tools for me, Ah, man, and this is my frame, maybe I needed to put on some different glasses or something. But the way I think about it is, of course, data, scientists need more domain expertise. But domain experts need tools and workflows to help them do the data science and machine learning as well. And I think that's probably as Jeremy Howard has said, like, people go to med school for a decade, lawyers study for however long, right. So getting that domain expertise into a data scientist is highly non trivial, if possible at all, as opposed to getting good tooling and good workflows in the hands of experts. So I think both need to happen.

hamel
Yes, I agree with that. And that's why I think it's important, at least, to try to understand the domain, because it's really so critical and being effective in like to be honest with you. So you might ask me, Hey, like, I have all this criticism about tools, or too many tools. And of course, I'm working on tools. In some sense, it's very meta, because the domain I understand is data science. And that's why I'm building tools. Because I actually, I understand data scientists, I may not understand lawyers very deeply, or doctors or anything else. But I understand data scientists, I can build tools for them. So I think in the sense that kind of, you can say the same thing about data scientists like they're essentially, if they're building tools for others, I think it's important to engage with the domain. And but it can be very tricky because that affordance is not always given to you, you kind of have to reach for it. Expectation sometimes is, hey, you're gonna sit down on a computer, start building a model right away. And that ultimately, can be disruptive in the sense that you may not be working on something valuable.

hugo bowne-anderson
Exactly. So where I'd like to go now is I think we have been critical, but constructively. So as well. I do want to end on a positive note, because you and I are both very excited about a lot of things in the space. So we discuss your excitement about auto ml. I'm wondering what else there is in a space that you just makes you happy or that you're excited about for the future of data science and ml?

hamel
Yeah, I mean, I do think that tools are going to get better and are getting better. You mentioned Python. I do think very long time horizon. I don't think anytime soon that there will be a better programming language that will be popularized, that will probably fit the data science workflow better. And I think it's really like a lot of fun actually to it. explore different programming languages. And I highly encourage that. And I think that's cool. I think there's some fascinating work going on there. But it's very, Python is still going to be here for a while, deep learning is really cool. It really unlocks. It's not about doing things at a massive scale necessarily, like unlocks different things you can do with ml, that you just couldn't do before. Like some things that are made a lot easier with deep learning. And so I think like, it's exciting to like, explore that

hugo bowne-anderson
two come to mind for me to computer vision. And now natural language stuff. And transformers are super exciting as well. These are the types of things you're thinking about, or

hamel
Yeah, and I think it's gonna be really exciting to see what is beyond transformers? Like, is there something as more research happens in the space? Can we go back to something has like, more tractable in it's compute over time, things like transfer learning and stuff help a lot with that, of course, but

hugo bowne-anderson
transfer learning is actually don't talk enough in public about how much I love transfer learning.

hamel
Yeah, it's great. I'm really excited to kind of hear more, I think one subject is not talked about enough is I don't

hugo bowne-anderson
want to just flag sorry, I think transfer learning also has very deep implications for privacy and security, that aren't talked out about enough. The fact that we think about privacy of data, don't do enough on it. But the type of data that can be encoded in what models and then using transfer learning. I'd love to find out more about that. So but my spidey sense, once again, tingles with respect to that, like, we're gonna end up in a place with like algorithms sold on marketplaces and stuff like that. Right. So thinking about the implications of this, but I'm sorry, I just wanted to flag that there are concerns about transfer learning as well.

hamel
Yeah, yeah. I think like one area that I think is pretty fascinating, that doesn't really get that much attention is active learning. With this, like, how do you the practice of efficiently using your data, and like labeling it in a smart way

hugo bowne-anderson
weak supervision, semi supervised learning, all of these things are fascinating.

hamel
Yeah, he's like, these situations where you really don't have that much data? Plus, you need to find a way to label it, or what does that even mean? That stuff? That's almost everybody who's doing ML is doing to some extent, but no one really talks about it. But I think that's really interesting. And I think it's like, continues to be like, pretty exciting. So yeah, another thing I mean, I'm excited about, I still remain excited about, even though it's not has anything to do with necessarily ml or data science, like literate programming, then that's super powerful to kind of see where that might go. And like bring more people into software development, and maybe enable different kinds of software development.

hugo bowne-anderson
I love literate programming. We've talked about this before, and I don't, but the fact my background is in scientific research, I work with biologists who have their lab notebooks, right, so what they'll do is, they'll write down their hypotheses in the notebook, then print out some data and stick it in there then put like a photo of a PCR run, then all of this and the basic, like one of the initial kind of inspirations behind the Jupyter, notebook. IPython notebook, of course, was to give people who are using their computers as laboratories, which is what we do in so many ways, right, we have an experimental interface right in front of us, you know, in our laptops to have a computational analog of these lab notebooks. And that's kind of part of the power of literate programming for me as well.

hamel
Yeah. And another thing kind of related to that I'm really excited about, you know, I'm working on at outer bounds. And we're open sourcing there is, is documentation systems based on literate programming. So let's say you don't want to do literate programming, you know, or it's too, let's say, esoteric for what you're working on. One thing that I think every everybody can relate to is the pain of code documentation. So like, how is code documentation done today, it's very backwards, like you copy and paste code into markdown files, you can copy and paste the output of that code into markdown files, then you maybe write some pros in that. And then oftentimes, like those code in Markdown files is not tested. There's not like, necessarily clear ways to test that. And the output of that code is, you know, goes stale and not generated. And what happens is, like, super painful, and a lot of times code, or documentation is out of date, or doesn't work. And kind of is very unnatural, because you have to copy and paste stuff. That's painful as programmers, copying and pasting stuff continuously is like a super painful experience is something that we know that we should try to avoid. It's kind of like using Jupyter Notebooks. Can we create a system where you have like, tutorials, how to guides, things like that with notebooks, and like turn that into beautiful looking websites that allow you to create nice documentation that you can also test. You can run a test on it and see if the documentation is still working. Like the code is doing what it's supposed to do and what it's up to date. And then all So like providing an avenue to do API documentation, and that to bringing in stuff you may already have in your code, like in docstrings, and stuff like that, but like showing it in a way, and then having more examples in your API documentation. So like, yeah, we're working on that at Hunter bounds. So we have something if you go to outer bounds, slash docs, is the repo which he'll he'll he'll share the link probably in the podcast.

hugo bowne-anderson
That's right. I'll share the link in the show notes. And I gotta say, Man, hearing you talk about that is great. But for our listeners, Hamelin iPad on this stuff earlier this week, and it kind of blew my mind, it was so much fun seeing the docs in real time updating in a beautiful manner, when all I was doing was what I do in Jupyter Notebooks. Anyway, particularly after the weeks of suffering with AWS and sage maker. Also, I'm not sponsored by AWS. That's super exciting man. And I do admire and I'm inspired by your dedication to documentation as well, as I made clear before, this, maybe as a nice way to wrap everything up the work that our stats, and the people involved in the tidy verse in particular, have done on documentation. And a couple of other things such as folding the terminal into our studio, for example, there are a few other very beautiful design choices they've made that have made it so practical for people to get up and running. I mean, you know, my background, I worked with a lot of biologists and seeing how they could jump in to the VR ecosystem. You know, there are quirks, of course, and people can come and say whatever about indexing, and blah, blah, blah, but whatever it is, right. But the truth of the matter is, it's gone at adoption and stuck because it helps people and reduces the barrier to entry. Similarly, with a lot of the PI data stack, there are a lot of packages, I could mention. Dask is very close to my heart, their documentation is beautiful scikit learn their process of documentation driven development. Psychic learn, of course, has been popularized for many reasons, including the API, but the documentation has meant that anyone anywhere in the world with an internet connection can suddenly get up and running with basic model building and that type of stuff. And the documentation is so so absolutely key there. And I think I mentioned this, but the practice of documentation driven development is incredibly important with respect to making sure that what they build, actually, that he is to good documentation protocols. So the fact that you're thinking about this and working on it and have been for years is awesome, as well.

hamel
Yeah, no, I think documentation is probably the most important aspect of a tools project. Because I mean, it is what people see and is the main interface that people have to your product. And so it is absolutely the most important thing, like that's where you can either welcome users or make users suffer.

hugo bowne-anderson
Exactly. Thank you so much for such a freewheeling wild conversation with a lot of your rich history as well. And diving into kind of all the details of your thought processes around what you've worked on, historically, from consulting, and everything that happened there to the auto ml data robot, everything at Airbnb, GitHub, and now outerbounds. I'm wondering if you have a final call to action for all our listeners, something you'd like to see them do?

hamel
Yeah, try this documentation thing. I think it can be a really cool. I mentioned fast pages as a blogging site. I think this documentation thing is can also be used for blogging. So if you're trying to create really nice looking blogs and you want to use notebooks, are you blogging about code in general that has code in it? Give it a go.

hugo bowne-anderson
Awesome. And we'll include the link in the show notes. So everyone, please do check out our github.com/out of bounds slash Doc's. Oh, and we'd love to get your feedback on that. So Hamill, thank you once again for such an awesome chat.

hamel
Yeah, it was fun, man. Awesome.


Transcribed by https://otter.ai