The following is a rough transcript which has not been revised by Vanishing Gradients or the guests. Please check with us before using any quotations from this transcript. Thank you.

===

hugo: [00:00:00] So Ines and Matt, I, a lot of people here and listening will know about what you work on and what you do, but I'd love just a brief introduction. When I get, have two guests together, I often get them to introduce each other.

So I wonder if you'd be up Ines for introducing Matt and then vice versa. 

ines: Oh, sure. I like, I have its story down, but,but yeah, Matt's my co founder and CTO, and he's also the original author of Spacey, which he started writing when he, left academia after becoming a bit disillusioned with the system and also seeing that companies were using his research code, in practical applications.

And it really wasn't, designed for that. and, We ended up writing some good models and systems that were topping the leaderboards, for a while, the fastest parser in the world. and, yeah, so there was really a gap in, software that wasn't designed for teaching and research and really for getting stuff done.

yeah, he left academia, had some, support from his family, luckily. So he was able to take some time off and really focus on writing spaCy and, yeah, that's around the [00:01:00] time we met and started working together. 

hugo: Incredible. 

ines: Matt, 

hugo: are there any gaps you'd like to fill in there or anything you'd like to correct?

matt: no. she's heard me give this many times, especially, yeah. so yeah, Innes and I met, as, just before Spacey was released or actually a bit before. and we talked a bit about the project and, Innes had done some linguistics at university, but. She studied arts rather than programming, even though she'd been programming since she was an early teenager, making websites and things.

But she never felt oh, this identity with it. And so it never really occurred to her to study computer science. But she always had that sort of background. And so when, When we met and we started talking about the project, I always thought visualization would be, something that was important and, be a nice feature for this to really communicate what people, what the software would do.

And so this was the first thing that Ines worked on and it was, really brilliant from the start. and, from there, there were just other things with,the project and [00:02:00] other things that we did that, she was more involved in until we've, founded the company together. And then,we, co developed Prodigy, together as well.

hugo: Fantastic. Innes, is there anything you'd like to add to? 

ines: No, I think, again, Matt has heard me do my intro before. I think that's a good summary. And yeah, actually when. when Matt first suggested that idea, I was actually, initially I was like, ah, visualize it. Like I understood what it was, but I'm like, ah, sounds a bit boring.

I'm really glad I did it because that actually turned, and it was a lot of fun. Once I started that turned into displacy, which, is still probably one of our. Most, popular vis visualizers to date and really what a lot of people actually associate with spacey, like the little,arcs, show, visualizing grammar and syntax.

matt: But it's beautiful. I think it's weird when I say this, but, she was the first person I met who really understood CSS because, I'd dabbled in front end and I just it was large and confusing and I just assumed that oh, it was just normal to not really understand this sort of find it awkward And, I had met many front end people. but [00:03:00] yeah, there was just like this, complete fluency,I would have some of the technologies that I know, and the solution that she proposed for, for display, see actually uses like CSS to,for part of the art calculations and things.

And, in the early version.Yeah. That's right. Yeah. It changed like since, but SVG, but it's 

ines: similar. Yeah. No, CSS is one of my favorites. Like I know some people, I don't want to get people being all like, actually it's not a programming language. even though I think, HTML5 and CSS is Turing complete.

there's anyway, people. Yeah. Yeah. I think I'm pretty sure. I'm pretty sure. I don't want to say something wrong, but I think it is, but I don't know, anyway, it's not, but language that I write code in, I think, it counts and it's definitely one of like alongside Python.

It's definitely one of my favorite. Language. I know a lot of people hate it, but, I'm actually good. 

hugo: You mentioned this, I wasn't going to bring this up, but I've actually always, since we first spoke in 2017 or something in us, I've always been a mixture of inspired and jealous from how like full stack digital native and front end native, you are as well.

And all the, I'm [00:04:00] actually just going to put a link in the chat to your website. and I'll include in the show notes, some wonderful posts you wrote about creating, slides for conferences. Yes. Which are, continue to be an inspiration for many people I know in the space as well. so that's super cool.

I do want to get into. So we've had, our quote unquote chat GPT moment. whatever that means. And it. It's brought a lot of attention to, generative AI, natural language stuff. And I think perhaps there are a lot of people who don't realize that natural language processing and applied natural language processing or industrial natural language processing, as you call it, has a very rich history that predates what we've seen now.

so as people who've worked in this space for so long, maybe you could just tell us what applied or industrial NLP means to you and give us some context around its history. 

matt: Sure. So at a fundamental level,language is the medium of human information exchange. so it's what we transact in for most economic activity or in the knowledge economy, certainly.

it's [00:05:00] really like the bedrock of, cooperation. And so this applies both in written language and then, spoken language. So look, in a way, you can see, library indexing systems as a language technology, because it's, that sort of, retrieval and, it's all, it, the program is, implemented by,the human who goes and,does a bunch of things, but really what you're trying to do there is program a way to retrieve information systematically or categorize it systematically.

and over time, almost as soon as computers were able to process text, we've, been wanting to make them better at these things. So Google, for instance, was a big advance in this and it's a language technology application,some of the, the early key things that it did was like, the links link graph structure and stuff and using that signal, but certainly the page content and the,an understanding of the language and being able to see two websites that have similar linguistic content is similar and like a range of space around those things is also really fundamental to how, search works.

And and then, you can go on to, [00:06:00] many more examples like, translation, speech to text to speech, et cetera. So these are some of the ways that, we've tried to make computers interface with language or manipulate it better, or just in some way. have computers take on more work, or complement the work that's happening, outside of it.

And so I think that as the digital revolution took hold and, there's software eating the world, concept. It was about this expanding range of things that you could get a computer to do. Tasks that you could,not even necessarily automate as such. But have some,technology solution that's like replaceable,or complimentary to labor and having, and, then as soon as machine learning started to get better, we got,we greatly expanded the range of, viable natural language solutions that we could, build.

So applied natural language processing really applies to a huge range of back office, activities within companies. if we look in the real economy, the sort of manufacturing stuff, everybody has [00:07:00] safety manuals. Everybody has safety reports. Everybody has, all of these things. If you imagine what the workers in these companies do.

They do well. they often are working on manipulating language, and so there's something like, okay, whenever there's this sort of incident, this is documented in this sort of safety report, and then we want to understand what's happened in those over time. But these have been written by four different companies that we acquired over the last 50 years.

that's, something that requires a lot of machine learning, in order to. to get that to work, and that's the sort of project which maybe wasn't viable before, and then it progressively becomes viable. so that's a lot of what there's applied natural language processing comes down to, and a lot of it,at its core, would be around categorizing, text or turning it into structured information in some way.

Because if you're turning the text Into other text or other structured information, and that's something that will be consumed by a human if you wanted to actually interface into a computational system this way, at some [00:08:00] point, you have to cash out these, dense representations or unstructured things into something structured.

And the activity of applying like some metadata or categorizing it or that sort of thing is pretty fundamental to this,sense of natural language processing. 

ines: Yeah. And it's really something people have been doing or companies have been doing since before, computers even. And I think it's easy to sometimes get sidetracked by all the new capabilities that we have now.

Oh, we can generate a lot of texts. We can do all kinds of things and it actually works now, but there's also no shortage of projects that companies haven't even, started yet because they didn't have the capacity. And that's it goes all the way back to categorizing things on index cards to computers and now to using machine learning.

and we also see that as like one of the biggest, yeah, use case areas in industry. And, Yeah, so we're really a lot of value is being generated. 

hugo: I'm really glad you mentioned, essentially library sciences as well and index cards, because I do [00:09:00] think,something software, of course, has in the world in a lot of ways, search has eaten the library sciences in a number of ways.

And I think they're due for hopefully resurgence generative AI has in the cultural consciousness.Eden, ideas of search, but of course, like exploring the fertile ground between search and generative AI will be like a very fertile ground moving into the future, wouldn't you say? 

matt: yeah, one thing I would add to that is that, there's, I think the library science sort of dying out thing is a instructive example in some ways about the trade offs of different technologies.

search certainly supplanted,this categorization system, but the experience of doing of using search and using the categorization system and not the same and,the browsing, ability that you get and the ability to discover horizontal content or to, know what you don't know.

when you're looking at a categorization system and, you're seeing information in that way, it's different than when you have to search for it and you presented this sort of blank text box and you just [00:10:00] have to know what to query. and it's a different sort of skill being able to navigate from queries and generate new ideas for what, how you should amend your query.

Search replaced library science, but it doesn't do quite the same thing and people who were, expert users of library systems, which was a small amount of people because this happened, at a time when there were just fewer people in academia and fewer people in,deep in university studies and things, search started to take over, the people who were expert in those systems, they definitely felt the lack and ah, this isn't the same.

And, I wish I could get more of that, all their experience because I'm missing out here. 

hugo: There are also design choices, right? Like I remember when I was a kid and used to go to the library and a librarian and chat with them about things I was interested in, they'd recommend a variety of things, they'd recommend a cloud in some ways and have a conversation around it as it, there's a design choice in search.

and Google search, but a lot of search to rank order things and have a top one item as opposed to a cloud of solutions. which impacts how we interact with information and knowledge. 

matt: Yeah, [00:11:00] before Google, there was actually this dichotomy between indexing search, between sort of information retrieval style search engines and indexes.

there was DMOZ and Yahoo and stuff were basically like indexes and you can browse these categories and stuff like that. they were trying to do the library science thing and, the DMOZ ontology is still quite interesting. It's this. Really deep, tree that's been built over time for categorizing information content.

but Google wanted to do things differently and the idea that they had was, they had a, an idea that would let them do the information retrieval style much better. of course, that's, what they did and that's what they built the rest of the business around. And, indeed, it worked very well.

hugo: Cool. I didn't expect to jump into library sciences, but I'm glad we did. I am interested in we've talked briefly about applied industrial NLP. I'm wondering how spacey and explosion just relate to this broader landscape of applied NLP and what role they've played in its evolution. So Ines, maybe you can, 

ines: yeah, it's definitely I think, we've definitely always seen that there is [00:12:00] really this kind of gap in the market or this need, for software that's easy to get started with.

you want it to work on the, out of the box, have reasonable defaults, but also being flexible enough, to really be used for all kinds of custom use cases, which. a lot of the things that are most valuable are usually the most specific. and then also,actually run fast, because we've, I think this was like, a lesson Matt learned from his like supervisor back in academia that like Moore's law won't really save us because,the text available that you can process, it's like growing much faster than computers are getting faster.

I think it's really the, this intersection of speed. Accuracy developer experience, that we're working at, and also,providing like a way to pipeline and, to combine all kinds of different techniques, depending on what you want to do. Like you might want to have one component where you do use a generative model to help you extract structured data.

And [00:13:00] then you might, combine that with, a component where you use a rule based system to improve certain cases that you know. and then you might combine that with, a small model that you've trained specifically on your data. and keep this tight feedback loop around it. and yeah, especially I think around structured data, there's so much, there's still so much to innovate and there's so such a need, for flexible software, and that's really where we see ourselves fitting in.

hugo: I am interested in for people who don't maybe haven't played around with spacey and prodigy so much, like explicitly what types of use cases there are for it. And also, if you can speak to, as you've written about, actually the choice of what to work on with prodigy isn't necessarily the obvious choice or what investors would want given what you've done in the open source space.

So talking about the differences there and what prodigy actually does would be super interesting. 

matt: Yeah,Spacey takes on the task of this,what in academia we thought of as a traditional or core NLP tools, and this is, adding annotations about linguistic [00:14:00] structure and the nature of the language that aren't, that are application neutral.

So this includes annotations about, word boundaries and then,inflectional morphology. saying wrote is the past tense of right and linking that up, even though they're different word forms, and there's no,suffix that you can trivially just split off in order to, make one versus the other.

and then. going deeper than that into part of speech and then, syntactic structure and, then entity recognition and that sort of thing. So this type of functionality isn't the answer to any application. there's nothing where it's oh, i'm going to make a lot of money by Showing people to pass the speech of words like this is not an, this is not an app.

however, there's a lot of text processing tasks where this type of neutral information is useful. And the fact that it,it's something that you can say about the language regardless of what you're doing, makes it, a useful thing to provide in that sense. And certainly for languages other than English.

Processing the text in any non trivial way requires this sort of statistical processing. And, [00:15:00] but this is always a pre process. There's always something else that you're doing. And so you always need this to be fast enough. because it's always going to, there's always going to be other things, in the pipeline and you don't want to take up all of the space.

So that's one set of functionality. Then the other side of functionality is training your own models in order to add structures. This could be text cutter categorization. it could be,predicting spans. people call this named into recognition, even though it's, what you're recognizing is spans aren't necessarily names.

It can be just arbitrary phrases that you're interested in. It can be, and then, going beyond this,recognizing relations between text and, arbitrary things. So you're adding. you can populate some, database of, company announcements or,product sentiment or something like this ,and train a whole pipeline to do that, which includes several processing modules, including ones powered by rules.

ines: Yeah. And then in order, and that's how we got, we initially got the idea of, okay, what kind of product can we add on top of our stack? That's [00:16:00] like the most useful. and, yeah, we saw very early on that data,engaging with your data was like one of the biggest bottlenecks and also with, at the time, like transfer learning was still relatively new, but it was clear that Hey, this is like really a trend that's going to continue, you will need less.

Data and it's more important, creating data, annotated data is not this, big data process that you outsource. Like you can train really good systems with very few examples and you always need to evaluate them. And, you can automate a lot of these things. So it really, becomes more of a development process rather than, yeah, this large, click work task, and that's what inspired Prodigy, which is really a downloadable annotation and data development tool for, machine learning developers and data scientists specifically, but also, for larger teams and you can, run it.

On without an internet connection, it,keeps it never connects to our servers. It keeps your data private, but also let, lets you [00:17:00] script with it in Python. So you can have powerful automations, like especially now you can use the large language model, to create a data for you when all you have to do is correct it.

And, also it really enables this, process of iteration. because just like with code, you need to iterate on your data and you need, yeah, often the first, idea you have is most likely not the one you're going to ship into production. And the hard part is not actually training the model. The hard part is breaking down whatever business problem you have into steps that are, as modular as possible, that are transparent that you can work on, and,that make the system less operationally complex.

That's basically what's hard. And that really is. That's what Prodigy aims to solve. 

hugo: That's great. I've linked to Spacey and Prodigy, and I'll put that in the show notes as well. I do encourage people to go and play around, with Spacey and check out Prodigy if you're interested. And in fact, I'm a huge fan of many things about Spacey, including your landing page.

I love the fact that you've embedded, you've used, I think, MyBinder, right? [00:18:00] To embed an interactive terminal, so people can actually play with spacey in the browser immediately, which is super cool. and my binders, but part of project Jupiter for anyone who wants to check that stuff out as well.

Yeah. Shout out 

ines: to them for providing that service, that really made a huge difference. 

matt: Basically it's also installed in Google Colab. it's an easy way to,to get started without this sort of software installation process and things. 

hugo: Awesome. yeah. What you were just talking about with respect to, I suppose figuring out what humans could do.

Can do and machines can do and how we can work together. that was really nicely with, I wanted to talk about, your wonderful S and P global case study, but really in the context of everything you're doing with respect to human loop distillation for creating smaller, faster, and more data, private.

Models. I do want to hear about that, particularly with respect to S& P and then maybe zoom out and think about in the broader space of what's happening with closed models as well. So maybe you'll give us some insight first into what even human loop, human in [00:19:00] the loop distillation is. And I'm still here.

I'm just turning a light on because the sun is setting. 

ines: Okay.yeah, I feel like I'm talking to the green screen now. Oh, you're back. Cool. Yeah, no, I don't know why it feels. So yeah, the S and P project, that was extremely cool. because, yeah, what they essentially, I think S and P global, most people are familiar with,their financial company.

And they've also, one of the things they do is they provide data products, that, For a variety of different areas. And one of them is commodities trading, which I have actually learned a lot about while writing up that case study. So it's if something like crude oil is traded somewhere, they get like that data in, in real time in a very specific format, but who participated in the trade,

price, all location, all kinds of things. 

matt: Maybe if you screen share to the case, like just bring up the case study and screen share, you can show the examples and things. Would that make sense? 

hugo: Absolutely. Yeah, that could be cool. Yeah. Yeah. [00:20:00] And let's just make sure to talk through what's happening because we'll release it as an audio podcast as well.

Ah, yeah, 

matt: sure. That's fine. But, there are people who can see, and we're 

hugo: working on things at the same time. And also, to your point, learning, Innes, I learned many things, including the term herds, which are trading activities. That market reporter, Steve Daley via phone, email, instant messenger and so on.

ines: Yeah. Yeah. I also hadn't, yeah, I had like very little idea of like how this all worked. I actually went to visit their office, in London and it was quite interesting to see how the analysts work. It was a bit of how you imagine,investment like bankers or something working with like lots of screens.

Yeah. Yeah. And then like tickers coming in. 

Yeah, so basically,yeah, this actually leads into one of the main requirements because this information that comes in that can significantly move or affect markets, and the economy. And,if. If you want to make,the mark, these markets more transparent, which is obviously one of their main goals with, their, data products.

ines: you want to make sure that it goes out as quickly as possible and, is not [00:21:00] shared with any third parties. So even, in the office, there's like this kind of glass, room that you can only get in with a key card. And that's where they receive that information. And that means that a, the models need to run.

Entirely in house. They can't call out to any, third party APIs, during, production. And it also means they have to be super fast and, that did require, some clever solutions, but, yeah, essentially in a nutshell, What the team, the relatively small team was able to do is train extremely fast and accurate pipelines for the different markets, that, run at, I think over 15, 000 words per second.

And the art with artifacts that are, under, around six megabytes. And, about, it took about 15 hours to develop, in terms of the data. that's one person. So basically if you had two people working on data, they would take one work day to build an entire pipeline.

For a market that, is this small and accurate with some clever automation. So if you think about, yeah, what else have you spent [00:22:00] like a work day on? like I was using the example of getting your GPU to work. we've probably spent more time trying to get CUDA up and running. then, the time spent.

by the team to create data and basically take an LLM, that they had access to only during development time, help it to create data more efficiently and faster, and then use transfer learning to distill this knowledge down into, a model that doesn't only,do the one specific thing that you use the larger model for, but it's also actually better, and outperforms it.

And that was super exciting to see this. We've been working on kind of some of these ideas around it for a while, but to really see this working well in a real world, application and really see the whole stack come together. And I think it also shows, Hey, it's totally, you don't, there are a lot of people trying to tell you stories around like one model to rule them all and what we just need up bigger and bigger models.

but, in fact, that's not how it works. You don't need to [00:23:00] compromise on. Development best practices and data privacy. And if you move the dependency to development only. There are a lot of problems,around black box models and, cost and speed that,or economies of scale that, have, that become like a lot less relevant.

that's basically the idea. I don't know, Matt, if you want to add something. 

matt: yeah, sure. I think it's, we can also talk a little bit about where the games came from, because I think that's it. Actually quite interesting. If you scroll up a little, if you go to one of the examples, I'll just read it out.

So basically,from the start when I,met with Chris about the project, the, these, messages are very short and they're like very information dense and very coded because they've been something that, brokers are basically like noted down,a trade that they've been,become aware of and, there's a system for doing this, right?

so the language is very specialized and it's, the annotations are extremely dense in it. you, the things that need to be annotated make up like a lot of the tokens in these short things. So it might [00:24:00] be, America's crude oil,colon TUPI colon May delivery heard off in July.

I, ICE in. capital's Brent plus 2. 50 slash B, comma, CIF, kinder. and then the annotations on this would be, America's crude oil is the market, TUPI is, a grade, May is a laycan, which I still don't understand what that is, even though Chris has explained it to me several times, Offered is the herd type, July is the pricing, pricing basis, or timing, ICE brent is a pricing basis.

then 250, slash B is a price. the, 250, without the B is the price. And then B is the unit of measure. CIF is an inco term and kindow is a location. So almost every, word, every token of this ends up being some sort of,entity token, which makes sense because there's like very few unnecessary words in this that only.

convey linguistic structure or that sort of thing. They're not here to chat. This is what this is. what a sort of basic idea of how you would solve this type of problem with machine [00:25:00] learning is, you, that's the output structure. So we need to train a model that has the output structure.

that produces that output. So I'm going to annotate all of these as training data, and this really sucks to do. You're clicking and dragging a lot. There's you're doing all of these types of things at once and it's super slow. so almost immediately what we said is look, the way that you should structure the annotation task is to be thinking about streaming the minimum amount of bits from you into the computer.

think of yourself as this. Computational device that you can call into, and it's an expensive piece of hardware. You want to transform the data as much as possible on your coprocessor, your CPU, and really craft that problem in a way that's suitable, that is most friendly for the human. And as soon as you start doing this, it was noticed that like a lot of these annotations can just be done with rules, or at least they can be done to a like 95 percent accuracy with rules.

And then, the correction of that 5 percent is, far less arduous to do. [00:26:00] And then also the information ordering, doing one type of annotation at a time, let's speed through it because, you're looking, you're focused at the start of the markets at the start, your eyes are there, you're clicking through and.

you even I often find myself clicking faster than, I'm actually recognizing. And, then when I notice a problem, there's sort of a reaction time where I stopped and I go back five because that's where I saw that there was a problem. And that's great. you're clicking through it more than once a second.

and as long as you structure the, the interface in the right way, you can get that to happen. And This is where, the big,advances in efficiency came from that type of structuring and using spacey well to, create those rule-based processes, which is super easy and space.

It's designed to give you these token annotations and parts of speech and stuff, which you can hook rules onto. and so you can say, okay,offered, that's, that string of letters is only a verb in English, but there are lots of others, which,you might have that as a verb or a [00:27:00] noun.

you want to only have the herd type if this is the verb. The verbal sense of that. And, this part of speech annotations are, 97. 6 percent accurate these days. and with much of that inaccuracy actually being annotation errors in test set. So you can rely on these things pretty well.

and,that's, so prodigy is also structured to make it really easy to. to do this type of transformation and to, really get the human to do as little as possible in order to produce gold standard annotations. 

hugo: Awesome. And I am interested in, we've talked briefly about how human in loop distillation can help address issues of modularity, transparency and data privacy.

I'm wondering if you could just say a bit more about, maybe Ines, you can give us some insight into that. 

ines: Yeah. yeah, one,obviously, if you, so one, one thing actually just to add to the SMP, example, like they also, for some attributes, they also used, I think GPT four or three, five that they have access to internally during development time to help them highlight, things as well.

they had to do less [00:28:00] work and,there's a lot of great things that are in,a lot of the bigger generative models. There's clearly a lot of knowledge about the language in the world, which has always been the main problem we faced in, NLP. Like how do we encode that and how do we bootstrap a system with that?

and you get good contextual results. There's really this huge, Advantage in prototyping previously, you had to do at least 40 hours of work until you had a model that was shit and that you can somehow improve. And, yeah, now you can start with something out of the box that works and go from there.

So there's, there are all of these great advantages, but on the other hand,you're essentially dealing with black box models. like we, we don't actually know what, like we have some ideas of how GPT 4 works, but,ultimately,we, we can't look inside and even with open source models that we can run, we actually know relatively little, about why a model is producing a certain result and how we can fix that.

And then, you also, often if you're only interested in a subset of things, you're still running the whole model. And, that can get really slow. That can get [00:29:00] expensive. especially at scale, if you really want to process like, meaningful, amount that's not really viable. and then finally, yeah, in order to take advantage of economies of scale and,make it work, because it, you can't, it's not, economical, in most, situations to really run a model entirely yourself in house.

you use an API and so you send, your data, to someone else, which is not acceptable for many use cases. And, similarly, like there is. over time as an industry, we've built up a lot of best practices around how we want our software to work. Like we want things to be modular. We want, to be able to refactor code,reduce the complexity, and so on.

And they're like a lot of,there are a lot of these best practices that we have built up, that, make sense and that like our ways we develop software for very good reason. And that also becomes, very difficult if approach it as Oh, you're going to have one monolithic,model that's supposed to do it all.

And that also actually doesn't match the reality of how people, even if [00:30:00] people use generative AI and do it successfully,that is not,how it works and how it's implemented. it might be, it might be like how some companies are trying to sell you, the use of, generative models.

But, even, if you look at, larger companies, if you look at companies who use these things in production, it's usually always an ensemble of different models and different techniques. and that's what makes it work so well. yeah, so basically that's that's the underlying.

idea. And that's where we actually see a lot of value. And that's the part that we found we find most exciting to work on, and making these workflows easy for people, because one, I think one reason that like, a lot of,generative, models and large language models, especially with the chat interface, like why this has taken off is that it's very easy to approach.

Like you just write a prompt and you can write. The way you would talk to a human. And that's cool. and then, yeah, even if you're more experienced and you know how you can,do this distillation process, how you can train smaller models, how you can get around all of these things. It takes time and you really need to know how to do it.

And, so the main question we're thinking about is how can we make that [00:31:00] process as approachable as writing a prompt? Because, it clearly has all of these advantages. 

hugo: Absolutely. and also before I've, I meant to say this earlier, So I've put the S& P Global Study in the show notes, along with your other blog post on human distillation.

it's a great project. I'm glad you did it. Also, I'm so glad that however you got S& P Global to allow this to be public and to share that knowledge with everyone else, as someone who's worked with In a lot of companies doing a lot of developer relations and that type of stuff. I don't think people appreciate how incredibly hard it is to actually get internal knowledge from companies with respect to their workflows and projects out,in the public.

So that's really awesome as well. 

ines: No, I think it's great. And also I do think, they're really, you could tell like their mission is really about providing this market transparency and that also, I think, it, extends to the technology choices and, 

matt: yeah. it's interesting, I would say that I find this quite exciting, which, a lot of people would find bizarre because on, the saving baby [00:32:00] scale of,clear,social benefit or like clear outcomes from the project.

It feels so dry,okay, you're providing this, feed of information. I like that. It's so concrete. And the, I do also really believe that if you make these markets more transparent, that there's a real world benefit to that. Because as soon as you have, information that isn't flowing well, as soon as there's a track, that means there's a trade to happen.

So somebody,The finance bureaus can go in and they can like, make a bunch of money by moving things here from things there. 

And in commodities markets, this creates a bunch of price instability because, suddenly, I don't know, Wall Street's piling into like Namibian cadmium or something like this, right?

Which might be a smallish market. And, there's a bunch of future contracts in this. And this can easily disrupt, the actual real economy and we saw this a bit in COVID, like there were lots of trades happening and this sucked for the rest of us. So if you make markets more transparent and information flows better, this I see a benefit to that.

And there's lots of these [00:33:00] sort of small, things that are real, that I find interesting and exciting. And I guess that's part of the taste of doing something like spacey as opposed to something like, a chat bot, which I'm. Remain skeptical about, as a class of applications.

And it's not, everybody's super motivated by I want to talk to my computer. but I, yeah, I've always I haven't felt that as deeply. 

hugo: Yeah. And also, it's great that last decade seemed like the decade of demos with sentiment analysis as well. So it's nice to Move away from that.

I just want to say, we've got an interesting question in the chat from crumb on a dove, who would like to hear your thoughts on the future of model distillation, for example, CPU instances are so much cheaper than GPU. will there be models good enough and run quickly enough on CPU? That would be very valuable.

These types of things. Yeah. 

ines: that's SMP model, like that project we talked about. the six megabyte, that's that's a, CNN based, model that runs on CPU, right? That is. Running on CPU and we actually have, there's another case study I'm currently working on also with a, larger company.

and yeah, they also, running a lot [00:34:00] on, CPU that's always, yeah. 

matt: Yeah, I think, a sort of classic concept that I think is super useful in,machine learning or natural language processing. Certainly it's the idea of whether a problem is AI complete or not. So what this means, it's by analogy of something being an NP complete.

in other words, there's a class of problems where, okay, in order to solve that, the solution is entangled with a deep well of knowledge that's going and capability. That's going to apply to a lot of other things. So the S& P project is an example of something that's very certainly not AI complete. You do not need to know, background knowledge about America and its relation to oil to recognize the string of characters.

America's crude oil as a market. That's in the data and that's something that can be, done symbolically, from the language. And, to a great extent, these linguistic annotations have been talking about a surface code of the language that mostly doesn't need deep background knowledge. And indeed, we could,tag these things accurately, even when we had methods that had very [00:35:00] poor ability to incorporate background knowledge.

Armed with this distinction of okay, our problem's AI complete or is what I'm doing AI complete. we can say something about how large models need to be. And the thing with generative AI,the big thing that came with these large language models is suddenly this class of AI complete applications felt open to us.

And there's all of these things which didn't work for shit before, because they were all AI complete. Now they feel much more possible and, the jury's still out on how practical some of these things actually are to do. But with, there's this like big crack in the ceiling and we see this light shining through and we see, a way forward perhaps for these things.

But if we're talking about applications that need that AI completeness, we're a long way from,that actually being available at this sort of runtime on a CPU or something. So the magic, the trick. Is to have easy problems and to try to have easy problems as much as possible. And so if you can structure your application needs or reframe the machine learning problem in a [00:36:00] way that is solvable from the surface linguistic structure, it is not AI complete, then you're off to the races and you can.

really,code with accuracy. you can break down the problems that humans are able to annotate it super accurately and super clearly. There's really crisp boundaries around the categories, and ultimately we'll be able to train models that are very accurate and very small. 

ines: I think it's something people often forget that like you're allowed to make your problem easier in that kind of ties back to the distinction between applied work and like research like,research problems are hard.

And that's the point of them and you're trying to solve it. Yeah. Whereas like in applied work, you want to,you want to make your problems easier and you're actually people do the same in programming all the time. And,you're trying to,find the best possible solution.

and,you're not the most complex one or the most clever one. and the same. 

matt: the food's bad and you just get dysentery. if you can stay back comfortably in these settled regions that are more developed, that's like [00:37:00] good for you. You don't have to be in the frontier.

Yeah. And so 

ines: you're allowed to make your problem easier. And that's actually, I think one, one of the absolute keys to success that we're seeing. 

hugo: I love it. you get to choose the question you're asking essentially. 

ines: Right. 

hugo: Incredible. So I do want to step back a bit. We've talked around,the space in which we have smaller models, perhaps even, focusing on, on, on specific, questions and challenges, and then these big, beefy, large language models, one solution that fits all.

I know Ines, you've given a talk many times. I've been at one of them, called the AI revolution will not be monopolized. and Matt, you've written a blog post called against large language model. Maximalism. I think both these speak to this tension between smaller bespoke models and these, big,LLM.

So why will it not be monopolized? Do you think? What do you see as the future for the space? 

ines: Yeah. I think,there's still,a chance we'll just end up like kind of accident or [00:38:00] not accidentally like gifting some tech company a monopoly because,we let them write their own regulation.

but, I think one of the core points, I had in that talk was around a, the distinction that they really need to make a distinction between products. AI products and the models, which are ultimately just software components. And if you're looking at products, like yes, open AI might very well dominate, the market of AI powered chat assistants.

that's, totally fine. And they have a lot of user data that doesn't make the AI better. It makes their product better. and,that's fine, but that doesn't mean by extension that, they'll don't, they'll monopolize and dominate AI products. more generally, and, it's also, economies of scale.

so basically, yeah, the fact that, open AI or Google can provide, much cheaper,API calls because, they have so much traffic and can batch up the requests, that does matter if you. make these models a runtime dependency and build everything on top of that. But, that's not the only way if [00:39:00] you move your dependency to say development, that really changes the calculation and means you can run your own models.

and, yeah, that's, that's basically, that's the idea. And I do think open source, provides this, interoperability, which is really the opposite. And a lot of the reasons companies choose open source, are really also the same reasons that, it makes a lot of sense to have modular, models and not just one monolithic,magical AI model.

and. Yeah, I think, probably linked to the slides for the talk. I go a bit, I have a few more examples in there, but that's, that's basically the idea. And I think it also, this, these ideas generally, like we have talked, I went to a lot of conferences and talked to a lot of people and it really resonates with developers and matches their experience.

and yeah. 

hugo: So I do want to drill down a bit into, let me say I work in tech. So let's double click on, I speak with too many VCs these days. Let's double click on that, Hugo. [00:40:00] I remember the first 

matt: time I heard that I'm like, Oh, that's an expression. And then suddenly it's, there was just this butter mine, half effective hearing that all the time.

I don't know whether it shot up in popularity, but yeah. I also love that at first, 

hugo: that it just reminds us that we're still in the point and click. Paradigm, which is so ancient in so many respects. you want minority report style. And anyway, I, what I do want to double click on is I've actually lost my train of thought.

Oh no. Yeah. So you mentioned open source models. I'm interested in what on earth does open source mean now? Cause there are so many dement is open white, open source. We used to be able to read source code, right? Do we have training data? Do we have the code that was used? do we just have. A set of weights.

So how do you think about open source in this new space? 

matt: so to me, it's important to think about the motivations of the people who are publishing different things, because this explains a lot of the distinctions and it lets us like, think about what we will be able to expect over time. So I think the article that, [00:41:00] will that Has been the best and it's been the best for a long time about this sort of topic is, commoditize your compliments by Joel on software.

So this was written a long time ago. And the idea basically is that in business, there's this advantage in making free the things which compliment your business because they're going to be supplies into you or,or they're things that are consumed alongside your product or something like this.

And if people, if those are available and good, then that's good for you. and it also stops them from doing the same to you. If the business like that's built around that thing suddenly starts to want to commoditize you, that's bad for you. so fast forward to Facebook, right? Facebook has no natural business.

That's going to be in providing LLM services. LLM services or, these technologies are always going to be an input into Facebook's monopoly on social graphs. from Facebook's perspective. The worst the market is for those things, and the more of a race to the bottom it is, and the like, more commoditized LLM technologies are, the better.

And this indeed is [00:42:00] why Facebook does participates in so much AI research. Because they don't want that to be locked up in other players. And then used against them and, become a really,expensive thing for them to consume. And perhaps even something which somebody else can later use as a unique selling point in order to maybe Particularly as players like Apple 

hugo: are increasing kind of their, the way they're flexing their moat against players like, like Meta.

matt: Yeah. So Apple's an interesting case in this because they basically. I'd say that Apple backed themselves to make a business out of almost anything. And so their bet is, okay, we can do this type of research and keep it closed and use some, some output of that as a unique selling point. So Siri, for instance, when everybody was developing like the first chatbots, Apple was also quietly developing their first chatbots.

They weren't publishing, things in, the commons like Google and Microsoft, et cetera. they were like, okay, if we can get Siri, that's great for us. And this will be, this will sell, help us sell a bunch of stuff. and [00:43:00] yeah, and in a way, actually what was promised 

ines: as Siri is only really possible now.

we're now seeing a lot of the promise. actually, sorry, I didn't want to interrupt. yeah. 

matt: So exactly. so with this in mind, the open sourcing of these models is driven by those sorts of commercial interests. And, it's not driven the same way as open source that's coming out of sincere desire to do academia or something like this.

There's some desire to do academia, but And not only 

hugo: a desire though, remember, like an actual, a serious need. I mean, your work, the, PyDatastat came from scientists who were like, I need this shit. Fernando Perez built PyDatastat. iPython because he really needed something like that. John Hunter started working on Matplotlib.

Wes started working on pandas because they were like, Wes was like, I need to read a CSV. So pd. readcsv was born, right? So yeah. 

matt: Yeah. certainly NLTK was like a teaching toolkit before it was, yeah. Absolutely. Yeah. Ines, what were you [00:44:00] saying? 

ines: no,that's how our tools,came about as well in some way, like we, we have all the tools around spacey, that were basically things we also needed and wanted to do.

But,yeah, but I think to go, to take, to go back to the open source models, like I think, yes, there's definitely, I think there's also a nice overview of a lot of models. I think that we can probably link. I forgot like what it was called, but that goes to models and whether the code is open source, the weights are open source, different licensing and so on.

But, I do think, there are a lot of options and I also think eventually like we're still in this phase where there's something new coming out all the time and it will take a while for the industry to converge on,the best type, the best model artifacts, the best, types, just like we did with,embeddings and transformers, like there was a time where people were training all kinds of, transformer embeddings on pretty much everything.

And,now we've converged to mostly what do people use? Burt Bayes, Roberta, like a few handful of other [00:45:00] things that work well and there's actually not such a significant difference, in all the variants. And I think we'll also be seeing that going forward. 

hugo: I love that you mentioned that because I do, we've been talking about why You may not need large model, like the largest models and that type of stuff.

But I do want to ask a very practical question around this. And it's based around one of your talks. I think, I can't remember which of you gave it, but maybe you can tell me. The question is how many labeled examples do you need for a BERT size model to beat GBT 4 on predictive tasks? 

matt: Yeah, 

ines: that was Matt's 

matt: talk.

Yeah,in some sense, and that was actually like a talk title, there was a little bit of clickbait in that because there's a sense in which this is how long is a piece of string. if you think about, some classification problem, the boundaries of that classification complex, of what you're classifying could be arbitrarily complex.

And so you might need,there's no, real upper bound in how many examples you might need in order to show a model. those boundary [00:46:00] cases, especially if the classes are imbalanced on and you've sampled from the same distribution is you'll see it test time. So if you've got, some phenomenon, it's only in 1 percent of the data.

you need to show your model at least a few examples of that, and so that's going to blow out the,the size of the training set. That said, the, for problems which, are of a reasonable character, and the sort of average problem that people have with, without that sort of class imbalance or where you're able to rebalance the data, in some way, You typically, you typically only need a few hundred or, examples or so to outperform,something like ChatGPT or a zero or few shot classifier.

because they're all, they also have the same sort of thing where if the problem's very, unintuitive or if it's got all these boundary cases and stuff, they're also going to suck at it.

hugo: Ines, is there anything you'd like to add to that? 

ines: no, I think again, the answer is often, it depends, but people are often surprised how little data is generally, needed for [00:47:00] these things.

And I hope, that, and I think tooling ultimately is the answer to that, to, really, help people get over this mindset of, thinking that creating data is somehow like this huge. address top task, that takes a lot of time and resources. It's actually, if you need 100, 200 examples, that's something you can do yourself in a few hours and build your prototype that outperforms, GPT 4 on your problem.

Like it's not, 

matt: And to be clear, you get the same sort of effect by fine tuning,some of these models, like Google Gemini now offers a very affordable. in practical fine tuning, a system on their flash model that, I sit, at work in, the, a conference in the data hack conference, in India a few days ago, and, so the technologies around that are improving and you can, use an LLM by a fine tuning with this way to make it like, to show examples and ultimately that's what it's about, like having some method That can scale up the number of the amount of data that you can bring to bear because data is key like examples are super powerful [00:48:00] and so some any method.

Which locks the, that method out of being able to leverage more examples has a huge limitation. And so the in context learning where you've, where you're not going to fine tune, you're going to embed all of that into, the text, is really fighting with an arm tied behind its back. And it works for the constraint where you really can't get those examples or, you don't have them initially.

But as soon as you're able to cross that trade off with actually it is worth this few hours of effort to do those examples. using methods which take advantage of them is going to be better. And you don't, you can fine tune, but that's also an oblique way to solve something like a prediction task, doing something like attaching a specific prediction head to a BERT model, or, using spacey or some other library that has an API built around the abstractions of like text classification, the abstractions of, entity recognition, as opposed to the neural network architecture is going to be better.

hugo: Yeah, makes sense. We've got another question or comment first. Thank you for the interview. You're very welcome the [00:49:00] question is and I don't So the question is, I don't know enough to know whether the assumptions underlying this question, how correct they are. But why is there so little research into base encoders nowadays?

It seems there is nothing really big after deter is the question. 

matt: so I guess I also am unclear about what exactly they mean by base encoders. so do they mean something like, I guess these sort of encoder only models birch or something like that? the fact is. Plateaus are pretty normal, and, it hasn't been that long, and we had a long plateau before, before the Word2Vec,Colabird and Western, neural network, thing really broke that in, in NLP, and, it was normal that, We weren't, just weren't really seeing advances in the parsing scores and things weren't going up.

people don't have a very strong idea of what bits of the architecture are important or not important. There's been a lot of papers trying to unpack this. It's still unclear. and so there's been tons of papers which, [00:50:00] have some variations of those things.

But then they evaluate this in a number of ways. You, you show some benefit on those things, and then it doesn't really somebody else tries. It doesn't really translate and people basically get sick of this. So it's a combination of,our measurement tools being blunt enough that you can't he'll climb your way out of it because we can't.

it's difficult to get a really principled understanding of these things. And then we haven't had the next advance. Instead, the next advance has been, you can just make them bigger. and, then, and so that, that trick worked and that's where all the attention has gone. But if we're planning this out over a number of years, this is, there was almost no time in between.

So we were still, trying to improve these encoder models. And large language models came about and people were like, oh, This trick's interesting. Let's see what's going on here. 

ines: And I think that's also, they're that shows like the different objectives that you have in research versus industry.

there are a lot of,the research problems are not necessarily the problems that are,gonna solve like [00:51:00] some specific practical use case. And it shouldn't be, like research is an entirely different. And it's I think one example I sometimes use is sure I can, I can improve absolute state of the art on many tasks by adding a few regular expressions.

that's true, but that's not a research question, but that's clearly, something you can do in a practical setting. And,yeah. And I think when that coupled with. where a lot of the research interest is at the moment. And I think the more competitive, nature just means that, yeah, people aren't not working on this anymore.

There's something else that people are working on just like how nobody's creating high quality corporate anymore. Like it's just how it is. It doesn't, but I think people often, especially people maybe new to the field or people who work in. industry often, I don't make the mistake of being entirely guided by what's happening in research or like assuming that this is what you should be focusing on and what's important.

and I think that's, yeah, how you get people looking at some papers or looking at what research focusing on and then assuming this is what I should use in my work rather than, a [00:52:00] lot of other things that are around and maybe better and that,are not what 

matt: I find academia incredibly hard to read and use at the moment as well, compared to, back in my day, when the field was much smaller and it was a lot easier to judge,the, there were fewer papers and there were many, which like, weren't going to be significant, but then you could see that some of them would be, and they would like, take off and there would be some of those ideas.

and you could. Now, there's so many papers and the evaluation is so unclear, and it's so difficult to show a really decisive advantage for something. and so it's really difficult to know what to pay attention to and to look deeply enough into the paper to understand it and to, really judge it on, at a deeper level and to take on any lessons that it has.

So I find this really difficult right now. 

hugo: The amount of amazing research happening in industry as well is phenomenal, and it always has been historically as well. I don't know, I don't know [00:53:00] if many people know the story of, the creation of the t test, but that happened at the Guinness brewery.

It was William Gossett who, he, they were trying to figure out which, which crop yields would result in different, Flavors and different amounts of sellable beer and that, that type of stuff. So he created a statistical test, which became students t test. So there's a huge amount that has happened historically.

Industry. I also love that you mentioned regular expressions. I've linked to one of the worst websites in the world, called reg RegX crossword.com. I don't know if you've done the regular expressions crossword. Oh, no. . But it's one of the most ity. it's, I actually don't suggest anyone does it. I troll myself in the world when sharing that.

It's really brutal. I also, I've always. I've been skeptical as we all have of the term and AI, and I think that's a polite way of saying it, but I always used to joke that I believe we had artificial intelligence when I no longer had to write regular expressions. And I think we've gotten there for the most part, at least I can get chat GPT to or whatever to help me a lot.

ines: [00:54:00] that's the meta step that people often forget. It's it's not you're not replacing. The process, you're not replacing rules entirely with the model. You use the model to help you produce those roles. And actually chapter PT is really good at writing spacey rules as well.

Like the,linguistic Metro rules. so that's the pro tip. Like you don't replace the workflow. actually it's there's an analogy here, right? To what we were talking about. Like you're not replacing runtime with the model. You're placing, replacing development with the model.

And that's, it's the same thing as like the distillation, idea. 

hugo: Yeah. And it also helps me with delimiter issues as well. which I think is fantastic, but I am now interested in, I think we've been taken over and the people doing the work don't necessarily think along these lines, but we've been taken over with thinking about gen, generative, models.

I'm just want to think more about and hear from you about how the modular systems which you work with and have built and talked about today allow you strategically place generative or predictive models within workflows [00:55:00] where each is most effective.

matt: yeah, so basically imagine some application pipeline,imagine that we wanted to monitor Twitter for mentions of Spacey and,see what people were saying about the library every time.

Now, there's a motorbike called the Honda Spacey. a thing which I would also, there's also people 

ines: who are, who frequently misspell Kevin Spacey. 

matt: Yeah. if you look at our Google, Search things. There's these big spikes around when there's news around, Kevin Spacey. because even if 1% of the people who you know, are Googling for him, misprint, misspell, that name it, it's still, that's a significant volume versus the number of people who are searching for the library.

so yeah, you're building this sort of model, right? you wanna know what people are saying. you want to turn this ultimately into some sort of data,you want some sort of trend around this and say okay, these are the types of sentiments to people that are extracting and,here's how to gauge it over time, but then you might also want it to summarize those things, and to give you some human summary of those things or to produce representative tweets of those things,that's [00:56:00] something which, degenerative technologies plug really neatly into those things, into these libraries.

And in many ways, Data science didn't live up to the promise that was made around,discovering insights from data and,maybe some of the answer will be, being able to plug in some of these, extra steps of actually generating reports or generating,summaries and things that help, consume the information that, was produced there.

So that's one type of thing. And then the other way is that you can flip this around and have. Applications, which are generative. some sort of chat application or,discourse application. And then there's a space for a huge amount of,NLP pipelines associated with that.

So you might have guardrail models, which are separate models where, you know, which run on the, the question answer pair and say okay, is the model giving up its secrets here? Or has the user tricked it into, telling them how to build a bomb after all? it's always difficult to do that from the inside, from the model, because that's the one that's been tricked.

So you can have somebody else who's, dumb and sitting on the outside and saying [00:57:00] ah, okay, that's the thing that I'm looking for. And it's like bare and I'm going to flag this. and then, in the preparation of data, there's a bunch of these tasks and trying to get clean data and then also for instead of retrieval augmented generation, you can have what I'm, now going to be calling like, retrieval via information extraction.

if you've got some, if you've got some data and the documents, describe some,something like financial reports or something, which you can extract into a database ahead of time. That's very advantageous compared to,having the model having to,first retrieve the right paragraph and then extract the information from it on the fly.

So if you can do that stuff as a pre process and then have the model translate a question into SQL or some other query system, that's a much better architecture. so there's a lot of,use cases for these,for turning text into structure,rather than,just, text into text, as you get from generative, purchase or structure into text.

hugo: Very cool. I am also [00:58:00] interested in when talking about, human loop distillation. we definitely briefly mentioned transfer learning. We may have even mentioned in context learning. And these are such important things, I think, for people to be aware of that. I'm wondering if you could just break down what transfer learning and in context learning are, and then talk about how they can compliment each other in bridging the gap between predictive and generative tasks in NLP.

matt: Yeah, so I guess there's the, new employee analogy, which Ines, do you want to explain that? Oh, 

ines: oh, I think, yeah, for transfer learning, we've often use the analogy of oh, imagine, you're hiring a new employee. you expect them to come in and already know, the language,know how to use.

I don't know, a register, know how to whatever your business is and, know how to talk to people like you, you're not going to expect to raise them from birth, but you're totally fine about teaching them the specifics of the job. And, I think that's a really fitting analogy for transfer learning.

Like you want to start off with general knowledge about the language and the world. And then on top of that, you can use, or [00:59:00] you can use that as, embeddings. And on top of that train much more accurate, predictive, models. For very specific tasks that you're interested in. And that's the only part, that you're training.

And I think just because people are now,focused more on in context learning where you provide, the natural language prompts, doesn't mean that transfer learning is somehow outdated or has been replaced. It's it's just a different technology, for different types of use cases.

And as Matt said before, examples are. incredibly powerful. they are like, they are downsides of having to,write rules or instructions and, showing things specific to your problem has like a huge advantage. it's mostly about, okay, how can you actually make the most out of that for your use case?

matt: And so for in context learning, this is, a step beyond the transfer learning. And it was, I found it quite remarkable when this was discovered to work because I certainly didn't predict it and still find it, very unintuitive. That,you can code the problem [01:00:00] into some sort of, into a natural language prompt, or, instruct the model to, do something rather than training it to do it.

so you might say, okay, classify these articles according to this, and then it will, you produce that classification. Or sometimes you can phrase the problem as a completion task, and it can, complete the thing with the right label. so then there's this distinction of okay, taking that fundamentally in context paradigm, but then you are going to have examples of the input output pairs, and you're going to fine tune the models so that you are showing it examples, but you're still, structurally, it's still a question answering thing.

And, the transfer learning is not that, what you get from the transfer learning is the ability to, have some specific neural network architecture. And this means that you can speak to the le to the model in the language it understands, which is the language of gradients. that's the only, language that, it knows you bid into submission with loss.

and,that's how you craft it to do what you want. And being able to [01:01:00] control that neural network structure lets you control the loss function. So this means that you can say, okay, what types of misclassifications are more important than others? What,exactly how should this task be structured?

What should, all of those things are, available to you and that can make the model much more accurate.

hugo: I now want to move to think more about, less technical things and more organizational and social things. and I'll link to this. You published some really thoughtful blog posts recently about how you transitioned, explosion back to a smaller, more independent company.

And I just want to say, I'm really grateful for you talking about this stuff publicly because we're all trying to figure out how to build companies that support open source, but also can monetize certain aspects of open source and productize these things. And we have an increasing number of data points on things that kind of work, things that definitely don't work.

so publishing all of all your learnings, I think, I think it's A lot of us are incredibly grateful for, but that's one of the reasons I wanted to have this conversation as well. So with this transition back to a smaller, more independent company, I'm wondering what, [01:02:00] what drove this decision and how do you see it shaping the future of what you're working on?

matt: people talk about startups very euphemistically,and I think that's not necessarily good. So to speak plainly, we ran out of money. and that's a great inciting incident for, transitioning the company as we did. for most of the company's history, we were very small and like running off revenues.

Then in 2021, we, we sorted, there was something that we wanted to do a product that we wanted to build that we weren't able to build in that mode. And so we. took venture funding to do that. We tried to be larger and to scale the spacey team and to, have more work on that with, the availability of capital.

And ultimately our success at running a smaller company didn't translate to running that sort of company. And we ran out of money before we could release the product that we wanted to do, and we weren't able to raise more. And we transitioned back to, back to our roots as we put it in post.

ines: Yeah. And that also means, we can't, we, we'd be able to really focus on kind of the core of what we're doing and that [01:03:00] is like our stack and we want to definitely, I think one reason we also published this post is we want to make sure, and we want to also make sure. People know that like our stack is not changing,it's core identity and we'll continue, maintaining the products and also continue working on the things that we think, make the most sense for the company, to work on basically.

And that's like a lot of the stuff we discussed today, for example,fall into that category. 

hugo: Absolutely. I am interested. in, in your post Matt, you make an A wonderfully insightful point that the saying hindsight is 2020 is actually not true at all. If you think about it, and these types of things,wonderfully and complexly multi causal as well.

But I'm wondering if you can tell us a bit about through the process of raising, then spending and building product,what do you think went wrong? Or if you were to do things again, how would you approach things differently? 

matt: Yeah, soI definitely, I've never. Believed in [01:04:00] hindsight in this way, and,reading these postmortems and things, you people are looking back at, the decisions that they made, and those were difficult decisions, and they picked the one which they thought was going to work out best.

And,ultimately, they might say, Actually, I should have picked something else because, I got a bad outcome there, but you're trusting the same person who made this, mistake in the first place, if it was even a mistake, who, demonstrably didn't have the, these right insights about this to, maybe their second guess is right, maybe it's wrong.

And so I think that, there's this sort of epistemic humility about this of saying,I shouldn't pretend to know what went wrong or to have like a. this type of analysis, because ultimately,causal graphs are hard to induce. I don't know why it's difficult. and certainly I don't know what could generalize from this situation to something else, because there's so many,specific factors around,what we did in the company, who we were as managers, who we were as, what sort of instructions we gave people, the COVID pandemic, like, [01:05:00] all of these things.

And so I really don't want to weigh in on what sort of open source structures can be successful or not successful, or what our example teaches people or something, because I don't think people should view it as teaching them anything. 

hugo: I, I agree with that. I've actually just, I'd forgotten I wrote this, but I wrote something near the start of COVID for O'Reilly called Decision Making in a Time of Crisis.

And I actually wrote specifically about that. we definitely shouldn't judge Decisions by outcomes, right? Yeah. And you can even think there's actually, I don't know if you remember the movie swingers with Jon Favreau and Vince Vaughn, but there's a scene where they're gambling and one of them does the right thing.

he like splits on whatever, and he loses and he's that was the wrong thing to do. And the other guy's no, you actually made the right decision. just. The probabilistic outcome wasn't what you wanted it to be.so in, in games of chance, which is of course where probability arose and [01:06:00] how we think about it, this is entirely obvious, right.

ines: Yeah. Yeah. And I think also when you, yeah, when you're running a company, like you also, You, on the one hand, you might have some ideas of, how, what you think you should do and how things, should work and your conviction. And I think we've always been like, they, opinionated about some things, but then at the same time, if you're like growing a company and, building a product, you also want to be open to, doing things.

in different ways or there are certain,adjustments you need to make if you have a larger team or if you're,even if you're building a SaaS product, and things like that, or, taking more advantage of the cloud. And, I think these are all great things.

very difficult aspects and you can't, it's very hard to,I'm sure we could come away and thinking, oh, we should have stuck to our guts more. We should have,done this. I think we always try to be very careful. Open to,on the one hand, Oh, really reasoning about every decision, but on the other hand, not becoming too, stuck on like our way of doing things, which doesn't necessarily translate, into say running a larger company or, think so.

[01:07:00] Yeah. Yeah. So I think that's also the. The difficulty there, when trying to evaluate this. 

hugo: I am interested, Matt, in, in your post, you do identify, several things. And one is, as you know, I, I live back in Australia now. and I actually, I work with people in the U S and Europe and stretching across time zones.

I've got ways of making it work, but it isn't always the most straightforward. what way of working? you did identify a concern with respect to not only remote work, but stretching across time zones.and how can we make that work? 

matt: maybe you can't. because, even if you, somebody works a night shift, this is still bad for them cognitively.

they're not going to be able to make the sun rise at a different time. and, they're not going to be able to build some artificial sun. there's no, no amount of sad lamps can, make up for living a night existence. Right. So that's going to be bad for your sleep.

And that's ultimately going to be bad for your work. there's going to be some contexts where working asynchronously,[01:08:00] is like totally suitable and it works totally fine because there's this sort of natural football effective. you've logged the thing over to the other person and they work on it for a while or they've got their time 

hugo: and you're able and the style of work with the person that can even be like, 

matt: yeah.

hugo: Gains in efficiency, actually. Yeah. that's what we were, 

ines: that's what we wanted. That's what we were hoping for. Yeah. 

matt: And there were times where you get the great, the baton, I call it the baton pass. you've got some people work a couple of points and then, they're able to pass this on to the next person and work happens while you sleep.

And, this, when you get into the right rhythm around that, it's awesome. but there's, you don't guarantee that sort of rhythm and it's, it's brittle. And you find that As soon as you start having problems being separated over time zones sucks. and, when we were having problems, in the team for the Europe, team pair programming is a great solution to, to figure out like, Hey, what's going wrong with, why people aren't able to like, do a thing or why they're having trouble or sometimes even just, [01:09:00] feeling motivationally stuck and,not feeling it so much getting that enthusiasm going by.

Pairing with somebody is,really valuable. And it means that the knowledge spreads, you, that's certainly not a tactic that you can use, when you're spread over time zones and it's only one tactic. It's not indispensable, but you find yourself with just fewer resources. there are a few times, apart from that sort of baton pass effect, that's the only time where you would wish you were in different time zones or, managing support load or, something like that can, help as well.

hugo: Do you want to say You can't force the sun to come up at midnight, you're right, and no amount of sad lamps can fix that. But I do want to make clear that both of you live in Berlin, where you may not see the sun in the three winters, one, two, and three, for a month or so. So we need to be careful about that as well.

matt: Yeah, it's true. Like second winter especially,the sun is a distant memory and I am like, subsisting on sad lamps to some extent. And,but Like I'm 

ines: fine with darkness, 

matt: the sun [01:10:00] that shines in winter is still like orders of magnitude, more powerful than you could replicate any other way.

hugo: this 

matt: weak,like pathetic, like trickle of sunlight that we get still overwhelms anything else. 

hugo: Yeah. And actually this is a total tangent, but I remember in 2020 when people online were like, this is the worst year humanity has ever had. And we're like, ah, and it was horrible.

don't get me wrong. But then I saw people like actually know historically, we think this is the worst year. And it's like, 556 AD or something when there were volcanic eruptions all around the earth and supposedly the whole earth was covered by a gray cloud of ash and the sun actually didn't come up for the whole year pretty much and there was plague and pestilence and famine and all of these things so 

ines: yeah that was objectively probably worse yeah that sounds pretty crap 

hugo: yeah so i'm grateful For what I have today, 

matt: if you get to think about it in a time and place, because, it's cheating a little bit to say the worst for the whole world, because you [01:11:00] really only get to do that when it's causes are globally connected.

So it's okay, something like these volcanic And or options and things, but otherwise you're limited to only the last,50 or 60 years or something, but certainly there have been a lot of times and places that have sucked much more to be in. if you just look at a time, place pairing, then, 2020 is the year.

hugo: Absolutely. so getting back to NLP and AI, I'm interested in just looking ahead. what do you think the key challenges and opportunities is? In NLP are particularly with striking a balance between large models and the need for modularity, transparency, privacy. 

ines: I think definitely like at Corbett tooling and, developer experience and basically making, a lot of these like expert workflows or things that we talked about today, really available to everyone.

and,to.cross functional, teams. and especially because we are seeing a lot of people coming into the field now who are coming from a very different angle than people have been, in the past or people [01:12:00] who start out like, Oh, I did some prompts and then go from there. And I think,really picking people up at that step and, giving them better tools, I think is very, very important.

And yeah, Matt, what do you, 

matt: No, that's exactly the sort of thing that I would say. this isn't like a specific technology or problem, but, We've long noticed in our stack that there's a sort of user profile and Chris from the S& P study is a great example of this. Somebody with a domain background who then,comes across into data science is very smart and enterprising and able to pick up tools and they're able to, drive things forward a certain amount and that get it.

And then figure out that, ah, this is what we need to do. And that's been at the core of so many successful projects that we've seen across industry. people who meet that sort of,that description. So the question is, like, how do we make that easier? Because,if you look at what somebody's had to actually do in their cross skilling in order to get that, it's pretty impressive.

And so how do we make it less impressive? how do we, arrange tools and stuff that are easy enough to learn? And, [01:13:00] we're proud of the work that we've done in spacey and prodigy to come this far, but there's still further that we want to go. So one of the example, one of the types of problems is around machine learning.

and, I guess the. In the literature, it's called like parameter free learning. So there's so many knobs to tune when you're,when you're doing supervised learning and, you get very little signal back, from the problem. You really only get this number that says, ah, shit didn't work. And, when you're stuck in this,shit didn't work stage.

You actually, that's where the knowing theory actually helps a lot because it starts to, you start to know what to try to gather more information and to unpack this a little. And, you're like, ah, I should look at whether my gradients are changing or I should look at the variance of my gradients over time or something like this.

but. so how do we make it so that's not, necessary? In space, we have tried as much as possible to have the architectures not be so hyperparameter sensitive and to make them scale down to a few examples, but there's always more to do around that. And in particular, how do [01:14:00] we get the sort of latest, benefits from,more accuracy while keeping them as robust as we can? 

hugo: I do, I have always been super, and I'm just concerned about complimenting you all too much. but I do mean it. like the work you've done on spaCy in particular and the abstraction layer, that you've created for the developer experience and the scientist experience has been a huge inspiration to me and a lot of projects that I've been involved in as well.

thanks. Yeah. And also, like the documentation and the MyBinder interactivity on your website, like clearly it's a very thoughtful, mindful approach. I also, Inez, you said something earlier where you mentioned that AI is software and we're talking about software. And I think this is something that gets lost a bit too much.

People are like, can AI do this? And I'm like, Remember, we're talking about software and it's data powered, ML powered, AI powered software, maybe, but we are talking about software. So I'm wondering how do you think the intersection of AI and software development will evolve, especially [01:15:00] when we're integrating more and more generative aspects into traditional software workflows?

ines: Yeah. So I think, it's definitely never been easier to get into programming and, to build things. you have assisted coding, you have there's so many resources. So I think that definitely, has a huge impact. And actually also, on what Matt was saying earlier, like helping people skill up.

and I think actually one. Big misconception, or misunderstanding that I think people have is that they think that, there's this idea that in order to make things simpler and easier, for non experts or non programmers, like whatever that means, you, you should add more layers of abstraction.

on top and that's how, you end up with all of these, magical one line of code or one click solutions that are marketed towards, people are not,programmers or, beginners. that sounds great, like in theory, but once,you're actually hitting custom things and things you want to do your,you're basically stuck and it becomes much harder, than any other workflow.

And, so I think instead of,[01:16:00] seeing, software development, and, yeah, in the intersection of AI as something that will make, programming less relevant. I think it's actually the opposite. And I think because it's, a lot more approachable and a lot easier. Descriptability and making things programmable will actually become more relevant because it's something people will be able to do, and it is still the best way of telling a computer what to do.

at least,that's how I see things. And, yeah. 

hugo: Absolutely. And there are two things that came up for me then. The first is Firstly, the ability to paste tracebacks into chat GPT and to have it abstract over my local file system, as opposed to Google, or Stack Overflow, is incredible.

The other thing, Claude recently taught me how to build like basic React apps and renders them as Claude artifacts, and This is wild. It's absolutely incredible. So I'm really excited at, how it can make me an ex engineer. No, I'm kidding. 

matt: yeah, I think I use it as a batch scripting infrastructure stuff.

it knows Terraform pretty well. [01:17:00] it knows the stupid fucking,completely nonsensical command line, thing that I need to do for AWS,this sort of stuff, it's, it's, I think 

ines: what you also need to consider. I think that's why I like also the, web development example is like that there is this difference between, lowering the floor and raising the ceiling.

And I think that's something, people. Sometimes forget that yes, technology helps, makes it much easier to get started. and for example, in the, the web is a great example because that has moved on, a lot further, like everyone and their mom can now basically create a website for their business, but,it's.

That does not mean that, that,web developers are obsolete. it's actually the opposite. There's so much work being done at the other end of the spectrum and, at the ceiling. And if you're Netflix and you can make your little,browser based player point, whatever percent faster, that's like translates to millions.

And,There is there is so much work on browsers, there's work on,these technologies. I think it's always, important to consider where on that spectrum are you,just the fact that [01:18:00] we're lowering the floor, does not mean, that there's not also, a lot of work.

Done on raising the ceiling or that like development becomes obsolete. It's the opposite. 

hugo: Absolutely. I like the Netflix ex example as well. I do think you and I will probably have similar concerns around the fact that such corporations can, make millions of dollars from micro optimizations as, as well.

But I am to, to your point as well, I am interested,how interested you two are in building and thinking through like more low code and no code solutions to all the things you work on. 

ines: Yeah, yeah, as I said, I do think,I am skeptical. like I do think, there's so many different things people mean by low code or no code that it makes,it makes it really difficult to talk about.

And I don't actually, people talk about it as if it's this sort of field. when it's 

hugo: to the web development example, I'm thinking like, what would, this is actually a horrible question and it's probably the question you don't want me to, but what would like the square space of NLP?

Look like,

matt: we've, I think we've at different [01:19:00] times talked about building that, something like that, which, essentially connecting dots out from, prodigy, and, keeping the spacey thing behind the scenes and then, you've exposed to text classification.

or, entity recognition and stuff. The problem at the moment is that the technology isn't there. and so the product will, suck in many ways. And, in most of the applications that I think are interesting, it's actually worth doing that little bit of digging and working.

At the level of the API. And so we, never saw it as actually at the moment, the thing to do, because, we couldn't see a consumer application in it. And as a business to business thing, usually.even if you don't want to, even if you don't need to make it like awesome and you don't want to spend forever on it, there's still, there's very few projects where you shouldn't spend at least like a week or something like this.

And then having to fight through the interface to understand, to get better classifications underneath or to, get your data out or something like this [01:20:00] doesn't seem better than,the tooling that we're able to provide. So it's possible that. You can have some sort of no code solution that maps directly into the coded solutions that you can transition.

And that's the sort of thing that we've been, we've, we're thinking that we've been thinking about with some of our products. 

ines: Yeah. But also we want to, obviously we do want to focus on the project that have the projects that are most successful and that have that really deliver value to, a company as opposed to, some stuff that kind of doesn't matter.

and I think also, in terms of,yeah, what you were asking about, yeah, the, these types of applications where you add like some abstractions, on top, like a UI, which can be useful. I think another big problem is that a, what Matt was saying, yeah, the technology, but also the, this isn't necessarily the hard part, like it adds, it makes one part easier, which, to some people is also hard.

If you're starting out like writing code. The hard part is actually happens before that the hard part is deciding what you really want to do in the first place. And before you start even writing code and yeah, if writing code is hard for you, that's another thing, but like [01:21:00] the breaking down the problem into what, yes, you can, I can, you can give someone, it's easy to give someone a UI, for, where you put in the categories you want to classify and upload your text.

and, that does something and, people have tried that, but the problem is that the. The really hard part of that really makes a difference is like coming up with these categories, coming up with what your texts are and coming up with what you want to do. And there is that is actually something even that we're like almost like the most interested in.

It's like, how can we translate this rather abstract concept of deciding which techniques to use, like how to,how to break down a problem and how, why you shouldn't,translate.a business problem one to one in many cases and like factoring out business logic, deciding what's like the part where you can use a generalizable, technique versus,something custom, like all of this is really, this is hard.

This is what people are most struggling with. And how do you trans abstractions on top of that? that is, that's, I think that, and yeah, if we can solve that, I think [01:22:00] that's, that would be like the, obviously the ultimate, goal. 

hugo: So what I'm hearing there is this is a form of like design thinking as well in a number of ways, right?

ines: Yeah. it's also like we've called it refactoring at some points. or, there's days reasoning involved in this, it's definitely not a tech, like a purely technical challenge. And, it's also something you can't,you can't really, decouple from the domain.

you're going to have a much harder time doing this if you're,if you don't really know. the kind of, use case. 

matt: Yeah. So it, it occurs to me that actually the SaaS application that we raised money to build prodigy teams could be used as a no code solution like this, but we, you get, it has the annotation integrated, you farm out the work to, your team, you get the app data back.

And then from the same interface, you train a model from, from that in a dropdown menu, , and then, you can export that, that, and, use the model. we didn't have the sort of mobile posting, side of that because we didn't think anybody would want it. And, [01:23:00] actually we, didn't think of this no code solution as the real nature of the product.

Instead, we imagined that most people would want to script their own recipes and, to make annotation better and doing, a number of these other things like this. as Ines said, yeah, the planning phase, is Where we see a lot of leverage and then also feedback from errors.

it's the other thing that makes it hard and the no code solution can't promise to do better at that. If we knew what the errors meant would tell you in the terminal.

hugo: . So to wrap up, I just want to, let's try to do some acts of prediction because we work in machine learning significant error bars because we work in statistics or variance as well.

So if we would have this conversation in five years from now, what type of new topics do you think we'd be discussing about AI and NLP and your work in the space?

ines: Yeah, it's difficult. I do get that asked sometimes. Yeah, 

matt: five years isn't that long. So what would we have been talking about this in 2019? we wouldn't know. Something similar. 

ines: Yeah. Just different terminology. People just always reinvent terminology. it's like a lot of the problems stayed the same.

And now [01:24:00] people talk about, or like LLMs and now everything's an LLM. People talk about agents when before it was like a classifier and a lot of cases. So it's just We 

matt: had GPT two, not GPT three, some of the same lessons were there and, we could see the scaling laws paper, I think is older than that.

GPT two 

hugo: is very open as well. 

matt: Yeah. maybe more closed models 

hugo: is something we've seen in the past five years. You'd almost think 

matt: that the organization was called something like, the AI would be available. 

ines: Yeah. yeah, actually we put together, we For a grant application recently, we actually put together like a research timeline that actually really led to, a lot of,led to the ideas that we, in the implementing, you know, around prodigy or for prodigy teams or the, actually a lot of human in a loop distillation type stuff.

And it was interesting to see when the trends were already coming up because, we got like a feedback question of but all of these, GPT4 wasn't, or chat GPT wasn't,available when you started that project or there were so many changes in technology, like what did you have to [01:25:00] change?

And then we were actually, we did that research timeline to explain like,there were always small adjustments, but actually the research trends were there in 2020 when we,really restarted, that project, it, it was all there. It was all like, pretty,

matt: I've thought of something.

so this is being a bit provocative and, only describe like one possible future, which I don't know whether this is how things will plan in. But I think there's a significant chance that in 5 years, we'd be talking, reflecting on the hype cycle and,saying, there was all this promise and it didn't pan out in this way.

The same way that, we might talk about big data technologies or something like this, except that hype cycle was much like steeper. So to, to put my cards on the table,I'm still hedging this a bit, but I think there's a significant chance that every. Non non fundamentally generative application of large language models.

So these things doesn't pan out. the, the use of in,in something that'sthis chat application is super clear. So that's, and chat GPT is a product. So we clearly see that type of product category works. It's [01:26:00] useful. I use it. Other people use it and it's tremendous promise in that product category.

what's, but what's claimed as a deeper thing is that this is a way that, transforms,these predictive AI workflows and that sort of thing. And, I'd say that there's a significant chance that doesn't pan out. and the success that people are seeing in this. At the moment, partly comes from problems that can be fixed with the workflow around training your own models and, using data rather than instructions and partly also comes from,there's some misclassification where people talk loosely about LLMs and say BERT's an LLM and so you're not actually talking about different approaches at all.

and then some of it is,also just focus on prototypes or focus on proofs of concept. And as soon as you go to actually translate that success into something,more stable, it's unclear how that will actually work. so yeah, I, I think that's one possible future in one way that, our discussion could be framed in, 2029.

ines: Yeah. Yeah. And I think also, it's hard to say what will happen, but like clearly right [01:27:00] now, especially if you're looking at structured data and using gen AI for that, it all feels like a massive, it's like a hack. There's like a lot of,it's this hack around it that like, it's good for, prototyping, but, it's all, a lot of it, it's all very hacky.

I do think, eventually there will be. a better way to actually, have models that you can interface with on, kind of computer to computer level, rather than, like actually solving a computer readable tasks by writing this question to this model that then responds in texts that you then pass into something structured so we can go back into the computer.

That's a hack. And I think we also, from what we hearing and seeing of the Larger companies or companies that actually,use things, successfully, or use,even chat applications. there's a bit, it's always a combination of models. You might have an 11, like this idea that Oh, you have this one model that does everything is not, not actually realistic.

and again, in practice people, people have a model that actually generates the text and a lot of the other things, if it's something you can [01:28:00] predict people will Build a model to predict it and then put them all together in this like big ensemble. I think that's, and that's how, chat GPT works.

That's, how,a lot of these products work as well. but I think, there's obviously. People have, some people have an interest in,promoting, blurring the lines, between, again, between the product and that's, it's all part of this, AI revolution, this monopolization point, as well between AI products like ChatGPT and the underlying, product.

actual model, which is GPT 4, for instance. so because if you blur the line between that, if you blur the lines, between that, then,for example, if you're arguing for, regulation, you could use,for regulation that favors you as in, OpenAI, as,Sam Altman is doing,he's not genuinely concerned about, the danger of AI and one's regulation,big tech leaders, one regulation that only they can meet.

and and what helps there is blurring the lines between products and models. because you can point to chat GPT to show, Oh, here's [01:29:00] all of this dangerous like stuff that the technology can do. And it's underneath it, you have just have this model. all of the rest of that is just product.

And a lot of that product isn't even AI. and, yeah, I think, did I lose my train of thought? you didn't lose, but Yeah, but I didn't 

hugo: lose your train of thought. Agreed. Or I didn't lose my train of thought. Within your train of thought. My human in the loop dis distillation of your train of thought.

I am. What I, and this is something we didn't touch upon, but I do think we've got around 10 minutes left that we scheduled, the idea, and this is something we've talked about before you've written about and spoken about. but the idea of, regulating, the model layer, as opposed to the application layer is deeply concerning.

And we're seeing movement on that front in a lot of ways and how this may actually impact, the ability of the AI revolution to be monopolized. So maybe. You could tell me your thoughts on that and things happening currently in legislation, that may impact. So I think 

ines: that the EU AI Act, I do think what I like about it, or what is,at least,less concerning is that [01:30:00] it does differentiate between actual use cases or, like risk level of use cases and applications of the technology and not the technology itself.

Because, again, people have proposed ideas where it's like about the models and,evaluating and, a model itself. Can be used for so many different things. And, it's about what you act like it's about actions. and, it's about applications and things that you build with it.

And, the type of things they're doing. So I think that is,reasonable and useful. way. Of approaching that and,definitely better than, I dunno, some of these ideas of where it's about the size of the model or something like that's,or the number of parameters or whatever else people have been trying to do.

That's 

hugo: I can't remember what happened in the U S it may have even been like a Joe Biden executive order or something along those lines, which had to do with, the number of parameters or how many, 

ines: Stuff like that. That's that is very concerning. and yeah. 

matt: Yeah. look, I think, there are things in AI safety to worry about.

it's I don't want to say the solution is to do absolutely nothing. The [01:31:00] solution certainly to any,problem like this isn't going to be well. We need to exhort the companies to act in a moral way. I am in principle fate, in favor of regulation to solve this type of thing and to set playing a playing field that, has clear rules for companies to abide by.

That said, it's really difficult to regulate this type of emerging situation. And the prospect of it really doesn't fill me with confidence. you, Look at the way that this is going to be done, then it's, a room full of people whose average age is like 102, and they, you listen to some of the hearings, and it's clear that they get their information from misunderstanding the same bad news articles that are available to us, or lobbyists, and, there was this, Panel where they were really trying to, be aggressive against this, ByteDance executive.

And look, if ByteDance were doing something shady in the U. S., he wasn't going to be caught by that panel. it, it just outruns them to straw because their questions are so uninformed and off base to anything that could be, like, actually problematic. [01:32:00] And so the decision making apparatus behind this is, something that, we worry about.

and,I, see little reason to doubt that players like,Sam Altman and,and such would be attempting to regulate themselves into a monopoly. that's what I, it's a cynical view of their motivations, but I think that, I don't see why I'm forced Yeah, this is the strategy of companies like have some charitable interpretation of 

hugo: if we look at the last 25 years as well and tech companies and also the revolving door between tech and government in the US is more than any.

Industry, historically, I think, as well. But also on the 

matt: Google spends enormously on lobbying. They've got an extremely effective lobbying,presence in Washington, Yeah, 

ines: no, but also in the, industry, there's not even,such a cynical take that, if you are in a, a lot of tech companies, are going for winner takes all opportunities.

And that's also, where a lot of venture capital comes in and,that's just how it is. And that's like kind of a fact. And if you are [01:33:00] in a winner takes all market, then,you're trying to create a situation where,you get rid of,the competition, and, That's, again, that's also just, the operating playbook.

And then of course they are different ways. you can do that. Like money helps a lot. if you have economies of scale, like in, a lot of the social things that works great. or, an alternative, that's very attractive as well. You can't, you can't compete because the government says so, which is regulation.

And these are some of the main things and this is surely You know, this is what companies are thinking about because that is how you navigate a winner takes all, market. that's not, it's not even such a cynical take. of course that's on people's minds because otherwise, they wouldn't be successful as a company.

hugo: Absolutely. And, we have people like, I'm not going to go down. Peter Thiel's like zero to one. He's Hey, monopolies are great for the world as well. And there is a worldview of that. 

matt: Yeah, exactly. It's not a worldview I agree with, but it's certainly well developed and it's certainly influential.

and, he was on the board of,of Y Combinator when, Sam Altman was,CEO, there's the ideas [01:34:00] have great currency between them and, I don't see any reason to believe that Sam Altman is like, aligned with Teal politically, for instance. I don't think that's the case, but I think that, the, these ideas about how do you make your business a monopoly in order to make it more valuable?

That's like Silicon Valley 101. This is like basic. and yeah, as I said, one of the, one of the best ways to do it is if you can get yourself regulated into a monopoly. 

hugo: I appreciate you both for this wide ranging conversation. I do that. We have a lot of listeners who are scientists, data scientists, people who work in machine learning, who I think have probably experimented with generative AI, large language models, perhaps pinged APIs, hosted a few things locally, perhaps, or fine tuned, big, large language models.

I'm just wondering for people who want to get started more with thinking about data, private models, reliability, affordability, modular workflows, trans transparency, explainability. So all the human in the loop distillation stuff we've talked about [01:35:00] as well. I've linked to your blog posts, and we'll do so in the show notes, but how would you encourage them to get started technically as well?

matt: You can use chat GPT pretty well. spacey has been stable as an API for a long time. And, it was developed when the internet didn't have so much generated content. And so there's a lot of Stack Overflow answers and things that chat GPT has been trained on. You can use it as a basic starting point to, how do I solve this sort of problem?

Or would spacey be a good solution to this sort of problem? You can chat through that sort of conceptual stuff pretty well. And, sometimes it'll just be dead wrong. and that sucks. But if you are aware of that, then, you navigate that in and, price in the risk or, accept that like the navigations might send you off a pier.

And, don't just blindly drive where the map tells, where Google maps tells you to go, that sort of thing. 

ines: And I think also you can, if you have some experience with NLP and the sort of traditional types of components and types of structured prediction tasks.

we do have, spacey, LLM, which is open [01:36:00] source and also, does integrate with,models that you can, run yourself. Or like you could start out with an API. And,basically you get components that mimic exactly how,trained the trained components would operate only that they, use, the LLM API or, generative model for it.

and that's A very easy step to prototyping and, it gives you a model. It gives you a pipeline. You can evaluate, which also, it's very important. If you're interested in improving your model, in any way, you need to have a stable evaluation and you need to have a stable way of telling how good is it actually, and, what impact do changes have?

it's like writing tests,yes, you can start out and get away with not writing tests, but eventually you want to have ways of checking if your software is doing what it should do. And that's the same. For models. And yeah, you want to have something you can start out with and then improve that further, and then.

Experiment with methods of how can you replace this component that does say named entity recognition using GPT [01:37:00] 4 with One or more components, that do the same thing, but better, faster, and more accurate. 

hugo: Super cool. And I'll include the link to spacey LLM docs in the show notes. And I'm just looking at it now and in classic spacey style.

So there are some beautiful examples. So people check it out. the first example is at a text classifier using a GPT three model from open AI, then add, named entity, Ricky add name. Entity recognition. I've been talking for several hours using an open source model through hugging face and these types of things.

So it's great how, there's a sense of, and then creating the component directly in, in Python. So all of these things are wonderful. So please anyone listening who wants to check this stuff out. Please do. that's it. I'd like to thank everyone for joining everyone for listening. Most of all, I'd love to thank you both.

we've been going for two hours and I really appreciate not only your time, but also your expertise, your wisdom and generosity as well. You've always been so kind and thoughtful to the community and for coming on the [01:38:00] podcast. 

matt: Oh, thanks. This has been fun. So yeah. 

ines: No, I need to go and feed the cats.

I felt sorry of those. if people have heard some meowing in I've think 

matt: before. Oh 

ines: no I didn't.if you, so no, just saying if you hear some, some meow, if you heard some meowing in the background that was, Stanis la Petro, who was asking where his breakfast was. Yeah. 

hugo: and I need to go and feed myself.

So if anyone heard my tummy rumbling, that's what that was as well, . thank you all once again and see you soon.