The following is a rough transcript which has not been revised by Vanishing Gradients or the guest. Please check with us before using any quotations from this transcript. Thank you.
===

hugo: [00:00:00] Super excited to be here with these. Three sharp, luminaries. I, it's usually my job to introduce people, but I usually get other people to do my job for me. So I'm actually going to get Joe to introduce Jepson. They're close friends and feel free to say anything.

Joe. 

joe: Yeah, I will. so Jepson, is an old friend of mine,he used to go by the name of Ben Taylor and then he changed his name. But, but in the machine learning space for a long time, I think it's how we knew each other back in, we're from Salt Lake city, Utah. I met you, I think you're trying to sell like a bunch of GPUs way back in the day.

try to offload them. And we're like, okay. We don't need him, but thank you. So anyway, but now you're doing some cool stuff. So yeah. 

hugo: yeah. And Ben, do you have anything to say in your defense or why don't you introduce Joe as 

jepson: well? I think Joe's an alien because he's not aging. So when Joe and I met, I didn't have any gray.

And then I went gray pretty quickly during the startup. so yeah, one of the things about, Joe is we're more likely to run into each other in Paris than we are in Utah, or just we travel a lot. [00:01:00] so Joe's, best selling author, you've been, did you work at Pure Predictive? we've crossed paths.

I think it was 

joe: employee number one there, yeah. 

jepson: Yeah, so we've bounced around in the local community and, yeah, been aware of each other for quite some time now. And, 

hugo: yeah. Cool. And Juan, would you tell us a bit about what you're up to? Yeah. Data. World. 

juan: Yeah, so I'm Juan Sequeda. I am the principal scientist and the head of the AI lab at Data.

World. a company here in Austin and we're a enterprise data catalog company but I'm a computer scientist. My research is all been about data. Combining relational databases and what we call knowledge graphs for all data integration and then now obviously combining knowledge graphs with LLMs have been a lot of the work that our lab has been doing for the last couple of years and, so yeah.

I, Joe and I were hanging out last night, going around talking about like the history of computer science in Austin and stuff and then, He was coming to this panel and I just showed up here two minutes ago and now I'm on the stage, The history 

hugo: of computer science 

joe: in Austin. you could have, I think the conversation last night, you could have gotten a computer science degree off of the topics we covered.

It [00:02:00] was 

juan: like four hours of just chatting. We were bar hopping on 6th Street last night. 

joe: They're really exciting 

hugo: people, so this panel, you're just going to be I do think the role this city played in, the establishment of Pi data and these types of things and the entire SciPi stack is actually quite incredible.

But we're here today to talk about Real problems versus real solutions versus chasing trends and the hype cycle in generative AI. So I'd like to start with a provocation. is generative AI as a nascent technology, more like electricity 120 years ago or more like blockchain 10 years ago? 

juan: No, it's, I wouldn't say it's electricity.

I'm going to, I compare it always to the web. And so Tim Berners Lee comes up with the web in 89, people started hearing about this very geeky, very, and then it exploded. And I think that explosion is happening much, much faster right now. But this is a web moment. So people are like, there are all these articles, all that web thing is going to not going to work.

And now it's everywhere. By the way, my pet peeve, it's the web, not the internet. When you say the internet, you really mean the web. So I [00:03:00] think it's a web moment. So it's here to change our lives. They'll be here around for the rest of our lives. 

hugo: And why do you think it's a web moment more than a blockchain moment?

And I don't mean to offend any crypto bros that are in, in the room. 

juan: it's, I think it's a network effect. And I think it's just impacting our day to day. suddenly we're going to be doing things that we're like, Oh, there's going to be a point in time. this is how I did my work before the web and how I did it.

Afterward, here's the way I did my work before ChatGPT. And after ChatGPT, we're already seeing that. So 

hugo: are we though? So let me be a classic contrarian. So actually I've started freelancing, this year full time. And for personal productivity, I'd find a generative AI one of the biggest game changes for me.

But this is an Upwork study data reveals 90%, 96 percent of C suite leaders expect AI to boost. Worker productivity, but 70 percent of employees report AI has increased their workload So as a provocation how do we see it? Actually providing value and utility beyond the let 

joe: me ask you [00:04:00] this.

How are you finding? It's affecting your own life and making that you more productive 

hugo: Yeah, so I A lot of ways and that's because I know how to use it essentially. It's not, I don't necessarily use it in scalable ways, but for example, getting, meeting transcripts, getting my to do lists, getting everything calendarized, all of these types of things.

Particularly as I have a, a bunch of clients now. being like, oh wait, what did this person say? Being able to interact with those documents. Of course, check the transcript. with respect to potential hallucinations and these types of things. but that's one of 

joe: the expectations are of the executives versus the, practitioners who are using it and what the incentives are for them to use it and get trained up on it.

if it makes me more productive, is that a good thing or am I better off like working on my job and burning the clock, which is what a lot of people do. So I'd like to know that it's the details of what this study had in terms What does it mean that people are,having issues with it, 

hugo: Yeah.

and what's your take, Joe, on what type of moment we're having, [00:05:00] whether it is as big as, and game changing as the internet? I don't 

joe: think it's that interesting as the internet, but I think it's, there's obviously utility to generative AI, but I look at it more sort of the epochs that we've had. Yeah.

over the years, like deep learning, I think was a game changer and I see this is the next evolution of it. I mean, it literally is a next evolution. It uses deep learning to do it. But I remember the hype around deep learning when it first came out that is going to replace all sorts of stuff.

and I think it settled into its own use cases. So that's where I see generative AI. it's funny last year. we're speaking a lot of conferences around the world. I think there were, A lot of prognostications. It was the year of imagination. Really? 2023 was more like what? How can we use this in our workplace?

And it's going to change everything and so forth and fast forward today. And how many people are running,generative AI in production in their companies? And are you, are you seeing productivity boost as a result of this? Yeah. Okay. Yeah. So that's cool. I think time will tell how this settles, but I think it's a lot of hype and then we'll see where the [00:06:00] use cases really are.

And I don't think we all know yet. 

hugo: Absolutely. What's your take, Jepson? 

jepson: I think this will be the, internal evangelist in me. So I was an evangelist for the last four years for DataRobot and DataIku and when I think of generative AI, I see this as being one of the most exciting technologies that humans have found because of the ability to extract knowledge and translate it.

so I was reminded reading technical books in college, which a lot of you relate to, they're terrible. They're not storytellers, like if you're learning about physics, chemistry, etc. They're terribly written. but we're trying to extract knowledge. And now with LLMs, you can translate it, into any analogies, any level that you need.

So I think it's game changing for education. the other thing I wanted to add is,I'm, I see a difference in the market. So a lot of engineers are, I put them in the stub their toe category. So they'll try it a little bit and then something that will happen. Catastrophic forgetting, it'll do something that's unexpected and there'll be [00:07:00] frustrated and they're busy and so they'll give up on it.

And then I think there's a small fraction of people in the market where they refuse to give up on it because you can fix all that. Like whatever problem you ran into, there's a fix for that. Yeah. I think you're going to get a very small fraction of the market that end up surprising people. And I think the last thing I'll say is it's a new tool, so use it.

That's how humans learn to become competent. we all started on the same starting line a few years ago. So I have two anthropic accounts. I might even need to get a third. just that's the level that I'm using it. , you just don't wanna pay for it. No, it's 

hugo: Could you use Claude in computer mode to set up a new account?

jepson: So I use the APIs for OpenAI and Claude and then just for my active development. Yeah. I've got, I currently get two accounts and then they get locked. And

hugo: and in terms of the type of utility we're actually seeing, I think it'd be interesting if you could maybe give a teaser for your keynote tomorrow on the type of things you are, you're working on.

jepson: Yeah. 

hugo: 'cause [00:08:00] that's pretty exciting. 

jepson: So when ChatGPT came out, it wasn't that competent, but when GPT four came out, it was actually capable of demonstrating runaway innovation within a very narrow scope. So what that means is it can actually invent new algorithms. and I think what you're going to see is this will become more and more competent.

So reinventing new machine learning algorithms that are competitive to XGBoost, eventually, as long as you have a digital twin, it can. And this is something we were talking about before is I think the mindset will quickly change from prompt engineering to goal engineering. or even for people in the audience, if we had a million dollar prize for who can come up with the best prompt for some challenge, it's actually a really stupid competition because if you can define the metric of success, AI will always beat us.

AI 

jepson: will always write a better prompt if it's able to digitally test that. And yeah, so that'll be the keynote tomorrow is focusing, walking through some of these examples where you can see runaway innovation happening, within very narrow domains. [00:09:00] 

hugo: Very cool. And Juan, what do you see as the major use cases,that you're excited about?

juan: So I think that the low hanging fruit on like productivity and I think what we really need to do is, look at it by different kind of business units or whatever it is. Senator company like at data dot world, we actually have a VP, AI ops. So one person who said we're going to go, he going off and looked at every single department and said, okay, what are you trying to go do?

What is the problem? We're trying to go solve. Like what are you complaining about? And yeah, let's understand the lay of the land. And I think that's the first thing that everybody should be doing right now in our organization. It's what are the problems? what are people complaining about?

And then see, okay, that's a problem now. Could I use this? not like I have this hammer and everything's in nails. Like these are problems now, how I can go see it. And is it actually worth it to go solve that problem and go invest? so like here's something we did that they showed yesterday to our sales team, which was they have a training app.

They're like, you're going to be calling somebody. We sell a data catalog, right? People can't find their data and stuff. So they actually have, we've [00:10:00] trained a system that can go off and saying, Hey, I am the VP of data at Coca Cola, whatever, right? You just on the phone, you have 15 minutes, go training.

And then it will go do the whole conversations and it will tell you how you did and that is a really helping people to go get better at that. So it's not just productivity, it's actually helping you be better at your job. So we just found a bunch of stuff like that. Like today, sending emails, just, I want to prospect, I want to go send a more detailed email to somebody like we can now do that in matter of seconds, which is before it took so much long.

So I think there's just, there's not one thing to go do. I think you need to go all out inside of your organizations and literally just go talk to people and understand what are the problems. What are the problems that you're hearing? How costly are they? And I call it the magic wand exercise. Like you got a magic wand, right?

If this was solved, what would you, what would that open up? And we don't know 

hugo: yet. So we just got to go talk to people. And so I think it may be useful. This is something you and I chatted about recently a bit, Joe, to just reason through if we do have a business problem [00:11:00] or an organizational challenge, What are heuristics we can use for, yeah AI might be useful for this as opposed to actually we just need some baseline search or something along those lines or another technological fix when, what type of problems should we encourage people to think about generative AI solutions for and what not?

I think 

joe: you both have answered it really. I think you need to test the edges of it and see where it's useful. It's equated to an alien being landed on earth recently and we're learning how to communicate with it.even, I would say, very prominent experts in the field I know are still poking holes in LLM trying to, figure out their use cases in their own companies.

Andrew Ing comes to mind over at Deep Learning, where he's, urged his team to figure out how to, incorporate large language models into their own work, and so forth. So I think, but for somebody like him, he would think that he would, grok this whole thing, no pun intended, and, think, trying to figure out the boundaries of this stuff.

So I think it's just a matter of experimentation and getting into it. I'll give you an example personally, [00:12:00] right? even at my son has a,issue with a reading. He's 11 years old. and he can read, but he, always seems to fail his reading tests at school. So I was like, what's going on here?

and I was like, okay, you're not getting the feedback you need. So why don't we,actually fire up ChatGPT and give you a reading assessment every day. a couple of paragraphs you read this and you answer some questions about the comprehension and his scores have gone up considerably because he's practicing and he's getting real time feedback and he's understanding where, the gaps in his,competencies are.

And I think that's a use case where it's that's a perfect thing, right? If you're trying to learn something, you want to get instant feedback. You don't want to read 20 minutes and then wait for, Nothing. so I felt like that, at just on a personal level, I thought that was really awesome to see.

So I would say just find pain points like that, where if you can increase feedback and iteration,that feedback loop much more quickly. I think that helps. but machine learning in general, right? You're trying to scale, you use that when you want to scale past what humans are capable of doing.

and I think you know what those are. Like computer vision is a good example, right? If we took everyone in this room and tried to have all of us look at images and classify them, I think we'd be. Gone [00:13:00] by the time that we're a spam 

hugo: filter. 

joe: Yeah. Our board, one of the two,where that's why you have a machine do it.

So I think that these are the kinds of use cases where I look at generative AI and say, okay, so what can we do to scale past or make improvements in? But again, the reading example, I think is a perfect example where that was just, there's a huge pain point. I don't think anyone who has kids wants their kid to just like really suck at reading.

So you're going to figure this out. but it ended up being a really good option. 

yeah. 

jepson: I've got a fun use case I was, just remembered. So I'm building a web application and when you put it online, you see a lot of hacker attempts showing up in your logs and with LLMs, it's pretty straightforward to build a hacker.

So you can build it, you build an agent and it attempts to, find vulnerabilities in your system and then the main system takes it offline and blocks it. But something like that is super fun. And it's with an LLM you can. Continue to put context in it and continue to encourage it to be more creative.

And so there's so many applications that show up. And I think that's one of the themes that started to come up on this panel is there, there's more [00:14:00] applications than have even been found. It's really about thinking, bring new perspectives to old problems. 

hugo: Cause I love your example of thinking about novel algorithms because David Donoho classically preferred the term recycled intelligence to artificial intelligence.

because it tends to recycle what's in its training data. But something you're hinting at is the ability to actually develop novel things that are maybe out of, of training data, right? 

jepson: it's the boiling the ocean, comparison. So as a human researcher, here's a quick example.

Let's say we give someone a billion dollar prize to, By this weekend they have to deliver a 10 percent improvement beyond XGBoost or a 10 percent improvement beyond GPT 

3. 

jepson: there's no human on the planet that can do that. Despite the prize, like an incredible prize, they can't. Because humans, we're limited to the number of ideas we can have.

I like to say we're always limited to our digits. How many good ideas can you have? And then the ability to test them and vet those ideas is also consuming. It's time consuming. So for this [00:15:00] example We're moving into a future where AI can have a thousand ideas, a hundred thousand ideas, a million ideas.

and it can, they're not stupid ideas. They're intelligent ideas. And then testing them in the digital space is now you can test them instantly. So I think it in to hit on a point that you said, a lot of our, a lot of our intelligence is compounded on the past, the random walk. And so is this the best way to design an algorithm?

Even though we're so proud of what we've 

hugo: done. 

jepson: So it, yeah, it's a very strange world that we're moving into. 

hugo: Yeah, and you've hit upon a really key idea there, which is, I think, that, we all know, the combinatorial explosion that can happen with LLMs when generating ideas. And the overwhelming amount of ideas, you can ask it for 500 or 1, 000 ideas in parallel, if you want.

But we haven't, At large started talking about its ability to filter and select from ideas as well I think that's going to be one of the big superpowers is not to generate [00:16:00] ideas, but then to help select them 

jepson: there's natural things that are well understood with genetic programming and evolutionary processes where you have multiple agents that are working together and then They're failing or succeeding and learning from each other.

So that's all straightforward. 

hugo: And we were just before talking about the example of chess, right? with AlphaGo Zero or whatever it was where, I play a bit of chess, but not a lot. But chess masters have been like, Oh, this thing has done moves that are counterintuitive and humans haven't really thought about, but it's opened up new spaces of games that are actually very interesting for us.

And I think, I like that example and with Go as well because it's, a confined, you get the combinatorial explosion, but it's a confined game with a set of rules where you can see it at play as well. 

juan: Serendipity. 

hugo: Yeah, without a doubt. 

juan: I think like a lot of these new technology, if we go back in, Go back in history, like this is just another thing that has happened.

There's so many, history shows us all the time things happen and just transform things, right? And I think this, these transformations are always something [00:17:00] about, I think about it like this. Serendipity lets us do something that we were not even thinking about or if we were thinking about it, it was like, I can't even imagine doing that.

And now you can do that, right? from Cars, the planes, like I could, We don't have to go get on a boat. Or smartphones. Or phones, everything. Exactly. And then we just take it for granted, and then that's our new, that's our new norm. And then, we're just humans, that's why I have faith in humanity.

Because humans, there's, I have faith that there's, the people will actually be coming up with ideas and new things and that, and the best ideas will always come forward, and that's how we advance, and and, yeah. I will say, 

joe: what you're talking about LLMs is interesting, because when I look at what Apple Intelligence just put on the phone, I actually want to turn it off right now.

Because it's pretty annoying, it just summarizes things, It's really the peak innovation that, we've been hoping for. I don't think so. but what you're talking about is, I think, a bit of a different game, right? Where,combinatorially, you're coming up with new novel things. I think all too often, we're on trend, pigeonhole, a use case, right?

In this case, it's summarize text. I'm like, I don't, my emails are [00:18:00] like, one word usually, yes, no, thank you. yeah, so it's just, it is interesting. 

hugo: Yeah. I also am interested in. Failure modes. A lot of people are putting LLMs in production and find it really difficult to escape, I suppose what we call proof of concept purgatory to actually get, you get this flashy demo, then it's, oh, hallucinations, then it's oh, I'm not getting my monitoring right.

I can't figure out my traces. And then it's all, how do I integrate into this, into my production stack? So what type of failure modes have you seen and how would you encourage people to start thinking about rolling up, rolling out production, LLM powered apps?apart from just hooking up to the open AI API.

yeah, but then you get what was he, I can't remember what was it. Ford or, I can't remember which car company it was, but somebody asked, Hey, what's the best car? and the response was a Tesla, but it was a different car company. And it was because it was backed by open AI or whatever it is.

juan: But I think, [00:19:00] yeah, we're just going to have, we're figuring this stuff out. We don't know these things. So yeah, this is the stuff that needs to happen, right? we, whenever new things were invented, like just, there's 99 ways how not to do a light bulb, right? You have to go, we have to go through all these things, right?

we're just figuring that out. And I think that, that's, that we're at the edge, people who are doing that stuff. And I think this is also just typical crossing the chasm, right? There's people who are going to be super early, right? Innovators, early adopters and that stuff. And then there's people who will just.

Are completely risk averse and they will be the lack of that's just how we always, that's how humanity works. 

hugo: I totally agree. I'm just interested in practical takeaways for people who haven't crossed the chasm 

jepson: yet. And there's some best practices that they can borrow from industrial engineering.

so it, AI has a bad reputation for saying oops. And it's when you're dealing with new technologies, I think it's important to have a meeting, talk about a potential problem. Potential problem analysis discussion with domain experts cross collaboration and come up with all the failure modes and how you're going to mitigate against them.

cause most of the AI failures that you've seen in [00:20:00] society that are in the news that did not happen. Oh no, we have a sexist resume model or we have this. And so a lot of the problems that have plagued applied machine learning You can prevent them just by being proactive. 

juan: So I think to be very specific, what we're seeing, even even for us internally building this in and for customers are doing is one, this is going to sound kind of the boring thing, but yeah, we have to go talk to people and create the committees and the G and the governance boards around that stuff.

But that's something we need to go do because I think it's, you want to have a diverse group of people coming in and thinking about possible failure modes. And if you just have the same group of people thinking about them, it's going to, you're going to have different ideas. Versus you could bring people from different, from different departments and stuff.

So I think that's one stuff to go do. second is, literally, we always end up in POCs, POCs. The reason why things get stuck in a POC is because you're not trying, you're not tying it to the end value for the organization. So it needs to be super clear. I simplify this always. How is this making us money?

How is this saving us money? How is this mitigating risk? If you can't tie one of the three things, then what are [00:21:00] you doing? So you have to be very direct to that stuff. That's how you get out of the POC thing. And then, don't boil the ocean, right? You gotta start really small around this stuff, right? And then iterate and learn.

it seems obvious what I'm saying, but I think these are just the things that we need to go do and it's just, I think the answer is right in front of us, but we just want to like, go test something with technology and throw it out there, and no, that's not, that's how we'll end up just in, 

joe: I think it's actually too early to, call any failure modes yet.

I think because everybody's still in kind of POC, there's some production stuff. And if you're in production, then of course you're probably succeeding. I hope, but I think it's too early to say that, we can assess,have a rubric for assessing the failure modes of this stuff universally.

It's just, it's too soon. 

hugo: What type of skill sets are important these days, do you think? How's that changing? 

joe: For what? 

hugo: To build this type of stuff and to deliver value to an organization. obviously, MLOps. Yeah, but not everyone needs to, be able to wrangle kubectl or kudokernel errors, [00:22:00] right?

Hopefully. 

Having said that, you kind of do, 

joe: right? That's what to say. I don't know if I'm going to get around it, but, I would say it's the same vanilla stuff we've had for ages, really. I think the difference is diagnosing when you would have a failure mode, quote unquote, right?

Because you do have hallucinations, so it's not exactly like model accuracy. As it used to because how would you know, how do you know it's wrong when you're well, I think this is 

juan: the so one kind of this on the social soft skills like this is what like critical thinking and problem solving. Yeah, even more important like I were talking about this last night.

I feel very bad for people who are taking a boot camp to learn how to code That is going to, writing code, the syntax, that's going to be completely automated. What's not going to be automated is the way how you think about solving the problem. Basically, algorithms. So I think that is something that's going to be very critical, right?

We're just like, how do we do agents? I was going to talk right before this about agents. All you really did was just, Broke down the problem of writing software into the world. The first thing I do is come up with this plan, and I do this. yeah, you just broke the problem to smaller pieces.

That's the algorithm to go, [00:23:00] right? and then I make it so small that at this point, it's like, Oh, it's a very fixed input and output expectation. Then I, that one, I can go do an LLM, and then I can do testing on that. I think what's going to change, which is for what I'm scratching my head is testing.

because we're used to like testing software and everything is this is deterministic. And now we're entering a world where it's, I'm testing nondeterministic systems and I, and even our mindset is changing. It's I have test cases for things and this is the inspected input and expected output. And I'm like, but not always.

How am I going to go deal with this? What is a, so I think that's a mindset that we're testing and 

joe: development in this situation is an interesting question. Yeah. I find this really interesting. It's interesting. I was doing a podcast yesterday with Chris Ricamini. he is a Hardcore systems engineer.

It was fascinating as he was using chat GPT, to,It was at scale DB, but it's all in rust, but he didn't really know rust. So he's been, learning it, but he said it accelerated his learning. But at the same time, if it spits out code that you don't know if it's right or wrong, I guess you're gonna find out [00:24:00] the hard way.

and then you can use it to debug, which you still need to know what to look for and still need to be able to read a stack trace if we're talking about like code that's problem 

juan: solving and being critical thinking of, 

joe: right. And in his case, he's building like a, an LSM,style structure, but you still need to know what that is.

for you. So it's so you need to understand the basics of what your craft is. I don't think that goes away. I think it's actually more important now because you actually need to be able to again diagnose what these things are spitting out. You've been doing some interesting research actually on text to sequel, right?

Where I think your results on text to sequel benchmarks were fascinating. who uses text to sequel in their work? Anybody? No? Didn't think so. Okay, but say that you want to write a prompt that generates sequel. How good is it out of the gate? 

juan: Yeah,early on when we were doing all this research, we were like, look, if you have zero business context, that stuff is just, it's going to generate SQL code or whatever, but it's 15, 20 percent accurate, what, given an expected answer, but then the moment that you actually, oh, I'm going to invest in that semantics, into that context of what the business means, that accuracy went up to 78 [00:25:00] percent of the stuff we're doing, and I'm like, Oh, so that's like that context that's understanding what things mean and that's also go talking to people around things and then codifying that and then so I think that's the, if I look at, if I look at this as a stack of at the bottom is data and technology and you go up the stack, you go more into what I'm calling like the knowledge and people and as technologists, we like to stay at the bottom because It's, measurable, right?

it's quantifiable, the stuff that I'm doing. this is better because of this, and so forth. That's why we have standards, and it's easier to, we can get to agreements easier. But the moment that we go up that stack, we hit this whole fuzziness of people, of politics of things. and if you're in the bottom part, and you want to go up, because you know that, for me to solve this problem, I need to go start talking to people.

You enter all that world, and you're like, oh, fuck, Screw that. I go back and that's why we end up so much in the tech space. And then people are like, there's all this gap. We can't communicate that because we're not bridging this. So I think that's something that [00:26:00] we, that, that need, we need to get more bridges and buy.

And I do think that these lms you were saying like you buy a textbook and this is bad storytelling, right? it's actually gonna help us now to, I think to bridge this get deal with all the politics and the people stuff. I think it'll be. It's a scalable way to, to capture what's in people's minds and be able to codify that.

hugo: Absolutely. I've been very selfish in asking a lot of questions because I want to find out the answers and all of your thoughts. I would like any questions we have from the audience because we've got around 10 minutes left. But it wouldn't, if any of you have questions for each other, I didn't encourage.

Joe, perhaps you've got to, you want to pick Ben's brain. 

juan: I've got a question for you. What do you think is the most hyped thing right now? Okay. Apart from generative AI. What was that? Sorry. What's the hype 

jepson: thing 

juan: inside of generative AI? 

jepson: come on. I think the large context windows are stupid. So they keep going bigger and bigger.

But what you'll find is, if you can refine the context, like with a semantic layer compression, the outputs are really good. Because I've seen it [00:27:00] first hand where a large context window, you're putting code in, and you're asking for an edit, and the LLM is suggesting an edit that you gave it. yep, that's how 

juan: stupid this context windows would be.

I'm 100 percent with you, and I think the large context windows is just, I align it to just, laziness. I don't want to think about how to break the problem smaller, so here's everything, go figure it out. I'm like, Or you can sit down and spend how to figure out a little bit and break the problem down.

I'm going to vote on, on, on, on breaking the problem down. Because also it makes me interested, gets my brain working, but there's lazy people in the world. And yeah, and guess what? I think those people who are less lazy are the ones who will be more successful. 

hugo: Do we have any questions? Is there a microphone to pass around or I can also repeat?

There's a mic up here. I can repeat because we're recording as well. I just want to make sure, everyone gets their 15 seconds of fame in Austin.

But you should ask in the meantime, you can come and speak into Joe's [00:28:00] lapel mic. Yeah, I'll repeat the question as well. So I'll just repeat the question. our friend from the Royal Bank of Canada, they put a lot of money into, machine learning, generative AI, these types of things. The results seem questionable, at the moment. So what do we think about, should we be putting more money and more research into these things, or what do we think about that state of things?

juan: yeah, so again, history shows us. I'm a big, you gotta look at history. We go back and forth, right? There's multiple AI winters. We are going to have an AI winter on this stuff. Because of course we've inflated all that stuff. And it's not going to help expectations. And that's a great thing because it's like, If you get so much funding, people are going to go do things.

And then when you have less resources, what happens? What do humans do? Get more, we optimize around that stuff. So yeah, it's going to be a great thing. So I'm looking forward for that headwinter come really soon so we can then focus on the shit that's actually going to provide value and just stop bullshitting.

All that. 

jepson: Do you remember KPMG announced a 2 billion investment in GenAI by the moment it came out, which I think is funny [00:29:00] cause it's like, Oh no, there goes our consulting business. 2 billion investment. Yeah. And I'd love to know the accountability on that number. I'm you work at a bank, I'm sure there's investments happening there, but Yeah, it's just funny.

I think you're right. 

juan: Yeah, yeah, it just goes on. 

joe: we travel the world and meet a lot of, people and business leaders and government leaders, not just here, but globally. And I think there's a sense that, I think last year there was a lot of hype, especially, I think it hit the crescendo, and I think we're on the down slope of it.

Last year, I remember one conference I was at, and fiber optic company was saying they're an AI company, which I thought was interesting. They just sell cables and it's but everyone's trying to slap AI onto whatever they were selling dog food. I'd love some AI powered dog food for my dog.

But what are you saying now? I think it's what you're describing it. Reality is setting in let's try this out. let's see how well it works. Okay. maybe it doesn't. and this is where the correction always happens. But I think it's a good thing that, you know, that this is, long term I'm bullish on it, but short term [00:30:00] I'm, not bullish on bullshit.

but this is just every cycle that we've been through. So I think you're absolutely, you're seeing things that I think I'm seeing,not just in the U. S., but, globally. there's a lot more investment in, AI in the U. S. though, I think, like an orders of magnitude higher in the U.

S. than other places. If you compare it to Europe. It's not even the same league and that's pretty high. I don't know what it is in China. There's China's got a lot going on, but it's almost like a tale of two multiverses at this point. I want to say, 

juan: but I still think it's still a web moment.

Because if, if you look at the web, there's, the web as a technology, and then we, what bubble did we have? The dot com, right? So that was, so we have, in generative AI, again, AI is a thing that's been going on for half a century, right? So this is not just, it just magically happened, like it's been building on things, right?

so this is definitely a, this web moment, but, building all these applications and stuff we're doing, that's the bubble, and then that's gonna burst, and then we're gonna go focus on it, and then, I 

joe: think we're also lumping in other things [00:31:00] that are not, generative AI as well. I think we've got to remember that there's other machine learning.

What are you talking about, man? Yeah, true. It all went away. It was all like,but it was fun. It's funny because now things like, time series forecasting and computer vision, everything's thrown under the AI label, whereas that was ML back in the days. I think when we say AI, we need to be very careful about what we're describing here.

Is it large language models? Is it image generation? Is it, Something else. like I was just talking to somebody back there, the support vector machine, right? On a, on an edge device. That's called 

juan: old AI. 

joe: Old AI, right? But all this stuff is still being done. Good old 

juan: fashioned AI. that would go back to like symbols and 

joe: But this is a discussion we were having with Paco Nathan on my podcast a couple of weeks ago where, it's do you really need to throw large language models at every single problem, or are there simpler solutions to use?

Like a small, large language model? it's yeah, I mean it's, that kind of hits the point though. It reminds me of back when deep learning was, this is 10 plus years ago, back when that was at the crescendo, and it's this is obviously, you don't need to do,just, you throw deep learning at everything, you don't need to process your features anymore, but then with [00:32:00] tabular data, you realize XGBoost still beats it.

Time and time again, right? And this is that's a, you can run XGBoost on a CPU. Even if you don't even need anything that fancy. So I think we're gonna come back to that realization that given the enormous cost,of these models and training and inference and what not, is there a simpler way to do,to get something that's probably just as good, there was some, there's a paper, can't remember the name of it, but it was basically talking about the scale problem of AI.

It came out last month, I think, but it talked about basically for,detecting epilepsy, like logistic regression, which is as good as, deep learning. In fact, I think it was really better in some ways, and you can probably run that on an Excel sheet. Yeah. 

juan: we just love doing this, right?

Everything's, I gotta hammer everything to a nail. I gotta go test it out. Yeah. 

jepson: It's frustrating that people keep making the same mistakes. So even with these new technologies coming up, if you're doing a big, ambitious AI project, this was also an issue with traditional AI, I like imagining the future.

okay, we finished in six months. We are in the future. It was amazing. everyone's [00:33:00] excited. The board's going to have you go in person. come up with some ridiculous scenario, and if they can't explain how that is, if, often times you find they can't even explain how they would map to value, that is a huge concern why you started the project.

if you can't even explain to me in six months. Why this is a win. 

juan: This is why I, this is like the magic wand exercise. yeah, it's done right? . And this is why I like the whole wizard, the Amazon that does the whole, write the press release. yeah, sit down and write the press release of this is the amazing thing that we did.

And this is the quotes by the people who did this. Just make this up. If you cannot write that, 

jepson: what 

juan: the fuck are you doing? 

jepson: . 

hugo: Sorry, I, no, not at all. I love we've got 1, 1, 1 more question up here and then we'll need to wrap up. Yeah. It's a, 

audience: it's a really simple question. I started to use, we'll decide whether it's just chat GPT for, just small things like snippets of code and like you mentioned earlier, if it's a technology that I've not coded in before, it's just, a good way to quickly start learning and practicing some stuff maybe.

but I ran into a really weird issue the other day. I was just trying to generate a holiday card. With some [00:34:00] text on it, and some, random images that I suggested that it generate. and for some reason it could not spell the word profitable, right? I was trying to say, may all your trades be profitable, okay?

And it would always spell That's a beautiful message, by the way. Right.it was a Diwali card for a bunch of Awesome. Group of stock traders. I was trying to mix Mixed elements of the Indian Diwali tradition with stock trading and charts and bulls and bears and stuff like that. And it did a pretty good card, but it said profitable, it would always spell it as profitable.

And I tried, I don't know, 10 times to tell it that the spelling was wrong, that you're spelling profitable as profitable. And it would say, sorry, let me try to regenerate that. And it would still spell it as profitable. what is so difficult about just. Having a digital or graphical representation of text that an LLM can't get right.

because we're 

hugo: doing something probabilistic in nature, I think it comes down to understanding what the [00:35:00] systems are. I'd be interested in all of your thoughts on this. But the other thing is, they do, several things happen. They tend to get stuck in local minima relatively easily and I would always encourage people to start new conversations with large language models when that happens.

The other thing is they're the world's best gaslighters as well. They totally quote unquote believe what they're doing and they're like, Oh no, this is what I meant. And they'll run you around in people pleasing gaslighting circles non stop. They're the perfect technology for the social media generation in that respect as well.

I can't count super well. I can't add super like we expect computers to be calculators or some odd computers in that sense, but because of the probabilistic nature of these technologies, they're actually very different to what we expect from computers more generally, right? 

joe: I think what you're hitting on is exactly why I'd be fearful of having these things run on their own in production rampantly, especially when they're.

I'm talking about agentic workflows, because They're, they, you wouldn't know unless you looked at [00:36:00] it.And then you keep trying to hit refresh, and it's no, you're the one that's 

juan: wrong, actually. Or now you learn, guess what, I'll just ask it to create an image with no words, and I'll add the words afterwards.

again, We're learning, right? And then people try to go work on it, but these things are non deterministic, they're probabilistic, that's why it doesn't work, it doesn't do that. And, again.we just need to use it for what it's actually for. we understand what the problem is and is this the tool for it?

sometimes you can just write the code yourself for some things, right? And it would be easier to go do that. 

intro: Thank you very much. Thank you to our panelists. If you have more questions, please find them during lunch afterwards. And 

hugo: I will just add, Jepson, you're giving a keynote tomorrow.

Everyone should check out Jepson's keynote. Do you have anything you'd like to promote? Joe or, 

juan: not here. Okay. . Oh, I have a podcast too. So,cataloging Cocktails. An honest no BS non-Salesy Data podcast. Oh, you mean Oh yeah. Oh, promote, sorry, I, 

hugo: yeah. Or promote your surfing YouTube channel. I dunno. 

joe: I, have a new course that just came out on [00:37:00] Coursera for data engineering.

So if you have anybody that's interested in data engineering, it's a 16 week specialization, in, partnership with deep learning AI and, AWS. yep. And I've got a new book coming out on data modeling, Q1 of next year. So this guy. 

hugo: Don't think I'd make you a good data engineer, but.

So I'll also, I'll be teaching a workshop this afternoon as well on escaping proof of concept purgatory. also, escape room. it's an LLM inspired VR escape room. I'd like to thank all our panelists once again for bringing back wisdom from the edge of generative AI experience. And thank you all for the great questions and for chilling and learning.

Thank you. 

jepson: Thanks everyone.