00:00.52 James Welcome back, everyone, to Merge Conflict, your weekly developer podcast about all things in software development and AI and all those goodies. I'm one of your hosts, James Montemagno. With whist me, I'm a little sick, Frank. 00:12.37 James With me, as always, Mr. 00:12.66 Frank ah 00:13.77 James Frank Krueger. How's it going, buddy? 00:15.49 Frank Oh, it's going wonderful, especially now that I know that you're sick. I'm doing. wonderful. I'm healthy. The room is 120 degrees because my neural network keeps training. I had a nice fire going this morning. 00:27.76 Frank um spoilers the Seahawks lost so it's a little depressing but aside from that life is good over here I have no decisions to make I'm i'm just running on autopilot everything's easy 00:41.01 James Well, I'm making a few decisions. One, I am sick. Everything is just like happening. It's like coming. And it's still going and that's not great. 00:46.03 Frank ah 00:48.07 James But I can at least talk and like I feel okay. I feel okay. It's just it's like ah up in there. um Now we are. We already recorded next week's episode. So if people are like, Oh, like, you know, some big events happened last week. 01:01.80 James Next week, we'll be having our big Microsoft hardware reveal podcast. So stay tuned for that. um So we have a whole bunch of stuff to say on that and also the Windows announcements and so much more. 01:13.65 James So we really wanted to to make sure that you're liked, subscribed, and done all this stuff to be here. And I'm currently having a decision. to where... to sell my Mac mini Frank Kruger. 01:23.52 Frank Oh. 01:23.79 James So that is. 01:24.36 Frank that's a tough one. 01:25.48 James Uh. 01:25.77 Frank What are your choices? 01:26.21 James You know. 01:27.08 Frank where Where do people sell computers now? Do you just like put a lemonade stand up in front of your house? I mean, you're in a pretty affluent town. You could probably just put a sign up. end. Within a day, sell it, I assume. 01:38.64 James Okay, so... I've sold stuff on I usually almost always sell stuff on Craigslist. That's like my go to but now no one's on Craigslist and everyone's on Facebook marketplace. 01:43.50 Frank Hmm. 01:45.56 Frank We're old Yeah. 01:47.05 James Right? It's like Facebook marketplace is like the go to but it's like all this junk on there. um However, I mostly just go. 01:52.82 Frank Hey. Some of us like drunk James. 01:54.08 James to like eBay. yeah that's true yeah i'd usually just go to ebay so i figure it's like It's... It's Facebook Marketplace Local. Or. 02:04.64 James ah or eBay. Now eBay has a 13 % fee. So it's like, Half D. 02:10.01 Frank Yeah. 02:10.04 James you know. However, 02:10.84 Frank Rough. 02:12.12 James Like I've sold stuff on Facebook marketplace. And usually these are items that are. and you know 50 bucks 100 bucks whatever not like a big deal this would be like a $2 ,500 to $3 ,000 purchase. 02:22.18 Frank E. 02:23.16 James So the question is, do you... And here's a little Mac mini that's like $3 ,000. You know what I mean? 02:28.82 Frank Yeah. 02:28.98 James Like and meet up with somebody at the local movie theater or gas station. You know, it feels a little. 02:33.67 Frank Do you print them a receipt, an invoice? Do you have a tax form you fill out, a warranty? Like, yeah, what services are you going to provide? 02:43.70 James Yeah. 02:43.85 Frank What if they say it's broken and you're like, no, it wasn't broken. now It's broken. it's a tough it's always scary selling something big but i mean it is roughly the price of a 02:51.82 James Yeah. 02:54.40 Frank car you're basically selling a car at this point 02:59.61 James Yeah, the smart car was $4 ,500. So I mean, Just saying it's not that much more than that. 03:03.75 Frank You're in the ballpark. 03:04.34 James ah Yeah, this thing has more RAM in it, that's for sure. But ah yeah, lots lots of decisions to be had. 03:07.63 Frank ah 03:09.98 James But what we wanted to do... uh this week is break down kind of almost a follow up. of a podcast we did recently, which was about a bunch of little tiny models. Like we talked a lot about Luna and things like that. 03:23.17 James However, Um, One of the models we talked about was Jev and Jev had it's like moment, you know, every model seems to have like a week. of of its moment and and jeff specifically is a 03:32.96 Frank Yeah. 03:34.74 James you I think you were like, this is just a decision. you know, a model. it's It's making decisions in and out. Why don't you give it a Jev? ah tldr and then jev update before we get into like jev lookalikes. 03:47.11 Frank Okay. Sure. um So Jev is still a large language model because it takes in natural language. So the idea is you can still fill it with a context, give it a bunch of source code or give it all your company's data or give it the collective works of Shakespeare or something. And then you can ask it. 04:06.01 Frank basically multiple choice questions. And it doesn't even have to be multiple. It could be just yes, no questions and things like that. So the nice thing about it, though, is it's tuned for that scenario. 04:19.06 Frank So it's tuned to just give you... you know Given options A, B, C, and D, choose which one of those options to output. The nice thing is there's no like JSON parsing of the output or waiting for the recursion to happen to generate a long response. Basically, it's. 04:38.78 Frank fast i mean that's one of the nice things about it and then it gives you an answer to your question And it also gives you a probability with it because that happens naturally with all these networks anyway. So they're just throwing that in. And then the nice thing is because... um Its output is so simple, you can actually ask it multiple questions all at once. And if your context is much bigger than your questions, then those questions become very cheap to have it answer. Uh... 05:06.73 Frank it's It's cool. It kind of took over the world because it's just another way to use these advanced LLMs that we've really... um Fine-tuned. 05:16.79 Frank Ooh, that's, can't use those words, but we've, we've refined. 05:18.67 James Specialize, specialize, specialize. refined. 05:21.06 Frank Yes. We've refined how LLMs work for the past few years. and it this is just a nice new mode a nice new way to use llms um To just. answer questions and make decisions and basically do it for cheap. 05:39.03 Frank Because you know what? All the other models are getting more and more expensive. So it was kind of nice to be like, hey, here's a new model and it's cheap. And you're like, oh, that's nice. I do like cheap. It's been a while since I've heard the word cheap. But people like it because it's fast. It's cheap. And it's it's one of those things where like... constraints breed creativity you're like well all i can do is choose from a multiple choice output so i wonder what questions i can pose to it that's basically a multiple choice output 06:04.36 Frank If we want to be reductive, we can say it's like the old classifier networks that used to be out there. Hot or not, hot dog or not, hums um ah sentiment analysis. It's like one of those, but a more general purpose and a more modern. version one of those. that can do a lot of things. 06:19.66 Frank Thanks. 06:20.49 James Yeah, and I've seen a lot of people do a whole bunch of crazy things like you know Drive. The games make like. Yo. different forecasting like monitor like decisions like when to purchase tickets or whatever because it's all decision based right so it's like here's decisions based on this historical historical part like when do we do these things xyz instead of these more general models or models that are fine-tuned for 06:33.86 Frank Mm hmm. 06:42.14 James uh coding for example that are out there So. Have you yet to find a use case in your daily? workflow to use a Jev type model. 06:53.81 Frank Absolutely not. 06:54.98 James Okay. 06:55.95 Frank I want to like every day I wake up, I'm like, could I use Joe for this? I'm like, I'm too lazy. I'm just going to pay $5 ,000 and ask GPT-7, Astra, whatever the heck, you know, I'm just going to do that. 07:11.73 Frank Um, That doesn't mean I'm not still intrigued by them, but... For a person like me that's been in the ML world for ah a while now. These classifiers are not new. things I know it's new because... People have only been introduced to AI through LLMs and natural language input and output. So this is kind of new to them. But to me, this is... 07:33.65 Frank Kind of old tech. it's an old It's an old thing with a new spin. Which is cool. Totally get it. But um I never had a use for that old tech, so of course I'm not going to have a use for this new tech. 07:45.42 Frank But that's not to say that, like, I'm not intrigued by things. like um You said you've seen it do some things like a fun one that I see it do a lot is do like video game input. And if you think about. 07:57.80 Frank Video games, what are you doing? Well, you're pressing, if you're ah if you're a keyboard, one of those disgusting keyboard game players out there, you're you're pressing your WASD buttons, your W-A-S-D, and... 08:09.58 Frank Think of that as a choice. Do I press W? Do I press A as D? Or do I press the space bar to shoot or something like that? um So there's actually like kind of fun games and. um this And back. 08:22.04 Frank It's almost AI. Look at that. But like NPC kind of AI. And I think about it for robotics and things like that. But there's problems with inputting data to them still. And there's still this latency of you're still having to go out to this. 08:36.40 Frank foreign network over the internet and you're going to have your Starlink latency and you're going to have the network providers latency and all that. So it's prevented me from doing anything serious with like robotics or anything, but I still think about it a lot because it is a creative use of a modern tech. that we think about a lot so to answer a question nope have no use for them But I'm still intrigued. 08:59.89 James I would say, while you maybe have not found a use case. And you have been intrigued at least. Many other people have been. extremely intrigued and It seems like people are getting to work because there's not just like one decision model. Now there's many decision models and it's only been like two or three weeks since Jeff was announced. So what I really wanted to do today was Talk about some of the... things that have come out because I actually didn't know like you sent me a note and I was like, I've heard of the open AI decisions, but I haven't heard a lot of these other ones. 09:29.76 James But the big... sort of news in the last week was microsoft decision one that came out that really took at least my twitter maybe there's some biases since i work at microsoft but like there's definitely some 09:38.54 Frank then 09:40.76 James Um, Really? um storm, I would say, of tweets that came out. and Uh, all relatively extremely positive for being even cheaper and even faster and even higher quality on it. 09:52.47 Frank Yeah. 09:54.25 James So I figured we would dive a little bit deeper. into... this new world of decision models that we have access to right when we do find that use case what it means today and what it means going forward. So Frank, what has actually happened? 10:09.36 James in the last three weeks because you have a list you have a bucket list and a whole bunch whole bunch of like. 10:13.77 Frank ah 10:14.77 James things here? What is happening? 10:17.09 Frank Well, basically, everyone who knows how to train a model has released a new Jev clone. That's what's happened, James. Because it is, it's intriguing. ML people find it intriguing. And we're all like, well, I can do that. Now, it's one of those things that's like obvious in retrospect. 10:32.41 Frank But it takes a smart person to like push it through and be like, hey, here's something cool. And then we're all like, yeah, that is cool. And I can do that too. Because now that you've demonstrated it, now I can be like, oh, I can do that too. And so we have a lot of the, oh, I can do that too's. 10:46.91 Frank come out. So I think actually um OpenAI i beat Microsoft to the punch. OpenAI released a Decisions. model. And an API, which is kind of nice because we haven't had a new API for these things for LLMs in a while. 11:03.19 Frank There's been the chat completions API that we all use. And so when and the joke is the entire industry just copied OpenAI's API. um And now we're all copying OpenAI's decisions API. 11:16.81 James Oh, geez. 11:17.09 Frank Yeah, exactly. So OpenAI i released a model. We're not really sure what it's based on, but it's it's a little bit expensive. it's 11:25.57 Frank it's It's open AI. You know what? It's going to be good. It's going to be expensive. It's going to be a tiny bit slow. That's open AI. So the cool news, the one that you mentioned is Microsoft. released one also. And I really appreciate Microsoft, because mostly because they wrote a good blog entry talking about what they wanted to achieve, the pros and cons they saw in the world. and Um, 11:48.60 Frank and how theirs compares to everyone else's because again Everyone's releasing these puppies now. And. As an LL person, it was nice. It was a proper breakdown. not even sure It's definitely not a white paper. It's not even really a technical report. 12:05.39 Frank because they didn't disclose a lot of information. But there's enough nuggets in it that we can see. How they were trying to compete. And I feel a little bit bad. I want to say this. 12:15.85 Frank first place like i feel a little bit bad for the jev people because it's terrible to be like hey we worked on this thing for two years isn't it cool and everyone's like yeah that's super cool and then everyone comes and copies it it's just a little disheartening it just hurts a little bit especially when microsoft comes 12:26.84 James Yeah. Yeah. Oh, I mean... I mean, I think to that point, though, like that's almost the world we. live in to some extent, not that like there's, there's copying, but it's like, Hey, listen, like this is a viable use case. 12:42.28 James And we are now able to. 12:43.30 Frank Yeah. 12:44.60 James to sprint, like we're sprinting. right out out of that at this pace 12:47.48 Frank Yeah. 12:49.52 James And. You kind of see this sometimes when. you know And there's all these articles. It's like, oh, and there's been this software that's existed for 20 years. And then, oh, someone just like. open source the thing or someone created some really cool thing i're like, well, I can just like clone it, right? 12:59.30 Frank Yeah. 13:02.02 James The question ends up coming down to 13:02.43 Frank Right. 13:04.39 James Okay, anyone can... at this point with enough tokens can clone a thing, can do a thing. like You can make it happen. 13:09.89 Frank Yeah. 13:10.41 James But what is the actual... like business behind it? What is the quality behind it? What is the long-term viability of... that piece of software or that product that's out there, like whether you're buying a car, we talked about this with the slate truck, for example, it's brand new, it's a brand new truck. 13:26.58 Frank Mm hmm. 13:27.21 James What is the long term? a viability of that company. 13:30.02 Frank Yeah. 13:31.50 James And of that, I was just talking to my... a friend who lives a few blocks away and they got solar. on their on their house and we just got solar on on our house and Um, 13:42.23 James The... company here has been around for a long time, which doesn't mean anything actually, but at least all the equipment is all end phase, which has been around forever again, which doesn't mean anything. 13:52.21 James And I was talking to them, I was like, Oh, what's your setup? And they're like, you know, I don't really know. But it's, it was this company. 13:56.19 Frank Eh. 13:56.81 James And then you know, they were here for a bit, and then they got bought out. And it's from some other company and something and you're like, like, so I don't even know if something goes wrong, what happens, right? 14:00.33 Frank Hmm. 14:05.61 James So it is, it it is the same thing when you go and buy a product in the store, right? you You know, when you're going to no shade on the dollar tree, when I go to the dollar tree, which I go to quite often, I know 14:11.24 Frank Yeah. 14:17.22 James what I'm getting myself into. And and there's there's a variety of quality metrics at the Dollar Tree or Dollar General, wherever you're at, right? When I'm buying something for $1 .50 compared to going to another store at $5, it doesn't actually mean that the $5 one is higher quality. 14:32.43 James What it means is like, I'm going to do a little bit more deeper research on this one that is cheaper and faster. And and fastrighted maybe it is just as good of quality as the $5 one I'm paying. retail markup because that one was first to market. 14:44.78 James Or for example, but that one has a brand name. If I buy a white t shirt, that's a Hanes versus one that's ah a Nike versus one that's a 14:52.50 Frank Hm. 14:52.99 James a Yeezy shirt, right? These are all different points price points. But you know what it is, at the end of the day, it's a white t shirt. Now, there's a lot of different elements inside there. So I think that this is why it's really interesting, because We're going to see more and more of this. 15:03.57 Frank Yeah. 15:06.55 James just like we've seen. the models. but have a huge variance of of low, medium, and and high, and extreme. You know you get your your haikus, your your sonnets, your your opus, your fables. You get your your your lunas. You get your your souls. You get your astras, for example, like you're talking about. 15:26.100 James and And now we have this. you know, not just here's the major LLM. So now we have a lot of different companies that are putting out things and now we have these new specialized models that are also coming through the same vein too so we we get this long one just for example of like hey we talked about on the the local llm one and and and what you're running and also oh can i just use a quen model or or uh 15:50.97 James um maybe an nvidia model it's like running on open router or whatever a deep seek model right and i could be getting the same exact quality you know, but at a different cost. Or it could be. 16:01.61 James but thinking I'm getting the same quality at a cheaper cost. You know what I mean? that is really It's really, we we have a lot of these different... um ah decision branches in our head that we can now evaluate deeper, if we will. 16:13.87 Frank Yeah, absolutely i 16:14.02 James So anyways, I just think I think it's interesting, because with these models, you can also apply it to things in real life, right? Because yeah A good example is this. and Think of... um If you're replacing the floors in your house, right? 16:26.22 James you you know If you're buying carpet, there's many carpets. But if you're buying ah floors, like hardwood floors, you have you have you know have like vinyl, luxury vinyl, manufactured hardwood, actual hardwood. I'm sure there's something ove and beyond that, right? you know There's all these these different tiers of it. And it doesn't mean that one's better than the other. It depends on your use case and your scenario, right? If you're a family, you got a bunch of dogs, maybe that LVP is the best for you in this point in your life compared to something where you're in a high humidity area where maybe that... 16:55.51 James that manufactured hardwood maybe isn't exactly what you need. I'm just saying, so. Anyway, let me play grass. 17:00.02 Frank You know, i'm I'm just saying large slab of marble. I mean, you really can't beat that. 17:05.48 James One. Just one huge one huge slab of marto marble. 17:06.44 Frank Okay, just big slab. 17:10.25 James That's it. That's all you need. 17:11.66 Frank Yeah. 17:12.12 James That's all Frank Krueger needs for his floor. Perfect. 17:15.99 Frank um yeah so what you're describing is called commoditization it's just surprising how fast it happened with this 17:16.21 James Anyways. 17:22.29 James Yeah. Yeah. 17:23.33 Frank um And it it always reminds me of that Steve Jobs quote. I think he said it about Dropbox. He says, this isn't a company, this is a feature. And I think, um sadly for the... 17:32.58 James Dude, that's... 17:33.03 Frank je It's hard. 17:34.01 James that ah That is a... 17:34.15 Frank it's It's just a knife right to the... 17:35.04 James ah Ha! 17:37.26 Frank i 17:37.40 James Ugh. ha 17:38.98 Frank Well, OpenAI and Microsoft just proved that. They're like, hey, um we have this thing called Azure Foundation. And guess what's on Azure Foundation now? Here's your own decision model. Look at that. Look how quickly we spun it up. But I want to talk about why they were able to quickly spin it up also. 17:55.72 Frank um A few quick things about it, though. um They definitely focused on speed. So it's interesting to see what people start competing on when it becomes commoditized. Like, OK, what are we going to discuss here? And I think this is. 18:09.74 Frank I like what Microsoft did because they basically established, here are the posts we're going to use to measure these things. And they released a bunch of bench um benchmarks. And we've had... 18:21.79 Frank We have terrible benchmarks for LLMs right now, but we have a lot of them. so maybe with a lot of them that makes you know um quantity over quality so there's there's something to there. But um Microsoft ah rated their model against 36 different benchmarks. Some public. 18:40.37 Frank some private. I hate private benchmarks because there's no way to prove what it did. 18:44.76 James Hmm. 18:45.30 Frank so microsoft don't mention your stupid private benchmarks like throw those out like it's public or GTFO, honestly. um and anyone can make up a private benchmark and say that you win at it you know come on Come on, Microsoft. 19:00.38 Frank But um to their credit, they they competed with a bunch of public ones. And there are some standard public ones. Like there's one named... Jev Bench. 19:09.13 James Of course. 19:09.60 Frank Um, Ironically, Jeff is not on Jeff Bench. 19:15.06 James Why? 19:14.73 Frank um for one very good reason um jev is still kind of a a beta kind of thing like i have access to it But part of the terms of service is... You can't benchmark us. 19:27.15 James Oh. 19:28.28 Frank Yeah, so Microsoft's in a tight spot. They can't actually compare themselves against Jeff. um So if you want to compare Microsoft's decision thing or OpenAI's decision thing against Jev, you're just going to have to do it on your own and you're going to have to not release that data. 19:42.41 James Yeah. 19:43.31 Frank So. But what? There is, thankfully, as a million other people have released their own decision models. So we do actually have a nice baseline for what's good and what's bad on Jev Bench. And there's a few that have trickled to the top. already. 19:57.47 Frank but um so there is the quality there that's fun But then there's speed. Because speed's a big deal. Speed is a part of this because some of the earliest demos of Jev were like, hey, it can control a robot. 20:10.50 Frank Hey, it can play a video game. So this isn't like your, I tell Astra to do something or Opus to do something. And now I go get a cup of coffee for a half hour and watch it think in circles. 20:22.71 Frank This is, I ask it something and bam, I get a response back. And that's kind of the delightful part too. You're like, oh yeah, computers can be fast if they're programmed to be fast. 20:31.55 James Yeah. True. 20:32.08 Frank it's a little refreshing and sometimes another steve jobs quote it's like giving a glass of water to someone stuck in hell um so it's um I got Steve Jobs on my mind for some reason. 20:43.11 James I see that. 20:43.66 Frank Anyway, yeah. 20:43.77 James I see that. 20:46.15 Frank so ah So it's fast. If you look at their charts, it's at the top. It's fast. 125 milliseconds to get a response. You can get eight responses per second. That's almost fast enough to play Unreal Tournament. 20:57.97 Frank No, but you know, we're getting there. 20:58.19 James wow Well, 60 frames per second, that means you're processing, you know... Oh, I guess it would be. 21:09.44 James Yeah, like 12. 21:09.58 Frank eight times a second it's actually pretty fast like um 21:10.20 James Yeah, it takes a second. Yeah. It's pretty good 21:14.83 Frank Yeah, like I just... 21:15.00 James That's faster than you can press the keystroke. That's faster than you can press. 21:18.90 Frank Yeah. 21:19.21 James Make a decision to press something you're shooting I wonder what the cognitive... 21:21.97 Frank It could play Wolfenstein. 21:24.53 James gonna play doom yeah i bet you know i saw it i saw it playing 21:26.22 Frank Oh yeah, sure. I think that was one of the demos 21:28.97 James I saw it playing space invaders and I saw it playing Tetris obviously is a canonical. 21:33.39 Frank Let's talk about that. 21:34.86 James Okay. 21:35.18 Frank Okay. let's Let's talk about it. Because these decision models, so you do give them context. And context is usually natural language, but we've advanced in the LLM world where you can actually give. 21:47.77 Frank images to llms now like here's a screenshot of my app tell me how to improve my app blah blah blah Um. 21:50.02 James Hmm. Yeah. 21:54.26 Frank So the... Jev can take in images OpenAI decisions can take in images. Microsoft One cannot. 22:03.29 James Hmm. 22:03.47 Frank So that's one little. Post. there um so like they have great numbers and good benchmark scores but it is an important distinction like it can't do video so then you're like Okay, but one of their benchmarks that they say they did awesome on. was playing Space Invaders. 22:23.32 Frank So what's going on, James? 22:23.42 James o 22:24.62 Frank What do you think they're doing? 22:26.09 James Well, you would just take the well, you know. You would take the grid. like every snapshot in time. Every frame. has you know, some sort of backing data to it. So you pass that data into the decision tree and represent the invaders, the ship. the asteroids that you're behind, or the bases you're behind, whatever it is. 22:48.09 James And you would also then send in all of the data such as its shots in your shots. So you should be able to near in real time, be able to pass in all of that data back and forth to make decisions on what to move left, right, shoot. on i would assume given the context that you're there. In fact, 23:05.77 James i have to imagine that's faster than sending it images in which it has to then parse through because the confines. So for example, the confines. of Space Invaders are... 23:18.48 James Like. deter most what they It's deterministic. 23:23.07 Frank Mm hmm. 23:24.04 James It. And even if, let's say you were playing. Um... Another game. such as Fortnite. And the... randomness, which is a human control here is that frame snapshot. 23:38.02 James the snapshot of the data That comes through, obviously, would be much larger. But that is still a defined snapshot in time. of the the state of the world. 23:48.53 James because the game loop itself. you know The game loop is processing that data every frame. So you would just be passing that data off to, I guess, MAI or Microsoft Decision One. in this case. 23:59.05 Frank Yeah. 23:59.98 James There's that. 24:00.44 Frank Yeah. 24:00.25 James Is that what they're doing, I assume? 24:02.55 Frank Pretty much. The biggest problem with this benchmark is it's a private benchmark and it's not released to the public. So we all have to kind of guess what they did. 24:09.40 James Hmm. 24:09.81 Frank So your guess is honestly as good as anyone else's guess. 24:10.28 James and 24:13.06 Frank But the fact is, you do have to take this graphical representation and represent it as text in some form or another. 24:20.35 James Hmm. 24:21.09 Frank The dumbest possible worst way to do that would be actually do like an ASCII conversion of the graphical screen. You know how people do like image viewers in the terminal? Don't do that. They're not doing that. They better not be doing that because that's dumb. These models are not trained on that. Models are trained on. 24:38.22 Frank natural language number one and 24:40.30 Frank Jason. Honestly. JSON is our universal data representation. So most likely what they're saying is here's a list of aliens. Here are maybe their spatial coordinates, X, Y position. Here's the player position. Here are the bullets moving. I forget all the parameters, but yeah, you got to capture your state space. So if I wanted to do a robot, I would have to capture my state space of the robot into some kind of structured data, most likely JSON and not just natural language. And so everyone's assuming that's vaguely how they're doing. it unfortunately that means it can't do the simple things of like Play an Atari game. 25:15.87 Frank um Whereas the other ones... can because they do accept vision input. And I do want to push back a tiny bit on what you said about. um text is guaranteed to be smaller than an image it's not actually guaranteed because um 25:29.85 James It's true. 25:32.27 Frank ah These numbers vary all over the place, but normally these networks will take an image, break it into like 32 pixel by 32 pixel cells. Each one of those 32... 25:43.15 Frank by 32 pixel cells becomes a single token, so your resolution of your image. will tell you how many tokens you have. Yes. So a high res image, if you're capturing every little bit of it, yes, that absolutely can take a bunch of tokens. but for like an atari screen it actually is not actually too many tokens most likely um But that's all guesswork. I just wanted to bring out the point that... um Our large language models. 26:08.43 Frank Language models. actually do more than just language. and They actually do vision also. 26:11.78 James Yeah. yeah 26:13.100 Frank And honestly, it's a little bit frustrating. I'm jumping a little to the end here, but um Microsoft was able to release this thing. Because all it really is, is a fine tune. of an open source model. 26:25.38 Frank It's a fine tune of the beautiful, lovely Quen model that I love because it runs on my cute little RTX 3090. I think it should get a bell like every time I mention it on the podcast because I'm in love with it. 26:35.66 James 59 ding. And Andy Dank. Thanks. 26:39.14 Frank And it's actually a small Quen. It's actually the 9 billion parameter, Gwen. So you can easily run the model. 26:43.46 James Nice. 26:45.66 Frank you could run Microsoft Decision One. on your own machine easily if they would release it. 26:49.06 James Yeah. Easily. 26:51.11 Frank publicly but they haven't it's just private But there are plenty of these decision models on Hugging Face if you want to go looking around and all that. 26:52.24 James Yep. 27:00.98 Frank And maybe we'll include some links to some of them on here. So I just want to put it out there that one of the reasons they're getting the speed is because they' it's just a 9 billion parameter model. and they threw out the vision side of it. So they're just doing the tech side of it, and it's a 9 billion parameter model. So. 27:18.19 Frank Microsoft making good engineering decisions, also benefiting from open weights, Microsoft. So maybe you should be contributing more open weights to the world also. You know, if you take something, you should give something back. Just saying. Hashtag just saying. 27:32.81 Frank um So ah the reason it's fast is because it's it's a smaller model. It just does. But one thing I really appreciate in their blog post. 27:43.42 Frank was how they spent a lot of time ensuring robustness. And this is something that we, I haven't seen too many other people test. And this is something that. um I think about a lot with these models of like, if you just change a small amount of wording in something. how much is that actually affecting the output? 28:03.48 Frank So this is actually an engineering definition in control systems. what we So this is a different field. This is my field in control systems. But we have a pretty strict definition of robustness, and it's very simple. It's small changes in input. should produce small changes in output. 28:21.03 Frank It just means it's not a twitchy thing. You know, you you change a little phrase here or there, you're still going to get roughly the same answer out of the machine. and microsoft basically turn that into a benchmark or whatever. They use that as a guiding principle for how to train this thing. 28:37.38 Frank so they've tested like Doesn't matter what order you give. um I said they're all multiple choice questions, so does it doesn't matter what order you give the answers to it. Does it matter if you rephrase the input a little bit? 28:51.43 Frank And to their credit, their model is not the best. at this. Other models outperform them. the models that outperform them are much bigger and much slower. 29:02.22 Frank So you would kind of assume that. 29:03.24 James and 29:04.11 Frank But it's it's nice to see um proper engineering benchmarks because I feel like a lot of the benchmarks we see in the world are... horrendous? And have absolutely no engineering foundation to them. This is the first time I've seen someone actually have like a serious benchmark. And it's it's robustness. This is a thing I care about. quite a bit about because i've trained plenty of ml models where as long as you don't mess with their inputs too much they produce great outputs but the moment you give them 29:33.43 Frank a variation on the input, or something it hasn't seen before, it it just acts like it. Idiot. And so to their credit. um They really pushed on that. And also in robustness is a variety of topics. Like, is it, did they train it to predict the stock market? Probably not. But you know, that's, that's one universe of things that they. 29:54.77 Frank train it to make good life decisions. You know, should I break up with Sally? You know, that's that's kind of an important question. Do we have a benchmark for that? 30:01.45 James yeah 30:03.02 Frank And so, you know, just be, yeah, exactly. 30:03.66 James ah Sally Benchmark. 30:06.84 Frank Just because it's a decision model doesn't mean it's a simple problem. In fact, it's an incredibly hard problem. If you're just saying like, there is a universe of things you can make decisions about and the training data is going to be. 30:19.48 Frank honestly very small. It's hard to get a good random sampling of every possible decision humans make. ever. That is a very large universe of open-ended questions. 30:28.30 James you know 30:32.63 Frank um So it was fun to see them tackle that and acknowledge that. um In some ways, it's easier than LLMs because you're not asking it to you know generate a book or anything. But in the same thing. It it really. 30:46.41 Frank shows you how hard LLMs have it too, because they are generating a book in this universe of possibilities. And if you just say, well, my universe is the universe of possibilities, plus this multiple choice thing, you're like, well. the input is still the universe it's still big and still difficult so it's fun it's just a good blog entry by the way also just well written so I hope everyone goes and reads it 31:06.31 James We'll put a link in the show notes, of course. And yeah, I think what's, what's nice is like, when you're going through, you'll see like, obviously, Jeff, quiet, rune, H2O lightning, obviously, Luna decision, which we talked about deck, there's all these different ones that are in here. So you get to learn about all of the other sort of decision. models that are out there. 31:23.11 James that are in these top of the the jev bench that are there too And I would also point out. Okay, I'm not sure about Luna exactly. if it's fine tuned off a Luna, I assume baby or something. 31:30.79 Frank Hmm. 31:33.84 James But you learn in the and the thing I was gonna ask, but then you you have gone ahead and solved it for me, which is how did these things come about so fast. 31:39.72 Frank Oh, sorry. 31:42.10 James And what you end up learning as you go through is that many of them are fine tuned off of other models. that are out there. and you know what's the important part here to to your part obviously i agree with you on on um you know, giving and taking, but also ah Transparency is also really important too. And many people ah called out. right after the post went out. 32:06.01 James um that it was, you know, fine-tuned on Quinn. um 3 .59 bit b and well that was right in the blog post it was literally like right in the beginning it's like here and here's what it is and here's how we got here right Yeah, and you see quiet is on Gemma for 31 billion parameter fine tune model and all these other ones that are here too. 32:13.08 Frank Yeah. Yep. Full disclosure. Yep. 32:21.67 James So it's not to say that that is the only way to get there. But that that does means that you're able to you know, fine tune on top of these models that are out there to do very specific things. And that is the the world. you know, that that we're we're we're leading into. to to to create some of these things faster than ever. um So it'd be really interesting to see how this evolves. I do think that it's... I think the only thing. 32:44.45 James The only thing I mean, I actually the one part in the blog post that I think is funny. but also is a good. i'm glad they i'm glad that they included it even though it seems a little bit like shade which is. they have in a lot of these graphs that that the team has on Here it is also against GPT-6 Sol. 33:03.55 James And. The reason for that is not to like... throw a bunch of shade on GPT-6 Sol. 33:08.08 Frank Right. 33:08.48 James But it is to point out that GPT-6 Sol is not a decision model. So if you are specifically trying to use GPT-6 Sol for decision making, then you are dramatically going to be. Getting slower. 33:21.56 James Um, much more expensive results. coming back and and maybe not even at the accuracy that that you desire. 33:27.64 Frank Yeah. 33:27.61 James So something to think about there. But so I am glad that they did include it. I think it's funny because it is the one that is like the biggest bar. um And then we're just funny. 33:36.79 Frank i wish they put more big llms in there you know honestly because i want to know like 33:39.02 James Yeah. 33:41.74 James Hmm. 33:43.14 Frank am I out-competing the bigger models? like 33:45.66 James True. small models. 33:46.66 Frank And. And the reason it is outcompeting them is because all these big models are fine-tuned. to... so to do a certain kind of task. And that task is usually. 33:58.74 Frank Chat with a human. and run agent junk. These are the two fine tunes that we roughly do on all these models now. So the whole... the the thing everyone liked about Jeff is like, oh, what if we... Fine tune on something else. 34:13.75 Frank Because... These are fine tunes, and that's important because we don't want to train these things from scratch. They have to understand all of English, all of German, all of Arabic, all of Chinese. 34:25.09 Frank You know, it takes a lot of money and resources to get them to understand all of that. 34:25.98 James Yeah. 34:31.15 Frank The question is, okay, now that they understand all of that or whatever, to varying degrees, throw on your own. verb there um what can we do with the outputs how can we um make the outputs more efficient what are more interesting tasks we can do with these outputs and i think that's why the ml people are all so excited about this it's not just marketing hype we're like oh yeah We have these very sophisticated things. 34:54.56 Frank Let's fine tune them to do something other than people please in a chat interface. Because honestly, as much as we all love chat GPT, like the chat interface. 34:59.78 James Yeah. 35:05.02 Frank It's cute. It's cute. but It's going to go away. we We all know it's going to go away eventually. um Everything will have a little bit of a chat interface. And the joke is these models can do the chat thing. It's just as you fine tune them to be. question and answer things. 35:21.35 Frank they're just going to get worse at chat because we're moving them away from that fine tune. 35:25.87 James Yeah, makes sense. Well, it's very cool. It's cool to to get down into it. And if everyone else is fine tuning, these can run. You just create one, Frank. 35:34.71 Frank Exactly. Well, because my GPU is currently busy doing something else right now. Have I mentioned how hot it is in this room? 35:43.14 James It's good. 35:43.22 Frank I think I mentioned it. 35:43.35 James It's cold. I got a jacket on. 35:44.74 Frank But yeah, um especially because it's a 9 billion parameter model, like you can easily go to like Google Collab. You don't even need a GPU. And in probably just a day or two, fine tune one of these puppies. 35:58.20 Frank The biggest problem in ah ML is data. So if you have a big question and answer kind of database. And I mean big. You need a big one. You can also go download some examples like on Hugging Face. You can fine tune these things yourself, which is really cool because they're small enough. They can be local models. So you might have some trouble running like. 36:20.66 Frank when 27 billion parameters with your small amount of video RAM. But a 9 billion parameter model, oh, that's runnable. A lot of people can run that thing. And then you can run your own decision maker. locally. And that's when you can really start. um 36:36.64 Frank Don't cheat at video games, people. But that's when you can really start cheating at video games and doing that because it's all just kind of running locally. You're not paying for external servers and all that. Don't tell Azure Foundry I said that. I try to stay in their good graces. But the nice thing about small models being fine-tuned is that you can run them locally. 36:55.78 James What girls have? Foundry local, I'm just saying. So you can just pull down those models. 36:58.10 Frank Okay. Yep. There we go. 36:59.04 James So you're totally good to go. Now it's all there. I mean, will we'll hear quite a bit next week when we follow up on the Surface and Windows event, talking about local models. 37:01.24 Frank Yeah. 37:06.46 Frank some instance sensation Yeah. 37:06.78 James We'll really deep dive. into these new devices that have 128 gigs of RAM and an RTX Spark. So pretty excited about that too. but 37:15.53 Frank What? 37:15.31 James I do think that this is good almost lead up to it because. um Well, we we're not going to talk about the devices until next week. But one thing that is really important is sort of this. even more and more local models running and making decisions on your behalf so We're only going to get better from here, smaller, tighter, faster. um as these continue to go so i do think that this makes a lot of a lot of sense. And like you're saying, to be able to run these. 37:40.83 James locally, just... Just, and why not? 37:43.62 Frank Yeah. And the one thing I want to bring up, and this goes back to your question of have I personally found a use for these. And the answer again is no, but... Open Router has released a skill. a skill .md. 37:58.83 Frank ah By the way, OpenRouter is fully in on these things. So they they have the same API endpoint as OpenAI. So you can start. um if you have an open router account you can start hitting these you can even hit microsoft decision with it because that model on Azure. 38:14.64 Frank Open router will go and hit that puppy. So ah there's a skill file, a markdown file that you can import, and it tells your big, fat, slow LLM, hey, if you have a decision point to make, here's a way that you can access a decision model. 38:32.01 Frank that And here, fill it with this context, give it these multiple choice questions, and then let that thing burn on it. 38:32.04 James Nice. 38:39.54 Frank So another good way to just preserve your own tokens and your own credits and all that kind of stuff. So I plan on giving this skill a little bit of... ah a little run, see how it goes. and all that. 38:53.33 Frank So I just I'm excited for that skill because otherwise I didn't feel like building a whole harness. You know, there aren't a lot of harnesses built around this thing because we're all still playing with big LLM harnesses at the moment. 39:05.34 James I'll put all the links in the show notes to the decision one blog and to that open router scale too. And I'll link to a few other things in there. Uh, which is super cool. But Frank, thanks for breaking this down for me. I appreciate you did all the hard work this week and you have a nice note here of all the different, uh, different, different links too. So I really appreciate that. 39:22.22 Frank Yeah, I love giving you links because i I just assume that you're not going to open any of them, but I think you actually read some of them this time. 39:29.20 James I did. I did my best. I did my best. 39:30.63 Frank ah 39:30.78 James Well, that's going to do it for this week's Merge Conflict. Do like, subscribe, do all the things. Next week, we have a banger covering the Windows and Surface event. We're going to get deep into the Ultra and the Dev Box, so you do not want to miss. 39:43.81 James That one, but that's going to do it for this week's Merge Conflict. Decision made, Frank Krueger. Boom. One in the chapter. I'm James Montemagno. 39:50.56 Frank And I'm Frank Kruger. Thanks for watching and listening. 39:54.02 James Peace.