VG-ZM-final - 18:7:2025, 12.22 pm
===

zach: [00:00:00] Let's say you're running a complex pipeline where you're running half a dozen agents with various degrees of sizes, right? We'll do a standard swarm approach where we have a dozen agents that go and pull things from different websites, and then we have bigger models that go through and compress that information down, report it back to the user, and so forth.

Each of these still need to be served in some capacity. It's a slightly different distributed idea rather than one model. 'cause we're serving many, right? But there's a world where you're serving half a dozen, 8 billion parameter models that all needs VRA to be stored on. It turns it more from a training problem to an inference problem, and especially.

Knowing the best way to serve those models locally, but it's still scale. 

hugo: That's Zach Muller technical lead at Hugging Face, where he works on the Accelerate framework and handles core parts of the Transformer trainer API among many other things. If terms like serving half a dozen 8 billion parameter models sounds out of reach, this conversation is exactly for you.

Scaling used to feel like something only [00:01:00] open AI or DeepMind did. But today, if you are fine tuning open weight models, experimenting with agents, or just trying to avoid wasting compute, distributed training and inference are skills every ML builder needs. Even if you're coming from a CoLab background, this was recorded as a stream.

So if you hear us mention questions from the chat, that's what's going on. And full disclosure, I've been podcasting, consulting, and teaching so much that week. I was actually losing my voice. Thankfully, Zach carried the heavy load in the conversation. So in this session, we unpack what scaling actually looks like in 2025 for individual devs and small teams.

No Kubernetes required. We talk about how to know when you need scale, what info trade-offs actually matter, and how to go from single GPU training to running multiple models in production without drowning in system details. So if you've been stuck between local scripts and cloud clusters, or just wondering how big models get trained and served.

This episode is your on-Ramp. Zach's also teaching a full four [00:02:00] week course on distributed training this September. Called From Scratch to Scale and he's offering a discount kindly to vanish in gradients. Listeners links in the show notes. Zach will also be running a guest workshop in my building LLM application course where he'll go deeper on fine tuning language models.

We're currently mid cohort with a hundred wonderful people and we've just opened up a new cohort starting in November. There's a significant early bird discount running through mid-August, and as always, students get hands on support, guest lectures, and plenty of compute credits. One of the reasons the course has really taken off lately among others, is a post I wrote called Stop Building AI Agents, which actually went to number one on Hacker News.

It was great to see that this message resonates, and I'll link in the show notes if you're curious. I'm Hugo Bound Anderson, and this is Vanishing Gradients.[00:03:00] 

Zach, we're here to talk about scaling. Right. And a lot of our viewers and listeners, myself included, have like a data or ML background and not necessarily a like, how do I really efficiently work with large scale compute and that type of stuff. So for example, I think a lot of people don't even necessarily know, like do they actually need to scale?

So I'm wondering if we could start at the beginning. And you could tell me how should someone know if they actually need to scale? What are the clearer signs? Their problems benefit from scaling? And to put it [00:04:00] another way, what are some cases where scale clearly delivers? 

zach: If it could add a memory, you probably need to scale.

Essentially, it's whenever you're finding the model you're trying to scale, or the data you're trying to train on, either is too slow or the model is too big. Too slow could mean even though your data pipeline is as efficient as possible, you know it's gonna take like three days to train. You don't necessarily want the patients for that or have the patience for that.

So you go spin it up instead of on one GPU on like a hundred and it finishes in a fraction of the time. Right. That is one instance of scaling where we don't need to worry about the model, we need to worry more about just getting to the data as fast as possible. The other version is we're not training like.

Let's say like 360 million parameters that can train on a single 24 gigabyte GPU easily. Instead, we're trying to train like llama. 70 billion. 70 billion, just the weights is 140 gigs. So you need a lot of VRA to fit that in. And so these are sort of the [00:05:00] two sides of that coin of is it fast enough for you and does the model fit?

And both of those play together in sort of this. How do you decide what needs scaling and how 

hugo: That makes a lot of sense. I'm just wondering, we do live in a world where a lot of people increasingly ping APIs or serve open weight models on cloud infrastructure themselves and don't train themselves. So is the need to scale only relegated to training or are there other cases?

zach: No, absolutely not. Like we consider these people who are hosting these APIs. Right. They're also. Doing distributed inference in some capacity. 'cause you can't serve LAMA four or 5 billion off a single G-P-U-G-P-U just is not big enough at this time. Even on the inference side, there is infrastructure in place to not only handle concurrent requests, but you have concurrent requests and serving the model as efficiently as possible to users.

A lot of the times will ping clawed or will wiling runway ml, or all these other folks that have the model [00:06:00] served for us. That being said though, do you want to just be someone that understands I ping something or do you want a mild mental landscape of what they're doing in the background? Because one of them, it's a black box, the other one, it's a black box, but you have a peak under the covers a little.

hugo: I'm just wondering, like with everything happening in the modern AI landscape from agents to all the different things people are building currently, how should people think about involving scaling and distributor compute in all of these considerations? 

zach: So the first thought is. How big are the models you wanna work with?

Right? Because how big the model is depends on what Jeep view you can conserve it on. So let's say for instance, you're running a complex pipeline where you're running half a dozen agents with various degrees of sizes, right? We'll do like a standard swarm approach where we have like a dozen agents that go and pull things from different websites, and then we have bigger models that go through and compress that information down, report it back to the user and so forth.

Each of these still need to be served in some capacity. [00:07:00] It's a slightly different distributed idea rather than one model. 'cause we're serving many, right? But there's a world where you're serving half a dozen, 8 billion parameter models that all needs VRA to be stored on. It turns it more from a training problem to an inference problem, and especially.

Knowing the best way to serve those models locally, but it's still scale, it's just the third axis of it. 

hugo: Mm-hmm. So there are many different ways to think about where your computes happening. So I think I, I told you this last time we chatted, I recently got myself a new MacBook Crow with 128 gigs of unified memory.

And bro, like, I'm trying to push this towards its limits, but it is really tough, man. Uh, people who listen to me often may have heard me say, one of the first things I did was, oh. Hugging face. I spun up a Gladio app where I used Alama in the backend and had like 10, 14 B models that I'd alama pulled working in parallel, and so I give it one prompt and then in 10 to 15 seconds just saw the output of all of these models.

So that was super fun. We have stuff locally we can [00:08:00] scale on laptops. And I just mentioned in our course, everyone gets $500 worth of modal compute for free, right? So we can think about scaling on platforms like modal, then we can think about having our own Nvidia GPUs and hardware and that type of stuff.

Or we can be using AWS and cloud services. So I'm just wondering how do you think about all of these different options and are there different approaches to scaling on each? 

zach: So let's narrow down a little bit, and I'm gonna start just with training, right? Because if we think about how you're possibly spending credit.

Probably your training models if you're serving, we have similar things. So the biggest waste of time when you consider comparing vendors. The biggest slowdown realistically when it comes to mixing and matching providers or even just having providers with local infra, is your communication time. That is the time it takes for, say, I am running LAMA four or 5 billion.

Across something on AWS and something on GCP because I have credits [00:09:00] on both, right? Well, computer A still needs to talk to computer B. And if they're in the same data center, great. If they're not, they're possibly talking across the world. That is your slowest point in training. Inference doesn't matter.

That communication time is the slowest possible scope when it comes to slowdowns in a training loop or in an inference loop. And so when you're considering what providers to pick, what GPUs to pick, you wanna look at, are they in the same geolocation? So this would be you're doing ping tests from A to B and seeing how long it takes for 'em to ping to each other.

And then also the hardware level matters too, because let's say AWS is serving H one hundreds. Great. They're running their H off of PCIE lanes. And then we have modal who is running H, but they're not using PCIE. They're using very fancy InfiniBand. So even though I'm serving the same chunks of a model, or we're doing the same training loop, doesn't matter.

This computer is gonna be faster than this [00:10:00] one always because it's faster processors. So these are sort of some of the things you need to think about when it's, well, okay, who do I wanna go with, right? How fast can they provide me the hardware? Because. If I'm running say BH two hundreds, but I have slow SSDs.

I have slow processors. Okay, great. I have to make sure everything stays on GPU as much as possible to make sure that they can run as fast as possible on top of everything else. 

hugo: That makes a lot of sense. Feel free to ask any questions there. I am interested, Zach, in just stepping back a bit, uh, I think a lot of people feel stuck between CoLab and writing local scripts and getting them deployed on cloud infrastructure, for example, and then needing to spin up full clusters themselves and handle that.

And I think it's one of those things that actually seems harder than it is, or it's more overwhelming than it is. So I'm wondering if you could give us that. Soft entry into this for people who feel stuck between CoLab and needing to spin up a full cluster, where can they start? 

zach: Sure. I was [00:11:00] actually just talking to one of my good friends about this yesterday.

My personal opinion don't run to Kubernetes. Researchers that I know use them and they like it, but it's a lot to learn very quickly. When you're starting out and you're learning this stuff, you want a very low barrier to entry. I can sort of say what my path was. We start with, which is single GPUT four, whatever have you.

Then you wanna go to say, two GPUs. If we want to stay in a notebook, Kaggle offers, I believe two T four still. So you can have a notebook interface if you wanna do distributed training in that way, accelerate or extension I made called I be distributed to launch and have a notebook interface running things now that gets you two GPUs and at least get your feet wet and it's free.

The next step is deciding how you want things to go. Because there's two ways you can do this. You can create your own infra at home or rent GPUs from individual providers that provide like renting out, I think like run pod and things like that where you get say, home clusters of two to four GPUs. Like mine right now has [00:12:00] five GPUs running on consumer hardware and sort of, you can play with that locally if you want to.

You don't necessarily need a full cluster like scaling and saying, I'm gonna go scale to say a thousand GPUs because I can is very expensive and a lot of infrastructure has to go in place to let that happen easily. The most common methods that I've seen where easy enough to handle. You either do kaggle's, multi GPU setup, or you hook into someone else's multiple GPUs, multiple systems, and usually that is run through a software called s lm.

Basically imagine s SLM as a controller for a whole bunch of computers. So you tell SM, go run this on a thousand GPUs. It will then go run your script on a thousand GPUs. Everything's replicated for you. You don't have to worry about data or anything like that. It's all handled in one central location and it just works.

Typically, these are gonna be resources that are monitored and it's a [00:13:00] little more intimate than just going to modal and saying, Hey, gimme like four computers to run on. The other option is to just go to and get four computers to run on and. Essentially roll your own distributed training. Essentially you have team UX open with four terminals, each terminal.

Going to one of those computers where you run the exact same command four times is kind of how that needs to work. 'cause all the computers need to know to run that thing. Mm-hmm. So that's sort of an overview of what the playing field looks like. Right. Each have their pros and cons, especially when it comes to skills that are useful in the real world.

S SLM is arguably the hardest one in my opinion, but if you ever get access to big clusters, the things you will learn by learning slum will carry you very far, especially because it changes how you think of in group settings. 

hugo: That's a wonderful summary, and I think there's so much for people to dive into and I'd love to hear for people who haven't.

Slum before or set up a distributed computer scaling on on Kaggle, as Zach mentioned. Love to hear I [00:14:00] if you do that and what you get out of it. It is interesting. I do agree that you do get a lot out of slum, but the overhead, as you mentioned, is not insignificant and we're any tight spot at the moment with just.

How many cool things there are to learn and do. So prioritizing slum may not be the easiest thing, but if you want to get in the weeds, I definitely encourage everyone too. I am interested in people getting started in this type of stuff. What's the number one mistake you see when someone tries distributor computing for the first time?

zach: Trying to do too much too quickly. When you wanna just go from one to two. GPUs. Find the easiest way to get there. Use whatever wrapper you have to and just see the thing go bur, because even getting that can be tricky 'cause you have to know what commands to run. You need to know how high torch distributed wants to be configured to make things work.

How deep speed wants you to call its launcher in order for it to tell everything, to do things. Setting up like passwordless, SSH if we're running through multiple [00:15:00] computers, that initial overhead of just getting from one to two is a, a steep chunk of knowledge. And so not trying to hyperfocus on, okay, well, before I touch anything, lemme go learn what Zero is, what FSDP is.

And run it from scratch and learn all of it from the ground up is the wrong approach First, just get something that runs like be that axel lot, be that TRL doesn't matter. Then from there, start tweaking things. Remove some of the layers that these rapper libraries are doing. Go run the Hello World Torch distributed example yourself.

Because what you'll find is you'll hit little errors like, oh, why did I hit a communication timeout? One of the most common errors that happens from torch distributed, why is this a thing? Why is certain processes hanging, going and investigating things like this? Are the biggest problems you're gonna find first.

And also, they don't go away. So it's important to find them first, experience them, and then move past them, [00:16:00] right? Because that's sort of how the landscape is distributed. Training's not gonna run flawlessly on the first go, and magically everything works. If it was, everyone would be doing it constantly, and we're starting to get there.

There's a lot of helpful libraries that help you get past that first step, but it's still a growth period. 

hugo: Hmm. Don't try to do too much at the same time as, I mean one of the reasons I'm losing my voice at the moment is 'cause I've tried to do, well, also successfully done too many things this week. Uh uh as well.

In the spirit of not doing too much too soon, would the idea be also to choose. Like a particular training job you want to do and learning all the things around it, or a particular inference task you wanna do, and do all the things around it in order to stand that up as well as possible. 

zach: Yeah. The, what's it called?

What that sort of learning is called is the T structure. I forget who did this, it's one of the lectures that got me started on becoming what I do today. [00:17:00] But the general idea is for you to differentiate yourself in the field, you have a high breadth, right? So you learn a lot of things. So that's, you understand at a fairly high level, like what are all the distributed methods?

How does PyTorch work? How does deep speed work? And then you pick one thing and you go as deep as possible into it. And so that is taking, Hey, I know this ran with axolotl using something called distributed data parallelism. Now let me take that string and explore it as far as I can. And doing so, you'll learn skills that you did to expect to learn at the start.

And also. Really steps you apart in this field of people that know their stuff, but also understand the landscape of it, right? Because, 'cause this scales, right? As you get deep in one thing, that leads you to another thing and then you have another T. And really that's, we call people like this, localized experts under the hood, people would probably consider them jack of all trades if they don't [00:18:00] know the tea part.

And in my opinion, that's on purpose because the T part helps you become a jack of all trades with everything. 

hugo: We've got a bunch of really interesting questions in YouTube. Bo Boskey has made a point that we should make more examples with Slum for the Accelerate docs and they found most of the useful stuff from HBC Accelerate part of the docs.

zach: Yeah. First he is been everywhere in all my courses. He's phenomenal. Yeah. Working on that slowly. Part of that to help is next week I'm also giving a free Lightning lesson on S slm, not to plug to Simply State, we need more resources on this stuff because it's a lot to get into. Like pretty much most companies have an internal, here's how S SLM works and here's how you run our cluster, because it's the nitty gritty Linux stuff.

No one finds fun and so we need, the main problem, in my opinion, is the teaching method for trying to describe what S LM does. Configuring it does not [00:19:00] work for static environments. Mm-hmm. Right. So you sitting at home. Not being able to physically watch spinning up eight GPUs via one S SLM command, and having everything work doesn't really translate well.

For example, when I was trying to learn S lm, I picked up four cheap raspberry pies off Facebook marketplace and created my own little mini cluster just to get experience with running jobs. 

hugo: Yeah, so Keval has some great questions in YouTube. Some of them are a bit detail oriented, we'll try to get to them.

But Keval, if you can ask them in Discord, we can answer them afterwards as well. But one of the questions is, what is the advantage over using Kubernetes for distributed training? 

zach: Headaches. Mm-hmm. Kubernetes, you require people that specifically are trained on Kubernetes to know how to run Kubernetes clusters.

Um, I can run up after a day of reading docs. It's a lock-in vendor situation, so a lot of times people don't necessarily get a choice in the fact of. Are they forced to use Kubernetes? But let's go with all of my experiences with it have purely been headaches compared to [00:20:00] just learning how to do stuff without it 

hugo: Makes sense.

And also, do you have a link to your Lightning lesson that we can just share? 

zach: Yeah, it's gonna go live, I think tomorrow I'll have that linked in the discord and everything end of day. 

hugo: Fantastic. So. Look, we've come in slowly. I do think there are just so many things at play in this space and there are too many bloody acronyms, Zach.

So I'm wondering if you can just give us a high level mental model for things like DDP and FSDP and zero and how do they differ and why should we even care? 

zach: What in the world are all these acronyms? And where are they all coming from? So we'll start with the shortest one and sort of go in increasing complexity.

So the first is what we call DDP, distributed data parallelism. Essentially, that translates as the entire model exists on every GPU and you distribute things through what we call a level of parallelism, a dimension by splitting the data. So rather than training on one GPU with a batch size of eight on [00:21:00] eight GPUs, I would be training with a batch size of one.

This is, for example, practical applications. Obviously you don't wanna do that, but what this essentially does is it speeds up the amount of data you can process in a single second. It is purely for speed. And then we have zero, which stands for the zero Redundancy optimizer. It sort of plays into effect with DDP essentially use.

Sit there and go, okay, well I have this big model and it doesn't quite fit on my GPU, but it does fit across, say, half a dozen. So zero splits the model, potentially the optimizer states, and potentially the gradients between all these GPUs and essentially makes them act like one big GPU when we think about the V Ram size.

Now all of this comes at a cost. That cost is communication, like I mentioned earlier. And so you sort of have to figure out what we call level of zero you wanna perform that gets you the fastest training. [00:22:00] Also, FSDP and Zero, same thing. One is what PyTorch called their version of it. Zero is the actual name of the algorithm.

And so sort of as I mentioned before, everything is. Determined based on throughput, based on communication time. 'cause that is your longest point. Pipeline parallelism, tensor parallelism, uh, F-S-D-P-D-D-P are all methods for dealing with training a big model and reducing that communication time as much as possible.

Then things go crazy and you can stack 'em on top of each other and certain nodes can do certain things 'cause they have faster things and whatnot. But at the end of the day, distributed training is. Take this model and train it as fast as possible on networks that potentially are continents away from each other.

hugo: MERT also has a, a really interesting question, and I think we need a bit more context for the question around, are there tools to handle everything around checkpoints like moving over to the next place, et cetera. How does this work at larger scale like the AI labs, like Open AI and Anthropic? I think for [00:23:00] our audience though, there are a lot of people watching who understand these things.

Well, maybe you can tell us a bit about what checkpoints are, why they're important, and then how they work at small scale and then at large scale. 

zach: Sure, absolutely. When we think of checkpoints, we essentially think of almost like a static image, like if you took a picture during model training, that frozen state that it gets photographed.

So that would be like the current way the optimizer looks, how the model weights look, the inputs, and how far we are in the data loader. That is what we call a checkpoint. It is a point in time that we check out and so we can resume training from this point if say, training crashes or at the end of training we can save a checkpoint that has everything.

These are sort of what checkpoints are now, what Merck's question is about is. When you think about a checkpoint, you sort of need a full state of everything in a way, right? If we wanna save the model and go serve it in the middle of sage, a gigantic training run. Well, my models currently split across a thousand GPUs.

I need all those weights. [00:24:00] So one option is you take all those weights and you do what's called a gather, where we gather all the weights together onto some particular GPU process. Then we save that away. We don't wanna do it on everyone. 'cause then we're like writing a 1 billion parameter model, which is a gig a thousand times.

No, we only need the one. So you do it on one process and then you keep going. The problem you run into though, is getting all those weights and saving them away. Everything is frozen at that point in the training room because everyone's gotta go send their weights to the first GPU first. GPU needs to go save it and then you can move on with your life now.

Sort of solutions people have been coming to. PyTorch recently wrote an article about this is what we call asynchronous checkpoint, which as it sort of sounds, training keeps going, but in the background, we're secretly storing away weights. The black magic basically happens by letting the GPU do its thing and then having a CPU version of the model that we go save at some point later.

That constantly gets updated, but it gets [00:25:00] updated in time so there's no wasted energy of like sending weights to places It piggybacks off of where that's already happening to update. That's sort of where the latest tech is, because otherwise you run into this problem of like, well, okay, I could save every five steps.

But if I do that right, it's gonna add potentially hours to my training run. 'cause of the time it takes to gather everything and the time it takes to actually write that to disc. Well, how much time am I wasting just storing weights and translated when you get into a lot of GPUs, how much money am I wasting?

Right? Because say, waiting five minutes to save my. 400 billion parameter model across a thousand GPUs. That's like wasting a thousand dollars per checkpoint on the low end because that's 10, 10 minutes times a thousand in billable hours. It's ridiculously expensive. So people at, people at the very big labs that have access to hundreds of thousands of GPUs are thinking about this and trying to come up with efficient ways to do it.

Now, for us, at a small scale, that's still helpful. Because we're not running a [00:26:00] thousand GPUs, right? If you're GPU, middle class or poor, then you're running on like two to eight a single node. Well, I can keep the training going and just save weights in the background, right? I still can benefit from this even though like my gains are a few seconds while theirs is like hours.

Think about the scale. 

hugo: So that's a really good question. I've never heard the term GPU middle class, which I like a lot and it is a sad irony that the GPU middle class is growing as the middle class itself is destroyed completely in our civilization. And I do apologize for like diving into some technical questions from the chat and then kind of pulling back.

To more general questions. I am interested if someone wants to scale like to two or eight GPUs or something like that to start with baby steps to go from zero to one, what's the simplest, most reliable way to do that, do you think? 

zach: Sure. My personal favorite is to use the Axel Lot framework. By wing and the Axel Lot folks.

Phenomenal. Axel lot's just phenomenal. You set a configuration and you [00:27:00] let it go back. It's a really good way to just experience. Here's how you write in Axel Lot's case and accelerate configuration, and here is how you start the training, right? Hardware agnostic. It's GPU agnostic 'cause it's running the trainer.

And we did a lot of work to make that happen. Axel Lotto wrapped it in such a way that you have the highest level possible way to run a training script and just tweak how GPUs work. Just tweak how zero operates out of the box. You don't need to change any code, you just change a YA configuration. And so doing that lets you start immediately with, okay, well here's a known thing that works.

Lemme go see what happens with the memory whenever I like. Try zero two. How does that work? What's the speed like? And it gives you quick ways to iterate and just mess with how these different applications translate to wall clock, for instance. How long it took for my script to run. 

hugo: That's really helpful.

So we've talked about kind of the different aspects of what you'd use scaling for, but it seems like [00:28:00] inference is something that's increasingly treated like a distributed systems problem, and I'm wondering why we're seeing that. 

zach: Sure. So if we circle back sort of to what we were talking about earlier, right?

We have like LAMA 4 0 5, right when that came out or Maverick this soon what, two a trillion parameter model. For me to run that locally, that's not a thing I can realistically do, even with my weird cluster mess that's happening upstairs, right? Someone else has to host that. Now the next question is, well, how are they hosting it?

Because they can't just host it in like serve one user a second. That's like one user per a time. That's not efficient. That's not how that works. On top of that, it doesn't scale. So how do you make that scale? You don't use zero or FSDP or data parallelism. They're using a technique called pipeline parallelism most of the time where it's very efficient for serving, because essentially no GPUs are running idle as the model is performing inference.[00:29:00] 

And the reason why this isn't needed is because. Again, if we have a four oh 5 billion parameter model, you're still looking at, I think it's 300 gigabytes of video card memory, just to run it in a reasonable state. Still gonna need a lot of GPUs to get there. And so, because sort of we had this explosion and hey, what happens if we just keep training bigger models?

Well, as that explodes, so does the complexity of serving these things. Let's say I am serving LAMA four oh 5 billion using this pipeline parallelism technique, but I'm using some of the stuff from Hamill's LLM course. Let's say we're running Laura's on top of it, which are low rank adapters. So I'm not fine tuning the model.

I'm fine tuning essentially a hat for it to wear. Well, now you have complexity and I have a dozen hats Specific users want specific hats, but also I need to make it work in this. Infrastructure serving that I'm doing. Mm-hmm. So it's quickly becoming a very tricky problem to figure out how do I make the model [00:30:00] available to my users as fast as possible, and also deal with hiccups and slowdowns and have redundancies.

GPUs will die, nodes can fail. And so in the real world, you have to think about how do I make it reliable? How do I make it efficient, and how do I make sure that my users don't even feel any of the effects that I'm basically stuffing into a trench coat to make sure that everything looks fine. 

hugo: So we've talked and we've touched upon a lot of different ways to think about and do distributed parallel training.

I'm just wondering. How I would, or our listeners would decide which one to do from a cost and throughput perspective. So how do I know what's the best for each situation to just make sure I'm using my very expensive GPUs the most effectively. 

zach: Sure. So generally, when you're thinking about running a experiment, right, you saw that like, oh, Quinn's new model came out.

I want to play with it. I want to train in all my data. There's a certain flow chart you run through of figuring out what you can and can't do. So the [00:31:00] first part is calculate your budget. This determines your GPU hours, right? If you can spend a thousand dollars, go explore half a dozen different GPU providers and see what they offer, and then figure out, okay, how much would it be for me to use?

Everyone, how much would it be to use provider A, provider B, provider C, figure out how big of a model you want to write. Ideally, you already know that, but just consider, are you training with Laura where you don't need as much VA and you can train for a bigger batch size, for instance, 'cause it has lower VRA for training.

Are you training from scratch? Are you training fine tuning involved? What are the sort of memory requirements needed for the model weight itself? Quick back in napkin math. Easy way to think about it is for every 1 billion parameters, you need about eight or so gigs of VAM. Well, eight's a little bit low.

Really it's higher, but it's somewhere around, actually, it's much higher. It's about 24 to 48 gigs I believe. I don't have my paper up in front of [00:32:00] me, but G, basically it's four times the amount of memory that the model takes just to train it. So even if, let's say you have a thousand gigs. You might only be able to train on 250 billion parameters.

Maybe just as one example, the math's off, but just to give you an idea of it's an exponential look at how things are going. I think almost quadratic at that point. So after you figure out your model size. Then you look at the token amount that you want to train on. This is how big your data set is. So let's say we're training on 2 million tokens, right?

We'll say 2 million tokens. So then what we do is we say, okay, I wanna train on 2 million tokens. With a model of XC size, and I have Y hours to get there, what collection of training paradigms get me to that point? The fastest and the easiest, right? Because if we're using every provider, we wanna limit the communication because it's gonna take a long time for those [00:33:00] different provider GPUs to get to each other.

But there's ways that you can handle that by running as much as possible. On each computer and then reducing that to like one communication per training cycle rather than like 50. So you basically go from budget to model size to data, and then from there you fit. Okay. Does zero two make sense because there's a lot of individual nodes, or does zero three make sense because the model is too big?

These are the flow charts of how you think about things. The LLM playbook by a lot of my coworkers has a wonderful chart that talks about, I'll drop it into Discord that talks about sort of the best parallelism strategies when you compare the number of nodes that is the number of computers relative.

To the size of the model you're training on in billions of parameters. Now they get into wild stuff called 3D parallelism. That's our parallelism strategies stacked up in a trench code and magically making things work. But like that's a level [00:34:00] of efficiency you can get to, and this is sort of all around what that flows chart says, right?

It is budget to compute. To understanding what strategies you need to use to train how you need to. 

hugo: Super cool. And I think certain heuristics, like the one you mentioned of four times the number of parameters, something along those lines are incredibly useful. Did you say there are papers which can help just people think through those type of heuristics?

zach: Yeah. The biggest paper is the, I'm gonna butcher the name 'cause everyone always says I pronounce it wrong. It's like Luther or EEU Labs. They have a paper that's. Gosh, I don't know. It has to be at least a year old or two at this point. That just talks about estimating memory needs of transformer models.

Do you 

hugo: remember what it's called? 

zach: Lemme look through transformer scale. 

hugo: I've actually got a a call T Transformer 

zach: Math one oh one 

hugo: Transformer. Math 1 0 1. Exactly. Yeah. You just link into Discord. 

zach: Exactly. Some other good [00:35:00] resources is the LLM Playbook, as I mentioned. It's a free book by hugging face. And then the last one is.

And if you really wanna go to the extremes and learn everything you possibly can. Sta Beckman has a book on distributed essentially everything from the ground up when it comes to things 

hugo: as LM engineering and STA contributes. He's at Snowflake book, contributes a lot to Transformers and deep speed.

zach: Snowflake actually is now in charge of deep speed 'cause all the deep speed folks. Went to Snowflake. STAs is absolutely brilliant and so he has been writing this book openly for the last year at least, I believe that just talks about everything he learned from training at scale. So there's, if you really wanna go into the weeds, it has things on figuring out how fast your M two drives run to learn how long it will take for you to save model weights.

You can take that into account when you're thinking about like how budgeting works. [00:36:00] So those three resources are absolute goldmines when it comes to thinking about these things. 

hugo: Incredible. And we've put them in the discord, so definitely everyone check them out. So we've discussed this briefly, but just to like double down on this.

We have all these different versions. How, so we've talked about cost and all of these things. Are there any other things you can say about how to figure out what's best for each situation to make sure I'm using my GPUs effectively? 

zach: Yeah, I mean, the flowchart handles most of that. Now, what I can say is the most important thing to do is just to play with the code, right?

Because unless you physically can see the speed ups in front of you and you physically can see like. Here's like a torch profile of my entire training loop. Oh, 75% of the time was just spent on communication, and that's because I use zero three rather than zero two. Just to throw words out there at people.

One involves more communication, so that's why we see this physically playing around with it [00:37:00] and learning sort of how these work really help you get a feel for not per se, how fast things run. I would more call it the. Ease of the ability to apply these techniques. So, 'cause on one hand, right, we talked about how you decide what these things actually are like.

You can use tensor parallelism, you can use pipeline parallelism, you can use these fancy words. The more fancy you get, the more into custom code you get. Mm-hmm. Tensor parallelism is literally parallelizing matrix multiplications in the model and running half the matrix on GP one and half the matrix on GP U2 only recently has.

Transformers helped you with trying to do that out of the box, but before you had to get really intimate with the model code to understand what matrices need to be ran as efficiently as possible by doing these techniques. Whereas DDP cares nothing about the model, right? We just care about the data.

Zero cares about model layers and say groups of layers to [00:38:00] know how we wanna split things up between GPUs. So I would start with what's the simplest implementation to work with that you can spin up and then analyze and increase complexity as you go from there. 

hugo: Mm-hmm. Awesome. So we're gonna have to wrap up in a few minutes, but I just want to reiterate, you've got a full course on this, which you're kindly providing huge discount to our listeners for which I'll link to in the chat and will in the show notes as well.

I'm just wondering what you want. Every student or listener to walk away with, whether they sign up or not, what type of things can they really get out of it, or what do you want people to do? 

zach: Really, the most important thing in my opinion, is just the fact that distributed training is here and it's going to stay and it's not gonna go away.

Getting into agents where we're running smaller models, but parallelism strategies still help training smaller models too. It's becoming more commonplace that. Having experience just with how this stuff works, makes you hireable. It makes you desirable. And so if you come take my course, I am ecstatic and I'm thankful, and I'm happy that you're [00:39:00] putting myself in a guiding place to help you get there.

But there's also a dozen free resources that you can do or more that talk about this stuff. The main point I wanna make is just go get those skills. Go use some of this compute credit that you're getting between Hugo and myself course, and go try distributed training. Just go see how it works. The weird quirks that come with it.

Get experience with just seeing what this code looks like because five years ago and now, like in the last. Two to three years. Distributed training has had an exponential boom to the point where like we're building larger and larger, just places to run GPUs at the detriment of the environment. And that's a whole different thing, but like it's becoming absurd the amount of scale that we're seeing in the industry when it comes to training models.

So get that experience, get that knowledge, at least play with something and get a model trained using whatever it takes on whatever platform, Kaggle, what have you, [00:40:00] just to say you've done it and you know the rough strokes of how it works. 

hugo: Thank you, Zach, for your wisdom and expertise and sharing it with everyone.

Absolutely. Thanks for having Mehan Yuko. This was a lot of fun. Thanks for tuning in everybody, and thanks for sticking around to the end of the episode. I would honestly love to hear from you about what resonates with you in the show, what doesn't, and anybody you'd like to hear me speak with along with topics you'd like to hear more about.

The best way to let me know currently is on Twitter at Vanishing Data is the podcast handle, and I'm at Hugo Bound. See you in the next [00:41:00] episode.