Harness the Power of Durable, Event-Driven Workflows with Joel Hooks
===

Paul: [00:00:00] Hi there, and welcome to PodRocket, a web development podcast brought to you by LogRocket. LogRocket provides AI first session replay and analytics which surface the UX and technical issues impacting user experiences. Start understanding where your users are struggling by trying it for free at LogRocket.

com today. My name is Paul and joined with us is Joel Hooks. He's a software developer, collaborator at egg. io, among many other things. And we're going to be talking about harnessing the power of durable event driven workflows and picking apart what that means. What is an event driven workflow? ~Love talking about paradigm.~

So Joel, we're excited to have you excited to dig in. Welcome to the podcast.

Joel: Hey, thanks for having me.

Paul: ~So egghead. io. I'm, I just have to ask, I'm kind of curious, when did you, uh, start that endeavor?~

Joel: ~I think we're about 11 years into, into that project. Um, the rest of the projects that I work on kind of stem from that same one project, uh, like a, like a iterations on the same idea, but yeah, it's been, been 11 years since we've been operating as a business.~

Paul: ~And, uh, on egghead. io, oh my gosh, that pronunciation is going to come back to haunt me. On egghead. io, do you guys use serverless? I mean, that's kind of the first thing I wanted to hop into our podcast today is we're going to talk about serverless. What even is serverless? And you said something interesting about it being a billing model, but,~

Joel: ~I mean, fundamentally, that's what it is. And people always make the joke. It's like, Oh, there's, there's got to be servers because ultimately somebody's running a server somewhere. Um, and to me it's, it's serverless means we're, we're operating on a billing model. We're not paying month to month for our own serve.~

~Is it, is it? Yep. Okay. I'll find it. I'll find another hand toy. No worries. Do we start over then?~

Paul: ~uh, how about we start with Okay. I'll start with asking you about serverless again. So~ one thing I'm really curious to know about Joel is serverless. I'm not sure if you use it on egghead.io or if you use it on some of your other successful endeavors that you've been on, but the main thing that's sticking to me that you said is that serverless is [00:01:00] ultimately a billing model.

That's it? ~Yeah.~ Can you talk to me a little bit what you mean about that? Being strictly a billing model? When we talk about serverless servers.

Joel: It's mostly a joke because I think people get mad about the term and ~it's one of those like tech ~It's a marketing term at the end of the day that's really representing how you pay ~Um, ~and how you host your website. We absolutely are serverless and kind of embrace that notion that we're not maintaining the server.

Like ~I'm, ~I'm a fairly competent Linux sysadmin but If I had to spend my days at that level ~of, ~of the stack, it ~would, ~would really impede me from ~kind of ~doing the higher level stuff that we need to do to, ~to, uh,~ focus on customers. So as a billing model, when we use serverless, it allows us to just ~kind of ~focus on stuff and reduce staff.

And you pay for that where ~you can, you know, you can, um,~ you can. Use a VPS 5 a month,~ uh, and, ~and host your own server and configure it top to bottom, or you can pay a premium and have somebody else,~ um, ~whether it's Amazon ~or, ~or one of the many kind of service providers that you [00:02:00] might use,~ uh, ~to go serverless where you are not maintaining.

A server. ~Um, ~and that's what we do. And it just really allows us to focus on building applications and doing that stuff instead of sysadmin. ~So, um, ~I love it. It does add complications. ~Um, ~and you know, like one of the fundamental things that we've discovered with that is right out of the gate where we're effectively building a distributed system because we're serverless, but we're, ~you know, ~using services across the internet.

~So, you know, ~like we have this distributed system that we're using. ~Um,~ and you know, like you're either doing that intentionally or ~you're,~ you're kind of doing that by accident. And in our case, we try to be intentional about it. And that's a ~part of ~part of the event driven benefits that, ~that, um, ~ we've been realizing lately.

Paul: Do you feel like it's an all or nothing with serverless? Do you see hybrid? Do you use a hybrid setup? Because I'm thinking about organizations that maybe they do have a piece of infrastructure tech as part of the offering. ~It's, it's, ~it's beyond the typical web application layer. Does serverless still play a role there?

~Uh, ~do [00:03:00] you guys use a hybrid model yourselves?

Joel: ~I don't, ~I currently ~maintain like at a,~ at an admin level, maintain no. servers whatsoever. So I would say ~we, we are, ~we are completely serverless. I can definitely see, ~you know, ~larger enterprises, especially, ~you know, ~legacy stuff,~ or,~ or there's particular things that you might want to use. ~Uh, ~it would make sense ~to, ~to stand up and maintain a full computer, ~you know, ~like at the close to the metal computer where you don't have ~that, ~that layer of abstraction, the API or whatever in between.

Paul: Do you feel like typically if you're starting an organization, if you're starting a new idea. It's a no brainer to go with serverless because of the thing you just mentioned. It's like, it lets you focus on the customers on the application.

Joel: Yeah. And ultimately it depends on the team too, right? ~Like, ~do you have somebody that is, is able ~to, ~to do that work? ~Um, there's, you know, it's, ~it's a whole skill set. It's a whole team at the end of the day. ~Uh, ~whether, ~you know, ~it's a team of one ~or, or, ~or many to maintain servers ~is, ~is a job. ~So, uh, ~do you have the capacity and then ~is it.~

~You know, ~like what's the financial benefit because there's the cost savings aspect and then there's control too and you might have Some sort of application where you needed to control if I was [00:04:00] operating a bank I feel like my decision making process would be different than ~um, ~what effectively,~ uh, my kind of ~My reductive look at what we do is a paywalled video blog.

~Um, ~and ~you know, ~like we are kind of doing low level e commerce and giving people access to video content, which is ~kind of ~like it's a low risk. ~If ~if there's security concerns or things happen, ~we're not, you know, ~Nobody's livelihood or bank accounts are at risk,~ uh, ~in terms ~of, ~of ~what ~the work that we do.

~So that, you know, like it, it's always, ~there's always scales in terms ~of, ~of what you're building and your capacity to do work. ~Uh, ~and for us,~ it's, ~it's always been,~ uh, mostly, ~mostly a convenient thing from when we started egghead. It was, it was Heroku, which I would ~consider ~consider serverless, right? ~I do not have, uh, um, ~it's a platform as a service ~and, ~and that allows You know, drop in and get infrastructure ~right, ~right out of the gate and just get started very quickly.

~Um, ~I built it from a, ~like a~ tutorial that was ship a rails app in one week or whatever. ~Um, and, ~and that was ~the, ~the kind of the mode back 11 years ago and it's changed these days. 

Paul: little bit about distributed systems. So you mentioned like we are in this [00:05:00] distributed sort of scenario, you're reaching out to different services, and we're talking about using serverless, this idea of serverless as a medium to focus on. Your application to focus on your customers, but it's not like, Oh, I don't have to think about it.

Right. So distributed systems, you have to think about those carefully. I assume as well. What type of considerations do you take into mind when you're doing that? And what sage advice, ~I guess, do you have to, ~would you give people like having run on serverless for so long about this distributed system design when you're using serverless,

Joel: it's almost ~kind of ~natural when you start blending in different service providers and that makes it distributed. So for us, our primary hosting, we build Next. js apps and we host them on Vercel. But I'll take ~like ~uploading a video as an example. When you upload a video, we actually use a service called UploadThing.

And it's just a wrapper around [00:06:00] S3,~ uh,~ and that's Theo Brown's,~ uh,~ ping. gg, that's ~his, ~his business. And it's great because it just, he gives us some components and it takes all the complication out of uploading. So when it gets uploaded,~ um,~ that calls back. So we get ~a, ~a webhook back to our service and then that kicks in,~ um,~ the transcript ordering.

So we use DeepGram for transcript ordering. So ~that, ~that ~gets kicked in.~ Kicks in on that service. ~Um,~ and at the same time in the background,~ uh,~ we're processing that video. We send it over to mux,~ um,~ who is our video host,~ uh,~ and they process that video. And once ~it's the, ~the deep gram transcript is ready that gets sent back.

And then we tell mux that the captions are ready and send those back to that service. And then once the video is already, it notifies us. And in the background, we're running a service called,~ um,~ party kit.~ That is a webhooks or excuse me. Um, That is, uh, Oh, my gosh, I just blanked out. Oh,~

Paul: ~comes back to~

Joel: ~it's, it's, uh,~ so part of kit is WebSockets and the WebSockets then notify our UI ~that,~ that the, ~you know, ~whole thing is done and the video pops in and the transcript pops in and now ~we can, ~we can edit it. And, ~you know, ~the alternative to that would be to have a server. ~Right. ~Like a single server that's running our next JS app [00:07:00] that has FFmpeg under the hood,~ uh,~ that's running Llama3,~ um, and, ~and all that stuff locally.

~And, ~and, ~you know, the, ~the sockets, all that could be contained in a single machine and it could literally sit on the floor in my office. And, ~you know, ~I've done that and maybe I could throw some Kubernetes into it,~ um,~ if I wanted to get distributed in my own house. But we don't have to do that. I think that's what's cool is these days we have ~this, ~this access to services ~and, ~and that comes at a cost though, because that really is a lot of, ~you know, ~it's a lot of communication over the network.

We have network issues that can, they can crop up. One of these services can go out and it's like, While that's a lot ~of, ~of action that is very distributed, it's distributed,~ uh, you know, ~across the globe,~ um,~ at the end of the day, I don't know or care where they're running these services and,~ uh, you know, ~I'll choose US East one,~ uh,~ when it's available, but, ~you know, ~that's not always the case, but in, in these particular processes, it doesn't really matter async, they can take five seconds or they can take five minutes ~and, ~and,~ uh,~ it doesn't really affect us.

Paul: I think to a lot of the folks out there who self host,~ like, you know, if, ~if you're [00:08:00] into some sort of servers or Kubernetes, maybe you run the various open source software is like paperless or whatever it is at home. ~I, you know, ~you mentioned running Kubernetes and those folks might be saying, Hey, that's great.

That sounds expensive, Joel.

Joel: ~Well, ~they're doing to run a business.

Paul: for a business, not expensive. Okay.

Joel: ~I, ~I literally have a Kubernetes box on the floor over here. Cause I think it's fun and interesting and it's cool to do that sort of thing. ~And, ~and ~like, ~I'm capable and able to do it. And I choose not to,~ uh,~ for my day to day work and for the business just because of the.

~It's just, it's just a, you know, ~it's just a pile of other stuff and there's so much to do and we're a very small team. So it's ~like, ~we just don't, ~you know, ~like we don't have the time to ~like ~build everything from scratch and,~ um,~ yeah, it ~just, it really, ~really solves a problem for us and a lot of people, I think, at the end of the day.

Paul: ~you feel like that margin gap is going to close of. Yeah. Here, let me, sorry, let me rephrase the question. So ~the reason why you would say, Hey, it's okay for a business is because the thing you're selling and the operating cost of whatever you're doing is that such there is enough to maintain the business and propel the business forward.

Do you feel like that gap over time is going to close and it's going to [00:09:00] be more difficult to be competitive using. the service and that service and creating an effectively in a distributed system.

Joel: So there's always, ~you know, ~like I saw somebody and they were like, Oh, this email service isn't working for me. ~Um,~ Amazon's email service isn't working for me. And everybody says,~ I mean, ~use postmark and they say, well, postmarks too expensive. And you're like, ~I mean, ~how many users do you have? And that's making postmark.

Too expensive. And could you just ~like ~wake up every morning and manually send your new users an email through your Gmail account? ~Um, You know, ~I don't know, like, is a quality service too expensive? And so you're trying to cut corners and save money by going ~kind of ~direct to the fountain of serverless, which would be AWS, right?

~Like, ~to me,~ like,~ that's ~the CloudFront or Yeah, CloudFront, uh, excuse me.~ CloudFlare and AWS ~and, ~and this is ~kind of ~the source of serverless ~and, ~and ~most, ~most of,~ uh,~ these services that I'm talking about ~are, ~are probably on AWS or CloudFlare.~ Um, ~but those are hard too. It's even hard, ~you know, ~like when ~you, ~you start talking about that, if you want ~to, ~to cut costs and I pay, upload thing the [00:10:00] $20 a month or whatever, and I could, I literally have ~the, ~the code to write directly to S3, but it makes it so much more convenient.

I'll just use, ~you know, ~that service and,~ uh, not, ~not do it. ~And it's always a,~ I think it's always a trade off. Like you're always, you know, it's like cost and time and UX and DX and all these things are weighing against us,~ uh,~ when we make these decisions.

Paul: ~Right, right. ~Yeah, it is a trade off. And it's just interesting to hear of your success with ~you know, ~the different businesses that you've done and the fact that you are such a ~good ~big proponent of using them rather than not using them. So it's interesting to hear That perspective from you and, ~you know, ~naturally people might also be saying, okay, distributed system.

Those are complicated. How do you manage that? And you mentioned kind of your video processing pipeline. You have ~the, the, ~the transcript generator, you have muxed, you have all these different things. And that brings me back to the title of our podcast event driven workflows.~ Can, ~can we dig into that a little bit?

So what is an event driven workflow? How do you leverage them? And how can you describe them [00:11:00] for our listeners and how they help your team succeed

Joel: So~ it's, ~it's frankly ~kind of, ~Kind of new to our system in an intentional sort of way ~for, ~for years, I've ~kind of ~built workflows ~and, ~and if you think about ~like ~a web hook, right? ~Like in a, ~a web hook is an event that happens and ~you are, you know, ~you typically will have ~like ~an API endpoint. ~Uh, ~and ~uh, ~I've done this from our Rails app to our next apps.

where some service that we want to have an asynchronous process will send us a webhook back. ~And ~another example outside of the video processing is Stripe. So when we use Stripe checkout ~and ~when, ~you know, like a, ~a payment occurs when the customer succeeds. That webhook gets sent and then ~we, ~we deliver the product or the promise that we've given them that they've exchanged money.

And that's so important, right? ~Like that,~ that first few minutes after a purchase, if the system doesn't handle it properly, that isn't great. ~And ~for the business or the customer or anybody, so it's like, how do you handle that? So we'd have a webhook and that would come into,~ uh, ~say an XJS. API route, it would handle that webhook and then ~you ~do things like,~ um, ~record the [00:12:00] purchase,~ uh, ~send the customer an email, ping the team in Slack that a new purchase was made,~ um, you know, like, ~like that stack of things that you need to do after a purchase.

And in a lot of ways ~that is, You know, ~when I talk about a workflow,~ that's, ~that's, that is a workflow. ~It's a, ~it's a function that's executing a series ~of, ~of tasks,~ um,~ that we need to happen after some event has occurred. So at the simplest level, that is, ~you know, like, ~like in my head and my definition is an event driven workflow.

I think you can get into like stronger, like you could probably argue with people about the definition of what an event driven workflow is because there's event driven architecture has been around for decades. ~Um, there's, you know, like~ workflow is also one of those terms where it's been like overloaded over the years.

And it's, ~you know, ~it's a simple term and it means a lot of different things to a lot of people, but ~that's, that's, ~that's really what I'm talking about. And the problem with that approach is, and it's been our approach for years is that, ~you know, ~like what if the Slack service. Throws an error or what if there's some sort of right issue in the database or what if we've introduced something in that code that then [00:13:00] breaks it so something doesn't happen so that ~the~ say the purchase isn't recorded and the customer doesn't get what they've paid for and that kind of stacks up and stripe like webhooks will typically ~like ~retry, you know, they do that ~that ~they'll retry for you if it's a good service.

Some services don't though they won't Send repeated webhooks,~ um, ~over time. ~Um, ~thankfully Stripe does that. ~We, ~I can imagine ~we'd, ~we'd all be in a horrible dumpster fire if they ~didn't weren't more, ~weren't as resilient as they are in terms ~of, ~of getting us that info, but you know, like what happens then, and that's, that's where we're at today,~ um,~ is we've introduced ~kind of, um, ~a workflow engine on top of that, a service that, ~that, uh, ~allows that event driven paradigm to ~kind of ~come into our system.

So when we get that Stripe webhook. Internally, we announced that with an event,~ uh, you know, ~a charge succeeded and that gets distributed across our system. ~Uh, ~and we're using a service called ingest with it's I N N G E S T. And it's been like,~ it,~ it absolutely ~like ~peel my face back. Amazing. Just in terms ~of, ~of how it's changed, how I even consider [00:14:00] building my applications.

And we've been using that for about two years now. And ~it's, it's like,~ It's event driven workflows, but ~it's, ~it's like the background jobs and cues and all that fun stuff. ~Um, ~and it gives us that sort of stripe level resilience in terms of how we handle work,~ uh,~ after, ~you know, ~from the video processing workflows to sending,~ uh,~ to handling like checkouts and,~ uh,~ sending customers,~ um, like, ~like a marketing campaign after a customer has purchased ~is, ~is things that we're able to do and think about and handle internally.

Ends. Actually not use external services. So I've been able to ~like peel, ~pull back on some of the services that we use ~by, ~by consolidating internally and thinking about them,~ um,~ from this event driven workflow perspective.

Paul: is in just also a serverless service. And I ask because it reminds me of temporal. io, which is.

Joel: So ~I tried to, I tried to,~ I've been trying to do this for a long time. ~Cause one of my,~ I love convert kit. It's one of the services that we use and it's an email marketing platform and they have pretty [00:15:00] cool automations that like just have frustrated me too, for hours and hours and hours, but I'm really good at it.

~I've, ~I've used them for a decade also, ~since we ~basically, since we started egghead, but~ I wanted like control over my, like more. ~I want to code level controller of my workflows and they have one of those, ~you know, ~like graphical interfaces where you connect lines and you build a build,~ um, uh, ~email automation.

And it's ~like, I've tried, ~I've spent so much money, like trying to build that internally and I wanted to eat temporal. I was like, Oh, ~this is, ~this is really cool. And I tried to install it for years. Literally a couple of years. That's where my interest in Kubernetes ~kind of ~sprouted from is because ~like, ~if you want to self hosts temporal, like good luck, you're going to,

Paul: Yeah. Yeah.

Joel: going to break out K8s and you're going to really ~like ~go to town and they've gotten better.

Like today they're better than they were two years ago when I started using ingest, but ~it's not, you know, ~it's not for the faint of heart and it never worked for us. ~Um, ~so it's similar. ~Like that's, they,~ they're in the same space ~as, ~as ingest for sure.

Paul: ~Got it. Okay.~

Joel: just a different, like ingest is a serverless product.

And it's one where like ~my, ~my question with temporal was, okay,~ well,~ I have a next app that's on Vercel and [00:16:00] how do I, like, how do I get temporal working in that? And they're like,~ um,~ I don't know,~ like,~ like host your next app inside of the same application container is temporal on, ~you know, ~something, ~you know, ~like something like render can do that.

Or you could fire up digital ocean droplets. Maybe I,~ like,~ I don't even know, like, I still don't know. And I spent ~many, ~many hours trying to figure that out where ingest was like. Oh, you can have all that. You can have this event driven workflow lifestyle, and you literally just need to,~ to, ~to sign up. ~And, ~and, ~you know, ~like you actually don't even need to do that.

Their development server ~is, ~is really fantastic. So you can ~like ~get it with a click and a command it's up and running and you have a development server and you can start sending events and debugging events and seeing it all just absolutely working 100 percent in your local machine inside of your next app.

And the difference with what they do with, I think,~ with,~ with Most other competitors is that the compute, the actual processing of the events, the functions that are executed,~ the,~ the work that is done in response to the events is all done with inside of my application. So the functions are being [00:17:00] executed inside of my next JS app, inside of the context of what we are actually building.

And it's like, ~ ~I kicked these events off. I handled the events. You can do things ~like,~ like wait. So if, ~you know, ~the customer purchases something and you want to wait a week to, ~you know, like ~send them a follow up or, hey, how's it going? You can do that. If you want to wait till they do like a key event, right?

Like makes first document or whatever it might be for the business. You can say, Hey, wait for them to do this event. If it doesn't happen within five days, send them this ~same ~email. But if they do the event, send them this email. And then ~kind of ~chain and build ~these, ~these complicated,~ um, ~decision trees, like these workflows over time, right?

That's why temporal is named temporal, right? Like ~it's, ~it's over time. ~It's this, ~it's this time based thing where if I want to schedule a follow up email for six months out, how do you even do that if you don't have, ~you know, like, like it's ~cron jobs ~and, and, um, ~database rows and that sort of thing. ~But that.~

And then you don't get the resilience. Then you don't get ~like ~concurrency and retries and all that, ~uh,~ kind of fun stuff, which is in practice, really great in terms of being [00:18:00] able to actually deliver to customers and have ~that, ~that kind of stripe level,~ um, ~safety in terms ~of, ~of making sure that ~the, ~the work actually gets done over time.

And if something breaks, I can go in and fix it, make a deploy and the event picks back up,~ um, ~and ~we'll, ~we'll actually then run with the new code. ~It's, it's like, uh,~

Paul: That's a dream. ~I mean, ~I don't want to run my own Redis of managing my application if I don't need to. We've all been in the hole of trying to figure out what the hell happened to your Redis queue. It's not a fun place to be. ~And, ~and this is also pretty new stuff. ~I mean, ~Temporal came out with their schedules.

~They're ~they're cron wrapper with all the stuff you were just talking about and ~like, ~was it January or something it was it wasn't like around for too long, at least for ga so there's still like a lot of documentation coming on but how to properly use this and that kind of brings me to my next thing I want to ask you about Joel if ~you've been using this ~you've been using ingest temporal whatever it is the ~same ~same paradigm we're talking

Joel: Same idea, right? Like at the end of the day, trigger. dev is another one that,

Paul: Trigger. dev, okay,

Joel: another one that people like, yeah.

Paul: ~If, ~if you're [00:19:00] designing an application, a distributed system, if we want to call it, what are some pitfalls? I don't want to focus on the good stuff. Now, the bad stuff is usually where you get to the nuts and bolts and maybe like a misalignment of the level of abstraction that you would suggest a team or a developer to think at and break down their events into.

Joel: So one of the like classic challenges I think with any event driven architecture is where did that event come from? What the heck is actually going on and why is that executing now? ~Um, ~and ~you, you, ~you know, like an event. It can be emitted from basically anywhere. And one of the cool things with ingest is actually it doesn't matter where I'm running it.

I can have like a Python service running somewhere that picks something up and sends an event out that is then handled inside of my next JS application. Like they all kind of play, or I can take my,~ um,~ I can move a piece of my application over to CloudFlare and out of Vercel and ~have, you know, like ~if I need to, like, I want to do some AI processing or whatever, and CloudFlare's environment and ~kind of.~

hop out of Vercel for billing [00:20:00] reasons. ~Um, ~I'm able to do that and make it portable. But once you start doing that, once you start spreading it out and you have events being emitted from all over the internet, it's like, where did that actually come from? And ~who's, who's, who's, ~who's the one emitting this event and why is it still executing?

~Um, is, ~is, I think a challenge and then you get the~ like ~naming things is hard. ~Um,~

Paul: Oh yeah.

Joel: so you have to name events and you have to like, like when events, ~you know, like, ~like event naming and how do we name an event and do we use, ~you know, there's, ~there's this like, what's a message versus what's an event where, you know, well, a message is something that's like, Hey, do this where an event is like, Hey, I did this.

~Um, ~and ~you know, like, ~like keeping to those patterns, I think ~is, ~is more of ~like ~a human discipline thing where, you know, like you need to be disciplined about that. ~And, ~and one of the places to take inspiration from that is something like, uh, amplitude. ~Um, ~which is or mixed panel or keep or segment or like product analytics.

And the way I think about this in a lot of ways is from the perspective of product analytics. And that's where I've seen most of the research. But what's the name of your events? If you think about analytics in your application, like [00:21:00] this is what just to happen. We're tracking what users do in our application.

And when you introduce something like ingest,~ um,~ and an event driven, Handler. So ~you, ~you have product analytics combined with the ability to take action on those analytics. So ~it's not just, you know, ~you're not just building graphs when this happens, then we're also going to do some work in the background.

And ~you know, like ~that structure, the taxonomy of your events is really,~ uh, it's ~challenging over time and ~they'll have, you know, like ~they have whole systems inside of product analytics products. That ~are, ~are built to both train and keep track ~of, of kind ~of that schema of events and how things are named and what that actually means in your system.

~Um, ~so ~that, ~that's a huge challenge. It's ~like a, ~a layer of complexity,~ uh, that, ~that you wouldn't probably have otherwise. It's a trade off ~again, ~because,~ you know, ~the complexity exists regardless, but ~you can kind of, you can, ~you can dig yourself a ~pretty, ~pretty long term hole with, ~with, uh, Uh,~ scattered events around ~and,~

Paul: Do you ever get,~ um,~ event. Anxiety or event size anxiety, and I don't mean like the size of the payload. ~I mean, ~the breadth you [00:22:00] talked about how wide your distributed system can get, because a lot of these services charge based on your action count, how many,~ you know, ~like workflows get invoked, or whatever it is ~that they ~that they choose.

If you maybe design something a little wrong, okay, well, now it's 10 times more expensive. Does this ever go through your head? And how do you deal with that?

Joel: that's, I think the, this gets back to serverless as a billing model. And I think that's probably highlights that,~ uh,~ more than anything. Cause I I've absolutely had ~like a, ~a react dot use effect run away and cost me like multiple thousands of dollars,~ um, in, ~in function execution charges on Vercel and, ~and that sort of, you know, it's like, ~If you're not prepared and you're not, ~you know, ~like capping your billing or have monitoring set up or ~kind of ~have some way to understand it, ~you know, ~you'll wake up one morning ~and, uh, you know, ~with a big bill.

And I think, so ~40, ~40, 000 is my personal record in terms of unexpected bills. And that wasn't run away. That was more like a,~ uh, that was a, uh,~ a leaked key AWS where they're, they send us 40, 000. We're going to shut [00:23:00] down your whole server, pass up. Kind of situation. So that's always an unpredictable thing.

And it's,~ uh,~ there's controversy over it. I think too, because, ~you know, ~like Vercels taking some heat ~where, ~where, ~you know, ~somebody wakes up to that bill, it's like, Oh, you've destroyed my business. ~You, you know, like you,~ and they're like, we just provided you with the compute ~and, ~and, ~you know, kind of ~tried to show you, ~but you had,~ Hey, but you didn't have controls and, you know, you didn't turn it off when that's happened.

Could you imagine though, if like they did turn it off and you didn't want them to turn it off? ~Like ~that's. Maybe worse. I don't know. It's like a difficult, it's really hard. ~Um, so, you know, ~it's like, how do you handle that? And how do you, ~you, you~ monitor that? And one of the cool things with ingest and probably most of these services is this ability to,~ um,~ like throttle and rate limit, like right out of the gate.

And like one of my rules in life is anything that can send an email gets rate limited period. Like ~any, ~any. Function in my application that will actually send an email. It gets rate limited. Cause if you don't do that, you're going to send a million emails. It that's like the, it should be some sort of law.

~Um, ~one of those law of the meter or whatever, right?~ Like some every, ~every one of those, the ~other, ~other one is anything that makes a call to like open AI. [00:24:00] Is it going to be rate limited because ~like, ~these are expensive services or this is, you know, in the case of email, you can get like your entire domain email blacklisted and you can't send emails anymore for basically ever, like until Google decides that you can again, good luck, you're not going to be able to petition anybody or whatever.

~Right. Like, ~so that stuff is like, how do I protect my business and how do I proactively do that? ~And, ~and a system like this and an event driven system. With ~a, ~a strong service that allows you to layer on those protections,~ um, is, ~is invaluable at this point in just a super reasonably priced. Actually,~ uh, I don't, I don't, I don't know what the, the, the threshold is pretty, pretty,~ they are very generous at the, ~at the, the~ bottom,~ uh, ~in terms of, ~of how, you know, like~ how many events you can have ~and you end up, you know, like people send.~

Some folks send tens of millions of events a day and that gets pretty expensive But ~you know ~until you get to ~that ~that volume with your ~like hundreds or ~hundreds or thousands. It's ~like ~very reasonable

Paul: ~Do you, uh, Oh shoot. I had this question in my head. I'm so sorry, Elizabeth. I need to ask you to edit this out. Um, I remember now. ~Did you hear about the Netlify thing that happened,~ uh,~ with the DDoS attack and the sky woke up and I think it was like a 125, 000 bill.

Joel: Yeah.

Paul: [00:25:00] And it was cause of some like one megabyte image.

Joel: Yeah, and ~it was like ~it was like a 404 that he kept getting charted ~like ~yeah, ~it was it was ~There's a couple situations i've heard of this ~the netlify ~and I don't know if it was the same incident, ~but ~but Serving a page that was a 404 but was still ~Like ~racking up the bill is another one that, that's like a situation where it's like, why would that happen?

~Like, ~I don't know, but ~they're, ~they're software developers like the rest of us. And ~you know, ~it's like these edge cases or some nefarious actor on the internet looking, they are always out there, right? Like ~it's, ~it's bot swarms all over the place that are looking for something to do. And some people just want to cause mischief and think it's funny.

~Um, ~And I'm sure to them, it actually is super hilarious, but if you're not, ~you know, ~if you're not the one ~getting the, ~getting the stick jabbed in your eye, then,~ um,~ it's one thing, but yeah. ~So, I mean, ~how the company handles that because the compute did not cost them 125 K, right? Like they had some sort of, of tangible cost on their side and like, When we racked up the 40, 000 AWS bill, they said it was mutual responsibility ~and, ~and we're super [00:26:00] generous and cut it in half for us, which

Paul: ~That's, ~that's pretty good.

Joel: cool.

Like, yeah, I mean, great. Like it costs them,~ uh,~ it effectively costed them 0, ~but you know, ~like it,~ it's,~ it's cool. ~Um, ~it is my, I should have protected my keys. ~So cool. ~But like how did the company handles that is always an interesting case and you'll see the people that take it to Twitter or whatever, and ~we'll get, get some, ~it'll get some attention and there'll be ~kind of ~a,~ uh, ~Social media news cycle about the incident that happened and how the company responds to that is really something that's interesting to see.

~And, ~and I think,~ uh, ~helps inform our choices about service providers, ~you know, based on, ~based on those kinds of reactions, because who could predict that,~ like,~ it's like,

Paul: Yeah. ~No, really. You're. Yeah.~

Joel: And who's ultimately responsible for that? ~And, ~and, ~you know, like, ~if it's just something ridiculous like that,~ I, ~I feel like that's ~like ~100 percent ~uh, uh, ~service provider, ~you know, like, and, and, like,~ my reaction would be like, hey, don't worry about that bill.

~Like, we, ~we got that. ~Like, we're, ~we're covered. ~Um, ~versus, ~you know, like, ~literally, if I leak an API key, then, ~you know, like, ~that's not really the service provider's fault,

Paul: ~Right. Right. ~Any,~ uh,~ last sage advice that you maybe weren't able to get out giving other,~ uh,~ developers out there who are looking at the event driven architecture, looking at [00:27:00] serverless and distributed systems. Yeah.

Joel: ~I, ~I would honestly just say ~like, like ~give ingest a try and just do it, ~you know, whatever ~they have flavors for all the major frameworks. But with next JS, it's like create next app,~ um, ~ hyphen T ingest. And ~you can, ~you can have it up and running. The development server is such an amazing thing. I use it ~for, ~for debugging.

~It's like ~if I'm building a simple service to send an email, But it is related to Stripe checkout. ~I don't have to~ like, once I do that, once I 4, 2, 4, 2, 4, 2, 4, 2, 4, 2, 4, 2 to do the Stripe checkout, it sends that event. I don't have to do the checkout over and over again. I just open the development server and hit replay ~and, ~and that's amazing.

Just being able to replay events and go in and you can edit the data when you're doing it. Hit replay, hit replay. So then you can, you know, like, is my markdown getting processed properly? Is this actually getting sent without having to go through the entire UX loop? I can, you can always do that again, or I can replay any segment of that.

Or if ~like ~I'm processing a transcript and I want to make sure, ~you know, like ~we're trying to get that right, instead of uploading a five minute video to deep [00:28:00] gram and then waiting for that response, I can just replay, replay, replay, and, and that ability, like I'll write scripts locally and then just run them through that and then, you know, like a migration script or something.

So I can build it incrementally and I get that.~ Those, ~those kind of fail safes and ~protection ~protections and, ~and, uh, event,~ event driven goodness,~ uh, ~even ~in, ~in something simple like that. I think ~it's, you know, like it, ~it really was a big paradigm shift for me and I'm relatively new to the space. So I'm ~not, you know, I'm not, ~not an expert, but I'm absolutely a fan.

~And, ~and it's changed the way that I think about building applications. ~Um, ~and also giving me an escape route. Cause ~you know, like ~one day if I want to ~like. ~Move my stuff over to a different service and save some bucks on our, ~you know, ~at scale from using a premium service provider like Vercel,~ uh, ~versus, ~you know, like a, uh, ~raw AWS.

I'm able to do that. ~This, ~all this business logic is now portable,~ um, ~because it's literally just TypeScript. It doesn't know anything about next or what framework I'm ~operating, ~operating. And it just is code. ~Um, ~and I can ~take it, ~take it wherever I please. ~Uh, ~and that feels good to, you know, a little flexibility because frameworks change.

Paul: ~Well, ~[00:29:00] Joel, this was illuminating for me as well, because. ~You know, I've like, ~like you've probably picked up. I've used temporal. ~I've ~haven't used ingest myself. It's great to know there's other services out there. It's great to know there's event driven workflows where you don't need to make a client tied on a server.

If that's what you need to do. That's really cool. ~Um, ~and there's trigger. dev and,~ um,~ I guess ~the, you know, ~if people want to go look at it, go check out the docs. ~That's, ~that's the end. I'll be like, go read about it. Go watch a video. ~Um, are~

Joel: ~I actually, I actually, ~I actually worked it in just for a few months last year because they had a DevRel job and I was like, I haven't ~worked it like ~worked a W2 job in a decade and I really love this company and I'll go give it a try. I lasted like three months and I started building a demo and didn't want to stop building my demo.

~Uh, ~and so ~like I, ~like I was, and I'm still working on the demo, to be honest,~ it's,~ it's our new platform that ~we're, ~we're shipping,~ uh,~ courses on. ~Um, I, ~we have Jack Harrington's pro nextjs. dev ~is, is, ~is our first like fully ingest backed,~ uh,~ app router react server components. Like ~we've got, like,~ we're absolutely [00:30:00] surfing the bleeding edge,~ um, with, ~with this thing, but ~it's, ~it's great.

And,~ uh,~ so it's like a ~demo, ~demo gone wild that,~ uh, uh, ~came out of ~my, my, ~my time at

Paul: You just couldn't stop.

Joel: I really couldn't like they, they legit wanted me to do other things. And I was like, man, I just, ~I like, I I'm, ~I'm an obsessive builder and ~you know, ~like that's my nature. ~So,~

Paul: That's a value success. Don't stop, man. You don't stop. All right. Well, Joel, thank you so much for your time. Like I said, it was illuminating for me. I hope it was illuminating for some of our listeners. ~If people wanted to, ~if people want to hear more from you, are you on Twitter? What's your most common social media outlet?

Joel: ~Uh, ~Twitter. And then lately I've been trying to spend more time on blue sky, because I think if there's something that's going to ultimately replace Twitter, that service feels like it's the closest,~ uh,~ it's not quite there yet, but they're getting closer. ~Um, ~yeah. And then,~ uh,~ badass. dev is ~my, ~my website where I write about,~ uh, uh, ~making courses.

Paul: Awesome. ~Well, ~thanks again for your time. It's been a pleasure, Joel.

Joel: Thanks, Paul.