EP049 Ensuring Stability: The Role of Code Freezes in E-Commerce
===

[00:00:00] Darin Newbold: Well, good day and welcome to Commerce Today. My name is Darren and as always with me here is Josh to talk to you about what's happening in the wide world of commerce and what's happening right now as it's, uh, as it's coming around. And I get a feeling there could be. Could be some interesting things happening 

[00:00:23] Joshua Warren: right now.

[00:00:24] Most definitely. So it's the day before Thanksgiving. I was thinking it'd be fun if we go out, deploy some new features to a client site. What do you think? 

[00:00:31] Darin Newbold: Well, Josh, as exciting as that sounds, that's a hard, no, that is a hard, no. Um, yeah. and that's why for this topic today, we're going to talk about that.

[00:00:41] Uh, well, it's not very exciting, but it is. Extremely important. And that is the code freeze. So, Josh, tell us a little bit more and kind of what prompted maybe even this, this topic in a way. Obviously the joke about day before Thanksgiving. Yeah, but maybe there's some other things. 

[00:00:59] Joshua Warren: Yeah, well, it turns out someone at Shopify decided that was a good idea, uh, that the week before Thanksgiving, they deployed some code.

[00:01:06] Shopify had a 45 minute outage last week. Ouch. Yeah. And if you look at their SLA, they promise 99.99 uptime. That's 52 minutes. So they don't have much margin to play with the rest of the year. They're basically guaranteeing your side's going to be perfect the rest of the year. So that is why I wanted to talk a little bit about code freezes today.

[00:01:27] And now you 

[00:01:27] Darin Newbold: know what it takes to calculate the two nines for 99. 9. If we go to the three nines, 52 minutes has got to be just right out. All right. So we're talking about code freeze, but what is it? Um, what does that mean? And. Why is it important? 

[00:01:43] Joshua Warren: So a code freeze is when you basically make a decision that you're going to make no changes to the code of your website.

[00:01:49] While we call it a code freeze. That typically also means you're not going to change database structure. You're not going to make any major changes. The only things that might be changing on your site are just people placing new orders, inventory, updating, that sort of thing. But not making major, disruptive changes.

[00:02:05] So you wouldn't make 

[00:02:06] Darin Newbold: any simple like, Hey, I want to change the color of a button maybe, or anything crazy 

[00:02:11] Joshua Warren: like that. I would not even do that. But we're going to talk in a little bit about how some companies will do that. And it's just important to have some, guardrails. And, you know, you already mentioned code freeze, not the like most exciting topic.

[00:02:23] Code freezes are probably what allow e commerce managers to actually get some rest over the holidays. So, I would say they're incredibly important to the sleep and well being, but yes, they're not the most exciting topic. Well, 

[00:02:38] Darin Newbold: I didn't say it depends on who you ask if it's exciting or not. Alright, so.

[00:02:42] Moving along here, you mentioned Shopify early on, uh, maybe kind of dig into that a little bit. And again, we're not here to, to dig into anybody that's happened. It's in the news. It's happening right now or has happened right now. We're going to talk about it. Guys, we're not trying to say Shopify is a bad, bad or horrible or anything.

[00:03:02] This just happened and 

[00:03:03] Joshua Warren: looking for the Shopify mug. But that is true. We're just reporting the facts. So Yeah, Shopify was down for about 45 minutes, um, affected quite a few stores, obviously not a good experience. And when I first heard they were down before they had done their post mortem and said why, I was a little surprised.

[00:03:24] And that's because Typically with ecommerce outages fall into one of three buckets. most common are changes were made to the site. So, code was changed, database was changed. Something didn't work the way you thought it would. Second most common is site gets overloaded. There's a spike in traffic, it can't handle the traffic.

[00:03:43] And then kind of the last bucket that is like maybe 1 percent of the time solar flares, which actually that can happen. Um, but typically unexpected errors, like really weird, obscure things we've, in all our experience, we have had cases where like Three weeks after a change was made and everything was working fine.

[00:04:00] If the magic number of people visited the site and seven of them clicked right here, just like those, those really weird. 

[00:04:07] Darin Newbold: Yes. 

[00:04:07] Joshua Warren: really weird cases. I remember that. Well, so, so I was just thinking, I'm like, wait, so Shopify's outage. It was the week before Thanksgiving, so traffic wouldn't be spiking. And Shopify being Shopify, there's no way they're going to be in code freeze So, so I was like, man, I really want to see this postmortem because it had to have been a solar flare. Like it had to have been one of these really interesting edge cases. Turns out it wasn't, turns out they deployed some changes to the entire Shopify code base that, um, and it wasn't like a major update.

[00:04:37] They had told people, Hey, we're going to roll out a major update. They deployed what they felt was a fairly small change, and it knocked Shopify offline. 

[00:04:44] Darin Newbold: So, wow. That's just crazy. And, at some point, just for those listening and, uh, paying attention, I do someday want to come back to the whole Solar Flare thing and maybe get into some depth.

[00:04:55] Because I'm sure Zombie Apocalypse falls somewhere in that category as well. But, around the, the, the code change This is one of those where, how do you define minor? What's minor? I mean, there's, you know, is there really minor brain surgery? I mean, any surgery on the brain is pretty major in my opinion.

[00:05:14] So, I guess, what does that 

[00:05:15] Joshua Warren: look like? Yeah, so in this case, if I understand their, their postmortem correctly, and unfortunately they have, Hidden it behind a login wall now. Um, so you have to be a Shopify customer to read it. But the initial version I saw, um, basically they said that they had changed the name of a database table, but had not changed some of the code that referenced that database table.

[00:05:36] So. To them, it was kind of like, Oh, this is a minor cleanup task. We just need to rename this one thing. Seems fairly simple. I would argue, yeah, that's not minor. That's still brain surgery. Like that's still, if you're changing the database that runs all the stores that run on Shopify, yeah, that's, a big change, but in their mind at the time, and I bet if you ask them now, they would probably say something different.

[00:06:01] But at the time they felt like that was a minor safe change to make. Interesting. 

[00:06:06] Darin Newbold: Well, we obviously know if they'd had a code freeze, this wouldn't have happened, at least not in this moment. It might have happened later on because they would have, could have done it, but the impact would have been significantly different.

[00:06:19] What do you do kind of going forward and then kind of the other big thing is, is, is how important are, are these code freezes besides letting the e commerce manager get some, get a TV timeout? I mean, really, they need to sleep. Come on. But anyway, yeah, so they can get some rest. how are they important?

[00:06:37] Because there, there, there's a, you're going to do code freezes more often than just, you know, before Black Friday or whatever. There's code freezes for major updates for. 

[00:06:45] Joshua Warren: You know, well, that's a little different. Like, that's more of a cut off date. Typically for all the new ideas for the new ideas. Yes.

[00:06:52] Yeah. We'll talk about that here. Um, so like in the Shopify case, if they had and they have now said, Oh, we're going to code freeze at least through cyber Monday, which I don't think is long enough, but they do. So, um, but basically they, yeah. If they had been in a code freeze, this wouldn't have happened then.

[00:07:10] And the reason that's so important, especially for a business like Shopify, is the week before Thanksgiving, this wasn't just you couldn't take orders. This wasn't just that even the customers couldn't access the site. Everything was down. So you also couldn't log in and change your marketing messages.

[00:07:26] You couldn't build out marketing campaigns, edit product pricing, all those things that teams, while they try to plan far in advance every year, There's always some stuff you're doing a week or two before Black Friday to get ready. So it locked out all of those merchants from being able to make those changes.

[00:07:41] And so that's why it was so important. You know, a lot of merchants, um, nowadays it seems like some merchants code freeze in October. typically, unless you're, a little more risky and risk taking, um, if you're a typical, uh, B2C business, you're going to code freeze around November 1st. And that way, It allows your team to be 100 percent focused on preparing the site for Black Friday and Cyber Monday and not worry about maintenance and is the site going to go down, site stability, things like that.

[00:08:11] The other thing is, typically you're going to want to do some capacity planning and make sure your site can handle Black Friday traffic. Um, You're in tears over this outage. Keep going. No worries. Um, this is the fun of a live stream podcast. So, you're going to be doing some capacity planning.

[00:08:30] You're going to be looking at how much traffic can the site handle. That's only valid for that version of the code. So if you then go and deploy an update a few weeks later, all that scalability testing you did, all that prep work you did, well, that's all out the window because the code's changed. 

[00:08:44] Darin Newbold: Well, obviously, uh, I needed to code freeze there for a moment, so Anyway, moving on, moving on down the road.

[00:08:53] So we understand kind of the importance, and we got, hey, what could Shopify have done? And done in a better way, it sounds like you kind of started to touch on it, but maybe we can dig in a little deep, a little deeper on the best practices for implementing code freezes. Kind of how do you look at it as far as how far in front of.

[00:09:15] A major event or a major thing, whatever that's going on, would you plan that? And those kinds of things. And then, sorry, the other thing is the communication 

[00:09:25] Joshua Warren: around it. Yeah. Yeah. Very important. Um, so one thing is we're talking a lot about Shopify. We're talking a lot about Thanksgiving and the, the typical selling season for B2C company.

[00:09:35] For B2B companies, code freezes are probably going to look totally different, and so it's probably not going to be around the holidays. I mean, it can be if you want to let everybody get a little extra rest. Um, a lot of B2B companies I see, though, this can be their most active time of year for deploying changes, and they'll have a code freeze other times of year.

[00:09:52] So I talked to a couple of great clients last week, one of which their busiest season is the summer, so So they actually code freeze for the summer. Um, another one. they have a big event each year in mid September that drives a lot of their sales. They're going to want to have a code freeze probably, mid August through that event.

[00:10:10] And so that's where don't just think, Oh, a code freeze means you don't deploy over the holidays. Um, really what a code freeze means is you're not deploying major changes to your site, um, during times of peak traffic or especially during times of peak revenue. 

[00:10:25] Darin Newbold: Okay. So, but. Two other quick things on that.

[00:10:29] How do you communicate that both internally and externally to the world? That whatever that world, if it's a consumer world or if it's another business world, that's, that's working with that. And then Josh, I'm guessing that innovation and cool new things coming forward. Don't just stop. Do 

[00:10:46] Joshua Warren: they? I mean, no, no, no.

[00:10:47] And you know, I love that's, that's the fun part of this job is the innovation. And so really, um, It's important to communicate, um, especially to your, either your internal development team or the agency you're working with when you want your code freeze to be and then plan around it and realize, okay, this is going to be our last great release before the code freeze.

[00:11:06] Well, then during that code freeze, that doesn't mean that everyone just stops working. That means that you can actually build probably a bigger feature during that time than you would normally build into a release because you know, We're not deploying anything for a little while. We can focus on this major feature and it can be the first thing we deploy after the code freeze.

[00:11:25] So we'll see a lot of, um, a lot of the B2C merchants really queue up a nice big new feature that they can launch in January. Um, when lots of times for a lot of our clients, that's kind of a slow month. Um, so they can get that out the door then. And it is really important when you're planning out your features for the year, developing an e commerce roadmap for the year.

[00:11:44] You put that code freeze in there because I'll see a lot of people say, Oh, yeah, and 365 days. Look at all the cool stuff we can develop. And then I'll say, Well, no, no, no. Call that more like, you know, 300 days, maybe 330 days and know that if you have some really awesome new thing and you time it, you say, I'm going to get it in just before code freeze October 31st.

[00:12:06] This is going live. What happens if the schedule slips and now that feature slips in the next year? And so that's why I would put your more innovative projects, maybe front load them closer to the start of the year away from that code freeze because things happen. Project slip. So yeah, 

[00:12:22] Darin Newbold: and that's That's all around this, this whole code freeze thing.

[00:12:25] It's so important. It's vital to how you do your project. The one last quick thing that I didn't even see in our notes, so I'm throwing this out just cause I know it always happens. At least it seems to happen to us. What about the oh shoot moments? We've code froze, code is frozen, but then there's that thing.

[00:12:43] It's not working or something, those How do you handle that? Is that the hot fix that comes zipping in 

[00:12:49] Joshua Warren: there? So when I was, when I was a young, naive e commerce consultant, I would just say, if you have a really important thing, we can still deploy it as a hot fix. It'll be okay. So don't worry, we'll code freeze, but if it's a really important thing, But all I ever said was a really important thing.

[00:13:07] Well, it turns out, my idea of a really important thing, a CEO's idea of a really important thing, a fulfillment director's idea of a really important thing are all different. So, you have to have a written agreement, written standards to basically say, it has to have this level of severity. And, typically, the way we would define it is we would say, the site has to be down or checkout has to not work.

[00:13:31] Like, it has to be something has happened to that level That's when you kind of get everybody in the room and nuclear submarine style. Everyone turns their key and says, yes, it meets this requirement. We are going to override the code freeze and make this one change. Oh, wow. 

[00:13:46] Darin Newbold: Scenes of a red October came to mind as you said all that.

[00:13:50] All right. Well, um, listeners, folks, we really appreciate you always being here, and we hope that you've enjoyed learning a little bit more and kind of digging into the importance of having that code freeze, taking that time out and making sure that, hey, things are in place. It's working. You get some time to Refresh, reboot, if you will.

[00:14:11] Any last thoughts? 

[00:14:12] Joshua Warren: No, just, uh, hopefully everyone that's listening, especially our B2C brands, y'all are all in code freeze and are enjoying the holiday season. 

[00:14:22] Darin Newbold: Absolutely. Well, if you've had an experience with or without a code freeze, we'd love to hear about it, please share that. Um, you can share that.

[00:14:31] Whether on, uh, Creatuity's LinkedIn page or our YouTube page, we would love to hear from, uh, hear from you and hear your experiences because that could lead to other, other podcast episodes, I'm sure. So with that, we know it's the holiday season. We want to wish you and yours a very happy holiday and a great time with, uh, with family and everyone throughout this entire season.

[00:14:53] And with that, as always, take care and have a great rest of your day.