Eric (0:16) Hey there, and welcome to the office of the IT guy, the show where we celebrate the people and talk about the technology that's changing our world. (0:23) I am your host, Eric, the IT guy, Hendrix, and my mission here is to share a love for open source and help build a stronger community. (0:30) I'm so thankful that you tuned in because the office is now open. (0:53) Hey there, and welcome to the IT guy show. (0:55) I'm your host, Eric, the IT guy Hendrix, And I'm really excited because this is episode number six. Eric (1:01) And took a took a short break there, family vacations, fixed some streaming infrastructure, rebuild my home lab, you know, all the all the usual things that that us nerds do. (1:11) So really excited to be back with you. (1:14) I'm currently as you're watching this, I'm currently prepping for an on-site meeting at work. (1:20) So that's kinda why we prerecorded this episode. (1:22) But really glad that you joined us whether you're with your live, I guess, quasi live, or whether you're catching this episode after the fact. Eric (1:29) Really glad that you you joined in. (1:32) So today, we're gonna be talking about ephemeral environments, and this is something that I really fell in love with over the last few months. (1:41) It's something I didn't really get to enjoy as a systems administrator. (1:45) And so I met Natalie at Kansas City DevOps days, or I guess DevOps days, Casey, just a few months ago. (1:53) And so we we chatted, and and she she gave an interesting talk at at DevOps days. Eric (2:00) And I really connected with it as a systems administrator. (2:03) I won't I won't spoil anything, but, really glad you tuned in. (2:07) And, with that said, let me bring on my guest today. (2:11) Natalie, welcome to episode six of the IT guy show. Natalie (2:15) Hey, Eric. (2:16) Thank you so much for having me. (2:17) It's great to be on your show. (2:18) And as I've told you, I really am a big fan of it. Eric (2:22) Well, I appreciate that. (2:23) You know? (2:23) Five episodes in and half of them, I think, had to, had to do with, DevOpsDate's Kansas City. (2:29) So Yeah. Natalie (2:30) Well, that's that's awesome. (2:31) No. (2:32) That was a great great conference. (2:33) Lots of great content. (2:35) I certainly learned a lot and, met a lot of great people, many of which who have been on your show since. Eric (2:41) Yeah. (2:41) It was funny. (2:42) It's probably the first conference I've been to in close to five years where I was actually an attendee. (2:48) I got to I got to sit in the audience and take notes. (2:50) It was unbelievably refreshing. Natalie (2:55) Oh, yeah. (2:55) Just the whole scope of, like, what what the speakers covered. (2:59) I thought that there was really something for everyone. (3:01) I feel like just like every stretch from things like development on an IC level to infrastructure to, like, adding value, there was it was really, really great lineup, super comprehensive. (3:14) Yeah. Natalie (3:15) Definitely learned a lot. Eric (3:17) Yeah. (3:17) Yeah. (3:17) It was it was great to get out in the community and connect with folks again. (3:20) So, you know, I'd it's not by mistake that I wore my my Casey DevOpsDay shirt today. (3:26) So if you do have a DevOps community in your in your area, definitely go check it out. Eric (3:31) I know Kansas City is is currently rebuilding after after the pandemic kinda blew up everything for everyone. (3:37) So we're we're just starting to get back into a routine of meeting. (3:41) I think think there's quarterly meetups right now, but each, a lot of the major cities across the globe actually have a DevOpsDays community. (3:48) So you might go and check out devopsdays.org and see if there's a community near you and get plugged in. Natalie (3:55) Yep. (3:55) Absolutely. (3:56) They're I've been checking them out. (3:59) There are quite a few. (3:59) Unfortunately, in my local area, there isn't one. Natalie (4:02) And the reason I was curious. (4:04) I thought, like, know, I'm in the Bay Area. (4:05) Like, why wouldn't you have a DevOpsDays community? (4:08) But the reason is DevOpsDays are supposed to be spaced out so that they're no closer than a two hour flight from each other. (4:15) So then that way, everyone has one, I guess, designated tier general geographic area. Eric (4:21) I did not know that. (4:22) You would think something as tightly clustered as the Bay Area, though, would would, be an exception to that rule. Natalie (4:27) Yeah. (4:28) No. (4:28) Exactly. (4:29) Like, if anything is gonna be an exception, that should be it. (4:31) But I guess we have LA and Seattle. Natalie (4:34) So I think those are probably each, like, hour, hour and a half flights from Bay Area. (4:39) So I guess it guess that's where we're supposed to go. Eric (4:44) Well, Natalie, why don't, why don't you introduce yourself? (4:47) Tell us, who you are and what you do and maybe even something you do for fun. Natalie (4:51) Yeah. (4:52) Absolutely. (4:53) So, my name is, Natalie Lundbeck. (4:55) I work at Shipyard. (4:57) We're an ephemeral environments, startup. Natalie (5:00) And right now, I'm, based out of SF, and I do a bunch of things at Shipyard. (5:04) I do, integrations, development, documentation, content, and speaking. (5:10) And for fun, one of the things why I have this this home setup here is I'm actually a volunteer DJ for a community radio station. (5:18) And I every Monday morning, I have a radio show that I do for two hours, and I just play some of my favorite, like, independent label guitar based music. (5:26) And so that's that's been a huge hobby of mine for I've had the show for four years, and I've been doing DJing for five years. Natalie (5:34) Then And in my free time, I also yeah. (5:36) Yeah. (5:36) No. (5:37) It's it's been it's been really cool. (5:38) It's something that I very much look forward to, and it's nice to have, like, a nonwork related side project to keep me excited about something that I love, which is the music industry and local scene. Eric (5:52) So years ago before before I had kids, I was actually a up and coming lighting designer here in Kansas City. (5:59) So I'd go to concerts and festivals, and and I'd run I'd set up and run lighting on the stage. (6:05) It was kind of funny because I love music. (6:08) Adore it. (6:08) There's there's hardly a genre that I won't listen to at least a little bit. Eric (6:12) And so during that time, I listened to all kinds of music, got to interact with with names, big and small, from local Kansas City guys that just hang out and play music to to some some big named country artists and and some pretty cool cover bands as well. (6:28) But it was it was kinda funny because lighting design modern lighting design uses a lot of the same technologies that a basic a basic IT network will use. (6:38) So it's it's like it it was like an adjacent hobby to to my day job at the time. Natalie (6:45) Yeah. (6:45) That's that's really cool. (6:46) I feel like there is so much overlap, like, the type of technical hardware stuff that is often, like, surprisingly underrepresented. (6:54) You don't really see a lot of, people having that crossover, but I think that it's such, like, a important kind of interdisciplinary, relationship right there. (7:03) It's super cool. Natalie (7:04) But that's I imagine that Kansas City I mean, I was there for the first time, in May at DevOps days, but it seems like the type of place that would have an incredible music scene. Eric (7:14) It is. (7:15) It's it's one of the fastest growing music scenes in the in The United States. (7:18) And, gosh, I'd I'd used to know the statistics. (7:22) We were, like, third or fourth behind some of the obvious ones like New York and Nashville. (7:26) And but Kansas City was was one of the fastest growing scenes. Eric (7:30) You you can go out you can go to downtown Kansas City and catch different music each night. (7:34) So it's it's a lot of fun. Natalie (7:36) Well, that's that's really incredible. (7:38) Well, I'm hoping, fingers crossed, for DevOpsDays Kansas City next year and getting to explore the city more and seeing that because that sounds like an incredible thing to be there for. Eric (7:50) Yeah. (7:50) Lot of lot of awesome culture. (7:52) And and if you're if you're listening to this and you happen to stop by Kansas City, feel free to to reach out and and give me a shout. (7:59) I'm always happy to take folks to dinner. (8:01) And and, I mean, that's that's kinda why I do these podcasts is it gives me an excuse to to network with folks. Eric (8:06) So that's how Natalie and I kinda got connected and and decided to to do a podcast episode. (8:13) But why don't we dive into our topic today? (8:16) I could talk about music and technology and travel all day long, but should probably stay true to the topic title. (8:24) So you you and I met at DevOpsDays, and you had not just your not just the company you work for, but also you seem to have a very special place in your in your tech in your tech heart, I guess, for ephemeral environment. (8:41) So why don't we level set with with the audience here and and define what is an ephemeral environment? Natalie (8:47) Yeah. (8:47) So that's a really great question. (8:49) Great way to kick this off. (8:51) So, typically, like, when we're talking about ephemeral environments, I guess, like, until until recently, there wasn't a lot of noninternal content. (8:59) No one had, like, a set definition, but a lot of these orgs kind of have this internal understanding of what constitutes it. Natalie (9:06) So over the past few years, we've just been kind of trying to figure out, like, what should an ephemeral environment have at minimum that distinguishes it from a static staging environment or a test environment, and what are best practices to kind of justify the technical effort that it might take to implement such a system? (9:25) So, typically, like, at least, like, when I'm talking about ephemeral environments, I'm talking about environments that are short lived on demand, but they're full stack copies of your app. (9:35) They're as production like as possible. (9:37) They should have pretty similar production like test data. (9:42) So then that way you can get realistic you you wanna make sure that your your database is pretty well approximated for realistic testing because one of the big use cases for them is end to end testing before staging and, you know, often before production as well. Natalie (9:58) Also be sharing Eric (9:59) a lot before production. Natalie (10:00) Oh, we hope it's I know that it's that's a controversial thing. (10:03) I I know that there there are smart ways to do it, but I I'm afraid to share too many opinions there. Eric (10:11) So short lived, as close to production as possible. (10:17) So what's what's kind of gotten shipyard and yourself interested in in ephemeral environments? (10:23) What was there some sort of market situation that came up? (10:27) Was was it just kind of right time, right place? (10:30) What's, what what's really pushing ephemeral environments right now? Natalie (10:34) Yeah. (10:34) So that's that's a very good question. (10:35) And I think that it's kind of been the ephemeral environments are often a a lift to implement, and I think that's something that we see, really large orgs have, entire DevOps teams, infrastructure teams dedicated to building these out in house. (10:51) And this is obviously great because if you talk to a developer at Facebook or, Google or Apple, they'll say, like, oh, yeah. (10:58) Like, of course, I I have that. Natalie (10:59) I have this. (11:00) I can anytime I create create a a feature feature or or need to solve a bug, I have the infrastructure right there when I need it. (11:06) But for the rest of us, you know, people at at large orgs, medium, even small orgs don't quite have those opportunities. (11:14) So companies like Shipyard exist to kind of exist, like, as a service to offload your environment management. (11:23) So, like, for example, just like a quick pitch, Shipyard, what we do is we take your existing Git repository. Natalie (11:29) If you have a Docker Compose file, that's what we we recommend. (11:33) But also, we work with Kubernetes YAML. (11:36) And we take that and we spin up we transpile it into Kubernetes and we spin up your application in ephemeral environments. (11:44) And it's it's super fast. (11:45) It happens within minutes. Natalie (11:47) It's usually triggered at the pull request level, and then you're able to interact with the environment, run your test suite against it, and then spin it down once you're once you're done with it. (11:57) So just kind of like finding the gap that people there there are never enough staging or test environments to go around when you're working with static infrastructure. (12:05) And I'm sure that you have a lot to say about that as well because that's such a a famous famous pain point. Eric (12:11) Oh, I I promised you a story. (12:13) And so I was probably a systems administrator for close to ten years, probably eight of which dedicated to Linux. (12:22) And I I'm sure I speak for a lot of people out there when I say that my my experience with nonproduction environments was even nonprod is production versus admin. (12:31) You've got a lot of times, depending on where I was at, I'd have dev, test, stage, Doctor, and prod. (12:39) And all of those had to be up all the time. Eric (12:45) Depending on where I was at or what point in the quarter we were in, would determine how available dev was to be taken down and worked on and and had maintenance take carried out, but you never knew who was using it. (12:58) Dev could be pushing a new feature. (13:00) Stage could be doing a load test. (13:02) So you could never just take an environment down or even part of an environment. (13:07) So a lot of times, I had to create yet another environment, which we usually affectionately call the sandbox, to to test out new patches, to to run updates, to build scripts against. Eric (13:20) Our our production environments were never you never touch production because everyone's afraid to, but you also never touch nonproduction, which was incredibly painful as a sysadmin. (13:33) So creating yet another environment was was was oftentimes required. (13:38) The problem was those were usually very small, very underpowered virtual machines. (13:43) And production at the time was usually bare metal racks upon racks of of, you know, Dell PowerEdge servers or h people, h chassis. (13:53) So it was it was never like for like, especially if you had, like, driver updates or something. Eric (13:57) You you never knew what you're getting into until you actually took that outage. (14:01) It's incredibly painful. (14:07) Yeah. (14:08) So one of the things that has kind of grown up since then is is not just virtual machines. (14:13) And now we can just spin up as many VMs as we want, and we've thankfully gotten past the time in in the IT industry where people believe that certain workloads would never be virtualized. Eric (14:26) Gosh. (14:26) Those those conversations were always frustrating. (14:29) My my my database can't be in a VM. (14:32) Why? (14:32) Well, it just can't. Eric (14:34) Anyway, that's that's probably a different topic for a different day. (14:36) But we'll have to have an episode where I bring on sys admins that we can just relish in in the pain of the past. (14:43) But one of the things that one of the trends that has spun up is is cloud, and that's that's been something that wasn't as prevalent before I moved out of IT operations. (14:55) So I imagine with ephemeral environments, you're looking at a lot of cost savings. (14:58) I mean, you talked about spinning up based on a pull request. Eric (15:02) So a lot of CI pipelines probably utilize ephemeral environments. (15:07) Would you would you agree with that? Natalie (15:09) Yeah. (15:10) Absolutely. (15:10) That's one of, at least, like, when I'm using ephemeral environments, that's one of the most valuable use cases that I find personally. (15:17) And, yeah, the cost savings are huge because when you have you have the entire life cycle, really, is automated. (15:24) So they're triggered by get hooks, get events. Natalie (15:27) Sometimes that's a pull request. (15:29) Sometimes that's a push. (15:30) And they should stay, up to date with your single source of truth, repository. (15:35) And, that way, you're only using them for I mean, when you're when you have a test environment, like, you can probably tell me, like, how long you need it. (15:42) But oftentimes, like, I only need it for maybe at most an hour or two. Natalie (15:46) I need enough time to run my automated tests, and I need enough time to, you know, poke around, test things manually, just make sure everything's looking up to spec. (15:54) But when you have a static environment, you know, that that's up $24.07. (15:59) It's extremely expensive. (16:01) And, you know, you're not even really getting your full value out out of that because everyone is waiting for it. (16:06) There's just and there's a lot of a lot of time that it's not being used. Natalie (16:10) But then, like, you know, during the forty hour work week, it's always in demand. (16:14) It can never, like, really serve its purpose. (16:17) You're not really getting a great return on it. (16:19) So when you do, like, the quick, like, napkin math, you find out with, ephemeral environments, you can have more environments and still save about 70% on your cloud costs. (16:30) Of course, varies company to company depending on what workflow you're using. Natalie (16:34) But, yeah, there's there's huge value in having environments only as long as you need them. (16:39) And I think that alone justifies implementing such a system for many people. Eric (16:46) Well, this would be huge for not only, not only my fellow sys admins, but for developers as well. (16:52) I mean, you can really start to build out some of the best practices within IT operations. (16:57) I mean, think about infrastructure as code, especially if you're on a cloud provider like AWS or even DigitalOcean or some of the others that are out there. (17:06) Spin up you can use API calls or scripts or cloud in it to spin up your exact infrastructure, and you can test that as many times as you need to until you get that golden image and then commit all that code into GitHub or GitLab, former GitLabber. (17:21) So it it'd be GitLab for me. Eric (17:23) But but then you hand over those images over to your development team, and they can spin up as many environments as they want. (17:31) Meanwhile, the sys admin can continue to iterate. (17:34) So maybe we've gotten a base image. (17:36) Now we wanna create a an image specific to database workloads, or maybe this image needs to have certain either health care or credit card security requirements, thinking about things like PCI DSS or HIPAA or a countless number of others that you can start to build multiple golden images and get to the this point where your developers can pick that image and then iterate on that even further from, from Python applications to Postgres databases to whatever the case may be and really just start iterating on that and not not step on each other's toes while you're trying to trying to work through that process. Natalie (18:15) Well, I really I really appreciate that. (18:17) That's a very valuable, anecdote for me just hearing the, I IT sysadmin, perspective of it all. (18:24) And I think that, you know, it just goes without saying that we keep finding more and more ways to to use these by by the day. (18:31) So absolutely, very, very valuable. (18:35) And I think that that's yeah. Eric (18:36) That's one of one for free. Natalie (18:38) Oh, thank you. (18:39) No. (18:39) I I I will. (18:40) I will gonna gonna write that down. (18:42) But, yeah, that is that is something that we've been trying to when we're explaining the value, we say that one of the other big aspects of that is the the collaboration and the shareability. Natalie (18:53) Like, that you're able to these they're not meant for wide audiences, obviously, but you're able to share them within Teams, within pairs of people. (19:03) If you have, yeah, something that you need to to run by, if you're working like developer, sysadmin collaborations, you're able to kind of bring each other in and have have shared access in a much, much, much easier way than you would with a static environment. Eric (19:24) So we we actually have a mutual friend that, that we made at DevOpsDays KC. (19:30) His name is Nathan. (19:32) And, you'd you'd mentioned that you wanted to talk a little bit about Dora. Natalie (19:36) Yeah. (19:37) Absolutely. (19:37) So, yes. (19:39) Just to to recap, Nathan Harvey, he's from Google Cloud, he works on the DORA team, DevOps research and assessment. (19:48) And he gave a really incredible presentation, a decade of DORA at DevOpsDays Kansas City. Natalie (19:53) And that was a great, yeah, great great primer on, yeah, DORA metrics have been around for ten years, and they've been, one of the, like, de facto ways that orgs are measuring their, delivery performance. (20:07) Not just their performance, but also their quality. (20:09) And then these metrics also translate into the well-being of your team. (20:13) Because when your developers are pushing fast pushing faster, they're facing less burnout, less frustrations, and, you know, quality is intrinsically tied to that. (20:23) So that goes up as well. Natalie (20:25) So I wanted to talk Eric (20:26) a little bit You you may not sorry to to jump in here. (20:30) You may not know of Nathan Harvey or the Dora team, but I'm I'm pretty sure if you have any interest in deployment and velocity and agile and any of those things. (20:41) You've probably read a lot of their work. (20:43) This team was part of the DevOps annual DevOps report. (20:48) And I think that split off to a different team, but they still do a lot of their own analyses and and have some some best practice guides and that kind of thing. Eric (20:57) So even if you've never heard of Dora directly, because up until DevOpsDaysKC, I hadn't either. (21:03) But having been, you know, formerly at GitLab and then working with working pretty closely with with the development with the developers team at at Red Hat, I do spend a lot of time looking at those reports to see what are the pain points. (21:16) So definitely check out the the the annual state of DevOps reports. (21:21) Sorry. (21:21) Go ahead. Natalie (21:22) Yeah. (21:22) No. (21:23) Those those reports are incredible. (21:24) I found some, just like when I've been doing my own personal research trying to find out, like, what are the pain points with, infrastructure, development. (21:35) There there have been some, yeah. Natalie (21:37) Absolutely great content there. (21:39) And then also, if you're looking for a really, really thorough research that resource that goes touches lightly, but it's very easy to to navigate. (21:50) The dora.webdev website has a great capabilities library of all these different things that you can implement. (21:57) Everything from, like, continuous integration to flexible infrastructure, also, like, a lot of proper team practices. (22:04) Really, really good stuff there. Natalie (22:06) But, yeah, I I wanted to talk about I think that, one of the greatest things about ephemeral environments is I know as we both know, DORA isn't something you really change overnight. (22:16) It's kind of a long tail effort. (22:18) There's there's a lot that goes into it, but we kind of found a way that ephemeral environments do are probably you know, I'm obviously a little biased, but they are one of the faster ways you can move your DORA metrics. (22:33) And do you mind if I give a a quick recap of the the four key metrics? Eric (22:38) Sure. Natalie (22:39) Yeah. (22:39) So the four metrics are deployment frequency, which is how often you push to production, lead time for changes, which is the time, I believe, from which you sit down to write a feature to when it actually hits production. (22:55) Change failure rate, which is how often, one of your deployments causes a regression, and then the time to restore service, which is if there is a regression or failure or bug, how quickly can you, remediate that? (23:06) And, obviously, this is a great resource for, not just your, efficiency, but also just the the quality of your deployments. (23:14) And I think it's really cool that you're able to distill this down so quickly, especially to something that you can kind of get a ballpark estimate of in less than a minute. Natalie (23:24) They have a great a great tool on their website for assessing that. (23:27) But we found out that ephemeral environments are huge impact for all of these because a lot of this a lot of the blockers that prevent people from having, you know what what do they call it? (23:40) Like, elite dura metrics. (23:41) Is that is that the the top performing term? (23:45) Right. Natalie (23:45) A lot of this happens yeah. (23:47) Infrastructure availability. (23:48) And I I feel like you can probably relate to to this, but teams that have true flexible infrastructure according to the Dorametrics website are 23 times more likely to be top performers in Dora, which means that they don't have that human factor. (24:07) They don't have that people don't have to knock on IT's door and say, hey. (24:12) I need the environment right now. Natalie (24:14) The the environment's just there when they need it. (24:16) And that cuts out, obviously, days, sometimes weeks of lead time. (24:22) If you're if you're able to make that infrastructure jump, you're able to write a feature and test it, like, right on the spot as soon as you're done developing it. (24:32) And then if it does if you do uncover any bugs in your tests, you're able to remediate them right on the spot and run your tests again, like, absolutely no downtime. (24:40) So you can see how that kind of impacts all four of those, and you end up, you know, really, really cutting a ton of time out, especially especially when you're able to and I I like to say, like, you're getting the full value out of your CICD pipeline. Natalie (24:57) You've spent a lot of time perfecting it. (24:59) You've spent a lot of time, perfecting your, entire, test suite. (25:03) But, what value is it if you're running it, like, maybe 10% of how often you could be running it. (25:10) So Eric (25:13) it's it's almost like there's there's a method to the madness. (25:16) We we could actually, gain the system a little bit. Natalie (25:19) Yeah. (25:20) No. (25:20) Exactly. (25:21) Yeah. (25:21) That's like, I think it's it's it's very, it's it's nice to know. Natalie (25:25) It's nice to know that, DuraMetrics aren't something that you have to, you know, dedicate six months of dedicated, engineering lift and major budget to resolve. (25:35) I know that it's it's you know, on paper, it it can be it can be tricky, but it's great to know that you have, yeah, something you can absolutely at least improve the general developer experience with, which in turn will such a positive impact. Eric (25:52) So I I wanna I wanna kinda bring this back down to back down to earth a little bit. (25:59) We we've kinda talked about what ephemeral environments are. (26:02) We've talked about a little bit about how they might have changed, like, my experience, and I'm sure that that experience exists for a lot of other folks. (26:12) So we've we've kinda talked in the hypothetical, and then we touched on how do you measure the success of deploying ephemeral environments. (26:19) But what do you have any examples? Eric (26:21) Do you have any anything solid, concrete that that you can point to and show that, yeah, these this worked? Natalie (26:29) Yeah. (26:29) So absolutely. (26:30) So when people are asking me, like, what are some real world examples? (26:35) Like, what are some companies that use ephemeral environments and have talked about them? (26:39) I like to talk about, Uber's internal solution, which they have a great article on their engineering blog. Natalie (26:45) And it's called SLAIT, which stands for short lived application testing environments. (26:50) And SLAIT is essentially, it exists to run tests earlier, more often, and also give developers power over these environments instead of, you know, kind of throwing code over the wall, having this all wait till staging. (27:05) They're able to kind of iterate, find bugs where they happen. (27:10) And, I mean, I I know that this is this is another controversial thing, but shift ownership left, which has been, yeah, very, very valuable. (27:18) But I will I will provide the the link for that. Natalie (27:22) That was a super valuable article. (27:24) And just showing that this is what these orgs who have the time and resources, they recognize the value. (27:30) They recognize just the the huge improvement to developer experience, and they have implemented such systems. (27:36) And, honestly, like, as someone who talks a lot about ephemeral environments, it's super valuable and reassuring for me to see that this is this is having such great attention in the engineering world. Eric (27:52) So are you you said Uber, which was which is kind of on that list of tech companies that a lot of people look at as as unicorns. (28:05) And I I think, like, the other example that comes to mind is, Netflix and Chaos Engineering, a lot of a lot of those big tech companies that have really made a splash in the industry. (28:18) But I think I think we can kinda get caught up in the hype of their story. (28:22) And I imagine there's some misconceptions that that creates. Natalie (28:27) Yes. (28:28) And I think that, in terms of, misconceptions, I think that, one of the biggest ones that I see is there's a lot of buzz right now around, CDEs, cloud development environments. (28:39) And a lot of times when I'm talking about a femoral environment, someone might ask like, oh, like, GitHub Codespaces. (28:45) And I I wanna articulate that GitHub Codespaces, Gitpod, Daytona, they're all incredible pieces of software, and they do absolutely have a place in the the development life cycle. (28:58) You're using them for development for iteration. Natalie (29:00) So what the important thing is that you have your maybe your local dev environment for I mean, I I I develop locally. (29:09) Maybe you then start work working on when you need additional services, integrations, compute, you iterate in your cloud dev environment. (29:18) And at the pull request level, we we like to say that that ephemeral environments are post commit environments. (29:24) So they're typically when you're done iterating at that point, then you would bring your testing earlier, distribute them out to your team, any shareholders, anyone who might need to do code review or product review. (29:37) At that point, that's when you bring ephemeral environments into the the equation. Natalie (29:42) So the the great thing is is is these tools all work together, but there is an important distinct difference in that, yeah, you you use them use them together. (29:53) And then also, like, if, you know, if something goes wrong, if you do uncover a bug in your femoral environment, you should be able to remediate that then and there, and it should rebuild very quickly. Eric (30:04) So putting my putting myself back into the position of of someone who doesn't work for one of the the big unicorns, I don't I I would be terrified to start this process. (30:19) That is just there's so much to build. (30:21) And we've thrown out so many different concepts from, you know, environments on on bill or on push or on pull, infrastructure as code, cloud based environments. (30:34) I mean, just all these things. (30:36) I'm probably this is probably breaking my brain at the moment. Eric (30:39) But it looks like, Shipyard started a website a few years ago, that can kinda help build out this process. Natalie (30:49) Yeah. (30:50) Absolutely. (30:50) So, when I started at Shipyard back in 2021, there like I said earlier, there weren't a ton of non internal resources. (30:58) Like, a lot of a lot of companies do have great internal documentation on doing this. (31:03) A lot of that is me me speculating that they do because everyone you talk to from a large org is like, oh, yeah. Natalie (31:09) This is how you do this, this, and this. (31:10) But there was really limited public facing content. (31:14) So that was difficult for me during onboarding to join this company and not have a solid enough understanding going in to my my interviews, to my my to just starting my job about, you know, what our our main product was. (31:29) So I talked with some of my team members, and we realized that there's there's this this huge gap. (31:35) Why don't we talk from our own personal experiences at Shipyard, what we're trying to solve for, and make this public open source resource guide, ephemeralenvironments.io. Natalie (31:47) And you can check it out. (31:48) That's the that's the link, ephemeralenvironments.io. (31:51) You can also try .com. (31:53) And we kind of established the six pillars that ephemeral should have at a minimum if you're building them from scratch. (32:01) And I think that this has been really and a really incredible learning experience for me just kind of finding you know, like, if you are architecting a system, you wanna make sure that you're you're fulfilling certain requirements for both your developers, but also not just those, but your product, your QA. Natalie (32:21) And, I mean, as as you were saying, your IT and sys admins. (32:25) So just making sure that if you are making this this investment that you're hitting all the major pain points and you're getting your money's worth and also, yeah, doing it doing it the right way. Eric (32:36) So in in femoralenvironments.io, this is a good good chance to point out that, when the show goes live, there will be links, to not only Natalie's work, but some of the, some of the topics that we've mentioned as well as chapter markers. (32:52) So if you wanna go back and and and kinda finish the episode, then maybe review what what we were talking about around DORA metrics, that kind of thing. (33:00) Makes it easy to bounce around. (33:01) So definitely check out the show notes for that that's your that's your free PSA for this episode. (33:07) So at least with ephemeralenvironments.io, you don't have to start this process from scratch. Eric (33:13) It kinda gives you some guidelines, some ideas, maybe cast a little vision, and and probably make it easier to start this process. (33:21) So when when you and I talked, we didn't wanna promote your employer so much as just kind of identify something that you're passionate about. (33:29) And when you're giving your talk at DevOpsDays, you really lit up when you mentioned minimal viable CD that this is this wasn't something you did for your day job, but it was more of a passion project. (33:41) And so while my passion project is talking, You're you're a major contributor to this. (33:50) So why don't why don't you tell us a bit about minimum viable CD and and what it's what it's trying to solve? Natalie (33:56) Yeah. (33:56) Well, thank you. (33:57) I really appreciate this. (33:58) Unfortunately, I wish I could say I'm a major contributor. (34:01) I've I've written one one page on it, which I'm I'm very happy about. Natalie (34:04) It was production like test environments. (34:07) But someday, I'm hoping to be a more more major contributor, but I can absolutely dream. (34:13) But this was a a project. (34:15) Yeah. (34:15) Well, I I appreciate that. Natalie (34:17) It's it's a project that I I respect a lot. (34:19) And when anyone asks me, like, how can I improve my CICD processes or, what might be, you know, what what's a great resource that you have for CICD? (34:28) This is always, like, first and foremost, the first thing that I show them. (34:32) And, this was started by, Brian Finster a few years ago. (34:36) And, when I spoke to him about it, I believe that he said it started as some internal documentation that he and his team had written, kind of setting the expectations for what your CICD should do at minimum. Natalie (34:47) Because this is something that, again, like with ephemeral environments, nobody has really been able to agree upon. (34:53) There's there's a lot of disagreement, like, what your CICD should do. (34:58) Because, you know, in theory, you could have a CICD pipeline but not be doing CICD by definition, which I'm sure you've seen you've seen that that quite a bit. (35:09) And I I really like this guide. (35:11) It's very, very opinionated. Natalie (35:13) I think it's it's very you know, it's it's something for all orgs to aspire to. (35:18) It a lot about what you need in order to do CICD the right way, and one of those things is trunk based development. (35:26) So, ideally, you should be, you know, working off of really short lived branches, pushing to main ideally, like, multiple times per day. (35:33) And then CICD is kind of an applied version of that. (35:38) So if and one of the prerequisites, which I think is so cool, it really ties back so well to ephemeral environments, is you when you have your branches, you should be testing them before merging to main and then testing after. Natalie (35:53) And there there's kind of an issue there because you realize, okay. (35:57) Well, if I wanna test my branch before main, I can do that in staging. (36:01) But then that might not be less than a day of lead time. (36:04) That could be a few days if I have a decent sized team. (36:08) But if I wanna maintain status as true CICD, I'm gonna need a a disposable on demand environment at that point in time. Natalie (36:16) So I like to say that, you know, if you wanna do CICD the right way, it's very smart to implement ephemeral environments because you're able to test things right then and there, have that lead time of less than a day. (36:28) And, yeah, continue continue pushing without that without those blockers to your staging environments or excessive human factors that might slow you down. Eric (36:40) And and and I think about this problem, again, from kind of the operation side where kind of the one of the goals, one of the one of the I don't know, don't know, one of the one of the facets of IT operations that many people strive for is if you have to log into production, you failed. (37:00) I don't know about failed. (37:01) I don't agree with that sentiment, but I I definitely I definitely agree with the concept behind it that you don't wanna have to touch production. (37:11) You should have enough automation, monitoring, and proactive management tools in place that you don't have to shouldn't have to log in, for the most critical of changes. (37:22) And having a true CICD pipeline would would help you get there. Eric (37:25) Now I'm I caveat all of that with the fact that no one is ever going to achieve a 100% of any of these concepts. (37:33) You know, we've talked about AI on this show. (37:34) We've talked about CICD pipelines. (37:37) We've talked about DevOps. (37:38) We're talking about ephemeral environments. Eric (37:41) And looking at these ideas, while it would be great if we lived in a world where you didn't my my ops guys didn't have to manage dev test QA production staging, d r, d r two, all these other environments, but just basically have production and ephemeral environments. (38:01) I I don't know of anyone who has, like, quote, arrived. (38:04) I I don't think there is a a finish line. (38:06) I think every every time you come into work, every time you log in, you look at your environment, go, what change can I make? (38:13) What piece of my job can I automate? Eric (38:15) What what can I build that will make my developers' lives easier? (38:20) Or what's what monitoring piece or or what what automation can I build into my environment so that I'm not getting paged out at two in the morning? (38:33) Some might say I'm lazy, but I just like my sleep. (38:36) Getting paged out is not something I'm interested in. Natalie (38:40) No. (38:41) Absolutely. (38:42) Not not at all. (38:43) So, I mean, it is it is great when you're able to kind of, you know, increase or decrease the possibility of something going wrong with just bugs going through a lot more exposure than they normally would. (38:56) And, of course, like, it is something like you said, it's something that you can't yeah. Natalie (39:01) Getting these things done a 100 of the way can be absolutely daunting, if not impossible for the vast majority of us. (39:09) But I think that when you're able to solve for, like, you know, 70% of it, you see you you see the quality of life improvements. (39:16) You see that you're not getting paged at 2AM. (39:19) You're seeing fewer bugs in production just because every single bug that might have occurred has at least been tested in isolation so you can quickly and easily narrow down to to where it has come from. (39:32) But I do think that there is, just a lot of value in in doing this. Natalie (39:35) And I'm also just like, I I really like this, the idea of leading your team with kind of a mindset of, continuous improvement and just realizing like, kind of benchmarking that. (39:47) Like, you can benchmark that with Dora, then it gives you kind of goals to work towards. (39:51) And, of course, like, the elite metrics are absolutely very difficult, but you can say, like, hey. (39:57) If I cut this out here, automate this, you know, have an environment here or there, for these people, I can get a little bit closer to where I want to be, and it makes it a little bit less intimidating of a lift. Eric (40:11) Oh, for sure. (40:12) So we could go on all afternoon. (40:14) But, in fact, there is an entire section about, developer experience we didn't get to. (40:19) But, I I I think we've covered a lot of ground, and I wanna I wanna kinda capitalize on on our conversation here. (40:27) If we've and and and, I mean, this episode is even called, you know, a quick look at ephemeral environment. Eric (40:33) So what if where where should people go if they wanna learn more? Natalie (40:37) Yeah. (40:37) So, absolutely, I mean, I I'd love to say go to ephemeralenvironments.io. (40:42) And, also, we're always looking for contributors, so we also have a GitHub. (40:47) It's github.com/ephemeralenvironments/ephemeralenvironments. (40:51) So if you have content that you'd love to contribute on ephemeral environments, yeah, absolutely, please please let me know. Natalie (40:58) You can also email me, natalie@shipyard.build, and I'm always happy to talk about that. (41:04) And then, of course, like, we we just talked about minimumcd. (41:07) So that's minimumcd.org. (41:08) That's also an open source community project. (41:11) And, the team there is, there are some really incredible maintainers. Natalie (41:15) They're always happy to help you talk through your content contributions. (41:20) And, I know that, yeah, really, really wonderful people to work with. (41:25) And then I think this this goes without saying, but dora.dev. (41:29) I love this site. (41:30) There are so many guides on it. Natalie (41:32) So much great stuff. (41:34) And also just, like, a great way to benchmark your org's current performance. (41:39) Yeah. (41:39) I think that that that that about covers it for for what I have. (41:43) Do you do you have any any resources? Eric (41:46) I've never had a guest ask me that before. (41:52) I I kinda circle back to to the DevOps conversation we had at the beginning. (41:57) DevOpsDays is an amazing group of folks. (42:00) I feel like probably next to to my LinuxFest crowd that that, you know, I only see once or twice a year at at a Linux fest somewhere in the country. (42:10) DevOps the DevOps community is probably one of my favorites to to spend time with. Eric (42:14) Sorry. (42:15) You know, one of the three pillars of the show is Linux, and we haven't we've hardly mentioned it. (42:19) So I had to I had to sneak that in there. (42:21) But but the DevOpsDays community is amazing. (42:25) I'd I'd feel like I have a DevOps family, you know, people that I only see at conferences. Eric (42:31) I mean, there's a couple of folks that even live here in Kansas City, and I still only see them when I travel to a show. (42:36) So it it it's really a tight knit group of people, and they're welcoming and friendly. (42:43) So if you're not involved in the DevOpsDays community, highly encourage you to do so. (42:48) But so, Natalie, I'd you and I have been connected for a few months, and you you post on LinkedIn from all over the place. (42:57) So you you do a lot of traveling. Eric (42:58) Where where's the best way for for people to get in touch? Natalie (43:02) Yeah. (43:02) So I think in terms of getting in touch, if so I I don't have a a Linktree or a Twitter, but I'm realizing after that, that would be nice. (43:10) But you can look me up on LinkedIn, natalie lunbeck, l u n b e c k. (43:16) And you can also reach me by email, natalie@shipyard.build. (43:21) And I check those fairly frequently, so I'm always happy to to be in touch. Natalie (43:28) But, yeah, always always happy to talk. (43:30) And this this has been really incredible. (43:34) Thank you so much, Eric, for for thinking of me for this. (43:37) And I'm really glad we get to continue the conversation we started at, DevOps days. (43:41) That that was, a really, really great event, and it was awesome to meet you there. Eric (43:47) Yeah. (43:47) It was a great time. (43:48) Really, really a lot of conversations. (43:50) And you mentioned mentioned Leaktree. (43:51) I I just set up one, right after the conference. Eric (43:55) My my previous guest, Laura Santa Maria, was there, from Dell, and, and she and I got talking about DevOps and, how it's a people problem, not a tool problem. (44:05) So catch episode five. (44:06) She actually turned me on to Linktree, and and so now I use it wherever I go. (44:09) So yeah. (44:10) Yeah. Eric (44:11) I don't I don't know about Twitter, but definitely Linktree would is is a great tool. (44:18) So yeah. (44:18) I think I think Nathan's going to be on the show to talk about Dora here here in the near future. (44:25) You connected me with one of your folks at Shipyard to to have an episode. (44:29) So I'm I'm getting excited because episode seven is kind of the the magic number for podcasts. Eric (44:37) If you if you make it past seven episodes, you've kind of arrived. (44:40) So one more to go, and I've I've still got content and guests lined up, including one of the members of the Matrix Foundation to talk about matrix and elements as a as sort of a nonproprietary communications platform. (44:58) Gosh. (44:58) I've got four or five episodes in in my back pocket and looking forward to to bringing those to you all. (45:03) So on behalf of my guests, I'm forgetting to say something. Eric (45:10) See, that's that's why I have an episode guide. (45:15) So if if you just cannot get enough of my sad stories and my pathetic sense of humor, You can also catch me on the Fedora podcast. (45:23) We'll I'll be live here in a few days to talk about boot c, which is bootable containers. (45:28) Might actually be interesting, Natalie, for you to check that out from from an ephemeral environment perspective. (45:33) Basically, being able to build out an operating system within a container and then just basically distribute that that operating system as if it were a container image. Eric (45:41) Might be interesting. (45:42) So that'll be on the Fedora podcast. (45:44) Next episode of this show will be in in two weeks, most likely. (45:48) I don't have a guest nailed down just yet. (45:50) It's kind of kinda weird time of year with with everyone going back to school, but also also live on Red Hat Enterprise Linux presents over there on the RHEL YouTube channel. Eric (46:00) So plenty of places to get connected with me. (46:03) If you if you need those links, just go to itguyeric.com. (46:07) And, Natalie, once again, thank you for joining me. (46:09) I really appreciate your time and and, like I said, continuing our conversation from from from our our conference here in KC. Natalie (46:17) Well, thank you so much, Eric. (46:18) This has been a pleasure, and I'm, really looking forward to catching more of your episodes. Eric (46:24) Well, next episode, you you can skip because, you you you kind of know the the content. Natalie (46:31) But for those Eric (46:32) of you that have made it this far, thank you very much for watching or listening. (46:36) Head over to my YouTube channel, or you can grab the show anywhere that you can get audio podcasts. (46:41) I'm still working on some of the infrastructure, still thinking about maybe shortening up the intro or trying to get some background graphics. (46:48) Little little tweaks here and there. (46:50) I'm, you know, kinda like DevOps. Eric (46:52) I'm never going to arrive at my ideal podcast, but I gotta tell you it's kind of breaking off and doing my own show has has really been kind of a boost that I needed. (47:03) So I'm really excited to do these episodes. (47:05) I just you know, once once I ship my kids off to college, maybe maybe I can do a weekly show. (47:10) But until then, we'll be back in two weeks. (47:12) So on behalf of my guests, Natalie Lempek, and myself, Eric, the IT guy Hendrix, thank you for joining us, and we will see you again next time.