Speaker 1 (0:14) Hey there, and welcome to episode 10 of the IT guy show. (0:17) That's right. (0:17) We are up in double digits now. (0:20) We'll wait on the champagne until maybe episode, like, 25 or so, but I'm really excited you you tuned in. (0:25) I'm excited to have these back on on a biweekly cadence here. Speaker 1 (0:30) So we've got some amazing topics coming up. (0:33) I hope you tuned in a couple of weeks ago. (0:35) We were talking about Lemiri and convergence and how I really, really wish I could take a tablet with me everywhere and then just plug it into a dock and have all the PC things as well. (0:45) It was an awesome episode. (0:46) Really appreciate it, Eric, joining me. Speaker 1 (0:50) Now this week, my next guest is Christian Hernandez. (0:54) He and I know this guy pretty well mostly because he and I worked for the same company for a while, and a lot of emails that were meant for Hernandez came to Hendrix. (1:05) So we exchanged a lot of emails, not on purpose. (1:09) But he's gonna talk about he and I are gonna talk about GitOps and what that means if you're a Linux systems administrator, kinda cut through the hype. (1:17) But we're gonna talk about what is GitOps and as as I like to do as as a marketing guy, we're gonna try and cut through the hype, cut through all the jargon, and we'll talk about what it really is. Speaker 1 (1:29) So with that said, let me bring on Christian. (1:31) Hey. (1:32) Welcome to the IT guy show, my friend. Speaker 2 (1:34) Thank you. (1:35) Thank you. (1:35) Thank you for having me on. (1:36) I'm I'm excited to be here for sure. Speaker 1 (1:39) Yeah. (1:39) This this will be fun. (1:41) When when you pitched the idea, I was like, yes. (1:43) We we need to talk about that. (1:44) We need to talk about it yesterday. Speaker 1 (1:45) So thank you for your patience. (1:46) This has kinda got all the production and everything back up and running. (1:49) But, for those that don't know you, why don't you, introduce Tell us a little bit about who you are and and what you're doing these days. Speaker 2 (1:56) Yeah. (1:56) Yeah. (1:57) So yeah. (1:58) So, like, Eric, as you said, my name is Christian Hernandez. (2:00) I am I'm currently working at Acuity doing community work, right, community management work. Speaker 2 (2:07) But my background really is in operations. (2:10) Right? (2:11) Systems administration, that sort of thing. (2:13) I think that was part of the reason why I wanted to come on. (2:15) It's like, well, like, I'm I'm like, I'm a get ops kind of DevOps kind of application delivery person now, but, like, my background is really like, my heart is in, like, system administration. Speaker 2 (2:25) If you can believe it, I started off with racking, you know, Sun Solaris servers, you know, making network cables. (2:31) And then, like, now I'm doing, you know, Kubernetes and declarative infrastructure with Argo CD. (2:38) So, you know, that's that's I I kind of ran ran the ran ran the gamut from everything from, like, you know, system administration, network administration to now, like, cloud native technologies now. (2:51) It's like, I I recently wrote a book about Argo CD that just got published. (2:56) And so, you know, I I think I think this was a a great opportunity for me to talk to, like, you know, system administrators and, like, operations folks about GitOps and, like, how it fits into into these sort of things. Speaker 2 (3:08) And for those of you are are that are watching this, right, so if you're if you're not listening to this, you can see that there's guitars in my in in my background there. (3:17) So right? (3:18) And I have some soccer scarves. (3:19) You can tell that, you know, in my off time, I either, you know, watching soccer on TV, yelling at the TV, or playing guitar being mad that way. (3:28) Right? Speaker 2 (3:28) So I have to get the get all this aggression out somehow. (3:31) Right? (3:31) So, you know, that's kinda like what what you know, kinda kinda mean in a nutshell and kinda what I've been up to. Speaker 1 (3:37) So what what you're saying is you'd rather automate your infrastructure so you have more time to play guitar and watch soccer than patching the systems at two in the morning? Speaker 2 (3:46) That's exactly it. (3:47) Right? (3:47) I think I think what was the there there was a quote by I think it was Bill Gates or someone that's like, I always hire the lazy person because they wanna get stuff done fast. (3:55) And that's kinda like my my motto. (3:56) Was like, yeah. Speaker 2 (3:57) Let's automate it and get everything done fast. (3:59) Get spend more time guitar playing. Speaker 1 (4:02) Actually, I told a manager that that I was lazy. (4:05) And he's like, no. (4:06) You're you're efficient. (4:07) He's like, no. (4:08) You're you're giving me too much credit. Speaker 1 (4:09) It's like, no. (4:10) I really don't wanna work on this twenty four seven. (4:12) I'm I'm lazy. (4:12) I'd rather just work. (4:14) Yeah. Speaker 1 (4:15) So similarly to to your background, you know, I kinda started out as desktop support, moved into the server side, specialized in Linux around 2011, and have never looked back. (4:27) I couldn't even spell Windows Server if I had to. (4:30) And it's it's been fun to to watch that progression. (4:35) I can remember all the way back to 02/2007 seeing vMotion for the first time. (4:44) The the ability to be running a virtual machine, which was just a complete mind bottle in the in the to to begin with. Speaker 1 (4:52) But to watch it running on one server and then over a few seconds start running on another server and having a terminal window open and running a ping against that server and it never dropping a packet. (5:05) I mean, we went from everything's mainframe to everything's standalone server to now we virtualized systems and now we can move them everywhere we wanna go. (5:14) And then after that, it was, you know, gosh, cloud and then containers. (5:20) Then there was that year where everything was going blockchain. (5:24) Don't don't forget the year that everything had to be green. Speaker 1 (5:27) All of our technology went green for a year, and then we we stopped talking about that. (5:31) So there's there's there's been all these different all these different ideologies, all these different movements within IT over the last twenty some odd years, and it's been fun to watch. (5:43) But there's been one thing that I want to see systems administrators adopt more, and so many of them are like, well, I'm not a developer. (5:53) And and that's that's this idea of GitOps. (5:55) So for start, why don't you why don't you kinda help tear through the the just kind of the bowl, to be honest, and explain what is GitOps? Speaker 2 (6:08) Yeah. (6:08) So and I'm and I'm glad you you put it that way. (6:12) Right? (6:12) Because around I wanna say '20 actually, pre pandemic. (6:19) About twenty seventeen ish, twenty eighteen ish, a few of us from the community, from the Argo community, from the Flux community. Speaker 2 (6:26) So this is, like, very vendor neutral kind of community driven. (6:28) It's like, hey. (6:30) You know, this term GitOps is coming around and a lot of people are just kind of using it misusing it incorrectly. (6:36) Right? (6:37) Also, we see people that are kind of just don't know what it is. Speaker 2 (6:40) Right? (6:40) It's like, okay. (6:41) Well, you know, storing my infrastructure as coding Git, like, that's been happening since forever. (6:45) Right? (6:45) Like, version controlling, you know, either your HCLs or your, like, your Ansible playbooks or in in in my case, way way back in the day, I was using, like, Chef and Puppet. Speaker 2 (6:53) Right? (6:53) Like, using that you know, those versioning those, like, has always been a thing. Speaker 1 (6:57) For that, it was it was your.com files. Speaker 2 (6:59) Yeah. (6:59) Yeah. (7:00) Exactly. (7:00) Yeah. (7:00) Or, like, SaltStack. Speaker 2 (7:01) Right? (7:01) Like, there's this it's always been that kind of infrastructure as code, you know, type of thing. (7:05) So what is this, like, get ops thing? (7:07) It's like, okay. (7:07) Well, like, you know, a bunch of us in the community set out to actually create a school of principles. Speaker 2 (7:16) Right? (7:16) Like a like a suite of, let's just say, axioms, right, for for lack of a better phrase, in which turned out to be the open GitOps principles. (7:23) Right? (7:23) So which is which should say, like, you know, how is a system like, how do I know if I'm running a system in a GitOps way? (7:32) Right? Speaker 2 (7:32) And so there's there's really four principles. (7:36) If you go to opengitops.dev, they're they're they're listed there. (7:42) Right? (7:42) And, you know, they're with with links with further explanation. (7:45) But, like, there's really the four principles is really, really kind of simple. Speaker 2 (7:48) Right? (7:49) We have, like, declarative. (7:50) Right? (7:51) Which says which just says, like, you know, a system manage my GitOps must have its desired state expressed declaratively. (7:58) Right? Speaker 2 (7:58) And so it's it's kind of like, okay. (7:59) That's like that makes sense. (8:01) That's the infrastructure as code kinda thing. (8:02) Alright. (8:02) That that sort of makes sense. Speaker 2 (8:05) And then the second principle that is version and immutable. (8:08) Right? (8:08) And I think this is where this is where I I wanna draw kind of or I wanna kinda, like, double click on this or I'll put a big big red laser pointer on this is that cause it states the principal state that a desired state is stored in a way that enforces immutability, versioning, and retains a complete version history. (8:29) So notice how I did not say get in any of that sentence. (8:32) There's no get in anywhere in that in anywhere in in the principles or anywhere in in the in the definition, and that is done on purpose. Speaker 2 (8:42) Right? (8:42) Because we actually in in in in the community, open GitOps community, when we're doing these these these principles, like, well, we don't really care where you store your your declarations. (8:52) It could be in git. (8:53) It can be in s three. (8:55) Right? Speaker 2 (8:55) It can be in in in SQL lite. (8:58) Right? (8:58) Like, as long as they're stored in a way that in one, enforce immutability and two, has versioning and retains that complete version history, then that's really all that matters. (9:09) Right? (9:11) That it's pulled automatically. Speaker 2 (9:13) So this is kind of where I I draw the differences between kind of like an event based kind of workflow. (9:18) It's like, oh, yeah. (9:19) Like, I make a commit and it triggers this. (9:21) Or I I, you know, I something happens and then something else, you know, triggers another thing. (9:26) And we're saying, well, actually, GitOps is more than just that. Speaker 2 (9:29) Right? (9:29) So GitOps is more than just, oh, wait. (9:31) In anytime I I make a change and it's you know, it triggers something. (9:36) It's like, well, what if that event fails? (9:38) What if you're in a disconnected environment? Speaker 2 (9:40) What if, you know, what if, what if, what if, what if, what if. (9:43) Right? (9:43) Mhmm. (9:43) So which leads to the excuse me. (9:47) Last last principle is is continuously reconciled. Speaker 2 (9:50) Right? (9:50) So it's, like, continuously looking for changes in the desired state. (9:56) So I always I always try to equate the GetOps principles to, like, a thermostat. (10:01) Right? (10:02) So, like, when I first got my you know, like, straight out of college, you know, first first apartment or, you know, your dorm room. Speaker 2 (10:08) Right? (10:08) Like, you have, like, that wall mounted unit, AC wall mounted unit. (10:12) Right? (10:12) Like, you get warm. (10:13) Right? Speaker 2 (10:13) You stand up. (10:14) Right? (10:14) You get warm. (10:15) That's an event. (10:15) Right? Speaker 2 (10:15) Oh, I stand up. (10:16) I go and I turn on the AC. (10:18) AC is running a while. (10:19) Maybe you're standing next to it. (10:21) You know, oh, you know what? Speaker 2 (10:21) I'm kinda cold right now. (10:22) Let me stand up and turn it off. (10:24) That's an event based workflow. (10:26) Right? (10:26) And it works. Speaker 2 (10:27) But GetOps wants to automate a lot of that. (10:32) So it's really more like a thermostat. (10:33) You set your desired state. (10:34) Right? (10:35) You go, I want it, you know, you wanna I want it at 72. Speaker 2 (10:38) And then once you set that, there's a system in place. (10:41) Right? (10:42) The if if if anyone wants to look it up, it's the in in control systems, it's called the control closed loop, right, where it monitors the the ambient temperature. (10:52) If it's too hot, it'll turn on the air. (10:54) If it's too cold, it'll turn it off. Speaker 2 (10:56) And so GetOps is really just that. (10:59) Right? (10:59) Kind of a continuous someone said someone said kind of like a a cron job on steroids. (11:05) And Mhmm. (11:07) Although I think I think that minimizes a little bit what GitOps is doing. Speaker 2 (11:11) I think it's a kind of like a a really good I think it it it's a really good kind of like a starting point where people kind of see that analogy. (11:17) Oh, in case an event doesn't happen. (11:20) Right? (11:20) Because you'll still have event based workflows even in GitOps. (11:24) But, like, in case an event doesn't happen or something happens, right, there is this controller that sits on the system that just basically runs that loop, constantly checks your source of truth, and constantly tries to make your desired state what try tries to make your running state what the desired state is. Speaker 1 (11:42) Mhmm. (11:42) And I I think that's been a problem in the sysadmin community with engineers in particular. (11:47) Just we we get so hung up on the tools. (11:49) I mean, like, even the term GitOps, people get hung up on the Git. (11:53) It's it's really just a desired state and putting checks and balances in place to ensure that you maintain that state. Speaker 1 (12:01) I mean, you you talk about a a manual window unit for AC, you know, and and kids these days have it so easy. (12:08) They just set their smart home thermostat on a on a on a schedule, and, you know, it's cooler at night, warmer during the day, that kind of thing. (12:15) And and I I think IT operations has gone the same way. (12:19) So you and I came up in in kind of the same way up through the industry. (12:25) So how did you go from being a Linux admin to being a GitOps evangelist? Speaker 1 (12:29) Now that we've kind of defined what a git what GitOps is, how how did that work for you? Speaker 2 (12:34) Well, it's interesting. (12:35) And it's actually I I always I always joke and I always said I even said it to you that I I I became kinda like a GitOps evangelist by accident. (12:46) Because on my on my team, you know, on we we we were kind of divvying up kind of responsibilities. (12:54) Right? (12:54) I'm like, oh, hey. Speaker 2 (12:56) You know, I'm I'm really, really good at, like, IT operations. (12:59) Right? (12:59) And so, you know, maybe I'll be like the the Kubernetes operations guy and, you know, I'll I'll, you know, create content about, you know, operationalizing Kubernetes and blah blah blah. (13:08) I was fairly new on the team, and someone else on my team actually raised their hand and wanted to be that guy. (13:14) And I'm like, oh, okay. Speaker 2 (13:15) I'm like, well, you know, that's my forte. (13:18) And I'm like, well, what else can I concentrate on? (13:21) And and and and I, you know, I I realized at the time, and I think my boss at the time was like, well, like, just take care of, you know, whatever, you know, splat ops, right, or whatever ops that goes on a Kubernetes cluster. (13:35) And so I think, okay. (13:36) Well, what's the you know you know, how do people and then then I stepped back. Speaker 2 (13:40) Right? (13:40) And then I stepped back. (13:41) I'm like, well, how do people really man how would I really manage a Kubernetes cluster leveraging everything that Kubernetes gives you? (13:49) Right? (13:49) So, like, you know, Kubernetes gives you immutable infrastructure, gives you, you know, automatic orchestration of workloads. Speaker 2 (13:55) Right? (13:56) Like, there's a lot of that stuff's taken care of for you. (13:58) And so at that time, you know, this new project was coming up called, you know, Argo, the Argo project from Intuit. (14:06) Mhmm. (14:07) And it's something that I joined. Speaker 2 (14:08) And the first time I saw Argo, the first thing I thought was this tool does everything that I wish I could do as a system administrator. (14:17) Right? (14:17) So looking back at, like, my system administration days, a lot of stuff that, you know, we do now with Kubernetes, we do now with Argo, we do now with GitOps is things we're just trying to do all along. (14:28) Right? (14:28) Let's say, okay. Speaker 2 (14:29) Well, like, you know, I wanna create, you know, I wanna create clusters And those clusters, I want them to be very, very similar or almost identical. (14:37) Right? (14:37) And so, you know, you're creating VM images. (14:40) You're creating, you know, you know, in in, I guess, in some in in some cases, making sure all the servers were, like, in the right patch set. (14:48) And then the exact same version of, you know, of of whatever Linux you're you're deploying and then, you know, the same version of PHP or, you know, whatever. Speaker 2 (14:56) Right? (14:56) Like, whatever stack you you want and, you know, just trying to do all that. (15:02) Doing that with, at that time, with the technology at the time was very, very difficult. (15:07) Right? (15:07) Because you always get configuration drift. Speaker 2 (15:09) Mhmm. (15:09) You know, there's people making out of band changes. (15:12) You know, one server inevitably, it's in a different patch set. (15:16) Right? (15:16) Or you think it's the same patch set, but this one has a dash one to it and the other one has a dash five or whatever. Speaker 2 (15:21) Right? (15:21) You're like, oh, there's just, like, something, you know, minuscule that is that is different that causes really a lot of headache. (15:29) Right? (15:29) And so when I saw our leveraging everything that Kubernetes gives you, but, like, in a uniform way, I thought, oh, okay. (15:38) Like, then I started applying kind of the same principles that I was trying to do a long, long time ago to to you know, using Argo CD. Speaker 2 (15:47) Right? (15:47) So using, you know, a GitOps tool. (15:49) Right? (15:49) And so I became by accident because it's like, well, there's some other guy that's more senior than me that's gonna do the Kubernetes admin kind of stuff. (15:59) So, like, what can I concentrate on? Speaker 2 (16:00) And I remember trying to deploy applications. (16:03) I'm like, okay. (16:04) Well, how how do I do this effectively now leveraging all of that? (16:08) So that's kind of how you know, my journey is really like, you know, okay. (16:12) I'm a I'm a sys admin. Speaker 2 (16:13) How do I apply those same things now with these newer tools, and how do I leverage these newer technologies to, you know, kind of do the things I've always trying to do, technology's finally caught up. Speaker 1 (16:25) Yeah. (16:26) I'm throughout my career, I'm I've seen different administrators try and solve those different ways. (16:32) Like, I mean, VM templates were were a huge boon, but then you always had to remember to keep your VM template up to date. (16:39) And how many of us did that? (16:40) I very rarely. Speaker 1 (16:42) Of course, the sixth or seventh time that I've tried to build a VM off of an outdated template, it's like, I really need to update this. (16:49) Instead, I'm running the same 16 commands trying to update everything or forgot to install that that security tool that we just added last quarter, you know, just different things. (16:58) And it's it's been funny to watch that progression because I went from that and and and storing .com files and and gits. (17:10) At one point, I thought I'd licked it by install by inserting symlinks from the operating system to a git repository that didn't work out very well with tools like SELinux or just Yeah. (17:22) Apparently day to day operations. Speaker 1 (17:25) Yeah. (17:25) And then in the in the Puppet, SaltStack was my first my first automation love. (17:31) And and then I came across Ansible and and fell in love even more. (17:35) And so nowadays, I tell people that YAML is my my preferred scripting language just because anything that I need to do on the Linux system, can do via an Ansible playbook. (17:47) And, yes, while it takes a lot of effort, how much effort does that save me down the road? Speaker 1 (17:51) Like, I'm I'm in the process of rebuilding my home lab to support multiple different distributions, multiple different versions for my day job where I'm doing product marketing. (18:01) And, so being able to update packages as needed, being able to, have an SSH key deployed, alongside a virtual machine. (18:13) Because as as a systems administrator turned marketer, I'd I spin up and tear down systems all day long. (18:20) And it's it comes with it with this effort comes a a a shift in how you think and the way you look at systems. (18:30) In fact, a lot of a lot of infrastructures code experts in the field tell you that if you have to log in to production with an account, you've you've failed. Speaker 1 (18:39) You've missed the mark. (18:41) And so as as you've made this accidental journey Mhmm. (18:44) What what are some of those minds mindset shifts that you've seen that that a a systems administrator needs to undergo before they can really embrace this this more ephemeral, more automated, more resilient approach to their infrastructure? Speaker 2 (19:00) Yeah. (19:01) I think the first mind shift and I actually made the mind shift actually really, really early on because I've always been I've oh, you know, in in my past as a as an administrator, right, on a small team, I was always looking at newer technologies. (19:17) And it's like, there has to be a better way of doing this because, like, you know, constant pain. (19:21) Right? (19:21) It's like, just has to be. Speaker 2 (19:22) Mhmm. (19:22) So I did so I think the first like mind shift that that I think a lot of people went through as people still kind of go through this is like the differences between like a VM and a container. (19:33) I think that's a very, very important distinction because, like, a lot of people say, oh, yeah. (19:38) Container's just like a small VM or a more portable VM. (19:40) It's like, actually, no. Speaker 2 (19:42) They're like you know, conceptually, yes. (19:44) Like, conceptually, yes. (19:45) But technically speaking, you can get a lot of trouble because you're running multiple containers on a Linux host. (19:51) They all share the same kernel. (19:53) And, you know, you get a vulnerability in the kernel. Speaker 2 (19:55) Now they all have it versus, like, in a VM, you're actually the the tendency is actually you know, you're fully virtualizing the kernel. (20:02) So you're running separate kernels. (20:04) So I think the that that's kinda like the the first shift, I think, is, like, the containerization aspect of, like, you're you're instead of, like, spinning up entire VMs for an application, you're isolating the application process. (20:16) I mean, like, oh, okay. (20:17) Like, this a a lot more efficient use of resources. Speaker 2 (20:21) Right. (20:21) You know, with with the trade off of, obviously, you know, you're not really, you know, running the different kernels and blah blah blah. (20:28) Right? (20:28) Like, there's trade offs there. (20:29) So but but you get, like, kind of, like, that that density. Speaker 2 (20:32) Right? (20:33) I think another mind shift also and I think I think you touched on it, but I think that's a very, very important mind shift is, like, okay. (20:39) Like, with with automation and things like that, if you are SSH ing into a host to make changes and you're doing, like, SSH loops. (20:48) Right? (20:48) Like, you're, like you're you're doing it wrong. Speaker 2 (20:50) Right? (20:50) I think that's, like, another thing is kind of, like, you know, at that time, you know, SSH I love SSH, by the way, but SSH I I've done, like, SSH, like, reverse tunnels and tunneling in here. Speaker 1 (21:03) Like, it's Speaker 2 (21:03) like a it's it's it's like a really nice Swiss army knife thing that does everything. (21:07) But, again, like, you know, everything looked like a nail, right, with SSH. (21:10) So I was, doing SSH loops and things like that. (21:12) So I think that's the first thing. (21:13) TMUX. Speaker 2 (21:14) Yeah. (21:15) Yeah. (21:15) TMUX, SSH. (21:16) Yeah. (21:16) Like, running, you know, commands and doing loops and and, you know, in batch scripting. Speaker 2 (21:21) But I think the the paradigm shift of, like, no. (21:23) Like, you know, I I'm I'm managing it from, like, a central, you know, d these you know, that like, it it became less about you know, they say pets versus cattle. (21:32) Right? (21:32) Like, they it's like, no. (21:33) Like, you know, these are just, you know, machines that run workloads. Speaker 2 (21:37) Right? (21:38) Like, I I should be able to tear them down and spin them up fast and efficiently. (21:41) That's like another one. (21:44) The one after that actually, I think is a bigger chasm. (21:47) It's like, you know, Kubernetes, you know and, you know, I'm a Kubernetes guy now. Speaker 2 (21:52) But, like, Kubernetes kinda says, like, okay. (21:56) I'm gonna take your SSH access away. (21:58) Like, not not even you shouldn't SSH, like, can't even SSH. (22:01) Right. (22:02) So, like, you get get you know, that's like a bigger chasm. Speaker 2 (22:05) Right? (22:05) Like, that's like I remember early on in in early on the Kubernetes days, I was, like, kind of one of the things when I was working with, you know, with various clients. (22:15) They're like, okay. (22:15) Well, like, you know, you know, they set up SSH keys. (22:18) I'm like, oh, no. Speaker 2 (22:18) You don't understand. (22:19) Well, there is no SSH ing. (22:22) Right? (22:22) And that's like a, you know, big paradigm shift in in a big, you know, kind of, you know a bigger chasm, I think. (22:30) And and I think the last paradigm shift and, like, you know, I I think it's a shorter jump, But it is a jump, like, going from, like, Kubernetes. Speaker 2 (22:38) Right? (22:38) Be you know, running kubectl or using, like, the APIs or something. (22:41) Because those are still very imperative imperative to running in a in a fully GitOps automated way where now now not only do you know how to have SSH, I I'm taking away KubeCTL away from you as well. (22:55) So, like, you're you're you're adding these layer abstractions. (22:57) Right? Speaker 2 (22:57) Like, you know, you're you're you're fully layering on. (22:59) It's like, you know, not only do you not, you know, have console, you don't have SSH, you don't have kubectl anymore. (23:06) Now, like, in a in a in a in a GitOps way is, you know, now you're you're treating Kubernetes clusters as cattle. (23:13) And you're just kind of like, you know, you get this Kubernetes cluster, you point it to this Git repo, that cluster goes away. (23:19) Okay. Speaker 2 (23:19) That's fine. (23:19) I'll just spin up another one and point it to the same Git and, you know, repo and it, you know, it's you know, Argo CD, you know, kind of takes it from there. (23:27) So that's kind of like I I wanna I don't wanna say steps to GitOps, but those are kinda like some of the paradigm shifts that, like, I myself had to make one. (23:35) And two, if those that are, like, moving towards more a GitOps friendly way or GitOps approach of managing systems, you know, you need to start, like, thinking about those things that are like, okay. (23:45) Taking away access for something else. Speaker 2 (23:48) Right? (23:48) Something else or something better. Speaker 1 (23:51) Is is that one of the reasons why sysadmins and traditional Linux organizations in in your opinion are so hesitant to adopt this kind of workflow? (24:00) I mean, if you think about it, you you you kinda left when when I was talking about SSH and TMUX. (24:07) Like, I used to have a bastion host that all it did was was basically a landing place for all my SSH sessions into my entire environment, which looking back, the the security administrator inside my brain is saying, why did you do that? (24:19) You've done it. (24:19) Yeah. Speaker 1 (24:20) But, I mean, that's what it was. (24:21) So, you know, I had a VPN, a super slow connection. (24:23) So just, you know, SSH into your bastion host, and then you've got terminal sessions in TMX for everything else. (24:29) But you you go from that and you go from control, and I'm I've I've had root access for years. (24:35) And and then when root fell out of favor with sys admins, then everything was in sudo. Speaker 1 (24:40) And and and and so you go from that in a sense of control and a sense of having your fingers on every server, you know, of course, the pets analogy there, to a lot of organizations now, like you said, don't even use kubectl, Kubectl, don't even do sudo d n f. (24:59) Now it's now whether you're talking about Linux or Kubernetes, I mean, I've I've seen a lot of systems administrators that are heavily invested in this space that can run a multi, cluster Kubernetes enterprise, you know, hundreds of nodes, dozens of clusters with nothing more than Versus code and, you know, a a YAML linter? Speaker 2 (25:23) Yeah. (25:24) It's it's I it I I seen it I see it as it's definitely a a big a big rock to move, I guess, or a a big, you know, a a big cognitive load. (25:33) Right? (25:34) I because because I I do remember. (25:35) I I do remember going back, you know, first adopting, you know, something like Chef. Speaker 2 (25:40) Right? (25:41) So, like, you know, Chef puppet. (25:42) First adopting that, it's like, well, I don't want things to make changes without me, you know, like, without me doing them. (25:49) Right? (25:49) It didn't fail it didn't feel like I was doing them. Speaker 2 (25:52) And that's kind of like the first paradigm shift or the the first struggle. (25:56) Right? (25:57) Because a lot of the times because and and and this is even true today with with with running Kubernetes, but, like, you know, even more so as like a I'm gonna call it old school Linux admin. (26:09) Right? (26:09) Like, kinda what I was is is like any change to the system, like, I am responsible for. Speaker 1 (26:15) Right? Speaker 2 (26:15) And so when I have all these tools making changes, right, like, you you can't you can't really blame computers. (26:22) I guess with AI, maybe you can one day, but but but now, right, before that time, right, you know, it it's kinda like, you know, something happened. (26:33) You know, my phone is ringing. (26:34) Right? (26:34) And so when I don't have that, you know, level of control, it it it is, you know you know, scary. Speaker 2 (26:40) Right? (26:40) Essentially, it's like, you know, I run one command and I'm touching, you know, 10,000 servers that, you know, I'm, you know, hunt you know, I'm I'm affecting hundreds of developers and, you know, one, you know, missing semicolon or, like, I guess, in the case of YAML, one, you know, one space too many. (26:56) Right? (26:58) You have which which you said, please, yes, do use a linter on Versus code. (27:03) It just saves so much time. Speaker 2 (27:04) But you that that could produce a lot of phone calls. Speaker 1 (27:07) One malformed for loop, all of a sudden you spun up a thousand, you know, EC two instances on AWS. Speaker 2 (27:14) Yeah. (27:15) Yeah. (27:15) Spot instances at that because so then you try to delete them and they keep coming back is yeah. (27:19) So you should have a war story panel one day as a party guys have a meeting. Speaker 1 (27:24) A lot of fun. Speaker 2 (27:26) So so yeah. (27:27) No. (27:28) It's it's it's I I I think that is that is that is that's a rock. (27:33) Right? (27:34) That that you kinda have to, like, boulder that you have to, like, kinda move out of the way. Speaker 2 (27:38) And and and you have to do it. (27:40) Right? (27:40) And you have to do it in, you know, in order to manage, you know, tons of systems. (27:44) It's like before, maybe you can get away with it. (27:47) But now everything's about scale and do more with less, that sort of thing. Speaker 2 (27:51) You you have to adopt these tools. (27:53) Right? (27:53) And and like and and like you said, you know, I don't wanna spend all my time managing systems. (27:59) Right? (27:59) Like, the system should be managed and I can go ahead and go and work on other things. Speaker 1 (28:03) Mhmm. (28:03) Well, and, I mean, in in the mid two thousands, you know, 20 in the mid twenty teens or so, I manage I was one of three Linux systems administrators in a medium sized business and we maybe had 600 Linux servers if you counted dev, test, stage, production, Doctor. (28:20) Mhmm. (28:20) And so, I mean, you figure that was two two hundred and fifty Linux servers each if if you, you know, split up the environment per administrator. (28:30) But now I I see one or two Linux administrators per thousand servers or more. Speaker 1 (28:36) You know, I've being in the enterprise vendor space for the last six, seven years, I talked to administrators that have tens of thousands of servers spread across the globe, and there might be five or six people. (28:50) And so that scale is unbelievable. (28:54) And, I mean, just the times where you had to go and change one argument in one config file across 600 servers, that that was a lousy day. (29:04) Thank you, TMux and the team that brought it to us. (29:07) But but I'm I'm really thankful for that time because it really gave me an appreciation for newer technologies, things like Kubernetes or infrastructures code tools. Speaker 1 (29:19) But, you know, I don't know what I would do without that experience of having manually administrate administrated hundreds of servers by hand. (29:30) So what about what about the the up and coming Linux systems administrator? (29:34) What about the guy today or or gal that that is in the space now, and they've got 10,000 nodes, and they have one other person on the team, And they're they're constantly fighting fires, and and they they hear what you're saying, Christian, and they're like, oh my gosh. (29:51) I I I knew there was a better way. (29:53) I didn't know what it was. Speaker 1 (29:54) But now this guy, Christian, is talking about it. (29:57) What what tools should should they use to to kinda get their feet wet? Speaker 2 (30:02) Yeah. (30:03) As a as a sis as a Linux system administrator, right, like, we're we're talking about that, you know, I think I think thinking about the GitOps principles abstractly, right, like the way the way they're presented is kind of a good map. (30:21) Right? (30:22) So, you know, the the these principles are and and, again, open getops.dev, you'll see what what what I'm talking about. (30:30) But the those those principles are written in a way that is, like, that is open. Speaker 2 (30:37) Right? (30:37) So you don't have to you know, it it it doesn't say anything about tooling. (30:41) Right? (30:41) It just kinda just says about, like, practice. (30:43) Right? Speaker 2 (30:43) And so that's kinda like one Speaker 1 (30:45) I do think that's important that the ideology and the technology are separate. (30:50) As engineers, we get obsessed with our toolkits. (30:53) So I I think that's a huge boon to to to that ideology is not having prescribed any tools. (31:01) But go ahead. (31:01) Sorry. Speaker 2 (31:02) Yeah. (31:03) Yeah. (31:03) No. (31:03) And and and that and that's, like, actually the point. (31:05) Right? Speaker 2 (31:05) So then that way, you know, if you're a Linux administrator, right, like, you know, you know, whoever's listening is Linux admin getting their feet wet, right, with with administration is, like, you have to look at, you know, tools that enable these things. (31:17) Right? (31:17) So I think you mentioned Ansible. (31:20) Right? (31:20) That's like I I was a I was a big big Ansible user as well. Speaker 2 (31:24) Terraform is is another one that you you can or OpenToFu, whichever one. (31:28) Right? (31:29) But, you know, functionally, they're the same idea. (31:32) Right? (31:32) I think those two those two tools are are are really, like, the the foundational tooling that that is needed in order to, you know, follow some of these principles. Speaker 2 (31:46) Right? (31:47) Some other things is that I think that you know, so one for, like, in certain in terms of tooling or, like, one, in in terms of, like, philosophically, right, like, open getops.dev, I think just looking at those principles, reading through them, reading through the glossary. (32:02) Notice we don't mention any tools. (32:04) And then as far as tooling, if you're a Linux admin administrator, you're gonna look at something like Ansible or Terraform. (32:10) But, also, you're have to look at something I think every Linux administrator should just learn to use Git. Speaker 2 (32:18) Yes. (32:19) And and and and that's not to say, like, I have any love affair with Git. (32:23) It's just it seems to be the industry standard. (32:26) Right? (32:26) Like, I've used, like, SCCS, RCS, Subversion. Speaker 2 (32:29) Like, I'm sure there's, you know, someone out there is that that there's a better tool other than Git. (32:35) I probably agree with them. (32:37) They're probably right. (32:38) But, you know, you need to learn to speak kind of almost the same language as the people. (32:43) Like, if you are supporting developers. Speaker 2 (32:47) Right? (32:47) Like, they're gonna be using Git. (32:49) Right? (32:49) And you're gonna be hiring people, and they probably know Git. (32:53) Right? Speaker 2 (32:53) Like, it just happens to be an industry de facto standard. (32:56) Right? (32:56) It just happened to be that way. (32:58) So I I think, like, in terms of tool adoption, I think at least, you know, you know, you don't have to be a genius in GET, but you have to, like, know enough to, you know, be able to, you know, like, roll forward, roll back, Speaker 1 (33:11) you know you know GetResults is a good one. (33:13) If Speaker 2 (33:13) Yeah. (33:14) Get get Speaker 1 (33:14) get If you're dangerous with git. Speaker 2 (33:16) Yeah. (33:16) Yeah. (33:17) Yeah. (33:17) If you wanna get, like, git stash or, you know, like, just kinda just you just start start learning git because even if you're, like, storing, like, bash scripts, like, that it's even useful for that. (33:26) Right? Speaker 2 (33:26) Like, even if you're just wanting even if you're just doing bash scripts. (33:30) Right? (33:30) Or like like me in my past, I did because I came from a Solaris background. (33:34) I did a bunch of corn scripts, but that's, you know, the that's kind of the, you know, kinda like baseline kind of things to look into. (33:44) You know, people right away want us to start saying, oh, look in containers and Kubernetes. Speaker 2 (33:48) I'm like, oh, not every workload requires Kubernetes. (33:51) Not every workload requires containers, but you can adopt some of these these ideas and ideologies to better help your Linux administration. (34:00) Like and and I say that as a as a as a Kubernetes guy. (34:03) I say that as a containers guy. (34:04) I say that as a GitOps, you know, Argo CD guy. Speaker 2 (34:06) Not Like, every workload requires it. (34:08) Right? (34:09) Not not everyone's gonna run, you know, Kubernetes cluster. (34:12) Like, that's just not gonna not gonna happen. Speaker 1 (34:17) That's it. (34:17) Yeah. (34:18) Ansible, Git, Terraform. (34:21) You mentioned OpenTofu, which is sort of an open source fork. (34:25) If I remember correctly, Terraform changed the license and so there was a fork of it. Speaker 1 (34:30) But there's still to this day more or less feature compatible. (34:35) I think OpenTofu and Terraform are starting to diverge a little bit. (34:39) But Yeah. (34:40) So if if you hear OpenTofu out in the community, that it's it's it's an it's an answer to Terraform licensing changes. (34:50) So so I I know you're a big Kubernetes guy, and I imagine a lot of Linux systems administrators out there are also being handed Kubernetes clusters, which funny enough is kind of the same way I got into Linux administration was I had a stack of active directory enabled Windows systems. Speaker 1 (35:08) And if I never see another group policy objects, I I will die a happy man. (35:13) But then someone was like, hey. (35:15) We've got these half dozen servers. (35:17) They don't have a web interface or a GUI or anything. (35:21) Maybe you can figure out what to do with them. Speaker 1 (35:22) Like, oh, this is a lot more fun. (35:24) But, I mean, that's kinda how I started in my journey. (35:29) But I I moved to the vendor side before that happened with Kubernetes. (35:34) It's like, hey. (35:34) I know you've got this pile of Linux servers, but now we've got this weird Kubernetes thing. Speaker 1 (35:38) I don't I don't know helm charts, all this kind of stuff. (35:42) I I don't know. (35:43) Can you can you fix it? (35:44) Can you do stuff with it? (35:45) So I imagine there's a lot of administrators that find themselves in that position because it it seems like some organizations, anything that had a power cord on it ended up under the purview of the Linux systems administrator. Speaker 1 (35:57) Looking at used storage, backups, networking, tape drives. (36:04) It's like, this this is IT gear here, admin. (36:07) You you go fix this. (36:10) Okay. (36:12) Anyway, may maybe some old wounds to discuss at a at a Speaker 2 (36:17) few Yeah. (36:17) Yeah. (36:17) Exactly. (36:17) But Speaker 1 (36:19) so I I imagine that there's administrators out there that have been handed the keys to Kubernetes cluster. (36:26) So much like much like you're talking about getting into tools with GitOps, what about what about the Kubernetes? (36:33) Because I know you're a big Kubernetes guy. (36:35) So how how would I, as as former Linux systems administrator, dip my toes into Kubernetes? Speaker 2 (36:42) Yeah. (36:42) So my favorite so my I'll tell you my favorite tool or my favorite thing to, like, mess around on Kubernetes is is a tool called KIND. (36:53) Right? (36:53) So it's a KIND. (36:54) It's an acronym sense for Kubernetes in Docker. Speaker 2 (36:58) And and, basically, is, like, you take Docker desktop, right, or or Podman desktop, I think, orb stack, I believe, works. (37:06) And whichever one that Suse created or Rancher created, I forget they created one. (37:10) Oh. (37:11) I forget what it's called. Speaker 1 (37:13) There's Yeah. (37:14) That that one. Speaker 2 (37:15) That one. (37:15) Yeah. (37:16) It well, that that one also works. (37:17) But, anyway, basically, you you run, you know, kind create cluster and it creates a Kubernetes cluster inside of Docker. (37:26) Right? Speaker 2 (37:26) So this gets you kind of, like, local kind of setups where you can, you know you you talk about HomeLabs. (37:33) You can spin up and test all kinds of various scenarios. (37:39) Right? (37:39) And so the team behind Kind is actually actually uses Kind to run regression tests, right, to run tests. (37:49) So anytime the the actual Kubernetes actual engineers, It was born from in inside of Google, but the kind was basically, you know, when they run their pipelines or when they create a new version of Kubernetes, they actually spin up a kind cluster to run tests. Speaker 2 (38:05) So it's it's it's it's actually is pretty robust. (38:08) There's other tools called there's a one called k k three d as well. (38:13) That one that one k three d they they wrote k three d because it's, a smaller footprint because they don't use ETCD. (38:25) Right? (38:25) If you don't know what ETCD is, the more of a reason for you to use Kine to kind of kinda investigate what I I I'm giving out homework now. Speaker 2 (38:34) But I think I think I think getting your getting your feet wet with, like, Kubernetes, you know, starting off with, like, with Docker and then running Kine so that way you can spin up and tear down Kubernetes clusters and run a test against that. (38:47) I think becoming and and you said this before, but, you know, I think I think it's worth repeating just becoming fluent in YAML, I think. (38:57) You know, you're you're you're you're you're moving away from, like, you know, running bash scripts, and now you're kind of just, like, declaratively saying, here's a container image. (39:05) Run it on these, you know, these, you know, group of clusters. (39:10) You know, getting really, really familiar with YAML, I think, is is a is is a good way to go if you wanna start, like, adopting and start getting, you know, really, really good with Kubernetes. Speaker 1 (39:21) I think YAML gets a bad rap. (39:22) I I wanna catch flack for this. (39:24) But Mhmm. Speaker 2 (39:25) You know what? (39:26) I'm there I'm there with Speaker 1 (39:27) numbing me infuriating? (39:29) Yes. (39:29) Yes. (39:30) But does it do a really good job? (39:31) Is it readable? Speaker 1 (39:32) Yep. (39:33) Yes. Speaker 2 (39:33) I actually prefer it over JSON. (39:35) I don't know. (39:35) People think I'm crazy as well. (39:38) Probably developers because they're used to doing I mean, it's so much easier to do a JSON payload and API call. (39:43) I get it. Speaker 2 (39:44) But, like, I'm not doing that, and the rest of us isn't doing that. (39:47) Yeah. (39:47) Almost so much easier to read. (39:49) Right? (39:49) And now that we have linters on Versus Code Yes. Speaker 2 (39:52) You know, even even if using, like, NeoVM, there's plug ins. (39:55) Like, there's some more I don't think there's any more excuses. (39:59) Yeah. (40:00) I agree. Speaker 1 (40:02) So I I want to you're you're you're big on on experimenting and, you know, trying things out in in code. (40:09) So I I kinda wanna wrap up today's conversation with with something I I want to I've been wanting to experiment with. (40:16) I wanna call it a lightning round. (40:18) I'm just gonna hit you with a few questions you just answer, and and we'll may maybe I maybe I'll get some music underneath it or something in in the future. (40:25) Yeah. Speaker 1 (40:26) But let's Speaker 2 (40:27) try this song and a timer. Speaker 1 (40:28) Yeah. (40:29) Right. (40:30) Although, sadly, I already asked you the first question. (40:35) So the first question was YAML, friend, or foe. (40:37) I think you'd say friend. Speaker 1 (40:38) So I I only have three questions in this lightning round. (40:41) My my bad. (40:43) Alright. (40:44) So favorite Linux distro for getting work done? Speaker 2 (40:47) Oh, that is Fedora. (40:50) I've been a big Fedora user since Red Hat nine, so I've been on it for a while. Speaker 1 (40:54) Fedora. (40:55) Love it. (40:55) Alright. (40:56) C l CLI tool you can't live without. Speaker 2 (40:58) CLI tool. (40:59) Oh, that is hard. (41:01) V I Vim. (41:02) Vim v I. Speaker 1 (41:04) Love it. (41:05) A GUI that doesn't suck. Speaker 2 (41:07) A GUI that doesn't suck. (41:08) I have always liked GNOME. (41:11) I know I'm probably in the few, but I've always liked GNOME. Speaker 1 (41:15) I don't know. (41:17) I I feel like GNOME is kind of the silent default. (41:22) But maybe it's just the Plasma users that I know are very vocal about the fact that they use Plasma. (41:28) But I think for the most part, no Gnome just works. (41:31) I mean, to be honest, when I came to Linux, I was coming off of off of Mac OS, and I wanted an experience similar to that. Speaker 1 (41:40) And with a few tweaks, Gnome looked and felt a lot like macOS. (41:46) So usually when I go when I when I do fire up a Linux Linux desktop, that's that's what I'm running. (41:52) And and it's usually running Fedora as well. (41:54) Yeah. (41:55) Gosh. Speaker 1 (41:55) I've been I've had a Linux desktop laptop, some sort of user device running Fedora since 2021, somewhere in there. Speaker 2 (42:10) Somewhere around there. (42:11) Yeah. (42:11) Same here around college days. (42:13) Right? (42:14) So, like, that I've always yeah, but I I've distraught, but I've always I've always come back to Fedora. Speaker 2 (42:20) I mean, I always I like Arch too. (42:21) Like, I've used Arch. Speaker 1 (42:22) I was gonna say. (42:23) Yeah. Speaker 2 (42:23) Yeah. (42:24) It might like, so it's either Arch or Fedora. (42:26) Right now, I've I've I've just been on Fedora. (42:27) Like, unless, like, they take it away, gonna stick to it. (42:30) But if they do take it away, Arch definitely is is is something that that I'm Speaker 1 (42:35) I definitely went through an Arch phase. (42:37) Actually, tried installing Arch on a Mac on a MacBook Pro, gosh, and my son's, what, almost nine now. (42:44) So, gosh, eleven, twelve years ago. Speaker 2 (42:46) That was Speaker 1 (42:46) a lot of fun. (42:47) Yeah. (42:48) Although at the time, I adopted Arch because of just how quickly all the software advanced. (42:54) So a lot of the bugs I was seeing running Arch on on a MacBook were fixed fairly quickly. (42:59) But, yeah, now nowadays, I I found that I have other things I like to do other than sit in front of the computer, like sleep or spend time Speaker 2 (43:06) with my family. (43:07) Yeah. (43:08) Exactly. Speaker 1 (43:09) But yeah. (43:10) So awesome. (43:12) Well, any anything you wanted to answer that we didn't cover today? Speaker 2 (43:16) Yeah. (43:17) Actually, no. (43:17) I think I think it's it's it's been been a good conversation. (43:22) Think I think it's a good starting conversation. (43:24) Anyone who's listening to this, you know, can you know, has has a lot to look look at. Speaker 2 (43:30) But just know that, you know, the GitOps Argo guy started off as a Linux admin, started off racking servers. (43:38) So it's not as scary as you think it is. (43:40) It's just a lot of paradigm shifting. (43:42) It's a lot of inward looking, I think. (43:44) I I think I think I I think that's, like, the main main thing I wanted to get across. Speaker 1 (43:49) Well, if if you've enjoyed listening to Christian today, please feel free to put that in and give this episode a like. (43:59) We'll we'll put Christian's contact information in the show notes, but what what I think I heard was him volunteering to come back in the future to talk more about this topic. (44:08) He he Christian did mention this was an introduction. Speaker 2 (44:12) So Yes. (44:12) Right. Speaker 1 (44:12) We'll have to dive in. Speaker 2 (44:13) I guess, implicit. (44:15) Yeah. (44:15) Implicit invite back. (44:16) Yes. Speaker 1 (44:18) Definitely appreciated getting to catch up. (44:21) I I I think our preshow was twice as long as as usual. (44:24) I I don't think we did any volume check or anything other than just kind of listening in. (44:29) But, yeah, it was great catching up with you and and, you know, back out on the conference circuit. (44:34) So, hopefully, we run into each other here soon. Speaker 2 (44:36) No. (44:37) For sure. (44:37) For sure. (44:37) Thank you for having me on. (44:38) I appreciate it. Speaker 1 (44:39) Yeah. (44:39) Well, thank you all for joining us. (44:41) I really enjoyed this time and really excited to be publishing on a regular on a regular cadence again. (44:47) Make sure you subscribe so you get notified every time I go live. (44:50) I'll actually be trying to build some kind of a some kind of a lab VM pipeline here in my home lab, just considering the number of distributions and versions that I'm currently currently creating content around. Speaker 1 (45:04) So looking to do that automatically using Ansible. (45:07) So I'll probably livestream some of that process. (45:10) You can see me bang my head against the keyboard when I forget a colon in YAML. (45:14) But meantime, join us again in two weeks. (45:18) I'll be hosting a conversation with Aaron Honeycutt. Speaker 1 (45:21) He is a big advocate and developer for Nick's OS. (45:26) So he was kind enough to answer the call when I said, okay. (45:28) I've been hearing people talk about this next OS thing. (45:31) What what's it all about? (45:32) Somebody come talk to me about it. Speaker 1 (45:34) So, unlike, a lot of our topics, I'm actually going to be sitting back and learning a lot about next OS along with all of you. (45:41) So, join me for that. (45:42) I'm really looking forward to, to our next episode. (45:45) Until then, y'all take care, and we'll see you next time.