Amos (00:00:11): Welcome to Elixir Outlaws, the hallway track of the Elixir community. Chris (00:00:18): Hi! Amos (00:00:19): Hi! How's life treating you? Chris (00:00:19): Oh, I mean, as good as it can be- Amos (00:00:22): When you're sheltering in place. Chris (00:00:25): Yeah. We're here. We're at the house. Kids are home, school's out for the year. We're just, uh, we're, we're, we're good to go. Amos (00:00:34): Are your school's closed for the rest of the year? Chris (00:00:36): I mean, essentially. Amos (00:00:38): Yeah. I assume I assume ours will be too. I think we're close to like, after Easter now. And- Chris (00:00:44): It's very, it's been very interesting to me because I have been working remote, not remote, but I've been working remote for years for a ton of years now, but I've been specifically working from home three to four years now. And I forget- it's been very interesting for me psychologically to go through this and see other people's reactions to it. And my immediate reaction, I'm sort of embarrassed to say it was like, guys, it's not that hard. Like, you just, here's the deal. Here's how it works now. You're going to do your work, but now you're going to be at home, right. Um, and that's a very, uh, I mean, it's, it's just, uh, that was my immediate reaction just because it's sort of like, I've been doing this for a long time. It's not that bad, completely forgetting about the fact that this was a conscious choice I've made and not something I was forced to do. Amos (00:01:39): Right. Chris (00:01:39): And it was also, it also took me like a year to get used to it and get good at it and figure out patterns for it. Amos (00:01:49): So, and what about the people who live alone though? Like you have family. Chris (00:01:52): Yeah. Amos (00:01:53): Like the people who are just like sitting in their house alone. Chris (00:01:56): I mean, I think, I think this is hard. I think it's, it's hard to do. It's hard to start working from home no matter what the circumstances are, right. And it's hard to do it under the best of circumstances. And when you, it's a thing that you chose to do, and this isn't, anything anybody's choosing to do right now, you know, all the people who have to do it right now and didn't spontaneously choose to do it. They like need to do it. They have to do it. And that's a huge, that's, that's wildly different and it's, and you know, it's hard to do under the best of circumstances. And this is obviously not the best of circumstances. So it's really, uh, yeah. So I, I was out as, uh, I had, uh, an embarrassing first reaction, which I will now share because this is how we grow as human beings. Um, and now, uh, to everybody out there who's having a hard time, like, I, I, I empathize. I feel, yeah. I feel, yeah, it's, it's tough right now. Even under the, even under, yeah. It's like, you're a programmer and you can easily do your work from like, wherever you sit with a computer and uh Slack and the internet connection, you know, it's hard, it's hard. Everybody's going through a lot, lot of stress and anxiety, you know, even, again, under the best of circumstances I deal with, I'm dealing with constant anxiety, which is. Amos (00:03:13): On yourself? Chris (00:03:14): I think just anxiety, you know, my anxiety levels have grown over the past basically four years, just sort of continually and the culminating and, and sort of, uh, in this where it's like, I I've never really had problems with actual true anxiety and the way that people have actual like general anxiety. Amos (00:03:34): Right. Chris (00:03:35): And, uh, yeah, no, I'm just, you know, just, uh, it's definitely a very difficult I've, uh, you know, having to, uh, be very, very, very selective about when I consume news, how I consume news, you know, how I consume certain conversations, walking away and from conversations, that sort of stuff, knowing when to ignore things. Amos (00:04:02): I normally don't read a whole lot of the news, but I find myself reading news every day, Read a, read a white paper yesterday from Imperial college, uh, on the coronavirus study and how long they kind of expect this to last, uh, looking at all the data so far and some modeling is, is pretty crazy. But I, I would say that my reaction was similar to yours at first. Cause it was like, okay, I've been working from home or like, uh, an office by myself for a long time. Like this is something I can do, but it was more like about like me being able to survive. And then the longer I sit around, I'm like, oh, I'm very extroverted. We have no more user groups. We have no conferences. Like none of all of that stuff has stopped right now. So I mean, and then, then immediately it was like, okay, well maybe we can do our user groups remote, like we're just use Google Meet. Chris (00:04:57): Right. Yeah. Figure it out other ways.Yeah. Figure out other ways to do it. Zoom calls or whatever. Yeah. Amos (00:05:02): Yeah. So I'm, I'm excited to actually see that work too. I know that whenever we can get out and start gathering in groups again, I'm going to be excited because that's just who I am like, oh yes, I get to go be around people. But in the meantime, I think it's going to be interesting to see how, how we like as a society, um, and, and in our career field go from a lot of companies still pushing for, we have to have colocated teams to through the course of this, they're going to learn how to deal, hopefully, learn how to deal with people who are not colocated. And then is that going to change the way that our society looks coming out of this? Are we going to start having most of these big buildings full of people kind of fade away? Cause that's a big cost for companies. And if they realize at the end of this, oh my gosh, all this cost just went away. Chris (00:05:59): Yeah. I, I that's, it's, it's super interesting. I don't know what's going to happen with that. You have to imagine that some number of companies we'll see, we'll start to realize, start to have that, that realization. Especially a lot of companies who'd never, would never have allowed this prior. So you got to imagine that that's going to shake up a lot of the industry. I think we're probably pretty safe to say that companies will, companies will become way more remote, friendly, they'll just, out of necessity. Like this isn't going to go away anytime soon, right. So we're going to, we're looking at, you know, yeah. I mean, there's all sorts of estimates around it. Some of them are highly depressing, but, um, Amos (00:06:37): Yeah. Chris (00:06:38): But, uh, and probably accurate. The depressing ones is the, probably the accurate ones, but perhaps. Amos (00:06:43): How long this goes? Chris (00:06:44): Yeah. But- Amos (00:06:45): Right now I've read four months to 24 months. Chris (00:06:48): Yeah. I mean, I think it's probably going to be close. I'm going to take the over on that one. Amos (00:06:53): Yeah, I think so, too. Chris (00:06:54): But anyway, all that said without getting, I don't want to get too deep in the weeds and just make this, uh, so topical, um, about, about the virus and everything. Uh, because I do think we should provide, maybe let's, let's call it a little bit of an escape from that, but I think that's, uh, yeah, it's all, it's almost certainly that that big companies are going to become slightly more remote friendly, uh, especially ones that you would never have considered that. And we'll see. And then some may just crave going back to the way things were after it's, after people are free to do so. But, um, I have to imagine some of them at least keep those, those things in place. Amos (00:07:30): And yeah, I think, I think at least some of the workforce isn't, is going to say, hey, I really, I maybe, I really enjoyed being at home. This is what I want to do. Chris (00:07:39): Some workers just aren't ever going to want to do that. I think, I think it's, it's hard enough that some people are just not gonna be interested in that and just want to be around people. People are, you know, some people are just social and that's fine. That's all good. I think even those of us who've been doing remote work for a long time. There's a funny phenomenon that happens. So BR engineering, especially backend, is probably almost all remote, right. From a percentage standpoint, I think, I think back in engineers we're probably like, I'm just doing the math in my head and we're probably 80, at least that's, that's a low, that's a low amount. Amos (00:08:19): Percent remote? Chris (00:08:20): Yeah. 80% remote. I think the percentage is higher. Everyone we've hired recently has been remote. And so we're, we're very, you know, we were meeting, manager and I were talking about this last night, like how just lucky we are , privileged, like all of it, right? Like we live in a pretty rural area. So we have a ton, we have a big yard, we have hiking trails around here to go get- so the kids have plenty of ways to like get out of the house without, you know, while still being isolated. Uh, we have a great job. Well, I already work from home, you know, uh, all these benefits and like the company's like already remote, basically remote, highly remote friendly, if not basically remote first. Um, and yeah, it's just, it's been pretty straightforward for us to transition into. It's been a lot harder for the people who did go to the office and, uh, who are used to that sort of, who are used to that. And so a lot of Slack channels are very noisy these days. Went from nothing to highly like just like the signal to noise ratio, flipped upside down, like, I mean, people are just, they're just craving that sort of, um, they're craving that, that communication and connectedness still. Amos (00:09:43): I do know that, uh, a lot of developers that I've, that I talk to on a pretty regular basis that go into the office, reached out to me and they're like, hey, where do I hang out with people online? Like, what do I do? You work remote? How do I do this? And um, so I just, I, you know, community wise, I just try to point them at like the Elixir Slack is at least a good place to start to talk to some people. Um, and then they're, I know find some groups that have maybe a smaller contingency of people, especially as Slack gets more noisy, like you were saying. So find some, a smaller set of developers that you're close to, or that you want to talk to from that you met at the last conference and just start your own Slack or Discord or whatever. Those are, those seem to be like the suggestions that people are like, oh yeah. And it makes them feel like they can do something about it. Chris (00:10:40): Yeah, I agree. Yeah. And just, yeah. Finding just those ways to connect. This is hard, you know, and, but yeah, doing those, uh, they're, they're useful. They make you, I don't know. I don't have anything novel to say about this. Um, you know, you feel connected to people. That's good. Yeah. Amos (00:11:00): So I have been talking to companies too, about just how to go remote and communication is the biggest thing. There's, there's so much whenever you're not remote that happens with with body language and context of like where you're standing or what you're looking at. And whenever you are in a chat, you know, you say "this "and they're like, what's "this "or it. And so there's like a lot of relearning how to, how to communicate really well, I guess. And I think at the end of this, even if it only goes on for a short period of time that there we're, hopefully we'll be a lot better at communicating. Chris (00:11:43): Yeah. I hope so. I don't know. I certainly hope so. I think people are going to have to adapt one way or the other and you know, one of the benefits, one of the things that you do, if you are going all to an office, you know, that sort of necessarily means they're all in the same time zone, that makes, that makes communication a lot easier. Cause you can just do a Zoom call. One of the hard things that I've learned is, uh, all the ways in which I typically fall back on communication, uh, don't work on a truly remote team. Like Zoom is not sustainable or, you know, uh, physical meetings are not sustainable, even remote, on, with a large enough team and uh, and distributed over enough times zones. Amos (00:12:29): Right. Chris (00:12:29): Right. You only get a handful of times zones. Uh, you only get a handful of hours in the day where you're all online, that aren't either over someone's lunch or whatever. And so you have to move a lot of that discussion into something asynchronous and there- and you just lose a ton of fidelity when you do that. Like the truth is, and I, and I'm pro remote work. I'm also pro fixing these things. You lose a ton of fidelity when you go remote, a hundred percent, just a ton of fidelity. Like you lose a ton of signal, not being in the same room as someone else because you're right. You have all this body language, you have all this stuff and you've just got, you know, a human, just, you just pick up on stuff like humans have an innate sense of that, right. Or, you know, most of them do, not everybody, not everybody's good at picking it up social cues, but most people are pretty good at picking up subtle social cues and they're often unsaid. And when you're on a, the minute you go out of the same room, you lose a ton of fidelity. So the, you know, you lose a ton of the ability to read that stuff on a call. And then when you go from a call to some sort of written thing, even if it's in like a pretty well-formed, you know, GitHub discussion or email thread or whatever, you lose way more fidelity than if you were on the call with each, because you just losing all like tone of voice, intonation, you're losing easily, laughter you're losing, you know, you're, you're missing out on all those things that you just pick up on, uh, when you're talking to somebody. Amos (00:14:14): Sarcasm is really terrible does not come off. Chris (00:14:17): Yeah, doesn't translate, doesn't translate. Amos (00:14:19): It comes up as you being a complete jerk. Chris (00:14:21): Basically. Yeah. And so you lose that. And then like if you go from some sort of long form well-constructed format to something crappy like Slack or IRC or any, any like very quick hit, you know, chat program, those suck, like I hate Slack's whole marketing thing of like, we got rid of all your emails, like screw you , like, email was so much better because it actually allowed people form real thoughts. Like - Amos (00:14:53): Yeah, I think people in Slack, so there's a lot- Chris (00:14:57): Slack is not communication. Let's be clear about that. Slack is not communication, no chat, and I'm using Slack as like the colloquial I'm using it the same way, I'd talk about Kleenex, right. Or Frisbee, right. It's just the brand name for the thing. Amos (00:15:11): But, but like the big problem there is, you know, you get, you can even have two or three conversations in Slack at the same time with the same person. I see it happen a lot where like somebody asks a question while somebody's typing the answer, they ask another question. That's like, the first one was about work, the second one's about family. And then a third one's about like, what are you doing this weekend? And they get answers, get intermingled in there. And then pronouns start coming out instead of nouns. And it just gets confusing to follow. And, and I, you know, that, I like in some ways that if you have something like that, if you actually sit and wait for the other person, instead of trying to have multiple conversations like you would in real life, you can't like, while they're getting ready to answer, you, can't just like jump in and ask them another question, well you can, but you terrible that intermingling. When people wait, I find that communication and chat works really well because there can be a back and forth. It's a lot easier than email and you get things, I think sometimes slim down a little more than the giant emails that I used to get. But if people aren't waiting, waiting, and they're using a lot of pronouns, then it gets hard to follow anything. Chris (00:16:25): But I think there's also, there's the timeliness of it, right? So if you're in a reasonably popular channel and you're not on like, there's this perception that you then need to be online all the time. Um, and if you're not on, and again, if you get getting back to the time zones thing, right? If you just happen to miss the important conversation where you decided on what the Kafka payload was going to be, and it's in Slack somewhere, that's not visibility like that, wasn't a discussion. Amos (00:16:50): And then you're in trouble because you didn't know. Chris (00:16:53): Right. Or, you know, whatever. And so it's like you have to move to higher fidelity things when possible, but finding ways to do that, that are asynchronous that allow people to get involved based on their time zone. Like that's really hard. It's really challenging. And I don't think there's a really good solution for it. I think it's a lot of effort. And so I do think that there's, it's, it's an interesting problem space because I like remote work a lot, but I think this is a very unsolved problem, like a highly unsolved problem. Amos (00:17:24): So what were the hard things, when you, you said it took you a year to transition, what were those things that hurt? And maybe like, like, it sounds like some of them are going to be Slack related, chat related. For me, it was like all communication. And I loved being around people all day. And then suddenly I was around like physically around nobody. Chris (00:17:44): I think that's part of it. I think the, the physical isolation from other people was hard. I think communication was definitely, is definitely still a thing I struggle with and I'm working on and you know, I feel like I'm getting better at that, but it's, it's a long, long process to get good at that kind of this sort of communication. And that was a big part of it. I think the really the really, um, draining things, the things that were really the hardest to learn was how to disconnect from work. Like when your work is at your house and your computer is at your house and you're at your house, you know, you really have to start thinking about modes that you're working in, like modes of your life. And you're either in work mode or you're in home and life mode. And you really have to get good at walking away from both of those things. Because at the end of the day, like you need to have a life still. Even if you're just sitting there by yourself, you need to not work all the time. That's just good for you. You need to sleep. You need to not be staying up late, just working. You need to do all these good things for your life. Amos (00:18:58): One of the things I've seen work for some people- your office, actually, in your house, would work great for this. Cause you have a door outside and a door that goes inside- is that to go from your, the rest of your house, to your office or your office to the rest of your house, you actually walk outside and around. Chris (00:19:13): Oh, right, sure. Yeah. Amos (00:19:14): Um, just to give you like that mental thing of I'm leaving home. Cause the biggest thing for me, well, so it was different when I worked remote, but I had an office that was like three miles from my house. It was just, I was alone in a room with no windows and nobody else in the building worked with me and it was mostly empty buildings. So that's how I first went remote. Um, cause I lived in the middle of the woods and didn't have internet that was even good enough to chat with. So, but then when I, when I moved to Kansas City last year, I started working out of my basement. And the hardest thing for me was like, I would run upstairs to grab a glass of water or something. And then I would be like, oh, there's like five dishes here. I'm just going to wash these real quick. Uh, oh, uh, let me stop by and talk to my wife. Oh, my daughter wants a hug. And by the time I didn't realize that I went to get a glass of water, which should take me two minutes, maybe, maybe five and I'm gone for a half hour. Chris (00:20:12): Right, right. Right. Amos (00:20:13): And then because of that, I also feel bad for clients. So then instead of working an extra half hour at the end of the day, I work an extra four hours a day. Chris (00:20:22): Right. Well, and that's the flip side of it, which is that you also have to work still. You have to get work done still, you know, your, your job, you get the benefit of being at the house. That is a huge benefit. Um, it's really hard for awhile. I would never go back to not working from home if I could help it at this point, but you still have to do your job. Like the privilege you get for working at home is that you have a job to do. And I think that that's reasonable. Um, and so it means that you have to have good boundaries about when you're working too. And that was actually the, you know, a lot of it, a lot of the challenging bit was , me and the, the kids and my wife all getting on the same page about what that meant. Um, and we have a really good strat- we have like a really good pattern or a bunch of really good patterns with it now, um, that allows me to be involved in the kids' lives throughout the day, without having to, you know, always sort of disrupt the work aspect of it. Amos (00:21:34): Your kids too are, whenever you started working remote, well, did you only have one then? Or did you have two, but they were pretty young. So they've, they've kind of like lived most of their life now with dad being a remote employee. Chris (00:21:51): Yeah, yeah. In some form or fashion. Yeah. So that's been interesting too. I mean, but like, but they understand now too that like we can't play video games in the middle of the day, even though they want to do that, we cannot play, we cannot build another Mario maker level right now because Dad's got work to do, but we will later, you know, but what we can do right now is we can eat lunch together, you know, or like I can take Alice to school in the mornings. Like there's all these other ways to be involved throughout the day. But I think you do, there is something about, you know, like you have a job to do, you have to do, you can't, you have to do your work. Amos (00:22:27): And do you find- oops, sorry. Chris (00:22:29): No, I mean, I that's it really, but you have to, you have to have time to be able to do that without it taking over your life. And so finding the, like when you, when you work, you're working and when you're not, you're not, um, and, and trying, trying to split those two things or bounce between them, for me, super doesn't work. Amos (00:22:50): Do you, do you keep a really strict schedule on when you start and when you end work? Chris (00:22:54): Not super strict, but I have a schedule, like I'm not militant about it, but I have a very, pretty defined schedule, and don't really deviate. Amos (00:23:03): Yeah. I would say that my beginning, my beginning and end of day are I try to keep them within a half hour, like I say 5:30, but that might be five to six. And my mornings too, sometimes just because like, maybe the kids need some extra help. So I try to leave that a little open, but my lunch, I'm terrible about keeping lunch schedule. Chris (00:23:26): Yeah. I'm bad about it. And I've noticed that whenever I, uh, I'm always happier when I have a really well-defined lunch schedule. Like when I take a lunch break basically at noon and then come back at one or whatever, you know, around there, sometimes earlier, sometimes later. But I typically, um, typically I'm typically in the office, like in my home office by nine plus or minus 15, 20 minutes. Amos (00:23:53): Okay. That's not too bad, Chris (00:23:55): Like in ready to work, um, and I, I typically stop it almost right at five, maybe like 4:30. Amos (00:24:02): So your, your kids, my kids partially homeschool at least. But your kids go out to school? Chris (00:24:10): Uh, yeah. Historically. Amos (00:24:11): Okay. So, so I was going to ask like, your kids are just starting, their home now, and you've also worked over summers and stuff, but not everybody who's doing remote has had to deal with that. Like their kids go off to school and they go off to work at the same time. So how, how are you dealing? Like what are the, some of the things that work for you in having the kids at home while you're working? Chris (00:24:41): Man. Yeah, it's tricky. It's really tricky. Amos (00:24:43): For me, the hardest thing was going, like when I would hear them like arguing in the other room or something, not like just knowing that my wife's got this, she, you know, I don't have to be there every time somebody goes, "Oww!" Chris (00:24:57): Right. Amos (00:24:59): Cause that was a big change for me. Chris (00:25:00): That's definitely, that's definitely really hard if it's really bad sounding, I'll go out there. But honestly, for the most part, they play outside or they're doing school at the table. They don't fight too much, uh, part of that's because we've been pretty honest about, with our oldest, whose typically the instigator of a lot of that. And just sort of told her like, listen, this is a serious thing. Here's what's happened. Here's why you're home. Here's why we're home. You know, here's why we're doing social distancing. Here's why we're doing all these things. And we're not trying to scare you, but this is the reality of our lives right now. If we do these things that will help a lot of people. And by being honest with her, she's not, you know, we're probably either giving her a lifelong series of, of, of therapy problems, uh, which will be treated in some way. But at the same time, it's been really helpful just to tell her kind of, you know, we're not like doom, we're not like spelling doom for everybody, but we're pretty honest. And I think that's really helpful for her to understand when we say like, we need you to do these things. She understands sort of the import of those. Um, and that's helped actually, uh, keep things- But at the same time, you just sort of, you just learn to tolerate a low grade level of yelling, you know, at any given time. Amos (00:26:22): I should get better at that. Chris (00:26:22): There's gonna be noise everywhere. There's going to be yelling, you know, their kids, you know, we're like I said, we're fortunate that we have a backyard. They can run outside and go play in and there'll be okay. Amos (00:26:37): Yeah. I feel for the people that are stuck in like third floor apartments. Chris (00:26:41): Oh yeah. No like that, that's so much worse than what I'm dealing with. Amos (00:26:45): I'm sorry for you. And if you want to, you can go out to our website and tell us about your coping coping mechanisms because Keathley and I, neither one of us have had to deal with that. Chris (00:26:57): We're not doing anything as difficult as what you're dealing with right now. Amos (00:27:01): Yeah. So if you have things that you would think you should share, tweet at us, something we would love to, to be able to share that with everybody. Like, that's my big thing. I'm just, I'm trying to figure out how, how can I help people keep that social distancing? And I did talk to my kids about like, look, you know, we're, we're doing this to protect society as a whole, like, like, I don't think that my children need to be scared that they're going to die. Like, the kids are okay. But it's, you know, we're just, we got to keep the hospitals open and stuff like that. Chris (00:27:35): Exactly, yeah. Amos (00:27:36): Anybody who's out there, who's listening, who's like throwing coronavirus parties, please stay at home, Chris (00:27:41): Yeah. Stop it, ya jerks. Amos (00:27:42): And if you want to get on a hangout on, on Google Meet with me so that you can have somebody to talk to, that's cool. Just let me know . Chris (00:27:54): Yeah, we should do that at some point.That'd be fun. Amos (00:27:56): Yeah. Chris (00:27:56): We should do an Outlaw party. Amos (00:27:58): Maybe we should do an Outlaw conference online. Chris (00:28:01): Let's just do like, okay- Amos (00:28:03): We'll start with a party. Let's start with a party. Chris (00:28:05): Let's, yeah. Maybe dream big on your own time. Okay. Amos (00:28:10): I like to dream big when other people around and hope that they take the reins and I don't have to do anything. That's that's the goal. Then I just do the fun stuff. Chris (00:28:20): Yeah. Okay. Well don't drag me into your giant, your giant dreams of like, let's do a whole conference, like, allZ I just wanted like a single Zoom call with a bunch of friends. Okay. Amos (00:28:31): Let's -okay. Let's do that. Chris (00:28:32): A single lunchtime Zoom call with friends. And you're over here like, you know what we should do, we should make t-shirts, throw a conference, we'll get speakers. Uh, we'll do some sort of send out, you know, gift cards. People can cater their own lunch. Amos (00:28:49): Now you're pushing mine to the next level. And you have great ideas. You should totally run with this Keathley. It would be amazing. Chris (00:28:57): You're over here, like, like all I wanted to, I just wanted a call and next thing I know I've got a week long extravaganza. I got a gala I got to go to now online. Amos (00:29:08): That's right. Bow ties only. Chris (00:29:10): Not cool, bro. Not cool. Amos (00:29:14): Oh, I think I, yeah. I think a lunchtime chat would be fun. Maybe, maybe record it. Maybe not. We'll see. We'll see. Chris (00:29:28): So that was fun. So topical. Lots of, lots of fun news. Amos (00:29:33): I think it's good though, that we talk about this stuff. Chris (00:29:38): Here's the thing . I'm happy to revisit this, uh, in weeks to come as long as it is, uh, we focus on constructive ways to do remote work better. That's what I'll say. Amos (00:29:53): Yeah. Fantastic. But I think, I think we had to just, I don't know. I felt like I needed to talk to somebody other than my family about everything that's going on. So it's nice to see that, you know, you guys are doing good. Um, almost everybody I talk to, everybody's doing okay right now. Chris (00:30:12): Yeah. Yeah. Same, same. Amos (00:30:14): Good, good deal. So what about Elixir world? What are you doing? Chris (00:30:18): Uh, what am I doing? I'm just writing a lot of code. No, just writing, writing tons of code. Amos (00:30:24): I have bronchitis leftover from that, uh, from the last conference I went to, so maybe remote conferences are sounding good right now. Chris (00:30:33): Um, I broke, I broke a thing in, I submitted a feature for, um, the Kafka library that we use called Elsa, which is really good. Check it out. It's maintained by friends of the show. I broke it because I, in the feature I submitted, the feature just doesn't work, which is typical of me when I don't write tests for things as much as I'm aligned TDD. I always, I, I always get details wrong. I'm bad at that. So- Amos (00:31:02): Should have written tests, Chris (00:31:03): I, you know what, I did test it. And then I changed something at the end about the API and uh, inadvertently broke it. So yes, so in this case you're right. I should have written tests. I should have written tests to verify that it works. A hundred percent accurate statement, a hundred percent accurate statement. So I'm fixing that, but mostly I'm just writing a lot of code. Like I'm writing just so much code in the belief that I can somehow solve problems that way. Most, a lot, a lot, lots of human problems that you can't solve with code, I'm still attempting to solve them with code. Amos (00:31:36): Nice. Chris (00:31:39): So we'll see. We'll see how that pans out. Amos (00:31:39): So me, me too. Um, I tried to get my wife to let me buy a farm bot, you know, cause I was like, Hey, we're going to need a garden this summer. Just, this is just my excuse to buy new toys. Like. Chris (00:31:53): You can just have a garden. You don't need a robot. I'm here to tell you. Amos (00:31:57): Shhhh. Don't tell people that. If you say that too loud, she might actually listen to the podcast. Chris (00:32:01): Here's the thing. People had people had gardens- Amos (00:32:03): She won't let me buy one. Anyway- Chris (00:32:04): People had gardens for a really long time. Amos (00:32:07): I know. I know. Um, but yeah, so like today's is my, I call it, we call it Noggin Day-n. Chris (00:32:17): Long before Ben Ben Franklin flew a kite in the rain, people had gardens Amos,. Amos (00:32:22): Wait a second. I thought we didn't start gardening until after- Chris (00:32:26): Oh, in a post Ben Franklin world? Amos (00:32:27): Well, I thought it was a post-Civil War world. Chris (00:32:30): He invented the bifocals and uh, the notion of growing vegetables. Amos (00:32:35): Yeah. Before that everybody was just hunting and gathering. Chris (00:32:37): Discovered electricity, invented the bifocal, I believe had syphilis, and uh, you know, invented gardens . Amos (00:32:48): And, and those things that you put on your feet. Chris (00:32:49): Early to bed, early to rise, also plant carrots. Amos (00:32:49): And the things that you put on your feet when you swim. What are those called? Chris (00:32:56): Flippers? Amos (00:32:56): Flippers. Yeah. They said, I just read somewhere that he used to strap planks of thin wood to make his feet longer and he would go swim nude. I mean, outside of- Chris (00:33:09): As you do. Amos (00:33:09): Yeah. I mean it sounds like a Benjamin Franklin thing, if you've read a lot about him. So yeah. So today's our, well, we call it Noggin Day at work, but it's just our day to do fun things. So I'm working with Connor this morning, looking at some Scenic stuff. Uh, I've been working on a lot of builds lately and you know what I can, I really appreciate actually the mix release stuff thatcame out way more, because, you know, I did everything with e- deliver before and I like e-deliver, it's got some, some good features, but I feel like mixed release, just simplified my life. And then I'm actually using circle CI for building. And as Connor likes to say that they give you free CPU, free build time. Just use it, like Chris (00:34:00): So much free CPU. Yeah. Well, this is, this is relevant to my interests because I am currently, uh, working on a thing where I, uh, unrelated to work just for fun. Something maybe we talk about in the future, where I, uh, I, I have the need to, to provision and build some, uh, some artisinal hand-rolled Bespoke, uh, servers and deploy code to them. Amos (00:34:23): Nice. Chris (00:34:24): So tell, tell me, tell me how it do Amos. Amos (00:34:27): So, so the circle? Circle CI? Chris (00:34:28): Nah, just talk, talk me through a deployment process that doesn't involve the words Kubernetes, Docker, all these things. I want to, I want to hear it. Amos (00:34:41): Oh crap. Okay. Chris (00:34:41): You're using the Docker. Amos (00:34:42): Well, because Circle CI, we build, we use Docker to build the box that actually- Chris (00:34:47): Okay. Fair enough. Fair enough. This is what I'm interested in. Okay. Tell me about your process. Amos (00:34:54): We have an article about it that Connor wrote too on the website. Um, Chris (00:34:57): Tell me how it do. Amos (00:34:58): So basically whenever we push a tag, is what we have it set up for, so whenever there's a tag on, on a production branch, Circle CI will pick that up and we have some Circle CI config. And we also have, we also have a Docker container already out there for you, for working with Elixir that we linked to in the article. Maybe we'll throw this in the show notes, but if you go to GitHub's, Binary Noggin's account on GitHub, you'll find that there. So we have the circle CI set up to use the Docker container that we have defined that has Git and Elixir and Erlang and all the stuff that we need. And it picks up our project only if you tag it, and it runs mixed release. And we always go to the steps on mixed release, and you have to have a symbol in your list of steps if you define them. But then after that, there's another one, tar, which is a beautiful thing. And it tars up the project. And then, well, after it builds it, so it builds a release and then tars it up. And then we have the keys, Circle CI has keys that connect to our production servers and it sends the TAR over there, tells it to un-tar, calls stop on the process and start on the process on the new one. Chris (00:36:17): Got it. Amos (00:36:18): And it takes a pretty short amount of time actually. Chris (00:36:20): Right. Yeah. Especially if you have a lot of this stuff already, like if you've got a cached, like building the releases really quickly or really quick. Amos (00:36:28): Yeah, yeah, yeah. And we do, we set up Circle CI caches for dependencies and stuff like that. So it doesn't pull those down again and, and I'm sure that there's ways that we could streamline and make our build better. Um, but it, it works pretty well- Chris (00:36:46): For what you're actually doing. Amos (00:36:47): Yeah. And then you have the release configurations are nice because they do give you some sort of like start time configuration for- Chris (00:36:57): Eh, we just, we don't use any of that. Amos (00:36:59): Um, I think, uh, I might be using it for one thing. And one thing only, and its a gem, or not a gem, a package. Chris (00:37:09): You, you listen, you gotta do what you gotta do. And I do understand that you've got dependencies that rely on mics on config files. You know, you gotta do what you gotta do. Cause people aren't aware, for releases, you have a, there's like a specific like release. exs file that you can add that will, uh, inject stuff into your application config. And it does it after the, after the app boots. That way you can populate it with like runtime variables. We largely skip that entire thing. And we use Vapor for all of it. And we just, the very first thing that happens when your appli- in your application start, you know, Def start, underscore underscore, cause you never need that crap. And uh, and then you, uh, the very next line is typically always like vapor dot load bang, and then load the configuration that you've specified. Amos (00:38:04): Yeah, I haven't, I haven't used Vapor, but I can tell you, I used Tomal before this and that was horrific. And I think environment variables seem to work pretty well, if you only have a couple. Chris (00:38:19): The benefit to Vapor is you can use whatever you want. So you can use JSON files or Yammel or Tomal, or. Amos (00:38:28): Aaah! Chris (00:38:28): Or actually Tomal is way better. Thomal is way better than Yammel or JSON, like way better. Not even close. Amos (00:38:39): Yes. I would agree. The thing that I like about the exs is that it's Elixir. And so I don't have to change my brain whenever I'm looking at different files. Chris (00:38:49): Sure. The problem with releases and the reason we use Vapor, uh, amongst other things it's really easy to add, um, it's really easy to do conversions because you can map, you can, for every binding you can basically say like, here's the environment variable I want , put it into this actual variable in the map, in the map you're going to give me at the end of it. And then that's called a binding. And when you have a binding like that, you can say like this has to be required. They're all required by default. So if they're missing it throws or you get an error back,so that the app won't boot, Amos (00:39:26): That is nice. Chris (00:39:26): Which is, that is exactly what you want in literally, almost every case. Amos (00:39:31): Right. Chris (00:39:31): You, you, you do not want to boot the app with a missing piece of configuration. Along those same lines, you almost never want defaults. Like, I don't think you ever want to use a default value if you can help it, because here's the thing, default values and configuration, they only help the developer while you're in development mode. Get over yourself. Like just set up your environment correctly, the way you would with production and always get the variable, because what will happen is you'll deploy, you'll not have the variable there, it'll fall back on the default. Everything will look correct. And it won't be, and then you'll get weird subtle bugs. Amos (00:40:15): So I, I, in the past, because of that, I've, I've set default, default values because they had to be there, but I set them to like, database not configured, so that when it actually blows up, I see a string that says database not configured. And I'm like, oh, crud. Chris (00:40:33): So Vapor forces everything to be there. It gives you translations like really easy to use, like you can just map, call like string.to integer on stuff, to convert it into integers and all that stuff. Amos (00:40:44): When you, when you run Vapor, does it give you some kind of a config map back or something that you pass out down to the other stuff? Chris (00:40:50): Yeah. It just gives you a map. Amos (00:40:51): Okay. Chris (00:40:51): Gives you a map with keyword lists. Amos (00:40:51): Is there anything built into vapor for, um, configurations changing while the system is running? Like notifications? Chris (00:40:59): No. That that does not exist. Um, Amos (00:41:04): It's a hard problem. Chris (00:41:05): And for a long time, Vapor's an interesting project for me because Vapor's the project I've struggled with the most to come to the right API. And I think it's finally pretty close to the right API. Although I think that there's- what's the right way to say it? I think that the building blocks are all there and now it needs an actual, like, easy to use API over the top of it. But all the building blocks are there to build that now. Cause it's just all structs and Elixir data structures. For a long time, I attempted to build an API that would do things like watch configuration and then update somehow and then emit that update to everybody. But the downside of doing that is one, it's a highly stateful thing, that's really easy to get wrong. Uh, two, you have to care about all these different errors semantics, and because you're caring about them, you're forcing everyone who uses your library to buy into those semantics, and that is often not really what you want because everybody's got, every service is going to have different ways to talk about this stuff. And so at the end I just ripped all that out. You know, it's like good design, right? It's taking things apart and pulling them all apart and just doing one thing really well. And so Vapor's really only focused at like loading the right variables. And if you want to load stuff from external sources or if you want to watch it and load it on demand, like you'd pretty much have to do that yourself, but then you're in control of it. So like the hard part of getting this layered together map of stuff where, you know, it's all there, that we do for you. Putting it in persistent term or putting it ETS or, you know, notifying processes to restart with new configuration or whatever. That's not really a thing that I think we can universally solve because you just have to buy into it too much. Like so many other things I have to buy into that process, if you do it that way, that it makes it less useful. So you'd have to do that yourself. But that I, I find a lot of benefit in doing it that way, because it makes your development process much closer to how your app actually runs in production, because like release. esx doesn't run when you're doing local development. Some other config file does. This way, its like, basically we don't have any, we don't even have a release. exs file. And typically we really only have like the, the config files we do have, you know, there's maybe like, I don't know, 15 lines of code between them when you cat them all together. Amos (00:43:37): Oh, that's my bad then. Chris (00:43:38): Yeah. It's like, you can look, and it's often, it is stuff like we're injecting certain types of modules into them, right. It's the stuff that you actually want to do in config. Like not compile time necessarily, kind of compiled at a time stuff, but often it's, you know, things like inject this mock module into the thing or, you know, don't actually try to call this real service while you're in development mode and that sort of stuff. Amos (00:44:03): Okay. Yeah. Yeah. Those are, those are probably a lot of the things that I see using configuration for anyway. So it is nice to have that as some runtime thing. And then I, the thing that I like most though, that you said the whole time there is that it requires everything to be set by default like that makes me happy. Like I don't find out, I don't find out 20 minutes into my system running that I forgot to configure something. Chris (00:44:30): So you can add defaults to the bindings and you can mark bindings as not required. That is possible to do. By default, everything is required. And it took me a long time to add that. In fact, like, basically Fred pushed me over the edge on adding it. Cause he was like, people are going to do it no matter what, they're just going to work around you. So you might as well like just support it. Like they just basically won't, they'll just ignore you, so you oughta support it because. Amos (00:44:59): Sounds like IBM in WWII. Chris (00:45:00): With a giant, you know, with a giant, uh, with a giant asterisk, like you don't really want to use these that often. So that's my asterisk, is don't use it that often, but they are there if you want them. So. You just don't want them. Amos (00:45:12): Do you have a giant asterisk in your, um- Chris (00:45:13): I do. Amos (00:45:14): In the documentation? Chris (00:45:15): Yeah. I believe there is. Amos (00:45:18): Here's how you do default arguments, but don't. Chris (00:45:20): Yeah. I want to go back to this, to the deployment stuff. So you build the, the release in Circle, you push directly to the server. Amos (00:45:30): Uh, yeah, I pushed to, well, we, we are the one that we have, um, it uses a Digital Ocean. Chris (00:45:36): Yeah, right. Yeah. I also ride the Digital Ocean wave. Amos (00:45:39): Yeah. I mean, for, for my projects internally, five bucks a month. Fine. Great. Um, so we, we push out to the Digital Ocean server. We un-tar right in the home directory cause like every project we just give it a user. So we un-tar right there in that home directory. Um, and the, the un-tarring it has a, a fantastic thing is that there's a releases folder that has every single one of your releases that you've un-tarred before in there- Chris (00:46:09): Oh, fancy! Amos (00:46:10): -under their own folder. Chris (00:46:11): And that's just so that it just, they just, you did that or you did that or it just does that Amos (00:46:15): No, no, whenever you un-tar, so it normally does, like, let's say you have a release zere point one point zero. It will put a releases directory. And then inside of that, it'll have a release zero point one point zero directory. And all of that stuff goes in there and then there's a start Url dot d- that file, that data, that dat, something that just kind of tells it what version to start out of all of those. So that means like if you have to roll back to a version, for some reason you just go change that start url.data file. And that gets- and it'll, and then restart your server. I haven't had to do that. So we don't have anything automatic set up to do that. The other thing is you could just redeploy an old version if you wanted to. But that takes a little longer than actually just changing that one file and restarting. Chris (00:47:06): And then are you, are you running all this under like, like supervisor or is it, or like system D or whatever? Amos (00:47:12): I should be. Chris (00:47:15): But you're not, you're just running it or like Upstart or something? Amos (00:47:16): Yeah, I should have made some kind of Ubuntu, it's, we're running on an Ubuntu box. Chris (00:47:20): Oh, okay. Amos (00:47:22): And should probably have done some moving to service level thing, I believe right now that- Chris (00:47:28): Well then you just stop the exits- the running one ,and start the next one. New Speaker (00:47:31): Right. Yep. Chris (00:47:33): And then it restarts. So there's like some downtime there, but whatever. Amos (00:47:36): Yeah. Uh, I am trying to think, I think there, there are ways to not do that. Like you can tell it to reload, I think, but. Chris (00:47:46): You can SimLink in there and you can also, there's a bunch of ways you can, you can kind of do the zero downtime thing, but what you're describing, it's like it's whatever. Amos (00:47:55): Yeah. Um, I think that if I wanted to take it to the next level, so what I do with some clients is , do like a red green deploy. We actually deploy to a completely new container, new server. Um, maybe you have two of them out there and then a load balancer in front of them. So you spin one up and then run some like basic little "Is the system actually up and running?" tests. And if they are, then you switch the load balancer over. Um, that I think is a good way if you don't want to have to do upgrade deployments. So I'm not doing hot reload. Yeah. Um, so that's a good way to get away with not doing hot reloads. Um, and it gives you, what I like about that version versus hot reloads. I think hot reloads are awesome when you need them and you should learn just enough about them that if you ever need to do them there, that you have some knowledge there. And I think you should try it. And I've, I've done that like, like a year ago playing around, but not with the new mixed release. I don't even know if you can. Chris (00:49:02): I don't think you can. I don't believe that's built in. Amos (00:49:04): Okay. Okay. But the thing that I like with the red, green, red, blue, whatever you want to call it, deployment. Chris (00:49:10): Blue-green. Amos (00:49:10): Yeah. Is that- Chris (00:49:12): Black and yellow. Amos (00:49:14): One- black and yellow, black and yellow._ Um, anyway, uh, um, one of the, the server that you're bringing up, if you want to, you don't have to switch for awhile. You could actually like go in there and play around with it and do more extensive testing and then switch if you need to, um, you could treat it as like a staging, if you wanted to, as long as your staging, doesn't create a bunch of crap data that you need to delete. It, it could be used as a staging that can be tested by QA , or yourself, and you could performance test on it. Like if you're working on performance, you could say, okay, now we have a production server up. Let's see what the performance on that box is versus what it was on my laptop. And then whenever you say, okay, we did make an improvement now switch to a load balancer. So the, um, so we, we do that. I do that with a client and it's, they're using AWS and AWS has a lot of stuff built in for that, where it'll spin up servers and wait five minutes to make sure that they're actually handling requests and not getting a bunch of errors. Chris (00:50:19): Right. And tear down the old one, and all that stuff. Yeah. Amos (00:50:20): So like, you can get super complicated with it. Chris (00:50:22): Right. Yeah. Yeah. But I mean, for, for my purposes, I just need a couple of Bespoke, uh, DigitalOcean boxes. And, uh, you know, just for when we'll slam, you know, some, some releases out there. Amos (00:50:35): Right. For $5 a month. Chris (00:50:38): Are you running something like an engine X as like a reverse proxy for your, for this stuff? I mean, I'm assuming these are web projects also. Or something that has an API. Amos (00:50:47): They, they are- all of them out there right now are web projects. Um, because it's a playground for me. A lot of times, some of them are not like my invoicing software is written in house, um, and deployed out there, but some of them are play projects. Um, and right now, yes, they are all web projects. And I do use engine X to proxy into them. Mainly like dealing with HTPs and, and that way- Chris (00:51:18): And like Encrypt and whatever else. Amos (00:51:20): Yeah. Chris (00:51:20): Yeah, yeah. That's, that seems like a reasonable stratedgy. Amos (00:51:24): And there's some other neat stuff that you can, and you can do some other neat things with engine X, if you need to. Like, like, if I actually wanted to take the app down for awhile, I can have it automatically switch over on engine X to a, uh, uh, Chris (00:51:40): Static site or whatever. Amos (00:51:43): Yeah, um, and, and, uh, I have not done this, but I've, I've talked to somebody else who used engine X almost like a load balancer. Um, so it's nice to just already have that in front so that when you want to add those things, you can, um, I also found dealing with SSL, for my brain, I've done it for so many years in engine X that I can just go do it in 10 minutes. Versus trying to put keys into the Elixir app. And like, I don't even know how. And I don't care to know how. Cause I don't need to. Chris (00:52:16): Yeah. Yeah. I know what you mean. Amos (00:52:19): And then, uh, I was told recently, I had no idea, but there's a way to, if you're using Let's Encrypt and engine X to have engine X automatically get those key updates. I had no idea, like mind blown. I was talking about writing that and somebody is like, oh, it's already there. Chris (00:52:38): Yeah. So it was like, you just do it. Amos (00:52:41): Yeah. So I, maybe I should do that today too. Part of my Noggin Day. Chris (00:52:45): It's very fun to go back to, er, to kind of go back to doing a lot of this stuff I have. It's been so long since I've actually had to care about this, uh, you know, for work, we use Docker and whatever. Just jam giant hosts filled with Docker containers. And that's fine for the thing I'm working on. It's like, I need like two things. It gets silly to, it'd be silly to try to like replicate some gianormous complex beast of a deployment strategy for two servers, right. It's already over engineered. Here's, like, here's the thing, right. Like me being me, you know, I'm using Terraform to like scaffold out stuff just cause. Amos (00:53:24): I hate Terraform. Chris (00:53:26): It's you know what it's, it's really the worst- like Amos (00:53:30): I don't know that I can find anything thats a whole lot better than Terraform. But I hate Terraform. Chris (00:53:35): That's the problem. Terraform is the Kubernetes, uh, which is the JIRA of tools, which is to say like, just because you're the most popular doesn't mean you're good. And, you know, Javascript Amos (00:53:43): Next time you have to do a for_each loop Terraform, you just get, gets makes me angry. Chris (00:53:49): Oh no. I mean, all of it's bad, but I do like the ability to sort of just like tear down and rebuild this stuff basically from the command line, that's pretty rad. Like the thing that it allows you to do is rad. I'm also using another one of my hated tools, which is Ansible, for like configuring a lot of stuff. Uh, it's fine. It's just configuration management, but it's like highly. It is, um, highly imperative, highly imperative. Just lets you run, run commands on over SSH on boxes, on multiple boxes. Amos (00:54:23): I like thatAnsible automatically allows you to communicate everywhere instantly. Chris (00:54:29): Is this a reference? Amos (00:54:32): I was trying, I was giving you a chance to explain the reference. Chris (00:54:36): Naw, I don't want, I ain't about that life. But anyway, so I'm doing this because that's already, that's already like way more complicated, like you shouldn't even be doing that for spinning up what amounts to like two servers and a database. But I'm doing it because- Amos (00:54:48): Why are you spinning, are you, why are you spinning up two servers? Chris (00:54:51): I got stuff I gotta do a dog. Amos (00:54:53): Are they both running the same thing? Chris (00:54:57): They're running stuff. Amos (00:54:57): I was like, I thought maybe you were like throwing it a load balancer in front of your little play project. and I'm like "What? " Chris (00:55:03): I mean well I got some customers coming up, but whatever. Amos (00:55:05): Nice, nice. Chris (00:55:06): Anyways, the point being- Amos (00:55:08): I think you needed a load balancer as soon as you have enough customers that they actually pay for it. Chris (00:55:12): -It's been a minute since I've had had to do this stuff. Um, and so it's a chance to go and relearn some stuff. It's good. It's good for you. Yeah. Good for you. Amos (00:55:23): It's a, you also get to relearn all the frustrating parts of it though. Chris (00:55:26): Yeah. I mean it is, it is one of those things where you're like, oh right, yeah, this is, I remember now why we don't do this anymore that often. Like I remember why. And it also, you know, the really funny thing is it makes me realize just how ridiculously under, uh, under priced like Heroku is when you really stop and think about all the things they're doing for you. Amos (00:55:47): Oh yeah. Chris (00:55:48): And you're like, this costs me nothing? Like actually nothing. How is that a sustainable business model? Like you know, I guess it's getin' bought by Salesforce, but anyway, Amos (00:56:02): I don't know. So why Digital Ocean over Heroku if Heroku's that easy? Chris (00:56:07): Oh, I mean, just cause. Why? I mean I could, I could turn around and ask you that question. I don't know. You want to come over here and mow my lawn? Amos (00:56:19): If I can tell you what to do, I will come mow your lawn. Chris (00:56:22): Oh happily. As soon as the quarantine is lifted. Amos (00:56:23): As soon as quarantine is over, I'll be there. It might be next year, but. Chris (00:56:30): I mean, most likely. But so no, I mean, we're doing it. I, um, Heroku should, if you're building a real thing and you want it done today, uh, you should use Heroku. It is the like, and I don't work for Heroku. I got no kickbacks from this. They're not a sponsor of the show. Amos (00:56:49): They should be. Chris (00:56:49): Because there are no sponsers on this show. Amos (00:56:50): They should, , they should, Heroku, can you go put some money in our Patreon? Chris (00:56:54): Right. You know, I mean, just really just a, you know, cross my palm, Heroku, ok? Wet my beak. I get no kickbacks from this, but like using stuff like that, right? Like you should, 100% be doing that. That's like a no brainer. There's no- And, and forget all the people who are like, well, you can't use distributed Erlang if you use Heroku, or you can't use hot code upgrades, if you use Heroku. And I'm like, yeah, but like, you don't want to do that anyway. Amos (00:57:23): But its free. Chris (00:57:23): Like, you're, you're, you're literally complaining about a thing that you don't need. You have no customers anyway. Just want to put it on Heroku, just run it. Amos (00:57:32): So why?- Okay. So Digital Ocean, you put the question back to me then, right? Why Digital Ocean over Heroku?Okay. Um, Chris (00:57:39): Come mow my lawn. Amos (00:57:39): Yeah. I don't get on, no, I, so for me it was like, okay, I can pay nothing. And sometimes when I go to hit the app, because it doesn't get a lot of traffic or it takes a while to come up, right. Like it's super slow because it has to actually start it and everything, and Digital Ocean is just always there. I also like being able to SSH in and do some other crazy things that I can't do on Heroku- Like, it's a lot easier to cut a release in SSH in, and then run app name remote and get a console. Um, and so for me, like $5 a month, I will pay $5 a month to have slightly quicker response times. Um, because my app doesn't get hit enough to stay in, in Heroku's hundred running cycle and SSH again. Chris (00:58:33): Yeah. Yeah. That's fine. Amos (00:58:35): But there's a lot of stuff that I also, like, I, you know, I should put this into the cost. Like you said, there's a ton that Heroku does, even at the free level, you don't have, that you don't have to deal with, so like. Chris (00:58:49): Just logging. Just collecting logs, like, it's a huge deal. Right. You know, that's partially to kind of why I'm, I'm personally doing it is because it's been a long enough time since I, you know, provisioned servers and I've never been good at ops stuff that I feel like that's an important thing for me to learn. Amos (00:59:11): If you're really good at ops, you probably wouldn't be really good at distributed systems. Like you have to choose. Chris (00:59:17): Yeah. You know, it's all, it's all trade offs, right. So I feel like this is an important, this is an important learning process for me, which is kind of a big reason why I'm forcing myself to go through it instead of doing the, the right thing, which is just to get it deployed. Amos (00:59:32): So, so have you used GigElixir? Chris (00:59:35): Uh, nah, man. I mean, I think it's actually probably a pretty cool service. I know people use it. Um, it's cool. I I'm, I'm, here's what I'll say. I'm really happy. It exists. I'm really, really down with the idea that it exists. Like that's really cool and it's really kind of amazing that anybody built it in the first place and got it running and working, like, that's also awesome. Like, like hats off to them for doing that. That's a ton of work. Like it's just a huge amount of work to get all, to get that stuff up and running. Amos (01:00:09): And I think it's just like this side passion project from my understanding too. Chris (01:00:14): Yeah, for sure. And it was basically because of like, just try to solve the, you know, the, the stuff with Heroku to have a Heroku like thing, but that let you do fancy Elixir stuff. Amos (01:00:24): Yeah. And I was going to ask is, is I thought that I had heard that it's like Heroku where you just kind of push to a, git repository and move on. Chris (01:00:32): Yeah, I think it is. I don't know. I haven't really looked that deep into it for my own- I have nothing against our service and I think it's probably really cool. And if, uh, you know, I was inclined, I would probably try it out. I think for me, I either want to not think about it at all, or I want to think about it. I want to control it. I don't know. That's a weird, it's a weird, uh, yeah, it's a weird dichotomy in my own head. I either want to not ever care about this in the slightest, or I want to care about it a lot. Amos (01:01:08): No, I feel that way too. There's a little bit of that thing on Heroku of like, I don't really know what they're doing. Chris (01:01:16): No. Yeah. And I don't, I mean, there's part of it, like you don't actually really get to . Amos (01:01:20): And there's probably a little bit of, I mean, there's a little bit of that at, at Digital Ocean too. Like I'm not picking up a box and putting it in a rack and like, so-so there, there's still a trade-off, but I feel like it's a little less, but I agree. Like I really also like the set it and forget it, mentality of Heroku that Heroku allows you to have like, and I'll just turn up the dinos. If this thing actually takes off,. Chris (01:01:46): Grab the dinos slider, drag it to the right until it starts working well, you know. And you're gonna see a pretty huge performance penalty for, for doing that, and uou know, maybe that's fine. I think, I think your bill would need to be very high, way higher than people think, before you start to actually break even on running your own infrastructure. Unless you have the skills to do that, which I personally don't, which is why I'm doing it, because I think that's like a useful skill to have, or, you know, it's funny, like we were, this came up the other day. I've been, one of the main things I've been working on for the last couple of weeks at work is rewriting, essentially rewriting, porting, um, improving, a bunch of Ruby stuff and moving it to Elixir. And it's just staggering what you can, what you can do still to me, like I've eliminated, I'm pre I did some rough math. And just in terms of what it takes in, in terms of hardware it takes to run the completely equivalent thing in Elixir, we're going to save about $6,000 a month. Amos (01:03:05): That that's, that's amazing,. Chris (01:03:07): Right? Like- Amos (01:03:09): You heard it, you heard it here. This is free advertising for the people who want to sell to their company that they should switch to Elixir. Here you go. This is it. Chris (01:03:17): We're replacing, we replaced, I don't know how many servers, it's like a single task, like a single Elixir instance replaced like some untold number of servers. And is it is I, it has way better performance still and we're not done. Like I've got a whole bunch of these things to like port over. And if, if current trends keep continue based on our, what our performance we're seeing right now, super back, super, super back of the napkin, math on this, right. But I think you're looking at a lower bound of like $15,000 a month. Amos (01:03:58): Wow. Chris (01:03:59): In terms of savings, like basically just from changing run times. That's $180,000 a year. Like, and that's not even including, that's just hardware that doesn't even include operational, like user like me operating at costs, right. The time, amount of time it takes to operate it, the amount of time it takes to deploy it. Like it doesn't account for any of the other, uh, accumulated costs that, which actually those costs completely dwarfed the hardware costs. The human costs are way higher. And now you're talking about just reducing all of that for everybody. Amos (01:04:40): Right. And like, well, even if you just looked at the, the hardware costs, you could even say, I'm going to hire an engineer that this is their whole job. And they should save me more money than what it costs to pay them to do that. Chris (01:04:53): And that's like, and it was like three weeks worth of work. Amos (01:04:55): Wow. Chris (01:04:57): Like, right. That's like the best dollar value ever. And so it is, there is sort of this, it is kind of amazing. I think if you can do things, like, if you're good at the ops stuff, right. Which I like as we've established, I am not. But if you were good at that, there's a certain amount of money you can save, assuming you have all those skills, assuming you have time for all that stuff, right. You know, assuming you have a team large enough to support that, right. You can save a ton just by running like a really big box or like two really big boxes and just doing your own ops on it, right. Cause like a really big box, a really big Digital Ocean box, is like what', 20 bucks a month, and you're going to run two of them. So 40 bucks a month, you know, and then you just do it all yourself. Like you can, you can save some coin doing that. Non-trivial coin. That's not gonna be worth it for everybody. A lot, most teams like don't have the time or in my, in my experience, a lot of teams don't have that sort of operational expertise to be able to do that and, and really do it well. And so it's worth your time just to like use something off the shelf, but it's pretty wild. Amos (01:06:12): And then selling that to the company. The other thing that I see with a lot of teams is they have, they have too much feature crunch to ever get the time to do that. Chris (01:06:21): Yeah, exactly. Amos (01:06:22): One of the, one of the big spaces for customers that I work with that are trying to convert from like Ruby to Elixir is I see a lot of like jumping through hoops where they're trying to do, I'm going to call it distributed map reduce. And I don't mean like distributed across multiple servers necessarily. But being able to distribute across multiple processes, your map and your reduce. So like something that fans out and comes back together. It's really hard to do with, with background processing libraries. But, but you can use something like flow and have it done in 10 minutes in Elixir and still be way more performance. Chris (01:07:02): Right. Right, right, right, right. Yeah. It's pretty wild when you really start. And I mean, obviously we're biased, I'm biased. The company is biased because we have a lot of expertise on running Elixir systems and doing tracing and production and diagnosing production problems and fixing them and moving on and all that kind of stuff. So we're biased in that way and you can get good at debugging anything, you know, people run like giant Perl systems out there in the world and they're good at it. You know, you can get good at diagnosing and debugging running systems and any assorted tooling. That said, the amount of tooling you get for free in this runtime to be able to like, make your systems operable, meaning you can get into them, you can trace through them, see what's going on, fix problems in production. You know, it's just, I don't need, I'm so broken. It's like, I look at other runtimes and I'm like, you can't do anything. You all, you're just so handcuffed on what you're able to do in a live running system. It's pretty wild, like what you just get for free. Yeah. I'm, I'm all, I'm just constantly staggered by it. When I, when I really step back and like do the comparison, it's pretty wild. Amos (01:08:24): We need to, we need to get some more conversion stories on here. Cause they excite me. Like I just, I see it every day at work, but I can't always talk about all of it. So we should find some people that get to, that can talk about it. Chris (01:08:38): Yeah. For sure. Amos (01:08:40): Well, we've been on here. We've been on. Chris (01:08:41): This is going long. We've got to go. Amos (01:08:43): It is going long. Um, I just want to say everybody out there, stay happy, healthy, and safe. Please stay home. Find, find some community online. Um, hang out, reach out to us. Um, maybe we'll see if we can help you out. I mean, I don't think we have enough listeners that we 'llbe completely overwhelmed with people reaching out to us. If we are, I'll make Juliet talk to them. Uh, anyway, um, I just, I just really wanted to say that everybody say happy, healthy, and safe and, and we love you and miss you. And we'll be back at conferences when they, they get going again and see everybody and in face in person to person. New Speaker (01:09:21): Take care everybody. New Speaker (01:09:23): Alright later Keatley.