DAVID: Good morning and welcome to The Hot Mess Podcast. I’m David Brady and I’m happy to host today. JESSICA: This podcast is not a hot mess yet, it’s a Podcast-Tron. CORALINE: I’m pretty sure that I got corrected on our first time out that the podcast is called Greater Than Code and I don’t know why I have to keep reminding everyone of this fact. SAM: Greater Than Code, what fun is that? JESSICA: It’s a lot of fun. DAVID: Jessica is with us here today — Jessica Kerr. JESSICA: Yes, I am, and thank you Coraline for the correction. Right, Sam? SAM: Absolutely, and also with us this morning is Coraline Ada Ehmke. CORALINE: Hi, everybody. I am very happy to also be here today with Astrid. ASTRID: Thanks, Coraline. And I’m happy to introduce our guest today which is Charity Majors. Charity is a cofounder and CTO of a new startup focused on making machine data explorable and delightful. Before that, she was a Production Engineering Manager at Facebook where she spent three and a half years working on Parse, both pre and post-acquisition, and she also spent several years at Linden Lab working on the infrastructure that powers Second Life. Hi, Charity. CHARITY: Hi, I am so happy to be here this morning. This is an amazing podcast. I had so much fun going through your archive. I love that you publish transcripts. This is a trend that I hope that more and more people will pick up. I’m super excited to be here and good morning. DAVID: Normally, when we start off with a guest, we like to get an introduction on them and Astrid gave a great bio on you. We also, in the middle of the show, introduce a guest question and Donald Plummer actually ties it all neatly together. He has a question for you which is, “What is your superhero origin story?” My understanding is that you were bitten by a radioactive bee and fell into a vat of acid. ASTRID: While playing the piano. JESSICA: Then giving birth to dragons. [Laughter] CHARITY: Not many people know that but now, it’s going to be on the internet. SAM: So it must be true. CHARITY: It must be true. I come from Idaho. I was a piano major. I accidentally found computers because I really hated women and that was where the women works. So misogyny got me into computers and thankfully, I kept the computers and the misogyny, more or less, left. So I started out doing music and I came down to Silicon Valley when I was 17. I’ve been doing startups ever since and just recently, co-founded my first, my very own [inaudible]. I can’t think of many role models in the industry who are operations people, who start companies. Can you think of any? CORALINE: No, I think most of them just want to imitate something that’s already been done or solve the Silicon Valley cishet white male problem. I don’t know. JESSICA: Operations is a Silicon Valley problem. Does that make it a cishet white male problem? CORALINE: Operations affects everybody. CHARITY: Yeah. This is the argument that I was having with some people the other day on the internet. SAM: Oh, this is going to be good. CHARITY: Julia, bless her heart. I love that girl so much. She posted something about operating software and how she wants to get better operations because she didn’t like her operating software. And I kind of jumped back. [Inaudible] opportunity, I can’t articulate, I think, what I really value about operations and why I get so pissed off when I hear people devaluating it because that reason has nothing to do with software. Every field has operations. It’s a way of problem solving that focuses on different aspects. When you’re talking about operations engineering in software context, it’s about building, designing, maintaining really complex systems over their life cycle and software is incidental to it. The nature of it changes according to the type of system you’re responsible for. So, my theory about why operations engineers tend not to start companies, like software engineers, it’s sort of self-selecting. I mean really comfortable being in [inaudible] brought on to kind of clean up the mess and make a company kind of grow up when it has proven its product market effective. Not used to do it from the beginning and then that’s really awkward — JESSICA: Now, you got to make your own mess. CHARITY: Now, I get to make my own mess and that’s really unsettling. Being the person that I know, I’m going to be cursing out, in six months or so. I think that if you’re going to think about operations as only the shitty parts of the software life cycle, then you’re thinking of it as a fundamentally uncreative act. Any role that is not creative is when it’s going to be automated out of existence. I do think the operations is a very creative role. CORALINE: Where does that creativity manifest, I’m curious? CHARITY: Where is your creativity when you’re writing code? You’re both reacting and you’re trying to automate yourself out of a job at any given time, and if the creativity manifests in response to unpredictable, unexpected problems that are coming at you. Operations are so exciting in a way that software engineering never was because I love the reactive role. I love the reactive role as well as I see my role as stomping out chaos wherever it arises. This is why I love platforms because platforms are just the type of software where you’re giving humans so much more creativity and flexibility in how they’re going to throw shit at you. You’re just like, “Hey–” However you can dream up, then it’s not just like a shopping website, where you have some predefined code pass, you make sure that your conversion rate is high, you alert if payments are sharply dropping. But you have a lot more control over the system than you do if you are a platform like Parse, where you’re just like, “Run your third party code on my servers and I will try to make it not infect anyone else.” Store your data in my databases. Oh, my God, you’re talking about being a [inaudible]. People have been writing random shit in your database from all over the internet and I think that’s really fun. SAM: That is not the conclusion I was expecting from that. [Laughter] SAM: You went from people writing random shit from all over the internet to ‘and I think it’s really fun’. JESSICA: That’s the creative part, right? You don’t know what they’re going to throw at you. CHARITY: I need to have something to rant about and has to be worthy of ranting, you know? I don’t feel like it’s just like ranting about, “Oh, I didn’t write a test to catch this error.” Like that’s just completely wrong. I’m not saying that I’m right. I’m just saying this is why I bring [inaudible]. Your kink is okay, too. [Laughter] JESSICA: Because heart of the creativity is experiencing this is fun. CHARITY: Yes, yes! This is why you find the best [inaudible] humor in the [inaudible] channel. It is not universally true. I actually look for this but I’m hiring people. You think about who do you want to be in chat with at 3 AM, when the world’s on fire, when your actions over the next five minutes could put your company out of business permanently or not. Who still has their sense of humor and they’re like [inaudible], like those are the people that you want to work with. DAVID: I like that you said ‘stamping out chaos wherever you find it or wherever it arises’. CHARITY: It’s an inherent paradox and like constant state of tension where you’re always trying to work yourself out of a job, and if you’re successful, you’ve been truly well. So this suits really well at area where the average job is like two years long. Like you’ve done your job well, there’ll be time to hand the keys over to someone who is more of a – what’s the metaphor of I say about you have the three types of the cavalry, the infantry and the whatever. There’s the people who like storm the banks but they will never be happy in a policeman’s role, whatever are the roles are. But I’m in doubt [inaudible] very much in the first camp. JESSICA: Like there’s the commandos who’ll storm the beach and just get something running. Then, there’s the infantry who come in and stomp out the chaos and make it okay, and then there’s the police who just keep order, keep everything the same. CHARITY: I’m like in the middle of the first two. I like ushering an organization into the more stable middle path. By the time it reaches policeman, I’m just long gone. CORALINE: You said something about like looking back on what you did six months ago and finding it to be a hot mess. I really think that if you are in a position where you look back on the work you did six months ago and don’t see what chaos you might have unintentionally wrought or didn’t [inaudible] to that that’s the way you could do things better, you are not growing as a professional. CHARITY: For sure. Oh, my God, totally. CORALINE: What are some of those mistakes that you’ve seen yourself make that you wish you could, or either you had to undo or wish that you could have done differently? CHARITY: Well, on a technical level, like working at Parse, we started out – well, we used Ruby in Rails. That was the choice. We used MongoDB in the really early days, that was a choice. And we started out by promising people the world. You know, all the regular expressions in the database. Now, you could argue that those were mistakes, you could argue that they weren’t. There are these set of things you have to do in order to have the problems of success. Most projects are never lucky enough to have the problems of success, where you have to start systemizing things. It’s hard to make something that clicks with enough people that you get to have the kind of problems that are worth complaining about. So on the one hand, you can say that those are mistakes because they were all painful choices that we, to some extent or another, had to unwind or had to recover from. But what we have gotten to that point if we had made those mistakes, you know it’s hard to say. You know we underwent an extremely painful rewrite from Ruby in Rails to Go for the API, two years in. It was much slower and more complicated to maintain the Go version. But we’ve been able to develop it fast enough in order to get there with Mongo. I like to think about it in terms of like getting to the problems of success. As a founder, one of the things that I am getting better and better at is compartmentalizing. And if they’re like, “Yes, that is a problem for future us,” and I cannot think about it now. That would be a problem of success we’ll hopefully have to fight with three months from now but we’ll just be paralyzed, we’ll just stop if we start thinking about those things. DAVID: I love that phrase. What do you mean by problems of success? CHARITY: The problems that you have when people are adopting you and is dizzying and the site is crashing because you have hockey stick growth. That’s one problem of success or you’ve found your niche, you need to hire. You found a thing that causes people to sign up. You need to start stepping on that gas pedal. You need to start creating flows that catch the problem of success making it cultural. You know, “Wow, we just hired a bunch of new people and they don’t all share the vocabulary that we have.” They don’t share the way we look at the world. We need to decide how prescriptive we want to be with our cultural stuff, whether we want to go the cult-y way like Stripe, or Linden, very cult-y companies, which is awesome. Like [inaudible] carry a lot of implicit values very, very quickly but they will alienate people, they [inaudible] so it’s a choice. The problems of success like Jessica, your blog post on the Spectrum of Risk with storage on the one hand, like you can be really super cautious about making big changes of sorts, or versus developer tools. Go nuts and exercise your creativity and your hunger from novelty on that side. In the beginning, you don’t yet know that you need to be conservative in your software choices, in your software sprawl, in your language sprawl. You know, go nuts! Like wait until you have the problems of, “Okay, kids. Now, we have to focus. Because we have customers we can’t let down. We have a pipeline that we need to focus on.” CORALINE: I see one thing in my experience that is a little bit counter to that and I’d love to get your perspective on it, and that is we have an incubator space here in Chicago that happens to share space with a [inaudible] and what I see happen a lot to those younger people that I have mentored, is that they get snatched up by these startups with business cofounders and maybe technical cofounder and get asked to build an MVP straight out of code school. I could see what you said being used as an excuse to say, “Well, the MVP doesn’t matter. We’re not successful yet. Let’s just do this.” So how do you respond to that? CHARITY: That’s a really good example of we have to be conscious to our audiences, right? I’m used to assuming that my audience is people like me who are in Silicon Valley, who are fairly senior, who have battle wounds and scars so I can just flippantly make these blanket statements that other people can then go off and wield in ways that I never expected or predicted. If I understand your question correctly, you’re saying these people right out of the hacker school get tossed in the business cofounder, they haven’t necessarily worked with anyone in a production setting, they don’t really understand what makes a software project sustainable, and they’re just [inaudible] MVP so that they have something that they can demo to get money. Is that what you’re asking? CORALINE: Yeah, and I’ve seen it happen at [inaudible] MVPs and that being the heart of a system when they start getting some traction. Then you have to bring in people like me to say, “It’s not working. What do we do to fix it?” It’s a big stinky mess because the course run. CHARITY: Yeah, absolutely. That’s a pretty predatory dynamic in my opinion. That may be true [inaudible] statement but — CORALINE: I don’t think so. CHARITY: So these hacker schools, I’m pretty pissed off at a lot of them because they’re promising the world and they’re not talking about what it takes to be a mature engineer. They’re talking about how you can get to make thousands of dollars a year. You should never be dropping to do some CTO role. I mean, this is their first job. This is not unique [inaudible] Silicon Valley. This has happened in Silicon Valley too. But if you don’t know how to build a piece of software that is sustainable — JESSICA: If you jump someone brand new into a CTO role, what are you doing to the people that you eventually hire? CHARITY: Yes, like you’re setting yourself up for failure on either way. If you continue to have this person in a position of leadership and authority responsibility, and they don’t know what they don’t know, it’s going to be so frustrating for anyone who is senior who tries to come in and work with them. On the other hand, if they do know what they don’t know, they probably wouldn’t take this job the first place because they know that it’s a terrible idea. Coraline, what would you like to tell the world about this? CORALINE: I think we need to spend a little time in boot camp situations, setting realistic expectations for people and also warning them that there are lots of people in the tech world who are predatory as fuck. CHARITY: Yes, so much. I go with that completely. SAM: I feel like as senior developers, especially for those of us who are more in the police role in this commandos-infantry-police metaphor, I feel like we have a responsibility too to respect that dynamic and recognize it, and recognize that we wouldn’t be in the company that we’re in if it hadn’t made it through that sequence. I’ve actually gotten myself into a lot of trouble by complaining about a lot of stuff and saying, “I don’t know why it was this way without actually establishing that trust first that I respect what came before,” so I don’t feel like we should put it all on the coding boot camps or on the people who are in them. CHARITY: Yeah, but I do think that humility is not a cardinal value in the engineering industry, let’s put it that way. We do this. We set people up to fail by not establishing realistic expectations and we also then put people in a situation where they feel like it’s not okay to fail. That’s the biggest failure of all because the people internalize that. While everyone else is succeeding, they’re not. That’s really tough to people who are already insecure, especially. It’s more toxic for the people who don’t have that sense of insecurity, who then just go out and stomp over other people. I just want to address the things that I said earlier about cult-y companies. I worked in a cult-y company. I’ve been accused of creating cult-y companies and I didn’t mean that in a bad way. When you were young and growing, it is the most effective way to build a really passionate team to have a lot of trust and empathy. But once you cross that Dunbar Number — are you guys familiar with the Dunbar Number? SAM: Some of our listeners may not be. JESSICA: It’s about monkeys. CHARITY: It’s about monkeys. Basically, it’s a hard wired limit based on the number of connected neurons in our brains, or the number of relationships that we can keep in our head. Grossly oversimplified, it is the number of people whose names and relationships to each other that you can keep in your head, and it’s about 140-ish. Like you see this pattern over and over, like in military and early tribal societies, etcetera, when you have a unit that’s over 140, informal methods of communication kind of fall apart and you need to establish some sort of hierarchy. You need to start thinking about people in terms of their role, rather the limit of individuality. JESSICA: — Innovation and coordination. CHARITY: Yeah, thank you. JESSICA: And straight totally consciously past that and even in the last year that I’ve been there, but now close to 500 or I don’t know, somewhere around 500, and when I started, I remember they said, “Oh, pretty soon, you’ll recognize everybody and know who’s a Stripe.” And I was like, “No, we won’t.” You started when this company was under — CHARITY: Especially remote, right? JESSICA: Yeah, no kidding, and it’s true. But we have really consciously moved away. One of our core values used to be trust other Stripe. But we passed the number of employees where you can know all the other Stripes in order to develop that personal trust. Now, we consciously do coordination, we do planning. We have teams assigned to own each piece of software because we can’t have that collective ownership anymore because like you said, we can’t all know each other as individuals. We have to abstract and go to roles and teams. CHARITY: That’s a really rough transition for a lot of people. It is for me. I really prefer the smaller ones so that’s probably why I’m not in a big company. CORALINE: You have gone through some other thing. When they started had no managers whatsoever, and then about two years ago, they saw that that was becoming really problematic for lots of reasons. A lot of people left because they wanted that original culture where there wasn’t any management, there was an oversight. You can [inaudible] whenever you wanted. But I think in order to get to a certain size, you need some of that structure. You have to ask yourself like, “Are the values that we were founded on still our core values?” Because now it’s shifting over time. CHARITY: Yeah, and it’s totally legit to what one thing or the other thing. It’s not okay to say this is good. This is inherently good in the way other people should operate because it’s really like an emergent property of how you function best at that size and that scale, and it has to change. You have to grow up in order to be effective and it’s okay to prefer the early stage which is a conscious effect of your preference and your choice, and you may have to all sort have to find it. JESSICA: Yeah, there’s also a challenge with the people who were there from the beginning do know each other and they think, “We personally, don’t need this structure,” and they don’t see that the new people coming in really do need a structure because there is always a power structure. It’s just a matter of whether it’s explicit or — CHARITY: For sure, and not explicit power structures are incredibly terrifying and disorienting. It’s like that there are rules and I don’t know them. They’re not written down anywhere. Other people seem to know them. How am I supposed to know them? JESSICA: It’s kind of like microservices. CHARITY: It’s kind of like microservices. That is an excellent transition. DAVID: Actually, that is a good point to transition on. Before we move on to the next point, though, I’d like to point out this show is dedicated to one of our $50-level Patreon donors, Thomas Peter Berntsen. He’s a software craftsman, designer, and consultant Aarhus, Denmark and is @tpberntsen on Twitter. Thanks Thomas. This one’s for you. If you want to be awesome like Thomas, visit our Patreon page at Patreon.com/GreaterThanCode and sign up. You’ll get immediate access to our private Slack community at any donation level. CORALINE: I should also point out that last show, I made a reference to brown M&M’s and I had said that I would give a shout out to the first person who got the reference. That person is @marshian on Twitter who correctly identified that not having brown M&M’s backstage was a rider in the Van Halen tour contract. There was actually a really good reason for it. They figured that if a venue couldn’t get that detail right, they wouldn’t get any of the important safety details, like stage strengths and being able to handle the wave of amplifiers and so on so that was a big red flag for them which is pretty cool. Thanks so much Marshian for getting the reference. CHARITY: So Conway’s Law is one of those laws in the internet which states that the organization will grow to resemble the hierarchy or human layout of the teams that built it. DAVID: Yeah, the software is constrained to follow that design, right? CHARITY: Yeah, and there’s this famous image that’s floating the internet. It has the picture of Microsoft being a circle of firing squad. I love that images. So when you start thinking about Conway’s Law as in a place to microservices, you might be inherently terrified, as you should be, because you’ve got– So, Uber, it has now thousand plus microservices and they transition to that over a six-month period, which is dizzying and mind-boggling. How should you structure teams where you’ve got a thousand microservices? Presumably you don’t want to have a thousand trained software engineering teams. That seemed incredibly inefficient and wasteful. JESSICA: Unless, you really like working by yourself. CHARITY: Unless, you really like working by yourself and not being able to go on vacation. So let me [inaudible] how to thinking about, and I haven’t to talk to so that would be cool to like, “You do some of the work for me,” but I’ll tell you about here. Yeah, so microservices, you can do this intentionally because if you don’t make these cultural transitions intentionally, they will happen unintentionally and your outcome will generally be worse. But what are the things that happens when you’re doing microservices is your trading engineering for politics. Your political interactions become much more important and much more challenging because of instead of just trying to test their own, instead of just trying to do get it all to build, you have to convince other people to make changes that you want. The personal aspect of it just becomes a lot more. JESSICA: Really, I usually express it the opposite direction. There’s been some great talks by Michael Feathers and there was spun by Erik Evans about Conway’s Law is a law, like gravity is a law. It’s just true. Your software architecture will follow the communication lines in your organization so plan your organization around how you want your software architecture to look. I have thought, people do microservices because they don’t want to have to talk to each other in person. They’re going to encode in API but as you point out, really we need to talk to each other even more. CHARITY: Yeah, [inaudible] isn’t sure. I agree that it seems like it should work that way and I think that’s resulted for a lot of the adoption. I think that it isn’t just work out that way. I do even work with microservices personally. SAM: Depends on your definition of ‘micro’ but, yeah. DAVID: Good point. We use it very heavily to cover my meds but the definition of micro comes into question. We have some pretty large microservices. CHARITY: It is fascinating to me how people will identify their architecture in a way that I wouldn’t necessarily identify it as an outside observer. But it’s more like we have chosen to adopt this way of thinking about it and we will apply it, even if it [inaudible] if you look at it. The fact about software engineering and operation engineering, you can pretty much assume that you’re not going to have — you’re a classic — software engineering team and operations team for every single microservice. There are many ways to solve this problem. There are many approaches that are good and can work really well. I’ve seen lots of them work. In Facebook, the way that they do this is you have to have a team of software engineers with embedded production engineers who are responsible for core parts of the stack, for ads, or for the internal [scuba?] metrics, and then there’s like the kind of roving bands of people who have a specialty like security who go around and service the dedicated spheres. There are a lot of things that I like about this model. I like that it puts the focus on software engineer being on call for their own stuff. I personally think that we should be moving as an industry away from operation as years being on call and towards software engineers being on call for their stuff, with ops people who are like expert consultants. I really like that model. If you think about it, all of us are already doing this. We just don’t think with this way. If you’re using AWS, or Google Compute, your ops teams is mostly AWS and mostly Google [inaudible]. We just don’t think about it that way. I think this is a trend that is going to continue and I think it’s good for everyone and should embrace it. So that’s one way of doing microservices from a team perspective. Another way, where we actually just outsourcing the operation stuff. Do anyone of you who do more software engineering still wanted to talk about what this looks like from the software engineer’s perspective? JESSICA: Yes. As a software engineer, if I’m going to run my program on AWS and I think this means I don’t have to do operations, well, I do. I just get to use AWS as abstraction of operations, instead of the hardware manufacturer and the operating systems and the virtual and all the email thing-ers and the SMS thing-ers and everything else that I can just say — Amazon, your stuff works together. But I still have to know that API and a lot of that stuff does not abstract away. A lot of those have complications. There are still memory restrictions, stuff like, “Oh, we’re running post-gres on this tiny box. Why does it pause every five minutes?” Oh because Amazon’s own RDS stuff can’t run on all of the boxes that it offers it on. CHARITY: Yes, so true. They keep inviting me back in these conferences and I don’t know why? Serverless is like explicitly promising people you do not have to care about these things and it’s a lie. It’s a lie and a trick. As a trap, it’s a very profitable trap. A lot of people can make lots of money off of it and lots of people see me making lots of money off the damage that it causes. If anyone ever tell you, you don’t have to — DAVID: It’s win-win. CHARITY: It really is. I kept threatening to get into the contracting business because someday, some glorious day, we may be made all around. Anytime, anyone is selling you something that sounds too good to be true, it is. SAM: I’ve heard the term ‘serverless’ but I don’t really have a good understanding of what it means. Would you mind unpacking that for a second? CHARITY: Oh, yes. Serverless is the newest term where people are promising that you don’t have to do the ops. It is basically saying, everything operational is bad. It’s so old like you have wrinkles, it’s so archaic, and if you’re a really good developer, you just don’t need it anymore. But in fact, it’s turtles all the way down. There’s always a server there. Have you ever used lambda and gotten the error disk full? [Laughter] CHARITY: As Jessica put it, it’s just an abstraction and you can change the set of problems that you have by forming a distraction and those are the better set of problems you have. Like Jessica want to be driving to the [inaudible] press reboot on a server. I don’t want to. I am so grateful for AWS and the people who do hardware ops because I remember the days when you didn’t have it. I want to use a better set of problems but I still have to think about physical reality. The speed of light is not changing. We saw they’re so much apart where I could tell you how operationally aware, any given mobile app developer was, just by looking at the code that they had written with the [inaudible]. A full table scan is still a full table scan. I would see people issuing [inaudible] in my database so like select all objects from this collection that are greater than X equal to X or less than X. [Laughter] CHARITY: You know, like you — JESSICA: They probably just select start too. CHARITY: Oh, you can. Or set of full table scan and they call us up and they’re angry at us because we have promised that we have struck away all of that and we do all the operations engineering for you. Yeah, we do but you’re still doing full table scans in [inaudible]. I can’t optimize as a way for you. I would like to personally address every single one of the developer and be like, yes, I can do this work for you but I can’t make it work well, and the more you think about the consequences of what you’re pushing out there into world, the better time you’re going to have. No matter who your provider is, even if they’re the best engineers in the world, you still have to know these things. You still have to think about them. The earlier you think about them, the better time you’re going to have, in general. I get personally irritated a lot when people just start slagging on ops, like ops is a dirty word. In my experience, generally, you don’t make this thing better by denying that it exists. You make the thing better by naming it, speaking to it, valuing it, and improving it. ASTRID: So, Charity, what do you think that software engineers should be doing to actually learn more about operations and how they should be bringing that into the way they write their code. DAVID: This is a great time to pivot on that topic. Before we move on, I’d like to point out we are offering sponsorship per episode opportunities for individuals, companies, conferences. Whether you’re hiring, selling conference tickets, or have a product or project you want to share with the community, you can take a 30 to 60 second ad out at the beginning of the show and put an ad out on the episode you sponsor. Get in touch with the show manager, Mandy at Mandy@GreaterThanCode.com for more details. CHARITY: So what can software engineers do to get better at ops and what should they be doing? It’s almost is as an individual question than it is an organizational question. I like to think of ops, and this is something I said at [inaudible] Conf talk. The reason is the emergent property of all of your values, practices, and habits, both implicit and explicit, around shipping quality software. This is going to be different for every organization. Like some orgs need five 9s. Most don’t and they just invent it. But it is way better to be like, “No, we don’t need five 9s because our [inaudible], we’re mobile.” Nobody who was in mobile should expect more than two and a half 9s. You just don’t going to get it because mobile sucks. I think, it’s way better to be explicit about your needs, then like having it to be sort of, you know, “Oh, we aim to be up all the time.” So your organizational needs around shipping quality software. This is something that everyone, from your tech support team to your CEO, participates in. Ops is something that everyone participates in. it’s not really a dedicated role. Even if you have a role that is 70%, 80% of their mandate is tending to your operational- it’s your job to make sure we hit it, whatever that takes. Sometimes, we need to do a better job of listening to our users because the metrics that we’re paying attention to don’t reflect their actual experience. Sometimes, it is well, our deployment pipeline is not catching on rolling back quickly enough when we ships buggy software. We were talking about creativity in operations but what I love in the ops is that it changes so much. You solved one thing, you solved it well, and I needed to put it down and go on to something else. There’s always something that is the biggest like [inaudible] and you’re not doing your job well if it’s the same thing. If you think about ops, it’s like a thing that we all participate in, then it starts with valuing quality, however you defining it for your organization. So for managers, how do you say that you value something? Well, do you pay people? So people are like, “Well, I can hire software engineers. I can hire good ops people.” I’m like, “How much will you pay them?” They’re like, “Well, you know, 80% of what you pay a software engineer.” I’m like, “Well, you’re not going to get operations people that are on a caliber with the software people that you’re hiring. You’re signaling that you don’t valued it as much.” Google, eight years ago or so figure this out. They were like, “We can hire software engineers. No problem. We have this great brand. We can’t hire SREs,” and they adjusted it so the SREs made more than software engineers. Suddenly, guess what? They had all the SREs they could stand because software engineers started converting to it. You had infrastructure-minded software engineers that are like, “Oh, this is valued just as much, if not more, cool!” DAVID: When you say SRE, what is the ‘R’? Software something engineer? CHARITY: Reliability engineer. DAVID: Reliability, awesome. CHARITY: — Which is Google’s name for it and I personally hate that but I hate lots of things. [Laughter] SAM: Well, that just says that reliability isn’t anybody else’s problem, right? CHARITY: Exactly, and it says that all you do is keep other people shut up, which, “Fuck you, no!” I don’t like the implication but what they’ve chosen to do. Okay, so as an org, teams are almost like a truism or pathological. Do you value the things that you value the most? But often those things are misaligned. The way that you’re expressing what you value is not aligned with what you say that you value, in terms of what you’re seeing in the world. Now, if you’re an e-commerce site, maybe the things you value most have nothing to do with ops, just bare minimum. Great, but then don’t complain about why you’re not getting world-class ops stuff. When you’re promoting people, like how much do you factor in? Do you start to [inaudible] data factor in operational effects? I personally believe that no one, even like mobile developers, no one should be promoted to have a title of senior engineer, if they don’t know how to give a shit about the quality of what they’re putting out there in the world, like the life-cycle their code. That to me defines a senior engineer. They do some kind of struck me, they’re going to be like we’re back [inaudible], I promise. But I really do think it starts with just like thinking about what do we value, what defines quality to us? You can’t expect people to just independently go off and value ops if you’re not signaling respect for it as an org. Some of the companies that are the most aggressive publicly about, you know, “We don’t do ops. We have transcended operations,” are the ones who I personally know have software engineers logging in as a sit in into the hosts individually and like looking at log lines. I was like, “Well, okay, that seems like a very sad, sad life to me.” SAM: Efficient allocation of resources. ASTRID: Now, I have a follow up questions. Earlier, you talked about not worrying about certain decisions until you have the really successful problems. So then, how does that square with thinking about the reliability in the operations like from the beginning as the software developer? CHARITY: Everything that we’re saying is contextual. When I say don’t worry about those problems until you have the problem of success and super-specifically talking about in the early days of your startup, in the first six months, in the first year. When you don’t even know if you’re going to be around for a year. This is your time horizon. When you start out and you maybe start it, your time horizon is a week. I guarantee you, from week to week, what you’re working on will shift or you’re doing it wrong. Most likely. We’re like approaching the one-year mark of our startup and it’s becoming a quarter. It’s like a month to a quarter so that change a lot. But our horizon is now much aware and we’re starting to think in terms of, “Okay, we’re getting a little more confident that we’re going to be around.” Now, if you’re lucky enough to work with senior engineers at the very beginning, they will automatically do this. This is what I’m talking about. It’s like you have internalized so many lessons that you’re not thinking about it. But you’re doing it anyway because you’ve seen this pattern in these mistakes. The amount of time I’ve spent choosing our stack was not a lot but I’ve done it a lot of times that advised a lot of people so I could do it quickly and I could anticipate out a couple of years. If I didn’t have that, I would ask the people because the biggest bang for your buck that you can get is just by sitting down and talking to someone senior. If you don’t have these relationships, you should always offer to pay. It’s incredibly presumptuous to just be like, “Hi, will you advise me for free, random person off the Internet, on how to like expertly do my thing?” This happens to me all the time and it’s just like, “You know, do you think that you’re the first person that asked me that? Fuck you.” [Laughter] CHARITY: Like to get back to the times that if you’re a three-year old company, like your horizon should be as solid half. You know, looking ahead, anticipating these problems because you will save yourself so much time and energy and pain down the line, just by asking the questions upfront. If you’re a company like Facebook or Google, your time horizon is like 5 years or 10 years. So I can very casually be like, “Oh, don’t worry about the problems [inaudible],” and that’s me talking from where I am now, in the pre-one-year timeframe. The older and more mature you get and the less your appetite for risk, the more you want your time horizon to be longer and the more work you want to do upfront to address that. Before we get to the next segment super quickly, you said what should software engineer must be doing? To educate themselves, to level up and first of all, put yourselves in the on-call rotation, whether you’re owning your little microservice, whether it’s you as a team, whether you used to leaning on your ops team, don’t. Ask them to put you in the rotation to some extent and we could have whole things that we could talk about. We could talk for hours of like the minutiae of how to craft an on-call rotation but this is about tightening the feedback loops. It’s making sure that the people who break the things get that feedback, not mediated through and through for layers because then they’re less sensitive. They are not exposed to the consequences and you don’t have to be exposed to it every time but occasionally. At Parse, we would have a rotation and this is not just technical on-call but also like on-call in terms of support. I’m a big fan of software engineers spending time in the support. Rotation, too. Every single time, a developer would be doing their day of support or their week of support. They would come out of it and re-prioritize everything on their plate. [Laughter] CHARITY: I mean, there’s nothing like being directly exposed to it — dog fooding. You know, Christine, my co-founder is amazing about the human and support side of this. She has a talk that she did recently about creating a culture that are optimized for developer experience, where developers are your customers. She was always [inaudible] create mobile apps using Parse. “Oh, hey, so our [inaudible] docs got a lot better. Oh, hey, so we were like half of code was updated because we realize how frustrating it was to have to go through so many early experiences. Expose yourself to the consequences, whether it’s operational or people. I know so much that will just flow from that and so much of it will be contextually appropriate to whatever that you’re doing and then value it, expose yourself to the consequences. I guess, it’s going to totally matter because you’re smart and you’ll figure so much out, you know what questions to ask after that, I think. JESSICA: Charity, you said something about problems that are just not a problem, until they are. Was misogyny in tech one of those for you? CHARITY: Yeah, for sure. I was raised very, very, very isolated, evangelical Christian home-schooled, did not know people. All I knew when I escape, I ran away to college when I was 15 because it was less rebellious thing I could think of to do. I was just like, “Fuck this.” I had this reaction to be treated like a woman because I grew up with this model of what being a woman was. That was having our babies and you were doing dishes and your life is the small, small thing. I just wanted to run as far away from it as I could. When I saw the engineering labs that were empty of women, I was just like, “Yes! I have found my place where no one can ever treat you like that.” This was, as you can imagine, eight years long, on-going process to [inaudible]. I didn’t think of myself as a woman in tech. I completely identified with the dudes. Been the only women on the team wherever I went. That’s where I was happy. That’s where I was comfortable. I still don’t recognize it, usually, it’s like you develop this incredibly thick callous and that callous was super helpful to me getting through the first 15 years in my career in tech. I did on call when I was 17. When I started becoming aware of reality, it’s like you’re shaving all the callous and suddenly you’re noticing things that you weren’t noticing before and it’s really uncomfortable. For me, I think that the inflection point came when I stopped going to other people for help all the time and I started being one of people keen to for help. That boosted my confidence. I didn’t feel as defensive and I started to feel more established. I consider myself deeply fortunate that I grew up believing the way of meritocracy. I completely believe that everything was within my control, and of course, was manifested as looking down on people who hadn’t succeeded. I have so much empathy for people who are in that place because I was there. It’s a lie but paradoxically, it makes you more powerful when you believe it and I would never ever achieved escape velocity if I hadn’t believed it. CORALINE: I think, one of the ways that we can help each other is by being there for each other and finding our own communities. It shouldn’t be you against the meritocracy. It shouldn’t be you again these biased power structures. It should be us against these power structures. ASTRID: Charity, I think what you’re talking about is the struggle and I feel like there are so many people who relate to that because on one hand, you don’t want to always be the only one because that is very hard. So you do want to go find somebody who can make you remember that you’re not just like an anomaly that there’s other people who are like you. But then on the other hand, you don’t want to be always off to the side in this little pocket of people because that’s also isolating. Even though you’re trying to find your own power, you’re still not a part of this overall majority, which is also a problem CHARITY: — Change the world, right? ASTRID: I think that what happens is that experience of feeling that you have to be two or three or four different types of people, depending upon the situation is actually happening to a whole lot of people. Really in reality, there’s only a small group of people who get to the [inaudible] everywhere and everybody else is trying to be the same person everywhere. But then nobody talks about how hard that is and how much it can kind of tear away at you because it makes you question who you are. It makes you question like, “Am I being authentic if I only want to be around like a woman’s group? Because that makes me feel like I have more power.” Or, “Am I being authentic if I avoid the woman’s group because I don’t want to be associated with, oh you’re a woman engineer,” because I experience the same thing. Like if I go to a meet up and I’m the only woman, then it’s like you don’t want to be talking to the only other woman because then it seems like you want to be in a corner with just women or if I’m the only person of color, then there’s maybe one or two others, then you kind of all stay at different points in the room because you don’t want people to be like, “Oh, we don’t want to talk to them because maybe they want to be alone.” I think, it’s really important for everybody who’s not experiencing that so understand that nobody wants to feel like that. Nobody wants to have to be two or three different kinds of people. They just want to be who they are and that’s good and accepted and that’s the struggle because you’re trying to advance the cause that you didn’t really start. Like you’re not the reason why people treat you different. But you also don’t want other people to go through that because you know how hard that is and how much it hurts people. But then you don’t want to spend all your time feeling like you’re an advocate and you just can’t be who you are. Sometimes you just want to be an engineer. You don’t want to have to be like ‘hyphen engineer’ so I feel like there’s a whole lot of people who relates that. I think it’s really important to talk about how hard it is because it’s not like here’s the answer. CHARITY: Yeah, that’s really, really well put. Thank you. Then, as I become more senior in my career. I feel the responsibility to pay for my sense. That’s like flippantly put. I made a lot [inaudible], but I do want to get back. I want to help the people who are coming the next generation. I want to help them survive. This is why I created a local group a couple of years ago. It’s like underground meetup and I love having this group because when I find the people, I have a place that I can vouch for. It’s safe. I have control over it. I can feel comfortable rapping people into the embrace of, “We’ll be your tribe.” It’s a mixed group and I love it because it gives everyone more power because they’re participating. In the meet up, that has some of the best engineers in the world, men and women, but we all share a philosophy of equality for all who are super excited about tech. SAM: That’s what I sort of saying, about being your whole self. Not every place is safe to your whole self, and there’s nothing wrong with banded together with other people who are either like you or different in ways that are different from you and being in intersection about it in finding your community and drawing strength and resolve from knowing that someone has your back. CHARITY: We all have to do whatever we have to do to make it through, for sure. CORALINE: At the end of our episodes, we like to take a little bit of time to reflect on the conversations that we’ve had, the things that made us think or the things that we’re planning on and taking away from it. So who’d like to talk about what they are taking away from a conversation we had today with Charity? ASTRID: My takeaway is that it is really important to understand operations. It’s not a side interest. It should be part of the main interest of building really good software. SAM: Yeah, I got something similar from earlier on in the call and it was surrounding me that like every so often, somebody in the Ruby world will write an excellent rant about the importance of actually understanding and evaluating the gems that you choose to include in your project. Today was like, and also a good reminder to me that that rule of actually understanding your dependencies still applies in ‘serverless’ architecture too and infects to everything we do. It’s a good reminder for me personally to maybe push back a little bit against my tendency to go, “Oh, that’s ops. It’s hard. I don’t know what to do.” CORALINE: I am reminded of something that came up very early on when we were talking about early career developers and creating spaces that are safe for failure, and realizing that we are all working to try and make tech better in our own ways. We all have a very unique ways of going about doing that that are informed by our life experiences. I think that’s something for us to celebrate and I’m really glad that we had the discussion we had today that touched on tech and touch with a lot of different things and ended on a very personal and human note. I’m very thankful that Greater Than Code creates an environment where we can have conversations like that. So, thank you all. DAVID: I’m in kind of a similar space. I’m really happy with how we got to a very human, very vulnerable point near the end. I love and hate at the same time and it’s a dichotomy so I think loving and hating is exactly the right way to approach it that if you want to change the system, you have to have power in the system or the ability to attack it from outside. Most of us take the inside power route but then, if you have power within the system, you often have to work within the system and that means being complicit with its behaviors. I love that dichotomy, I mean I hate it but I also love it that once you do find one route through, that doesn’t prevent other people from finding their way through but at the same time, there’s a certain amount of hubris that we have to have to get up and move every day and there’s a certain amount of hubris that we have to be very careful about not clobbering other people with. I love that dichotomy. I love the fact the system is the source of the power that we have to use to break the system because the system is broken. I’m absolutely 100% ambivalent about it. I love and hate it at the same time. CORALINE: Charity, is there anything about the show that you like to reflect on? CHARITY: I really appreciate that you folks have created a space to talk about the intersection of tech and human issues. Giving me the chance to talk about all the things that we’re doing in the space of like making observability more human-centered. Like this is what Honeycomb is all about. It’s taking complex systems, centering the human, and remembering that as engineers, we’re actually people too and subject to the same biases, the same short cuts, the same [inaudible] and tendency to overlook things. The old way of doing things is you put all this stuff and you’re creating dashboards and you scroll through them and this falls apart, when you can no longer predict all of the failure modes. You kind of have to give up control and be like, “No, I accept that I can’t predict all the failure modes,” and instead, let’s design a system that is resilient and then empowers us as humans to how we diagnose things. On a mental level, that [inaudible], I appreciate the things that came out today around being mindful of how we start with Conway’s Law — the human systems manifest themselves in the software systems that we create and how the feedback loop goes both ways. This is an awesome podcast and I’m really honored to have been part of it. I really enjoyed talking to you all and thanks for having me. CORALINE: Sam said a few episodes ago that Greater Than Code is also cheaper than therapy and I think, we’ve seen that idea realized here today. Charity, thank you so much for being our guest on the show. It’s always fascinating talking to you and viewing your perspective. Thank you so much.