NOEL:  Hello and welcome to Episode 46 of the Tech Done Right Podcast, Table XI’s podcast about building better software, careers, companies and communities. I'm Noel Rappin. 

Today on the show, we have two repeat guests. We've got Avdi Grimm, head chef or Ruby Tapas and Sarah Mei of Ruby Central and Salesforce. I recently realized that all three of us began our professional careers within a couple of weeks of each other in August and September 1998, which is almost exactly 20 years ago as I record this. I wanted to talk to Avdi and Sarah about what's changed and what's stayed the same. We will talk about open source and Agile and dynamic languages, distributed systems, some things you'd expect and a couple of things you might not and how they've all changed or haven't changed the developer’s experience. I'm really glad I got the chance to talk to Avdi and Sarah about this. I hope you enjoy the conversation.

Before we start the show, I have a few quick messages. Table XI is now offering training for developer and product teams. If you want me to come to your place of business and run a workshop on testing or Legacy code or Agile team structure or a couple of other things, that is a thing that we can make happen. For more information, email us at Workshops@TableXI.com or hit our website at http://tablexi.com/workshops and of course, if you like the show, tell a friend, a colleague, tell your social media network. Letting me know that you like the show would be great if only for me. All of that is very helpful and also, reviewing the show on Apple Podcasts helps people find the show, which every podcasts says but it is still true. We would really appreciate it and now, here is my conversation with Avdi and Sarah.

Our two guests should be familiar to you because they have both been on the show before and also because you should just be familiar with them because they're interesting people to follow and listen to what they have to say about technology. I have Sarah Mei and Avdi Grimm. Sarah, would you like to introduce yourself?

SARAH:  Yeah, I'm Sarah Mei. I help run Ruby Central. We run both RubyConf and RailsConf. I'm the program chair for RailsConf coming up and I work as a software architect at Salesforce.

NOEL:  And Avdi?

AVDI:  I am Avdi Grimm and I run RubyTapas.com, which is a screencast service and create various other educational stuff for programmers.

NOEL:  The reason why I have Sarah and Avdi on here today is that in various ways, I became aware that all three of us got our first professional jobs, more or less, at the same time in August or early September 1998, which was exactly 20 years ago as we record this and I thought it would be interesting to talk about how the industry has changed and how the industry has not changed and how being a developer felt 20 years ago and how it feels now. Hopefully, this will not turn into a, "We walked uphill in the snow both ways," or old people yelling at clouds.

SARAH:  But we did walk uphill in snow both ways, just saying.

NOEL:  My first software job was in Boston, meaning I think I was statistically much more likely to be walking in the snow.

SARAH:  That's probably true. Mine was in Seattle.

NOEL:  Maybe, let's start with each of you, sort of briefly talking about how you started and what your first job or first couple of jobs were like.

SARAH:  I have fairly traditional background and I did a computer science degree. I graduated in spring of 1998 from UCSD and I took a job that August at Microsoft up in Redmond. I moved myself from --

NOEL:  A scrappy small startup?

SARAH:  A scrappy small startup but at that point, we're only 40,000 people, not a whole lot of presence in the Bay Area at that point. They had a couple of acquisitions, including Hotmail that were in the Bay Area but basically, everything was in Redmond. I moved from San Diego to Redmond, which was a bit of a shock, let me tell you, weather-wise. I got myself a little apartment in Seattle. I commuted over the bridge every day to Redmond and I worked on a team that was doing web technologies and that's why I joined it. I thought it would be interesting, which was kind of a new thing, at that point.

Most of Microsoft, and I know a lot of the industry, was doing C code. This was before Microsoft had any .Net or anything like that. It was all COM, that kind of thing but they had a couple of web projects and I was working on one of them. I actually ended up writing cross-browser JavaScript at Microsoft in the late-90s, which is that was the era in which they were more likely to talk about destroying things. It was definitely an odd duck of a project. The job was you got to put a lot of IF statements in it. "If you're on Netscape, do this. If you're on..., do that."

NOEL:  Avdi, you go first but I just want to say, I spent a lot of time in the late-90s telling people not to write cross-browser JavaScript because of Microsoft.

SARAH:  Fair.

NOEL:  Avdi, what was your first job like?

AVDI:  My start was slightly less conventional. At the time, I was 18 and I was taking some community college classes part time with an eye towards computer science but I really had a grand total of three classes and I was on my last three C and C++ classes and I needed a final project to tie up my grade. At the time, my dad was working at Raytheon, one of the big defense contractors and they were transitioning an air traffic control radar project to using more C++ code and they were having some growing pains with moving... transitioning into the C++ world and the sort of pseudo object oriented frame of mind that that required.

At the time, I was studying C++ and so, that was like my whole world and my dad had the brilliant idea that he could bring me in and I could consult with his team for a couple of days and then, I could use that experience towards my final project. Somehow, he actually managed to convince his boss that this was a good idea. I'm like this 18-year old kid in a tie dye t-shirt with a bunch of middle-aged engineers.

Basically, what I remember is just saying 'polymorphism' a lot and they're all sitting around going, "Oh, yeah. Polymorphism. Good point," and a couple of months later... It made for an absolutely terrible class final project. Nobody knew what I was talking about but I got my grade anyway and a couple of months later, I got a call from the engineering manager there that said, "Hey, do you want to come on as a contractor?" and basically, never looked back. That was the beginning of my professional’s software career.

NOEL:  Mine's a little bit... I don't know, I guess both more and less conventional. I came out from a graduate degree so I guess, I'm a little bit older than both of you. I had actually worked at summer job kind of stuff. I actually was an intern at Apple the summer of 1995. I was an intern at Apple the summer that Windows 95 launched, which was very strange. The least good time to be there in the entire history of Apple was probably the summer of `95. That was when I was there.

But my first real job after I graduated was actually doing web consulting, which we didn't call it then... But I somehow found a small company in Boston that had been documentary filmmakers and then, documentary CD-ROM makers and kind of stumbled into building websites for healthcare companies in 1998 when this was a very easy thing to set up shop for and convince people that you could do. There are all these companies looking to get on the web.

They hired me because they thought my graduate degree meant I knew something about building software, which was theie naïveté and it was like a 12-person company that was, for weird historical reasons, had like nine people in Miami and three people in Boston. In some ways, my career is very strange because I never really had a junior level job because I came right in out of grad school into a small company, where I was basically the most experienced software developer already. They were all people who are like film editors who had learned HTML or were learning ColdFusion.

I had a weird situation where we were building web projects for major Fortune 500 healthcare companies and I was somewhat terrifyingly the person who knew the most about building software at that point. Then after I left there, I was unable to get a senior level job at another company on my years of experience. I was unable to get a job doing what I had been doing because I nominally didn't have the experience to do it, which was also unsettling and weird.

My first job in practice was web consulting for healthcare companies, doing a lot of stuff that's not all that dissimilar to what I'm doing now. What sparked this for me was looking back on that project -- my first really large project -- which was in about the year 2000, which took five people like three very intense months, probably four normal months and now, with the tooling and the framework, a single person could do it in a month and a half or something like that pretty easily, I think.

It was interesting to me to think about like what in the day-to-day practice has changed and where that improvement comes from and where it doesn't, so I was kind of wondering what you thought has changed in what's easier, what's harder, what is not as good as you thought it would be now, what is easier than you thought it would be now, anything like that?

SARAH:  One thing that I've been thinking about is that a lot of times, my team has quite a few folks that are new grads or are fairly early in their career. When I say things like, "Oh, yeah. Java was super cool back then," like they literally don't believe me. But I remember like at that point, I had done a little bit of Java in college. There was that one cool professor who had operated a multimedia course, I think they called it or something and it was basically like Java applets. I took that at my last quarter there and I was like, "This is awesome," and all the people at Microsoft were like, "You know, you can never run a virtual machine in production. They will never be fast enough. You will never be able to run real code on a virtual machine with garbage collection. You're always going to have to do memory management yourself because that's how you make it really fast."

Now, it's just like kind of an accepted thing. Right now, it's in the water. It's a piece of the scaffolding that we build on and we're really don't think about it if you're building apps. Of course, there's multiple layers of virtual machines now, between me and the actual memory. Not only is there a programming language but there's probably a container or probably several containers. Who even knows? It doesn't matter. You just don't even need to know about it anymore because it's fast enough. That feels like one of the big shifts to me.

AVDI:  One of the things in talking about Java that kind of reminds me is just that, maybe one of the things that didn't go the direction that we expected, I feel like I remember a lot of predictions that everything was going to go the way of one particular technology or one particular framework or something like that. Along the way, there were different ideas of what that was going to be but I think one of those sort of long term takeaways is that things are always fragmented and it never stops.

There's always somebody saying that in the future, everything's going to be CORBA or everything's going to be Java or everything's going to be this or that but it never really seems to pan out that way. Although looking at larger, broader technology stacks, it does kind of happen that way. Now, everything is done with web technologies and viewed as a one really large technology stack. That has kind of changed.

One thing kind of did pan out, as predicted by some people, was that when I got started, there were a bunch of people getting really excited about dynamic scripting languages, which were widely viewed by people of the establishment as being toys and there were a few people saying, "Hey, we're going to actually build real things with this," and lo and behold, these days, a lot of the real things out there are in fact built with languages like Ruby or Python or JavaScript or PHP. Sorry, PHP too.

NOEL:  I built some PHP and started on ColdFusion --

SARAH:  Yup.

AVDI:  I have to remind myself to say that because that powers an enormous amount of the web. 

NOEL: It really does.

AVDI: I forget sometimes.

NOEL:  Yeah. I remember having to convince my boss at that first company, having to talk him into using open source tools, in general. In particular, I wanted to use Python and he was concerned that he didn't know who he would sue if something went wrong, which --

SARAH:  Wow.

NOEL:  I mean, it sounded a little crazy to me even at the time.

SARAH:  But it was not an uncommon reaction to open source at first because not to sue, but who do you call when something goes wrong.

NOEL:  Right and he wasn't even the technology person who had the technologist’s objection to open source. He was like a salesman and a business person with the business person's objection to open source, which originally was, "There's something fishy about this. Why are people giving this away?" so he was just skeptical of it.

SARAH:  I still encounter people, especially in the Salesforce ecosystem, which definitely a little walled garden of its own, who don't really understand open source either, so I'm still having these conversations, which is kind of fascinating.

NOEL:  I think that one huge change, not just open source in the sense of the ecosystem having all of these tools available, which I think is a huge change but also, open source in the sense of having all of this code available to see and to learn from.

SARAH:  I think that is huge.

NOEL:  Yeah. Even before, when I was growing up and learning to program source code for something large was a rumor, like you never saw it and to be able to go on to GitHub and see the Rails source code or anything like that, is just amazing. It's a huge change.

AVDI:  Yeah and also, there's a connection to the technologies that have changed too because, like I was saying, like the advent of scripting languages. It used to be even if you were using some open source components, with a lot of the technologies back then, the idea of actually being able to, you know, in your stack traces, in your debugger, drop down into the framework code, the idea of being able to see the frameworks code and see how that was working was incredibly novel. It was just not a thing. Usually, that was just a black box. The fact that that code is both A, open and available and B, available in your development environment and not in some opaque compiled form, has definitely made things a lot easier to understand.

SARAH:  Yeah, it's easier to understand and easier to debug and easier to work with. Honestly, I feel like at some point, the fact that there wasn't anyone you could call became an advantage almost because that means that I can go look at it myself and figure it out right now. I don't have to wait for somebody to call me back.

NOEL:  Yeah. That was one of my selling points in 1999 when I was trying to sell Python was that I actually was getting better support just through the message boards than we were getting from Allaire for ColdFusion. I've never quite managed to convince them before I wound up leaving.

AVDI:  You know, something that I'll say about that though, is that I've sort of come, maybe not full circle but maybe, halfway around again or something on this because now, as somebody who runs a business, I actually do have the concern of who am I going to call if this code doesn't work for me because I realize that my time is not free and my time is often very ill-spent in trying to debug other people's code. It's a very low leverage use of my time. I'm often when choosing technologies and choosing software, looking at that same kind of consideration, not who am I going to sue but who am I going to pay to fix it if this goes wrong and that's actually something that I run into, as a difficulty with open source now, that there is a fair amount of code out there that I actually don't know the answer to that question.

There’s also code that I do know the answer to that question. Bringing it back to PHP again, the WordPress community has done a really good job of putting code out there that's open source but which has a professionally supported version that you know who you're going to call and they're actually there for you and can work out the kinks.

SARAH:  Yeah, I think Sidekiq, Mike Perham has done a great job with that model and he is also being really helpful, I know, in working with other people that help them think about that model for themselves as well.

AVDI:  I'd really love to see more of that model outside of just the WordPress world, which is where I've seen it the most. Sarah, you brought up Mike Perham and that's a great example but the notable thing to me about that is like in the Ruby world, he's practically the only person I know using that model and something that I rant about periodically on Twitter is that when you believe that all code and documentation and stuff like that ought to be free, that assumption comes with some implications.

I think it's brought wonderful things to the world but at the same time, that assumption comes with some implications and one of the implications is if you're expecting not to pay for anything, then the community that evolves is going to be skewed by whatever moneyed interests are actually paying for all that 'free labor,' and whatever part of the community happen to have an excessive amount of time on their hands.

SARAH:  I agree. I also think that they're sort of talking about something I've been thinking about recently which is the fact that the source code is not the intellectual property of a company anymore and it feels like open source has pretty much surfaced that idea in a way that is kind of subtle but I think it shows itself sometimes in these things, where it's like, of course, the source code is going to be available but then, you can charge for other pieces of support like documentation or being able to call somebody or email somebody or file a ticket or whatever. The source code itself is not really the hard part of any projects. The real valuable part is what's in people's heads about the knowledge that they have about a problem that they've been solving.

NOEL:  I definitely feel like the attitude towards that is something that has clearly changed over the 20 years, that source code as the thing you lock up in Fort Knox and throw away the key was a pretty common attitude when I first started and much less so now. It's just all across the board: big companies, small companies, frameworks, that kind of thing -- a huge difference in how we see the industry and where value lies.

I think that the idea that code is a liability and not an asset is much more prevalent like Avdi, like No Code kind of things that you talked about last time you're on, that kind of thing was pretty nonexistent 20 years ago too.

AVDI:  Yes, it's definitely true that back in the day, I remember on Ruby message boards, the recurring question was, "How do I lockdown my code? How do I obfuscate it so that nobody can steal it?" and you still see that occasionally but it's definitely a lot less common now.

SARAH:  Yeah, that was a question that came up in my first project at Microsoft too. It was like, "How do we keep people from stealing our JavaScript?" It turns out that it doesn't really matter if they steal your JavaScript.

NOEL:  Yeah, not in my first job but I had a Python project later on, where I had it as a concern and in particular, they had a proprietary algorithm that they didn't want people reverse engineering from a Python application.

One other thing that's changed, in this timeframe, Agile and XP as concepts both essentially started just a little bit after we both started. My first serious web projects were done in complete lack of knowledge or understanding that that kind of project management even existed and now, at least within most of a large section of web community, it's ubiquitous at least to pay lip service too. How do you feel like sort of the way projects run has changed over the time?

SARAH:  Honestly, not as much as I would have expected. Like you said, there's a lot of people that are doing --

NOEL:  The vocabulary has changed a lot.

SARAH:  The vocabulary has changed. That's a good point. I feel like what happens day-to-day has not changed as much as I would have expected and maybe, I have a somewhat cynical point of view here because I started at a large company and then, I lasted about a year and a half there and then, I fled to a series of smaller and smaller and smaller companies. It's only in the last year that I've come back to any kind of company that had more than 100 people and now, here I am back in my 40,000-person, monolithic, huge company. It's been really interesting to see the difference having been gone from big companies for the most of the last 20 years and now back to see that like, "Oh, yeah. Well, this is pretty much the same."

Back in Microsoft in the 90s, we each got our own office. Nobody does that it anymore, which personally, I think is a good thing. But really, there's still a lot of solo ownership of code and projects. There's still not as much collaboration between teams due to sort of just organizational drag and big companies seem to have done a reasonable job at adopting the language of Agile without actually having to adopt the spirit of it.

NOEL:  They have the lyrics but they forgot the tune.

SARAH:  Yeah and they're trying but there's organizational limitations in place that need to be addressed and that's a long term project. Certainly, it's better than it was in the late 90s when we all sort of toiled away in our individual offices and then that meant that we really didn't talk to each other very much. We each had this little piece of it that we owned and then, it came to the date when we were supposed to do start our integration work, we all showed up with our code and of course, none of it worked together because we hadn't really been talking to each other about what these interfaces meant and what actually was supposed to be passed back and forth. While technically, they would all hook up to each other but nothing actually worked in the system.

NOEL:  It does seem like continuous integration as a practice is a lot more prevalent now.

SARAH:  That's true. That's a lot better. It is a lot better. That's a good point. Sometimes, I think maybe it takes an outsider to point it out but before, it was like, "Here's the date where we're going to start our integration work," and then that was when we had to talk to each other and usually, they were like sitting in the conference room for a couple weeks to make us work together. But I think now, it's more like, "Of course, there's continuous integration. Of course, when I push my commit, it's going to go run all the tests" and you can find pathologies on that side as well, but I think that definitely the supporting technology is more mature.

NOEL:  At my first job, a little bit later on at a somewhat larger company, one of the first things they did to me was ask me to review a hundreds of pages of design documentation for a project. The company spent like five people, six months and their output was like a bunch of UML software design documentation. I don't think that happens anymore, at least not casually.

Comparing the small companies that I started out with the smaller companies that I've been at more recently, I think one thing that's changed, it's hard to say because my first company was so sort of innocent of software knowledge that their practices were they'll just sort of throw stuff at the wall and see what worked. I think that in some sense, it's harder to do that now, now that there's like, if you're talking just about web programming, 20 or 25 years of practice to look at, which in 1998, there really wasn't.

SARAH:  Where are the pieces of the industry today, do you think, that are still throwing stuff at the wall? There’s always going to be something like that, right? Is that in VR? Is that --?

NOEL:  I think in terms of how projects structure works. I think that like small... I don't know, I don't want to overstate this but I feel like when I started, we didn't even have a vocabulary to talk about what we were doing. We didn't have a vocabulary to really talk about stories or pieces of work and part of this was because, like I said, these were people who did not know software at all and I was pretty much in that boat as well, except that I had a degree. I think that one of the things that has changed is that vocabulary, at least for how you structure projects is much more readily available.

I remember, our first major project was not run well. We put in a ton of work on it and also, I don't want to get into the things that went wrong on the project but I read the original XP white book in the wake of that project and it was the perfect time because I just had all of these problems and here was this thing that's claimed to have a vocabulary for talking about what my problems even were and had an idea for how to solve them. I feel like the vocabulary kind of diffused through the industry but the solutions didn't necessarily.

SARAH:  Yeah. I do think that some of that now is in the water. It's the idea that you brought up about like infrastructure, automated tests. People didn't even write their own automated tests in the late 90s, usually like Microsoft developers --

NOEL:  -- But testing was something testers did.

SARAH:  Testers do that, so you just --

NOEL:  Was that what the government contractors was like, Avdi?

AVDI:  Oh, yeah. Where I was, I think I probably did one of the first projects that was ever test driven and had 100% TDD test coverage. It was unheard of.

NOEL:  The very idea that developers would be able to write tests, obviously the testers were threatened by it but the developers were threatened by it too.

SARAH:  Yeah. I worked with a lot of people who felt it was sort of beneath them.

AVDI:  Yeah, well what Sarah said, it was kind of beneath them and also, it took a long time for the idea that you could write a test first, to make any sense to people. There are still plenty of people that don't do it but it's a lot more accepted now. It's accepted that that's not a completely insane thing to do, that you can do it either way but that was a big shift -- the idea that a runnable specification could actually drive development as opposed to you just bash out a whole bunch of code and then, you run some checks to make sure it works the way you think it works.

NOEL:  One of the things that I think does stand behind all of these things is just how much faster and faster the hardware has gotten, which makes the containers more easier, which makes containerization more viable, which makes the dynamic languages more viable, which makes testing first more viable in a way that I think if you had said 20 years ago like, "This is how much faster computers are going to be," imagine how much faster the web is going to be. The web is not that much faster --

SARAH:  But it sure does a lot more.

NOEL:  Right. I always think about, there's that idea and I don't know what it's called but the metaphor here, as cars got safer in America, Americans by and large took that safety and spent it on being able to drive faster, so the number of auto deaths didn't decline as much as you'd expect because people used that... were able to drive the same relative level of safety but faster. Does that makes sense?

AVDI:  Yeah.

SARAH:  Yeah, absolutely. It's like people have a budget.

NOEL:  Yeah and I kind of think about like what we have spent processor speed on. You know, we spent processor speed on containerization. We spent processor speed on language design. I don't know, what else goes in that bucket?

SARAH:  We certainly spent it on JavaScript and user interface improvements on the web.

NOEL:  Yeah, I think that's definitely true, I feel like to some extent, in a weird way, a much broader amount of people who are developers, certainly then from 30 years ago as the hardware becomes cheaper and it becomes more accessible to become a developer.

SARAH:  That's an interesting thing. Definitely, the core of developers is very, very different now than it was from 20 years ago and in a very positive way in my opinion. I think when I started, it was probably... I don't know, four or five years before I ever worked with another woman who was a developer. There were women who were testers, there are women who are project managers but it was quite a while before I ever encountered another woman who was a developer. These days, that just doesn't happen anymore.

Of course, there are always going to be teams where that's still the case, where they're all male but it's really not that unusual. Like at lot of the small companies I worked on, I was the first woman there and now, it feels like that's less common and because the core of developers has just diversified as it has been able to bring in people, I think like Avdi you mentioned earlier, where instead of just being people that have a lot of free time and can do open source, now there are other avenues in.

I think, the computer is being cheaper and faster, like you're saying Noel is also, definitely, one of those causal factors as well, where if everyone can have a laptop that's fast enough to run Ruby or Python or PHP or even Java, you can run Eclipse not very well but you can still run it on like a Chromebook. You can really do a lot more and especially, with open source tools, if you don't have to pay $900 for an MSDN subscription in order to get a compiler, that's amazing too.

NOEL:  Yeah, that was the other thing I was going say about open source, was that they used to be all of these software-associated costs related and we still have software-associated costs related with starting a business but a lot of them don't kick in until you get a little bit bigger, that you can start for almost nothing. You don't need to buy a compiler. You can start on Heroku for relatively inexpensively, like you can develop on a Chromebook credibly. The barrier to entry there is much, much lower than it was.

SARAH:  It's an interesting thing to see how the venture capital industry has adapted or somewhat failed to adapt, I think. It used to take a lot of money to start a website. You know, you need to rack servers somewhere, so you actually had to buy hardware and so really, the least amount of money you could spend building a website was on the order of hundreds of thousands of dollars. Like you're saying, right now you can get going for the amount of money you might spend on lattes over the course of a month and that's a huge difference.

In some ways, the venture capital folks are still giving these people money, as though it takes $300,000 to launch a website. You need venture capital when you're in an industry where startup costs are so high and we're not in that industry anymore. But somehow, we still have a funding model that thinks we are.

NOEL:  Yeah, that's interesting. I don't know enough about the VC world to think about why that is.

SARAH:  I think that's why you've got so many companies that aren't VC funded these days, that are opting out of that whole cycle.

NOEL:  Yeah. It's much easier to opt out.

SARAH:  Yeah, for sure.

NOEL:  Let's see, what stayed the same? Emacs?

SARAH:  Yeah. Vim is sort of the same, although I guess, the 'M' is new.

NOEL:  If you're a web developer, you're still mostly effectively writing HTML. You're still mostly dealing with relational databases that you would have been. It's funny how much sort of incremental change was made against a very, very similar paradigm.

SARAH:  Yes, that's true.

AVDI:  This is not exactly in the vein of what's stayed the same but something I've noticed that's kind of come around in a way. When I got started, I wasn't building web applications. I was working on things like embedded systems and networking middlewares, low level stuff like that. One of the systems that I worked on, like I said, was an air traffic control radar system. It was composed of many different physical components in separate racks. Some of them could be separated by rooms or even miles, then you had the antenna tower itself and it had its own computers inside and then you had other computers inside where the controllers could actually interact with them. Many different components running different architectures, all communicating with each other over various sort of home grown service busses and networking protocols.

An interesting thing about that you couldn't really do isolated tests of these things. There was no way to run all the different bits of software virtualized. It's very difficult. You had to do a whole lot of connecting hardware together. You had to have entire labs to even begin to try to get everything together, and run it, put it through its paces and make sure it was all working together correctly and handling failures correctly and stuff like that.

One of the things that you saw with that kind of software was that there was a whole lot of integrated testing. There was a whole lot of production time testing. These systems, individual components of the system, they would typically would operate in kind of a cyclical fashion, where they just go through a loop every second or so with various bits of that second allocated to different responsibilities.

When you're processing constant returns from radar, that makes a lot of sense and a piece of each second that the software was running would be devoted to testing itself. It would be devoted to running itself through some pre-set scenarios with some pretend radar tracks and making sure that everything was working right and just doing that again and again and again every single second. Then the whole system, at a larger level of granularity, would do the same kind of thing, where one of the components would send test signals through the whole system and make sure everything was responding correctly.

What’s interesting now is for a long time, it felt like this web stuff is so much simpler and easier than what I used to work on because basically, everything is just a simple request response cycle. It's practically a function from the browser down through our software and then returning some data back. But nowadays, more and more, it's starting to look exactly like the kind of systems I used to work on, just at a different level of time granularity but the same kind of thing where maybe, multiple services you yourself control which are running on multiple machines.

Functioning independently and failing independently, you have numerous external services, external APIs which are an important part of your business. You have to be able to talk to them. You have to be able to handle when they slow down and when they fail and stuff like that and so, it's sort of come full circle where it really feels very similar again. I think one area we might be trailing behind a little bit is that we haven't really shifted yet to a production time testing frame of mind.

SARAH:  I do see that happening to a certain extent, especially with like you're saying, when you've got constellations of services, some of which you control and some of which you don't, it's pretty much impossible to spin up a realistic test environment that will actually make sure that the system, as a whole is working as you expect. I am starting to see now a lot of companies, mostly larger companies who have this problem, do a lot of testing in production and Facebook does a lot of this and other companies do as well. Facebook has talked about it publicly, I think and they basically, they will make a very small change to code and push it out to a small number of people -- small for Facebook, maybe only a couple of million -- and then, see which metrics change. They'll use these metrics and these monitoring. This is where you see a lot of monitoring and observability stuff that's coming up.

The reason that is becoming a thing now is because it's more important to see what the system is doing in production because you can't see it any other way. I think, like you're saying Avdi, there are some lessons we can draw from these older style of things where maybe we do want constant test loops, maybe we do want something that runs a diagnostic of a system once every 10 minutes. We're not all writing in air traffic control systems but in this complicated world of systems that may be up or down or not working or who knows, you need to be able to understand what these systems are doing as a whole. It seems like some of those models would work really well.

AVDI:  I feel like we also need to look harder at this point, less about preventing the failures and more about structuring our system, so that they can tell the story of the failure better because as I've gone on, I've realized that I've poured an enormous amount of time over the years into trying to reproduce errors in isolation. So many tickets that I've addressed over the years have been like, "This thing is happening in production," so I spend a day trying to figure out how to reproduce it in development.

SARAH:  Yeah. That was certainly possible with simpler web systems.

AVDI:  Yeah and I think, maybe a while back, we hit a point of diminishing returns on that.

SARAH:  I agree. I think that it was always an illusion but the illusion is becoming a bit more transparent now.

AVDI:  To me, it's a symptom of a much larger tendency for us, as programmers and maybe just as humans, where we tend to always grasp the illusions of control and of predictability. If somebody comes along or some architecture comes along and says, "Yes, you can actually have that. You can actually have the predictability and the reproducibility and the reliability and isolation and all that," it's a very tempting kind of siren song.

SARAH:  Yeah. I think ultimately probably not possible, though.

AVDI:  Yeah.

NOEL:  Yeah. To me, seems to me the selling point of the Docker world is that you can do that but... I don't know.

SARAH:  I think the best you can do is kind of divide the complexity right and try to put it somewhere else but it's not going to go away.

NOEL:  Yeah. I feel like you can manage it to a certain point but the last mile of it is maybe irreducible.

AVDI:  Yeah and something else that I think is a general truism is that the more you try to pursue that kind of rigidity, the harder things fall when they do fall. The harder they fail, and the harder they are to diagnose when they fail. I think there is sort of a continuum, like a slider between more predictable but also falls apart messier and less predictable but maybe, it can tell its story a little bit better.

NOEL:  I think there's a Douglas Adams quote on point. It's something like the difference between things that are supposed to break and things that aren't supposed to break or things that can never break is that it's much harder to fix things that you can never break when they break.

AVDI:  Yeah. To me, this is also the kind of continuum where we find between the dynamic languages and the static languages.

NOEL:  Right. At least in the web or at least in the part of the industry that I have visibility into static typed languages are growing again.

SARAH:  Yeah. The pendulum is swinging back again.

NOEL:  Yeah and I'm not entirely sure what's driving that particular pendulum, unless it's just that the complexity of the system is getting to the point where the advantage of a type system is beginning to outweigh the costs, especially because the newer languages with type systems seem to impose a much lower cost on them than C or even Java.

AVDI:  Well, yeah. Typed languages used to be complete garbage and so, it was obvious why you wouldn't want to use them. Now, typed languages actually aren't garbage anymore and so, it's much less obvious when you might not want to use one.

SARAH:  Types for me are always been a communication mechanism that you don't need, unless you are in a large organization. The usage of type systems by smaller organizations never made a ton of sense to me because in those situations, what you want is more flexibility and in order to get that, you can give up the safety of types because you don't really need it at that level but there's a certain point at which automatic communication between teams is a really useful thing to have. We do that in Ruby too. We make types when we create APIs around our gems. That's essentially typing that interface and we're saying, "Here is something that is solid and is not. Here’s documented and here's the test."

Testing is almost a form of typing to a certain extent, so there's a lot of ways in which we erect these areas that make the communication between pieces of the system more or less automatic. It's more of a spectrum now. It feels like sometimes, a typed language isn't a huge leap from certain ways of writing in un-typed languages.

NOEL:  Right. The first time I played with TypeScript, even just by myself, I got a noticeable advantage in the first thing I wrote in TypeScript, even accounting for the fact that I had to install TypeScript and figure out how to get set up.

SARAH:  Refactoring in TypeScript is really nice.

NOEL:  Yeah. The tooling that you can do in a typed language is something that I don't think the dynamic languages is ever really caught up to. I'm interested to see where that winds up going.

Okay, one thing you think is going to change in the next five or 10 years.

SARAH:  I feel like one thing that is going to change is that, it feels like over the last 20 years, I don't know if you remember a book called 'Code Complete.' 

NOEL: I do.

SARAH: I actually went back through my Amazon purchase history and that was the first book I bought from Amazon and I bought it from them in 1996. That was my first Amazon purchase.

AVDI:  I'm staring at my copy right now.

NOEL:  I bought Code Complete before I started grad school because I was in a panic that I would not know enough programming compared to the people who were in grad school, who mostly had engineering school undergrads and I did not. I think that was, in some sense, where I really learned to program.

SARAH:  A copy of that was waiting on my desk when I showed up at Microsoft on the first day. I read it again, pieces of it anyway, a little while ago and it's a bit dated at this point but the interesting thing about that book is that it is very focused on things you, the individual, can do that sort of relate to the other people around you. But the other people around you are never really mentioned explicitly as even existing. It's very focused on sort of like what do you need to do with your code in order to really ship software.

It feels like over the last 20 years, one of the things that has changed is that we've realized that software really is a team sport, that the interconnection between code and the people who write it is much stronger than it may appear to be just when you're sitting there looking at it and that it is really important to understand how to interact with the people on your team, both through your code and in-person and through other mediums like text and email and what have you.

This is a book I keep thinking is going to exist and doesn't seem to exist yet, which is sort of like the Code Complete for the next generation, which is like how do you do this on a team. You've learned how to do this on your own, how do you do this with other people? What are the skills that you're missing? If you come out of school or a boot camp or what have you, so you know how to code, that's great. What else do you need to know? What other skills are you missing in order to be able to write software in this modern world?

One of the things I think is going to change over the next five or 10 years is that we're going to see much more explicit acknowledgement of that, both in the industry and then, I think it's going to start filtering down into the educational system as well. We're going to see computer science programs shift to be more oriented around communication and code being one medium of that communication.

NOEL:  I believe you about the industry. I am skeptical about the computer science programs changing in any way, shape or form.

SARAH:  They do use Java now, instead of C++.

NOEL:  That's true, I suppose.

AVDI:  I don't know how it's going to change things but I think to me, one of the most delightful things that I'm seeing lately in how the industry is changing is just, love them or hate them, one of the impacts of the boot camps has been a much wider variety of people getting into programming. In my work, I've encountered so many more people in the last few years who came to programming after some kind of other life. They had already been doing something else for a while and then went into a boot camp program and switched to being a developer and I just see so much good perspective coming from people like that, people with much wider variety of backgrounds and a much better understanding of life out in the real world, apart from just the software part of it. That's something that's very hopeful to me. I think, it's going to change the industry in good ways.

SARAH:  Yeah. To add onto that a little bit, the first dev boot camp cohort is graduated in 2012 and so, right now those folks are just getting into solid mid-career. In another five years, those folks are going to be the seniors and it's going to be really interesting to see what happens when we have a much different group of people in senior technical positions.

NOEL:  Yeah, I agree with that. I agree with both of you and I can't add anything. We are, sadly running out of time. Sarah, where can people reach you on the internet, if they want to talk about this some more?

SARAH:  Probably best to reach me on Twitter.

NOEL:  At @SarahMei?

SARAH:  That is correct.

NOEL:  And Avdi?

AVDI:  You can find me at @Avdi on Twitter and Avdi.codes.

NOEL:  Great. Thank you both for being on the show. This has been great.

AVDI:  Thank you.

SARAH:  Thanks for having me.

NOEL:  Tech Done Right is a production of Table XI and is hosted by me, Noel Rappin. I'm at @NoelRap on Twitter and Table XI is at @TableXI. The podcast is edited by Mandy Moore. You can reach her on Twitter at @TheRubyRep. Tech Done Right can be found at TechDoneRight.io or downloaded wherever you get your podcasts. You can send us feedback or ideas on Twitter at @Tech_Done_Right.

Table XI is a UX design and software development company in Chicago, with a 15-year history of building websites, mobile applications and custom digital experiences for everyone from startups to storied brands. Find us at TableXI.com where you can learn more about working with us or working for us and we'll be back in a couple of weeks with the next episode of Tech Done Right.