Brian:
PodRocket it is sponsored by LogRocket, a frontend monitoring and product analytics solution. Don't know what that is? Go to logrocket.com. Thanks.

Ben:
Welcome to PodRocket. Today I'm here with Gleb Bahmutov, to talk about Cypress and testing web applications. How you doing, Gleb?

Gleb Bahmutov:
I'm good. Thank you, Ben, for inviting me.

Ben:
Yeah. Well, I'm happy to have you, and maybe you could give a quick background on yourself and then we can talk testing.

Gleb Bahmutov:
Absolutely. So I had a long journey from computer vision, image processing, Google's street view, MathWorks, until I arrived at Cypress.io, where I worked for over the last four years. I recently have left Cypress.io, but I'm allowed to work on e2e test runner, which is open source, which is kind of unusual. Working on tools for our developers is my passion. And now I'm here to talk about testing.

Ben:
Yeah. So maybe you could give a quick introduction. What is Cypress?

Gleb Bahmutov:
It's a free and open source test runner for end-to-end tests, for anything that runs in the browser. If you have a static side or web application, and you want to visit the URL, find a button, click on a button, a certain message appears, you're writing an end-to-end test and better to use Cypress for.

Ben:
Got it. And so how does Cypress... I mean, I think probably a lot of people are familiar with Selenium. So curious, how Cyprus compares to Selenium? What's the difference?

Gleb Bahmutov:
They're all, including Selenium, WebDriver, Puppeteer, Playwright and Cypress and Testcafe, are all end to end test products. They can open the URL, find a button, click on it. But all these tools implement interaction with a browser differently. Selenium implements Selenium WebDriver protocol, meaning... But test runner sits outside of the browser and sends through HTP connection to pick an action. JSON commands find a button, so it sends a JSON object into the driver that sits inside the browser and the browser executes finding a button. Then another command comes, click on it. So it travels down the wire to a web driver and web driver inside of a browser executes the click.

Gleb Bahmutov:
Puppeteer and Playwright create a websocket connection. So they can control the browser through a proprietary interface, but the commands and the responses are a lot tighter, because you're not doing HTP calls, you're actually using a websocket by directional connection. And when Cypress does something else differently, we actually load the end-to-end test into the browser page. So it sits right there. Your app is iframed in the window, and the spec code is iframe separately. But it has direct access to a page, so it can find the element, it can click on element and it can interact with the page directly. So to the user, these are all end-to-end test runners. But under the hood, the way they control the browser and interact with application are very different.

Ben:
And so what's the advantage to the developer to go with Cypress and how does that architecture help the end developer who just wants to run their tests?

Gleb Bahmutov:
So, think about sending commands to the browser. If you can only send, let's say, individual chunks of commands. It's like trying to throw tomatoes and trying to hit another room from a window. Yeah, you can... Imagine if it was a person and they tried to hit a person with a tomato. If you just... Throwing through a window, sometimes the person moves, right? The element is disabled while the app is fetching data. And so in Selenium and WebDriver, because you send a command from outside, the page might change, right?

Gleb Bahmutov:
Playwright and Puppeteer have a much tighter connection. So you are inside the same room throwing tomatoes at someone, because you're closer, there is less flake, right? But because it's a proprietary interface, fewer browsers implemented. Selenium WebDriver protocol is now a W3C standard, which meaning most major browsers implemented and use it to test the browser. So it's not going anywhere. But using Puppeteer, Playwright, Chrome Debugger Protocol is much tighter.

Gleb Bahmutov:
Cypress is like holding someone by the arm and then throwing a tomato at them. From much tighter control, right? But not as cross-browser, but a lot less flake. If you find an element, right? And you want to click on it, you want to check if element is covered by another element, but it's not disabled, it's actually there. It's much easier to check both properties before you click on it from Cypress tasks, which leads to less flake.

Ben:
Got it. So basically for developers, it's just less flaky tests, less brittle tests. And any impact on what type of tests you can run? Or can you run any type of end-to-end tests? And what test... What's it called? Test scripting languages do support for writing your actual tests?

Gleb Bahmutov:
Cypress is almost like for modality. You can write your test in any language you want, as long as it's JavaScript, right? Well, in reality, JavaScript is what actually executed by a browser. But there is no other languages, right? Flash is dead, Silverlight is dead, VBScript is dead. So if you want to actually load your tasks directly into a browser, you have to use JavaScript. Now Cypress supports TypeScript and CoffeeScript, but those languages get transpiled into JavaScript. The good news is that most of that web applications are actually written in JavaScript, and developers are really familiar with using JavaScript to program web apps.

Gleb Bahmutov:
So it makes sense to actually use the same language to write the end-to-end test. And one other thing that I want to point out, what kind of tests can you do? Because you run the tests inside the browser, your tasks can access any object in your application code. You want to disable some method, just go to that reference and use built-in sign on stuff from Cypress test to disable. You want to access some stuff, browser API like I don't know... Like navigate your location, you can directly do it. Our test runners cannot do that, because you're only sending a command to a browser. You don't have direct access to the JavaScript and objects in your web application.

Ben:
And I'm curious. That all makes sense and more broadly, I've seen a lot of these tools pop up that aim to kind of make tests less brittle in general. So it sounds like Cypress has some aspects of just being a better technical environment for making tests les brittle, but the perennial problem of you change your application a little bit, it breaks your tests, but you didn't want those tests to break. Some things like Testim or Testim, if you've heard of that one. A couple of these products I've seen over the years that use machine learning to try to make tests less brittle. I'm curious if Cypress has anything on the roadmap or any ways that Cypress also helps address that problem of brittle tests that just discourage developers from running them in the first place.

Gleb Bahmutov:
So finding the element and checking it before you can click on the button solves individual command flake, which is a big problem in Selenium based test. What you described, a brittle test, right? Where a developer can modify the structure of a page and all of a sudden you cannot find the button because its idea has changed. So we actually think of this problem holistically, but problem is the disconnect between your code and the tests. And all the machine learning startups and professional companies, they sell you this thing where someone else writes the test. They can maybe click around on a page to define the test, but it's really is disconnected from the developer working on the page. We believe the developer should be writing end-to-end test. And we make it super simple for developers to actually do this. You can run Cypress... And you should run Cypress locally, first of all.

Gleb Bahmutov:
Meaning if you're a developer working on a new feature, as you work on a code, you write the test and the test keeps exercising your code, right? When you open a pull request, it has your code, changes and it has new tests or change tests. And it if full request passes all the tests, you know the test are not brittle, or they have been changed with the code when you merge report requests and you run the same updated test and they should pass. So we believe that the tighter feedback loop is the answer to brittle test. Particular strategy for selecting elements, various.

Gleb Bahmutov:
We have best practices at Cypress, which I agree with. We're using data test ideas. So, but with this particular attribute is specifically for testing and doesn't depend on CSS classes. You can use area labels, so that you hit two birds with one stone. Your task can find the elements without being brittle, and accessibility is present in your page by design. And if you remove accessibility label, then your test break from a right reason, right? So we solve this problem by just eliminating this disconnect between the test and the code.

Ben:
So Cypress you mentioned is free open source software. I'm curious what's the business model around the open source software? Do you offer hosted testing or kind of what is... What does that look like?

Gleb Bahmutov:
Oh, we sell cryptocurrency called CyCon. No, I'm just kidding.

Ben:
I know.

Gleb Bahmutov:
This is a great question, because if you want an active open source project, you have to invest money into salaries, investing infrastructure, and so on. So Cypress, when I joined, has decided, "Let's open source, because selling subscriptions and licenses, it doesn't really work, right?" You don't want to write test in this particular tool from a startup, but your startup fails, that time and investment is all gone. So Cypress is completely open source. There are no future limitations, nothing. The company Cypress has additional dashboard that can store test results and Cypress records videos of everything by default and screenshots. So Cypress sells a dashboard where you can privately keep all the test results, allows you to run tests in parallel, has GitHub, Bitbucket, GitLab integrations and other related tools like analytics. Okay? So if you were a solo developer, use Cypress by yourself, on CI, everywhere, no limitations.

Gleb Bahmutov:
If you're a company that successfully writes a lot of tests, you will get the volume of additional services. And that's what allows Cypress as a company to thrive and invest money and resources into the open source projects. So I think there's a very nice balance of open source and commercial offering. And by the way, a lot of companies now actually offer similar dashboards for their open source test runner, like BrowserStack, Perfecto.io, AWS Amplify, you can run Cypress open source, test runner and get all the results in those private dashboard and we at Cypress welcome it within the Cypress.io Dashboard stands on its own merits and it's pretty strong case for businesses that write a lot of tests.

Ben:
So what does the future of Cypress look like? What are you most excited about? Anything you can share about roadmap or things that are kind of in the future for Cypress.

Gleb Bahmutov:
I will say, since I left the company last month, the company is doing great. When I joined, there was five people. When I left, it was 50 people. The interesting things that we always suffered while I was there were technical limitations of this, inject the spec into a window model. If you want to control the browser, you really need a browser that could be controlled. For example, Apple Safari. The browse is popular, but we cannot control it from Cypress right now, but it's work on actually supporting WebKit, which is the agent behind Safari. So once that lands, that will be the last major desktop browser that Cypress still doesn't support, because we added Firefox, the new Edge is supported, and Chrome always was supported. So WebKit and Safari is exciting feature.

Gleb Bahmutov:
Another thing what many people struggle with, a test that traverse multiple demands. Imagine you go to your web application page, but you have to login with Facebook. So you have to jump to Facebook, then jump back. Cypress, because it injected code in your original domain frame, would lose you as soon as you go to another domain. So far as now, multi-session support coming on the roadmap and it's actively being developed. And another thing that now has became part of a core that I started two years ago, is component testing.

Gleb Bahmutov:
So I said, yes, it's for anything that runs in the browser end-to-end test where you visit the URL, click around. But when you are developing frontend, you probably work with framework components from React Vue Angular, right? And so Cypress now has adapters and the mode where you can just instead of a URL, model that component and see it and interact with everything just like a normal page and use the full battery of Cypress commands and plugins to test the components before you put them into the page, which I'm excited about.

Ben:
Yeah. That component driven development, component testing is something we talked about a long time ago on the podcast. We had Zoltan Olah from Chromatic, and we talked about some of the component level, component driven development, component testing stuff they're working on, so definitely an interesting area. Looking more broadly, tests have been around since, I guess, as long as developers have been writing code and there's so many different people who share different best practices about what they believe the ideal amount of testing, the ideal type of testing you should be doing, whether you should be doing unit tests or integration tests, all the different types of tests. What's your belief for a modern frontend application today? What do you think is the right way to think about how much testing you should be doing? What different types of tests you should be writing?

Gleb Bahmutov:
I strongly believe in end-to-end tests, because if I put myself into the user's shoes, the user only worried about the full deployed page works. They don't care if you do unit testing. They don't care if you use super cool fast test runners. They care, "Can I go to a URL, see the page, do everything that I have to do, and click say checkout, what happens then?" So to me, the closest to giving you confidence with the whole thing works after it's been deployed, after every APIs, with first party services, and all those things, and the database is configured correctly, is to test it, prove it real browser by interacting with page through an end-to-end test runner. So I would start with end-to-end test run, right? You can still run unit tests, right? It was like little support things for logical flows for my particular piece of code.

Gleb Bahmutov:
But to the user, the end-to-end is probably the most important thing. So I usually recommend starting with that. The second thing that I usually recommend is to use visual testing. So Cypress is a functional end-to-end test runner. Find a button, click on it. As long as the button is there and it's clickable, it will succeed. But if you forget to log your styles, if there is a problem, if everything is screwed up on the page, the button can be still there, can be findable and clickable. But to the user of a page will look completely broken. Cypress will not catch that. So you should add visual testing, right? So you do end to end test, prove a whole user flow, and the major steps, just take a whole screenshot of a page and make sure it looks exactly like it used to look before.

Gleb Bahmutov:
So you approved it once and then it stays until you really want to change it and you approve a new visual look. But then there are additional things. And that's where I usually say, "Hey, at this point, manually keeping track of all the tasks and features is probably not going to work because you will have a lot of user storage and a lot of end-to-end tests." So it might be parts of your code not covered. In that case instrument your application and measure code coverage. And the code coverage will tell you, "Hey, out of all your application code, this lines were exercised during the testing and this lines were missed." So you never reached them. And now you play a detective. You're like, "What is that feature? Oh, I never tested deleting a user." And then you write end-to-end test for deleting a user. And then you see if there are any more edge cases lab that you haven't tested.

Gleb Bahmutov:
So to me, the code coverage nicely solves this problem of how many tests should I write? Do I have enough tests? Am I missing something? Right? Use code coverage, write end-to-end test and fill little gaps. With like unit, component, maybe API tests, right? I call this, and people laugh about it. Usually people call a testing pyramid the pyramid, because at the bottom you have unit tasks, then integration tests in the middle, and then just a few end-to-end test on the top. And I like to say, "Okay, take a whole pyramid, but just fill it with end-to-end and visual tests. And then the bottom have little legs of API test, component test, unit test. So it looks like a crab, right? Let's call it, a testing crab is my metaphor.

Ben:
So a couple of questions I'm curious, there are a bunch of tools out there that aim to help with the automation of visual testing. Tools that will compare the pixels and the screenshots and things like that. Do you have any recommendations for what tools you recommend alongside Cypress for visual testing?

Gleb Bahmutov:
Absolutely. So the biggest choice, right? The first question that will come up is, do you want to do it yourself, for visual testing? Or do you want to use third-party paid service for visual testing? Now my background is image processing and image teaching. So with that said, I don't want to do it myself. I would rather pay someone like Applitools, HAP, or Percy, right? Hundred, two hundred, it depends on how many tests you run, dollars per month, rather than me doing the whole thing.

Gleb Bahmutov:
Because as soon as you spend even two, three hours a month working on open source system for keeping track of images, for approving the changed images, for all the jazz, you already blew your image processing or [inaudible 00:20:18] budget. So try using off the shelf, third party paid visual testing service, right? All of them have Cypress plugins nowadays. See how useful it is, right? And if it's not useful, then either drop it or do open source. But remember, if you use open source tools and you implement this whole system, it will cost you money and time in implementation and maintenance every month. So kind of calculate your budget. It's not worth it. So I use third party service.

Ben:
And I think one of the things that I've always found is an interesting challenge when writing end-to-end tests is how to think about writing meaningful tests? And not just tests for tests' sake. And tests that are likely to catch the kind of regressions that a test should catch. So any advice on how people should think about that?

Gleb Bahmutov:
We strongly believe at Cypress that that should be readable. If it's hard to understand what the test does, then it's probably is not doing what you think it's doing or what the original author meant for it. So we suggest, because you keep a code in the test usually together, you should review the test as carefully as you review your production code. Does it make sense? Are you hard coding something? Why is that necessary? Wait for 30 seconds. That means you're not understanding your own application. So for you to write maintainable tasks, make sure there is a second pair of eyes, right? And review your test changes just like you review your normal code. I think that will guarantee longterm good testing.

Ben:
So how do you recommend, if someone wants to introduce the practice of end to end testing at a company and maybe the company hasn't written a live test yet, what's a good starting point and how can you, A get started from a technical perspective, and B get buy-in from the team and the organization?

Gleb Bahmutov:
So imagine you have an organization, could be small, could be large, right? You might have some manual testers already. You might have a combination of manual tasks, maybe some flaky end-to-end test and unit tests. The way I would start is by honestly, looking at the current situation, do you trust your end-to-end tests or people just ignore when they fail? Do you get confidence from writing a lot of unit tests, low level tests? Or do you still think, "Oh, I have to manually check the page before and after each deployment." Right? So after you do that, show a tool... Let's say Cypress, right? I'm partial to Cypress, because I think it's the best. Show Cypress to developers and QA people, just do a demo, right? I'd say here's how we test to do [inaudible 00:23:13]. Here's how you run it. Here's what it gives you.

Gleb Bahmutov:
If there's a mistake and here's how useful it is. See what they say, right? The best thing I have found, and I had a lot of webinars done with other companies that use Cypress and who was successfully introducing but the company, who decided it's bottoms up approach. A developer or QA starts with a single separate staff, that just visits for page. As soon as they have a test and it's library arounds on CI and locally, they decide, "Hey, can I write another test?" And all of a sudden the developer working on the feature will say, "Hey, writing texts to a test as I'm writing code, is actually useful thing. It actually helps me be more productive because I can see what that's running and executing. And if I changed something, it catches it right away." So all of a sudden, developers especially will say, "The tool is making me productive."

Gleb Bahmutov:
It's no longer a time sink to write test, but it gives me confidence makes me develop faster. On another hand, the quality assurance engineers, when they see that there is automateable way of writing end to end tests that used to be manual, will say, "Oh, thank God. Finally, we don't have to spend hours after each deploy going through Google docs, clicking the button and thinking, "Oh, I have to check everything." Right? So there's a little bit of, I would say, tightrope you have to walk. You cannot show the tool and say, we're going to automate your jobs and everyone who is QA today will have to find a new position. People who are doing manual QA right now could do so much more and look at other types and higher types of testing that they don't have time right now for, right?

Gleb Bahmutov:
Look at visual testing. Look at accessibility testing. Look at performance testing. Look at security scans, right? All these things right now are completely forgotten because you manually test every user workflow after deployment by automating. And Cypress is not perfect, right? And it's not a magical wand. You can only automate pretty much simple kind of [inaudible 00:25:37] steps, right? It can not do performance testing for you. It cannot do security scans for you, right? You have to actually be a human being, an expert, to do those things. So you've got to explain to your QA team, "This too will just take the boring part out of your day life. So you can do interesting stuff. And for developers, you are more productive by not doing it yourself manually as a push code. But instead this test will confirm that what you've written actually works." So, that's how I see companies successfully introducing end-to-end tests into their businesses.

Ben:
So, if someone wants to learn where... Or if someone wants to get started with Cypress, what's the best way to go about doing that?

Gleb Bahmutov:
If you want to just try Cypress, see what it's about, go to cypress.io and go to documentation. Cypress documentation is... And I mean, I'm praising myself a little bit, but every engineer at Cypress invested about 25% of their time updating and writing documentation. It's the only technical company where I saw that emphasis. And I was Director of Engineering, so I actually forced it a lot of times, right? Documentation is what makes or breaks projects, and Cypress documentation is exceptional. So go to Cypress, look at the introduction doc, give it five minutes of your time of your day and go from there.

Ben:
Well, Gleb, it's been awesome to have you on and learn about Cypress and testing. So thanks so much.

Gleb Bahmutov:
Thank you so much for inviting me. Best of luck and happy testing.

Brian:
Hi, thanks for listening. Please remember to like, subscribe, email me if you want, even though none of you do. Go to logrocket.com and try it out. It's free to try then it costs money, but yeah, we'll see you next time. Thanks.