As web applications become more complex and closer to native apps in their capabilities, end-to-end (E2E) testing is critical in ensuring they behave as their target users expect.

What is end-to-end testing (E2E testing)?

E2E testing is the testing of an application’s entire flow from start to finish, under simulated real-world conditions, to ensure it behaves as expected and that all of its underlying components work in tandem. It may be horizontal or vertical:

• Horizontal: Verifies common flows, such as navigating a site and making a purchase, from a user perspective, primarily at the UI layer
• Vertical: Involves testing at every layer of the application’s architecture and may include additional types of tests (unit, integration, etc.) beyond the above.

We’ll focus primarily on horizontal E2E testing here.

This type of E2E testing is important because a modern web app can potentially frustrate its intended user in many possible ways, ranging from generally slow loading times to more specific problems such as a poorly implemented birthday picker. The point of an E2E test is for testers to catch such issues and make sure the app is usable, by examining:

• Whether someone can reliably complete their journey through the app, for example, by sending an email or updating an account.
• How the web app looks and feels — for instance, if its UI enables intuitive navigation and if its APIs work as intended.
• What happens to the application, in terms of functionality and performance, when any code changes to it are integrated.

This is not an exhaustive list. E2E testing ultimately encompasses all the tests necessary for verifying that a particular application will work properly for an end user. For example, a tester might open an e-commerce web application, log in, add and remove items to the cart, enter payment information and attempt to check out.

End-to-end testing, the testing pyramid and the need for automation

E2E testing sits atop the classic “testing pyramid,” with integration testing (middle) and unit testing (bottom) below it.

Unit testing is the testing of individual modules (units), while integration testing is the testing of those modules as a group. In the past, E2E testing — that is, of how the application flow works, end-to-end — was often done manually, but now there is much greater need for automation, especially as continuous integration and delivery (CI/CD) become widespread.

Test automation, at multiple levels of the pyramid, is a fundamental component of CI/CD and a defining characteristic of high-performing DevOps organizations, as documented in the Accelerate State of DevOps 2019 survey. Being able to perform automated E2E tests allows for easier identification of defects, broader test coverage and, ultimately, a better end-user experience. This is where testing frameworks come into play, by enabling test automation.

Choosing a test framework: What should you look for?

At a high level, a web testing framework automates what happens in a browser. Where this automation occurs, how easy it is to drive and what protocols,which browsers and languages are supported will vary by framework. Generally, a good testing framework for web apps should be:

• Straightforward to set up, integrate into a CI/CD pipeline and subsequently use.
• Suited to the project at hand, e.g., with the necessary programming language and protocol support.
• Equipped (or equippable) with a test runner and test reporter, to evolve each test case and understand test results, respectively.
• Enriched with modern features like automatic waits for UI elements and network traffic interception.

Choosing the right framework is pivotal given the growing overall importance of web apps. Progressive web applications (PWAs) are a prime example of the high stakes here: PWAs have become more advanced over time, particularly on Android and Chrome, and offer distinct advantages such as low cost of development and maintenance. Without reliable automated testing, though, these benefits are lost.

There is no universally right choice for a framework. The best option will depend on a company’s intended pace of development, if it’s pursuing test-driven development (TDD) and the levels of browser support it requires. At a more technical level, it is imperative to think about whether testing needs to happen directly within, or outside of, the browser, as this will affect both the speed of the testing and the amount of information provided by those tests.
‍

Selenium, DevTools and Electron.js/Node.js approaches to web test automation

For example, the Selenium Webdriver protocol automates tests outside the browser, which provides flexibility but requires an intermediary to translate its commands. In contrast, the Chrome DevTools protocol works inside the browser and accordingly is faster and more detailed.

There are tradeoffs with each of them, as well as when using a framework built from the ground on Electron.js and Node.Js. Here’s a quick overview of these three major technical approaches to the automated testing of web apps:

Via the Selenium WebDriver protocol

Selenium WebDriver is effectively a remote control for the browser. Tests happen outside of the browser. WebDriver’s own low-level API may be used for automating testing, though the protocol is often used in conjunction with frameworks like WebDriverIO for richer functionality. That’s because WebDriver itself is simply a library and does not come equipped with a test runner and assertions.

Pros

• Numerous supported languages and browsers.
• Extensive SDK, plugin selection and user community.
• Mobile device testing available via Selenium-based frameworks like Appium.

Cons

• Slow, since testing happens outside of the browser.
• Relatively limited set of features.
• Lower reliability, with less information about tests.

Using Chrome DevTools, Safari WebKit, etc.

These protocols are built into their respective browsers and execute their tests internally. Like Webdriver, they are supported by multiple standalone frameworks as well.

Pros

• Fast, because they work inside the browser.
• Lots of features, directly from browser makers.
• Extensive information about tests.

Cons

• Each browser has its own API.
• Documentation gaps are common.

Working within an Electron.js/Node.js-based architecture

The Node.js runtime is the core of numerous popular web testing frameworks. Combined with an Electron/Chromium browser, as in the Cypress framework, it can enable comprehensive real-time testing of web app events and network traffic.

Pros

• All tests happen in the same event loop as the application, via interaction of a Node.js server process and a browser.
• High reliability, speed and level of detail, due to the above.

Cons

• JavaScript-only testing.
• DOM manipulation model doesn’t reflect how users actually interact with web apps.
  ‍

The major E2E web application testing frameworks to consider

Keeping up with the ongoing evolution of different frameworks for, and approaches to, web app testing can be complex. To simplify matters, let’s examine a few of the most popular web frameworks, which between them offer viable options for each of the three technical approaches outlined in the previous section. Each of these frameworks is open source and freely available.

Cypress

What it is: A frontend web testing platform designed for applications built with modern JavaScript frameworks. Cypress consists of a locally installed test runner and a dashboard service that records and displays test results. It uses an Electron.js/Node.Js architecture (so only JavaScript is supported), interacting with its Node server process to respond to application events in real time.

Pros

• Fast and reliable, with tests performed in the same run loop as the web app, in the browser.
• Can read and alter network traffic on the fly.
• Direct debugging from within frameworks like Chrome DevTools.
• Local installation means it can use the machine’s operating system for certain automation tasks like taking screenshots.

Cons

• No multi-tab support or ability to drive multiple browsers simultaneously.
• Delivers the most value only if a team is pursuing TDD approach, as it’s more suited to unit and integration tests than E2E testing per se, even though it’s billed as an all-in-one solution.

WebDriverIO

What it is: A highly customizable browser and mobile application test automation framework for Node.js. WebDriverIO supports a variety of modern JavaScript frameworks (React, Vue.js, etc.). Moreover, it can automate tests with either Selenium Webdriver or native API integrations to Chrome DevTools.

Pros

• Testers can use WebDriver or DevTools protocols as needed.
• Extensive API documentation is available.
• WDIO command line interface makes it easy to get started.
• Numerous features, plus compatibility with Puppeteer.

Cons

• High customizability creates a steep learning curve; easy to learn, hard to master.
• Javascript-only testing.

Playwright

What it is: A cross-browser testing framework that evolved from the Puppeteer Node.js library. Whereas Puppeteer is Chrome-only, Playwright works with Firefox and Safari as well.

Pros

• One API for automating all major browsers.
• Auto-waiting for UI elements and other rich features.
• Multi-programming language support.
• Copious documentation.

Cons

• Relatively new, with a small community.
• No Webdriver support.

Puppeteer

What it is: A Node.js library for automating headless Chrome/Chromium using DevTools. It can perform key tasks such as generating PDFs and screenshots of pages, automating form submissions and testing Chrome extensions.

Pros

• Streamlines the setup of an automated testing environment within the latest version of Chrome.
• Capable of creating a timeline trace for a tab.
• Can work with other frameworks such as WebdriverIO and Jest.

Cons

• More of scripting library than a true testing framework.
• Limited compared to Playwright in particular (no cross-browser support).

TestCafe

What it is: A cross-browser E2E testing framework that runs on Node.js. TestCafe does not control the browser, nor it is a browser itself. Instead, it functions as a proxy server between the browser and Node.js, injecting the code under test with the necessary scripts.

Pros

• Simple setup and configuration; no WebDriver or manual timeouts required.
• Works with multiple desktop and mobile browsers.
• Fast and stable performance.

Cons

• Cannot automate all user actions since it doesn’t directly control the browser.
• Webpages are changed before testing and their events are simulated, not native.

Protractor

What it is: An E2E testing framework for Angular and AngularJS applications. It is built on top of WebDriver.

Pros

• Runs tests in actual browsers, using native events.
• Allows for automatic waiting for events.

Cons

• Angular and AngularJS are declining in popularity, meaning Protractor may have less utility over time.

Nightwatch.JS

What it is: A Node.js-powered E2E testing framework that uses the WebDriver protocol. It includes a command line test runner that can run tests sequentially or in parallel.

Pros

• Easier to set up than WebdriverIO.
• Clean syntax, with support for JavaScript, TypeScript, CSS and Xpath selectors.

Cons

• WebDriver only.
• Support is difficult to obtain.

CodeceptJS

What it is: One E2E framework designed to rule them all. CodeceptJS supports Webdriver, Puppeteer, Playwright, TestCafe and others for E2E testing.

Pros

• Rich functionality, including the ability to run tests across multiple browser windows.
• Testers can switch between different frameworks as needed.

Cons

• Adds complexity, with another layer of abstraction and the need to jugge between frameworks.

Moving forward with end-to-end testing

Among the above options, we typically recommend:

• Cypress, if your top priority is speed and you have implemented a TDD approach.
• WebDriverIO, if your team is developing in JavaScript and requires maximum customization and protocol support (WebDriver or DevTools).
• Playwright, if you don’t need WebDriver support and need to test across multiple browsers.

As you think about how to ensure your web applications are expertly developed and tested, the experienced Trascenda team can help. Connect with us to learn more about how we can guide you on your next project.