articles

Popularity of the 5 web-app. testing tools

Introduction

In this section of “Benchmark of web-applications testing tools” article, we will compare popularity of the 5 tools chosen above:

  1. Cypress.io
  2. Nightwatch.js
  3. Playwright
  4. Selenium WebDriver
  5. WebdriverIO

Methodology

We will use several indicators to compare the popularity of those tools:

Summary

The table below gives you an overview of the relative popularity of these 5 tools, with a few indicators and their evolution:

Trend on the NPM packages

The Node.js versions of those tools are available as NPM packages.

On npm Registry, and for each package, we can take the number of weekly downloads as an indicator of their popularity.

As for 17/12/2023, we have, for each package, on last 7 days:

  Cypress.io
Nightwatch.js
Playwright Test
Selenium
WebdriverIO
package name/url cypress nightwatch @playwright/test selenium-webdriver webdriverio
number of weekly downloads 5197350 142971 2779815 1726049 1142691

Thats good, but we can have a better view of the popularity of each tool, by comparing them on the same graph. And see progression over time.

This was made with data from npm trends (www.npmtrends.com). On this site, we can see the popularity of each tool, by number of weekly downloads from NPM, with different time ranges.

Before looking at graphs, let’s see the evolution of each tool, compared to last year.

Evolution compared to last year

Comparing downloads on last 7 days, between july 2022 and july 2023:

  package name 3 july 2022   1 july 2023 evolution
cypress 3673636 4865643 +32%
nightwatch 171522 179653 +2%
@playwright/test 629646 2140160 +240%
selenium-webdriver 2395860 2013031 -22%
webdriverio 924956 1141885 +23%

We can see in the graphs below:

NPM trend - graphs

Graphs copied from npm trends the 25/12/2023.

period npm trend graph
over the past year 1 Year
over the past 2 years 2 Years
over the past 5 years 5 Years
since beginning All time

Trend on GitHub

Each of those “open-source” tools is hosted on a GitHub repository.

Like for all GitHub repositories, we can take several indicators as an indicator of their popularity:

List of repositories:

Tool GitHub repository name
Cypress.io https://github.com/cypress-io/cypress
Nightwatch.js https://github.com/nightwatchjs/nightwatch
Playwright https://github.com/microsoft/playwright
Selenium https://github.com/SeleniumHQ/selenium
WebdriverIO https://github.com/webdriverio/webdriverio

There are several ways to compare the popularity of those tools on GitHub :

indicator meaning
number of stars how many people “starred/bookmarked” the repository
number of forks how many people “forked” the repository as a starting point for their own project
number of watchers how many people “watch” the repository, to be notified of new commits
number of contributors how many people contributed to the project
number of commits and/or pull requests the activity made on the project
number of issues how many issues were opened on the project
number of dependents projects how many projects depends on this project

👨‍💻 Number of contributors

We can take the number of contributors as an indicator of their popularity.

Please note that contributors can be:

Comparison of number of contributors on GitHub (december 2023):

  Cypress.io
Nightwatch.js
Playwright Test
Selenium
WebdriverIO
GitHub repository cypress-io/cypress nightwatchjs/nightwatch microsoft/playwright SeleniumHQ/selenium webdriverio/webdriverio
Number of contributors 464 122 488 726 522

Note on top contributors, for each tool, in last 12 months (Jan 1, 2022 – Dec 25, 2023):

⭐ Number of Stars on GitHub

We can take the number of stars as an indicator of their popularity.

Allmost all those tools has more than 10k stars on GitHub. Some more quickly than others…

Comparison of number of stars on GitHub (december 2023):

Tool Number of stars on GitHub + 5k stars in… + 10k stars in…
Cypress.io 45401 3 years 4 years
Nightwatch.js 11507 3 years 8 years
Playwright 57606 1 month 4 months
Selenium 28459 4 years 6 years
WebdriverIO 8433 8 years not yet…

“Aligned timeline” view

The “aligned timeline” view indicates how many “stars” each tool has, after N years of existence.

Star History Chart - aligned timeline view

“by date” view

The “by date” view gives how many “stars” each tool has, at a given date.

Star History Chart - by dates view

Google offers “Google Trends” as a data exploration tool that shows what terms people are searching for on Google (search engine) in real time.

It can be used to measure the search interest in a particular topic, location, or language at a particular time.

We can use it to compare the popularity of each tool.

For this, Google Trends offers a “interest over time” metric, that shows the popularity of a search term over time. “Interest over time” number is defined by Google by:

Numbers represent search interest relative to the highest point on the chart for the given region and time.

We gave to “Google Trends” the following search terms to Google Trends: Cypress, Nightwatch, Playwright, Selenium JS and WebdriverIO.

note: the “Selenium JS” search term is used to filter results on Selenium JavaScript version, and not on other languages.

And we filtered results with the parameters:

The 2 below graphs represent “interest over time”, for each of those tools, over the years 2022 and 2023 (excluding last 2 weeks of december).

We can see in the graphs above:

2022 2023

source: https://trends.google.com/trends/explore?cat=32&date=2022-01-12%202023-12-31&q=Cypress,Nightwatch,Playwright,Selenium%20JS,WebdriverIO&hl=en

captured on 31/12/2023


Notes :