articles
Benchmark of web-applications testing tools
| Benchmark of web-applications testing tools | |
|---|---|
| Author | Alhusaine NEMER – test’n’dev |
| Date | 24/05/2023 |
| Version | 0.5 |
| Licence | CC BY-NC-ND 4.0 |
Introduction
In this article, we will compare some of popular tools used for web testing automation.
We will focus on tools having those characteristics:
- open-source tools
- executable on Node.js platform
- with implementation possible in TypeScript
- mature and quiet popular
We will begin with 4 tools (in alphabetical order):
Indeed, in this article, and to be fair in the comparison, we will take JavaScript/TypeScript versions of Playwright and Selenium.
Quick presentation of the compared tools
⚠️ to be completed
Popularity of the 4 tools
TL;DR:
- Cypress is today the most popular tool, among the 4
- Playwright is gradually gaining popularity (see trends on NPM/GitHub/Google “fame” indicators)
- Selenium remains popular, but has serious competitors
- Nightwatch seems to be less popular than the 3 others
To have an overview of “popularity” of each tool, we will use 4 indicators retrieved on NPM, GitHub, Twitter and Google.
As per end of may 2023:
| indicator | Cypress | Selenium | Playwright | Nightwatch |
|---|---|---|---|---|
| Number of weekly downloads on NPM [^1] | 4908680 | 2088040 | 1304640 | 180000 |
| Number of stars on GitHub | 43434 | 26651 | 51509 | 11370 |
| Number of followers on Twitter | 21000 | 23000 | 9000 | 3000 |
| Interest over time on Google Trends | 84,71 | 2,65 | 26,29 | 1,71 |
this graph shows the relative popularity of each tool, according to the previous indicators (NPM downloads, Github stars, Twitter followers, Google Trends).
See more details in “popularity” sub-page.
Technical comparison
| aspect | Cypress | Nighwatch | Playwright | Selenium |
|---|---|---|---|---|
| Other than JS/TS? | No | No | Yes (C#, Java, Python) | Yes, many |
| Protocol | Inside Browser | WebDriver | CDP (debug protocols) | WebDriver |
| Open-source & supported/developped by… | Cypress | BrowserStack | Microsoft | Community |
⚠️ to be completed
Installation 🚧
⚠️ to be completed
Performance 🚧
We will use our code hosted on testndev/web-app-testing-tools-benchmark repository for our technical benchmark.
⚠️ to be completed
See also
There is many articles that compare web-app test automation frameworks (especially Selenium vs Cypress vs Playwright since 2022), presenting their features, trade-offs, etc.
You can read for example:
- “Which web testing tool should i use?”
(in english, 24/04/2023, by Andrew Knight)
- “Choix du logiciel et avis - Framework : Selenium vs Playwright vs Cypress”
(in french, 13/03/2023, by Mr Suricate)
- Playwright vs. the world
(in french, 31/01/2023, by Jean-François Greffier)
- Cypress vs Selenium vs Playwright vs Puppeteer speed comparison
(in english, 04/03/2021 by Giovanni Rago)
Conclusion
⚠️ to be completed
Licence
This article is licensed under the terms of the Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) License
Credits :
…
Photo Credits :
- …