Leanpub Header

Skip to main content
Go to Leanpub.comLeanpub

Practical Continuous Testing

make Agile/DevOps real

Zhimin Zhan

Continuous Testing (CT) is a process of executing automated end-to-end (UI) tests frequently as regression testing on the new builds. If all tests have passed, the software is ready for a production release. If there are test failures, the team must act quickly on the feedback. "Test Early and Test Often" enables "Release Early and Release Often".

The author is letting you choose the price you pay for this book!

Pick Your Price...
PDF
EPUB
WEB
About

About

About the Book

The trend of software development is moving towards frequently releasing backed up automated functional testing. This is Continuous Testing.

  1. Unit tests ⇒ Functional tests
  2. Adhoc test execution ⇒ Repeatable process to run all automated tests
  3. Programmers only ⇒ The whole team
  4. Testing for better code quality ⇒ Testing for overall quality and frequent releases 

This movement shall not come as a surprise, as being able to push out software releases frequently with high quality is what every software project team is pursuing, i.e, “Quality at speed”.  In my view, this is the true objective of Agile and DevOps.

Despite all the hype of CT (and previously CI/CD) and DevOps, most software projects failed on functional testing, hence, on Agile or DevOps. Not convinced? You can try to answer this question: "Does your team have the capability to push software updates to the production every day?"

To succeed in Continuous Testing, software projects need to overcome the following four technical challenges:

  • Functional Test Automation
  • Continuous Execution in a CI/CT server
  • Scaling with Parallel execution in multiple build agents
  • Maintaining test execution (scripts and infrastructure) with high efficiency

In my opinion, the key to overcome the challenges is to courageously adjust the approach based on the feedback from CT.

  • If the team spends significant time maintaining Cucumber tests, why not use a much simpler and better BDD framework RSpec?
  • The testers and business analysts find it difficult to read test scripts in Java, why not change it to a good scripting language such as Ruby?
  • If the execution time of automated functional tests is too long, why not try parallel execution?
  • The current End-2-End test suite fails in a CI Server (such as Jenkins) every day with high failure rate, why not change the CI server to a proper CT server?
  • ...

In this book, I will share my 14 years of experience in CT to help you to implement real Agile and DevOps for your organizations.

What's unique about this book? Being practical. By following the instructions in Chapter 2 (included in the sample), you shall be able to set up a CT server and run a set of Selenium tests on your machine within one hour. How can that be? The answer is to use a real CT server, BuildWise. BuildWise is a free, open-source CT Server that I have created from the ground up to manage executing automated functional tests better. AgileWay (my company) has been using BuildWise since 2013, which enables our DevOps process: releasing all our (Web and Desktop) apps to production daily. BuildWise won the 2nd prize of the prestigious Ruby Award in 2018.

Learning CT techniques with a real CT server is much easier, more engaging and more fun. Very soon, you shall be able to set up a CT process at work.

Share this book

Categories

DevOps

Feedback

Email the author

Price

Pick Your Price...

Minimum price

$20.00

$30.00

You pay

$30.00

Author earns

$24.00
$

All prices are in US $. You can pay in US $ or in your local currency when you check out.

EU customers: prices exclude VAT, which is added during checkout.

...Or Buy With Credits!

Number of credits (Minimum 2)

2
The author will earn $24.00 from your purchase!
You can get credits monthly with a Reader Membership

Author

About the Author

Zhimin Zhan

Zhimin Zhan is the founder and principal agile testing coach of AgileWay Pty Ltd, Australia. As an advisor and coach, he helps organisations implement test automation using Continuous Testing with open technologies, such as Selenium WebDriver / Appium. Zhimin is the creator of TestWise, the next-generation functional testing tool that supports functional test refactorings, and BuildWise, an international award-winning Continuous Testing Server. Zhimin is a frequent speaker and author of 14 books on software testing and programming. He shares his test automation and CI experience on Substack and Medium.

Contents

Table of Contents

Preface

  1. My Continuous Testing journey
  2. What’s unique about this book?
  3. Who should read this book?
  4. How to read this book?
  5. Send me feedback

1Introduction

  1. 1.1What is Continuous Testing?
  2. 1.2Continuous Testing is the trend
  3. 1.3Continuous Testing vs Continuous Integration
  4. 1.4Separate CT from existing CI/CD
  5. 1.5Continuous Testing vs DevOps
  6. 1.6Reality Check
  7. 1.7Why will this book help?

2Set up a CT server to run Selenium tests in minutes

  1. 2.1Objectives
  2. 2.2Prerequisite
  3. 2.3Install CT Server
  4. 2.4Create a Build Project
  5. 2.5Trigger test execution manually
  6. 2.6Feedback while test execution in progress
  7. 2.7Build finished
  8. 2.8Common Errors
  9. 2.9Exercise: Create a new Build Project from another repository

3How Continuous Testing Works?

  1. 3.1Terminology
  2. 3.2CT Process in detail
  3. 3.3Install BuildWise Server for production use
  4. 3.4Understand build working directories
  5. 3.5Exercise: Fix a failed build
  6. 3.6Review

4Why is CT important?

  1. 4.1Continuous Testing is the key to Agile
  2. 4.2Detect regression errors quickly
  3. 4.3Reduce/eliminate the needs for Defect Tracking
  4. 4.4Try new ideas / upgrades
  5. 4.5CT is vital for the maintenance
  6. 4.6Training
  7. 4.7Benefits for All Team Members
  8. 4.8Review

5Set up your own build project

  1. 5.1Prepare your test scripts
  2. 5.2Create a new build project
  3. 5.3Trigger a build manually
  4. 5.4View a build in progress
  5. 5.5Cancel a build
  6. 5.6View Change Log
  7. 5.7Build history
  8. 5.8View test failures and screenshot
  9. 5.9View test script content
  10. 5.10View test execution history
  11. 5.11Build report

6How to succeed in CT?

  1. 6.1Definition of Success: AgileWay CT Grading
  2. 6.2Functional test automation and continuous execution are interdependent
  3. 6.3Success Factors (test automation)
  4. 6.4Success Factors (infrastructure)
  5. 6.5Success Factors (continuous execution)
  6. 6.6Success Factors (human)

7The Magic - Build Script

  1. 7.1What is Build Script?
  2. 7.2Use Build Script in BuildWise
  3. 7.3Configure Build Tasks in BuildWise
  4. 7.4Execute build tasks from the command line
  5. 7.5Functional Testing (Sequential) Task
  6. 7.6Functional Testing (Parallel) Task

8Manage a project

  1. 8.1Build Step Management
  2. 8.2Common Pre-Functional-Testing Tasks
  3. 8.3Clone an existing project
  4. 8.4Build Artifacts
  5. 8.5Enable the project’s API key
  6. 8.6Notifications
  7. 8.7Project Statistics
  8. 8.8Hide a project

9Manage builds

  1. 9.1Trigger a build via API
  2. 9.2Schedule a build
  3. 9.3Add Build Summary
  4. 9.4Invalidate a build
  5. 9.5Delete a build
  6. 9.6View and download build artifacts
  7. 9.7View build logs

10Sequential E2E Test Execution

  1. 10.1Sequential Mode
  2. 10.2Pre-requisite
  3. 10.3Decide what tests to be included
  4. 10.4Select test script files in the build script
  5. 10.5Define a build target
  6. 10.6Task Configuration in BuildWise
  7. 10.7Test Execution Order
  8. 10.8Feedback during the build
  9. 10.9Analyse test reports
  10. 10.10Anti-Pattern: Split tests into multiple sub-builds

11Sequential Test Execution Best Practices

  1. 11.1Limit the test count
  2. 11.2Show test results immediately
  3. 11.3Capture the error stack trace
  4. 11.4Capture the error screenshot
  5. 11.5View test script content
  6. 11.6View test output
  7. 11.7Customize test executions with Environment Variables
  8. 11.8Clone build project for different purposes
  9. 11.9Dynamic Ordering

12Parallel Test Execution

  1. 12.1Build Agents
  2. 12.2Prerequisite for Test Scripts
  3. 12.3Multi-Agents against the Single Server
  4. 12.4Multi-Agents against the Multi-Servers
  5. 12.5Set up a parallel build project
  6. 12.6Monitor the build progress
  7. 12.7Assess the benefits of parallelism
  8. 12.8Common Issues

13Parallel Test Execution Best Practices

  1. 13.1Optimal agent count
  2. 13.2Intelligent test execution ordering
  3. 13.3Auto retry failed tests once more
  4. 13.4Manually rerun a failed test
  5. 13.5Cross-Platform Testing
  6. 13.6Headless or not?
  7. 13.7Distribution rules
  8. 13.8Delay completion

14Parallel Testing Lab

  1. 14.1Hosting option: Cloud-based or On-premises?
  2. 14.2Cost
  3. 14.3Advice: starting small, grow gradually
  4. 14.4My preferred setup for CT Lab
  5. 14.5Set up BuildWise Server
  6. 14.6Set up Build Agents
  7. 14.7Best Practices
  8. 14.8Wrap up

15Why do most projects fail in CT?

  1. 15.1Failure Factors - test automation
  2. 15.2Failure Factors - infrastructure
  3. 15.3Failure Factors - continuous execution
  4. 15.4Failure Factors - Human

16CT with web app testing in other frameworks

  1. 16.1BuildWise supports multi-frameworks
  2. 16.2PyTest (Python)
  3. 16.3Mocha (JavaScript)
  4. 16.4Cucumber (Ruby)
  5. 16.5Parallel Build with multi frameworks
  6. 16.6Review

17CT with Native apps and Microservices

  1. 17.1Will my tests run in this CT server?
  2. 17.2Desktop App Testing with Appium
  3. 17.3Non-UI Functional Testing

18Other Uses

  1. 18.1Load Testing
  2. 18.2Cross-Browser Testing
  3. 18.3Prepare Application Data
  4. 18.4Requirement Traceability
  5. 18.5Execute specific tests on Server via Web Interface
  6. 18.6Utilities
  7. 18.7Wrap up

Appendix 1 - CI Steps

Resources

  1. Books
  2. Tools

References

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

Download Sample PDFDownload Sample EPUB

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

Now, this is technically risky for us, since you'll have the book or course files either way. But we're so confident in our products and services, and in our authors and readers, that we're happy to offer a full money back guarantee for everything we sell.

You can only find out how good something is by trying it, and because of our 100% money back guarantee there's literally no risk to do so!

So, there's no reason not to click the Add to Cart button, is there?

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $14 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub