Continuous Testing with CI/CD, Rings and Impact Analysis

The key to building quality into a piece of software such as To-Do Studio is making sure we can get fast feedbacks on the impact of changes using a deployment pipeline with continuous testing.

In the last few months, using the Azure Pipelines service, we have worked hard to put this infrastructure in place. It relies on practices such as Continuous Integration and Delivery (CI/CD), deployment rings and impact analysis.

Continuous Integration is about automating build and test processes to make sure the resulting software is in a good state, ideally every time a developer changes code. Continuous Delivery goes one step further by automating a software release, which typically involves packaging the software for deployment in a production-like environment.

Deployment rings were first discussed in Jez Humble’s book. Rings promote a production-first DevOps mindset and limit impact on end users, while gradually deploying and validating changes in production.

Deployment rings are used to advance a software release, built in the initial stage of CI/CD, through a sequence of target users.

We identified four types of users:

• Ring 0 – Staging : Developers and testers who validate the release before deploying it to real users.
• Ring 1 – Alpha : Close circle of employees and volunteers who test new features as they become available.
• Ring 2 – Beta : Early adopters who voluntarily preview releases, considered more stable than the ones in Alpha.
• Ring 3 – Users : Users who consume the service, after passing through the alpha and beta early adopters.

Let’s look at how a change goes through the deployment rings.

1. A developer commits a change (checkin) to our source code repository.
2. The checkin triggers a Continuous Integration (CI) build.
3. The last step of the continuous integration launches a continuous delivery trigger, which automatically starts deployment in the staging environment.
4. The deployment publishes a new release to the Staging ring. Only the developers and testers are impacted by the change.
5. The deployment publishes a new release to the Alpha ring. Only the Alpha users are impacted by the change.
6. The deployment publishes a new release to the Beta ring. Only the Beta users are impacted by the change.
7. The deployment publishes a new release to the Users ring. At this stage, everyone is affected by the change.
8. It’s key to realize that the impact (“blast radius”) increases as your change moves through the rings. Exposing the change to the Alpha and the Beta users, is giving two opportunities to validate the change and hotfix critical bugs before a release to everyone.

At each ring, an impact analysis is performed. In case of insufficient quality, the deployment is rejected, and the developers must correct the problems. The process must then start over again. Impact (also called blast radius), is evaluated through observation, testing, analysis of telemetry, and user feedback. We rely primarily on the Application Insights and Azure Monitor services to perform the impact analysis.

Configuring Cypress in CI with Azure DevOps Pipelines

Cypress is a front-end test automation framework built for the modern web. It is open source and written entirely in JavaScript. It addresses the key pain points developers and QA engineers face when testing modern applications:

• A rich yet simple API for interactions with automatic waiting
• Mocha, Chai, and Sinon bundled in
• A sleek dashboard with automatic reloads for Test-Driven Development
• Easy debugging
• Network traffic control for validation and mocking
• Automatic screenshots and videos

It is a very powerful tool that enables developers to write End-to-End tests entirely in JavaScript, directly accessing everything within the browser. It is also a versatile tool that can be used to test REST APIs.

At To-Do Studio, we use Cypress to test End-to-End scenarios, as well as REST APIs.

On developers’ computers, we install the Cypress Test Runner and write tests locally. Everything works perfectly. However, for CI/CD testing, we must configure Cypress with Azure DevOps Pipelines. The recipe for configuring Cypress is not as simple as it may seem at first. I’m writing this post to make sure that a configuration recipe is documented somewhere on the web to explain how to configure Cypress in Continuous Integration (CI) with Azure DevOps Pipelines. Thus, with the help of Google search, those who will have the same goal in the future will find the right answer easily.

Here is diagram presenting an overview of the major elements of the solution:

First, ensure during the build phase that the cypress tests files are zipped and published to the artifacts drop.

Second, during the release phase hosted on an Ubuntu agent, extract the zipped test files,run the tests using Cypress and publish the tests results. Here are the detailed steps:

1. Extract the zip file containing the tests

Extract the zip file that was published to the artifacts drop by the build pipeline.

2. Run the tests

Start by creating the Cypress config file. Define the location of your tests with the integrationFolder configuration value. Do not use the testFiles configuration value or –spec command line option.

{
  "integrationFolder": "tests/e2e/specs",
  "baseUrl": "https://info-staging.to-do.studio",
  "projectId": "<insert your project Id>",
  "reporter": "junit",
  "reporterOptions": {
    "mochaFile": "tests/test-output-[hash].xml",
    "toConsole": true,
    "attachments": true
    },
  "video": false,
  "pluginsFile": "tests/e2e/plugins/index.js",
  "supportFile": "tests/e2e/support/index.js",
  "env": {
    "urlEnv": "staging"
  }
}

Follow by running the tests with Cypress using the command: npx cypress run –record –key <insert your record key>. npx install and run Cypress in a single step. npx is a npm package runner (x stands for eXecute). The typical use is to download and run a package temporarily. Please note that if you intent to record screenshots and videos with Cypresss Dashboard Service, you need to add the unique projectId into your cypress.json and pass the record key into the command. If you do not need visual results, simply omit the recording option and projectId.

3. Publish the tests results

Here you will publish the junit mocha files created during the tests run. Make sure that your merge test results and that it fails if there are test failures.