Continuous Testing with CI/CD, Rings and Impact Analysis

The key to building quality into a piece of software such as To-Do Studio is making sure we can get fast feedbacks on the impact of changes using a deployment pipeline with continuous testing.

In the last few months, using the Azure Pipelines service, we have worked hard to put this infrastructure in place. It relies on practices such as Continuous Integration and Delivery (CI/CD), deployment rings and impact analysis.

Continuous Integration is about automating build and test processes to make sure the resulting software is in a good state, ideally every time a developer changes code. Continuous Delivery goes one step further by automating a software release, which typically involves packaging the software for deployment in a production-like environment.

Deployment rings were first discussed in Jez Humble’s book. Rings promote a production-first DevOps mindset and limit impact on end users, while gradually deploying and validating changes in production.

Deployment rings are used to advance a software release, built in the initial stage of CI/CD, through a sequence of target users.

We identified four types of users:

• Ring 0 – Staging : Developers and testers who validate the release before deploying it to real users.
• Ring 1 – Alpha : Close circle of employees and volunteers who test new features as they become available.
• Ring 2 – Beta : Early adopters who voluntarily preview releases, considered more stable than the ones in Alpha.
• Ring 3 – Users : Users who consume the service, after passing through the alpha and beta early adopters.

Let’s look at how a change goes through the deployment rings.

1. A developer commits a change (checkin) to our source code repository.
2. The checkin triggers a Continuous Integration (CI) build.
3. The last step of the continuous integration launches a continuous delivery trigger, which automatically starts deployment in the staging environment.
4. The deployment publishes a new release to the Staging ring. Only the developers and testers are impacted by the change.
5. The deployment publishes a new release to the Alpha ring. Only the Alpha users are impacted by the change.
6. The deployment publishes a new release to the Beta ring. Only the Beta users are impacted by the change.
7. The deployment publishes a new release to the Users ring. At this stage, everyone is affected by the change.
8. It’s key to realize that the impact (“blast radius”) increases as your change moves through the rings. Exposing the change to the Alpha and the Beta users, is giving two opportunities to validate the change and hotfix critical bugs before a release to everyone.

At each ring, an impact analysis is performed. In case of insufficient quality, the deployment is rejected, and the developers must correct the problems. The process must then start over again. Impact (also called blast radius), is evaluated through observation, testing, analysis of telemetry, and user feedback. We rely primarily on the Application Insights and Azure Monitor services to perform the impact analysis.

Bracket Show

As an entrepreneur, to launch a service as ambitious as a To-Do Studio, one must work on many fronts at the same time. In parallel to reviewing the user experience, over the last few weeks, I’ve been putting some efforts into engineering issues such as DevOps and testing strategy. I prioritize this work to respond to an invitation from the Bracket Show, a YouTube channel about software development (in French), led by my good friends Eric De Carufel and Bruno Barrette.

They asked me to share with their viewers the new cloud testing practices that we intend to apply for our service To-Do Studio.

For French-speaking readers who are interested in DevOps and new cloud testing practices , here is the link to listen to the video.

Tester en production

Do not hesitate to subscribe to the Bracket Show.