Mario Cardinal

"The real voyage of discovery consists, not in seeking new landscapes, but in having new eyes" – Marcel Proust

Continuous Testing with CI/CD, Rings and Impact Analysis

Leave a comment

The key to building quality into a piece of software such as To-Do Studio is making sure we can get fast feedbacks on the impact of changes using a deployment pipeline with continuous testing.

In the last few months, using the Azure Pipelines service, we have worked hard to put this infrastructure in place. It relies on practices such as Continuous Integration and Delivery (CI/CD), deployment rings and impact analysis.

Continuous Integration is about automating build and test processes to make sure the resulting software is in a good state, ideally every time a developer changes code. Continuous Delivery goes one step further by automating a software release, which typically involves packaging the software for deployment in a production-like environment.

Deployment rings were first discussed in Jez Humble’s book. Rings promote a production-first DevOps mindset and limit impact on end users, while gradually deploying and validating changes in production.

Deployment rings are used to advance a software release, built in the initial stage of CI/CD, through a sequence of target users.

We identified four types of users:

  • Ring 0 – Staging : Developers and testers who validate the release before deploying it to real users.
  • Ring 1 – Alpha : Close circle of employees and volunteers who test new features as they become available.
  • Ring 2 – Beta : Early adopters who voluntarily preview releases, considered more stable than the ones in Alpha.
  • Ring 3 – Users : Users who consume the service, after passing through the alpha and beta early adopters.

rings

Let’s look at how a change goes through the deployment rings.

continuous-testing

  1. A developer commits a change (checkin) to our source code repository.
  2. The checkin triggers a Continuous Integration (CI) build.
  3. The last step of the continuous integration launches a continuous delivery trigger, which automatically starts deployment in the staging environment.
  4. The deployment publishes a new release to the Staging ring. Only the developers and testers are impacted by the change.
  5. The deployment publishes a new release to the Alpha ring. Only the Alpha users are impacted by the change.
  6. The deployment publishes a new release to the Beta ring. Only the Beta users are impacted by the change.
  7. The deployment publishes a new release to the Users ring. At this stage, everyone is affected by the change.
  8. It’s key to realize that the impact (“blast radius”) increases as your change moves through the rings. Exposing the change to the Alpha and the Beta users, is giving two opportunities to validate the change and hotfix critical bugs before a release to everyone.

At each ring, an impact analysis is performed. In case of insufficient quality, the deployment is rejected, and the developers must correct the problems. The process must then start over again. Impact (also called blast radius), is evaluated through observation, testing, analysis of telemetry, and user feedback. We rely primarily on the Application Insights and Azure Monitor services to perform the impact analysis.

Author: mariocardinal

I am the co-founder of To-Do Studio, a software publisher offering online collective workspaces extended with automated guides. As an experimenter and an entrepreneur, I like to seize opportunities that emerge from the unexpected. Since 2004 (15 years in a row), I am a recipient of the Microsoft Most Valuable Professional (MVP) award. MVP status is awarded to credible technology experts who have shown a deep commitment to innovation, passion about technology and a strong community spirit. An experienced DevOps and Scrum practitioner, I have spent nearly 30 years designing large-scale information systems. I am the author of the book "Executable Specifications with Scrum" and the host of the Visual Studio Talk Show, a podcast about software architecture.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s