Mario Cardinal

"The real voyage of discovery consists, not in seeking new landscapes, but in having new eyes" – Marcel Proust


Leave a comment

Continuous Testing with CI/CD, Rings and Impact Analysis

The key to building quality into a piece of software such as To-Do Studio is making sure we can get fast feedbacks on the impact of changes using a deployment pipeline with continuous testing.

In the last few months, using the Azure Pipelines service, we have worked hard to put this infrastructure in place. It relies on practices such as Continuous Integration and Delivery (CI/CD), deployment rings and impact analysis.

Continuous Integration is about automating build and test processes to make sure the resulting software is in a good state, ideally every time a developer changes code. Continuous Delivery goes one step further by automating a software release, which typically involves packaging the software for deployment in a production-like environment.

Deployment rings were first discussed in Jez Humble’s book. Rings promote a production-first DevOps mindset and limit impact on end users, while gradually deploying and validating changes in production.

Deployment rings are used to advance a software release, built in the initial stage of CI/CD, through a sequence of target users.

We identified four types of users:

  • Ring 0 – Staging : Developers and testers who validate the release before deploying it to real users.
  • Ring 1 – Alpha : Close circle of employees and volunteers who test new features as they become available.
  • Ring 2 – Beta : Early adopters who voluntarily preview releases, considered more stable than the ones in Alpha.
  • Ring 3 – Users : Users who consume the service, after passing through the alpha and beta early adopters.

rings

Let’s look at how a change goes through the deployment rings.

continuous-testing

  1. A developer commits a change (checkin) to our source code repository.
  2. The checkin triggers a Continuous Integration (CI) build.
  3. The last step of the continuous integration launches a continuous delivery trigger, which automatically starts deployment in the staging environment.
  4. The deployment publishes a new release to the Staging ring. Only the developers and testers are impacted by the change.
  5. The deployment publishes a new release to the Alpha ring. Only the Alpha users are impacted by the change.
  6. The deployment publishes a new release to the Beta ring. Only the Beta users are impacted by the change.
  7. The deployment publishes a new release to the Users ring. At this stage, everyone is affected by the change.
  8. It’s key to realize that the impact (“blast radius”) increases as your change moves through the rings. Exposing the change to the Alpha and the Beta users, is giving two opportunities to validate the change and hotfix critical bugs before a release to everyone.

At each ring, an impact analysis is performed. In case of insufficient quality, the deployment is rejected, and the developers must correct the problems. The process must then start over again. Impact (also called blast radius), is evaluated through observation, testing, analysis of telemetry, and user feedback. We rely primarily on the Application Insights and Azure Monitor services to perform the impact analysis.


4 Comments

Configuring Cypress in CI with Azure DevOps Pipelines

Cypress is a front-end test automation framework built for the modern web. It is open source and written entirely in JavaScript. It addresses the key pain points developers and QA engineers face when testing modern applications:

  • A rich yet simple API for interactions with automatic waiting
  • Mocha, Chai, and Sinon bundled in
  • A sleek dashboard with automatic reloads for Test-Driven Development
  • Easy debugging
  • Network traffic control for validation and mocking
  • Automatic screenshots and videos

It is a very powerful tool that enables developers to write End-to-End tests entirely in JavaScript, directly accessing everything within the browser. It is also a versatile tool that can be used to test REST APIs.

At To-Do Studio, we use Cypress to test End-to-End scenarios, as well as REST APIs.

On developers’ computers, we install the Cypress Test Runner and write tests locally. Everything works perfectly. However, for CI/CD testing, we must configure Cypress with Azure DevOps Pipelines. The recipe for configuring Cypress is not as simple as it may seem at first. I’m writing this post to make sure that a configuration recipe is documented somewhere on the web to explain how to configure Cypress in Continuous Integration (CI) with Azure DevOps Pipelines. Thus, with the help of Google search, those who will have the same goal in the future will find the right answer easily.

Here is diagram presenting an overview of the major elements of the solution:

Deployment_overview

First, ensure during the build phase that the cypress tests files are zipped and published to the artifacts drop.

build

Second, during the release phase hosted on an Ubuntu agent, extract the zipped test files,run the tests using Cypress and publish the tests results. Here are the detailed steps:

1. Extract the zip file containing the tests

Extract the zip file that was published to the artifacts drop by the build pipeline.

extract

2. Run the tests

Start by creating the Cypress config file. Define the location of your tests with the integrationFolder configuration value. Do not use the testFiles configuration value or –spec command line option.

{
  "integrationFolder": "tests/e2e/specs",
  "baseUrl": "https://info-staging.to-do.studio",
  "projectId": "<insert your project Id>",
  "reporter": "junit",
  "reporterOptions": {
    "mochaFile": "tests/test-output-[hash].xml",
    "toConsole": true,
    "attachments": true
    },
  "video": false,
  "pluginsFile": "tests/e2e/plugins/index.js",
  "supportFile": "tests/e2e/support/index.js",
  "env": {
    "urlEnv": "staging"
  }
}

cypress_config

Follow by running the tests with Cypress using the command: npx cypress run –record –key <insert your record key>. npx install and run Cypress in a single step. npx is a npm package runner (x stands for eXecute). The typical use is to download and run a package temporarily. Please note that if you intent to record screenshots and videos with Cypresss Dashboard Service, you need to add the unique projectId into your cypress.json and pass the record key into the command. If you do not need visual results, simply omit the recording option and projectId.

run_tests

3. Publish the tests results

Here you will publish the junit mocha files created during the tests run. Make sure that your merge test results and that it fails if there are test failures.

publish_tests


1 Comment

How to clean up the wwwroot folder on the Azure Web App using PowerShell

I’m writing this post to make sure that a PowerShell recipe to remove all the files in the wwwroot directory on the Azure Web App is documented somewhere on the web. Thus, with the help of Google search, those who will have the same goal in the future will find the right answer easily.

Recently, when deploying the To-Do Studio website with Azure DevOps, I needed to clean the wwwroot directory before deploying. It is usually a simple configuration setting when running the release on a hosted VS2017 agent. When using the Azure App Service Deploy task with the Publish using Web Deploy option, there is an additional option to Remove Additional Files at Destination. Unfortunately, in order to roll the automated E2E tests with Cypress, we must run the release script on a Linux agent. This Remove Additional Files at Destination option is not available anymore on Linux. So I had to automate the cleanup  with an Azure Pipeline task.

At first, to fill this gap, I thought about using the “Azure WebApp Virtual File System Tasks” available in the Azure DevOps marketplace. Unfortunately, this task did not work. So I had to automate the cleanup with a Powershell script.

The non-automated (manual) way to delete files / folders in Azure Web App is to use the Kudu console. For those who like me want to automate this operation with PowerShell, there is KUDU Virtual File System (VFS) Rest API.

Without further ado, here is the script.  You can download it from my Personal GitHub repos or you can copy and paste without restraint from the script below.

$WebAppName = “insert Web App name”
$slotName = “insert Slot Name”
$username = ‘insert the username from the publish profile’ #From the publish profile
$password = “insert the password from the publish profile” #From the publish profile
# Initialize parameters for Invoke-RestMethod
if ($slotName -ne “”){
    $apiUrl = https://$webAppName`-$slotName.scm.azurewebsites.net/api/vfs/site/wwwroot/”
}
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes((“$($username):$($password)”)))
$headers = @{
    Authorization=“Basic $($base64AuthInfo)”
    ‘If-Match’ = ‘*’
}
$userAgent = “powershell/2.0”
# Define a reursive function to delete files
function DeleteKuduDir ($content, $dir)
{
    foreach($c in $content)
    {      
        if($c.mime -eq “inode/directory”)
        {
            # Get listing of directory as an array
            $childContent = Invoke-RestMethod Uri $c.href Headers $headers UserAgent $userAgent Method GET ContentType “application/json”
           
            # Delete directory
            $newDir = $dir + (Split-Path $c.href leaf) + “\”
            DeleteKuduDir content $childContent dir $newDir
        }
        # Delete file
        $file = Split-Path $c.href leaf
        Write-Host “Deleting” $dir$file    
        $result = Invoke-RestMethod Uri $c.href Headers $headers UserAgent $userAgent Method DELETE ContentType “application/json”
    }
}
# Get listing of wwwroot as an array
$rootContent = Invoke-RestMethod Uri $apiUrl Headers $headers UserAgent $userAgent Method GET ContentType “application/json”
# Delete files and directory in wwwroot
DeleteKuduDir content $rootContent dir “\”
Write-Host “Done!”