Rollback Kubernetes deployment

This guide will walk through rolling back a Kubernetes Deployment. The example application used in this guide is a containerized version of PetClinic that will create the following pods:

• MySQL database container for the backend (deployment)
• Flyway database migrations container (job)
• PetClinic web frontend container (deployment)

Database rollbacks are out of scope for this guide; refer to this article, which discusses methods of database rollbacks and when they are appropriate.

Existing deployment process

For this guide, we’ll start with an existing deployment process for deploying PetClinic to a Kubernetes cluster:

1. Deploy MySQL Container
2. Deploy PetClinic web
3. Run Flyway job
4. Verify the deployment
5. Notify stakeholders

Zero-configuration Rollback

The easiest way to rollback to a previous version is to:

1. Find the release you want to roll back.
2. Click the REDEPLOY button next to the environment you want to roll back.

That redeployment will work because a snapshot is taken when you create a release. The snapshot includes:

• Deployment Process
• Project Variables
• Referenced Variables Sets
• Package Versions

Re-deploying the previous release will re-run the deployment process as it existed when that release was created. By default, the deploy package steps (such as deploy to IIS or deploy a Windows Service) will extract to a new folder each time a deployment is run, perform the configuration transforms, and run any scripts embedded in the package.

Simple rollback process

The most common reason for a rollback is something is wrong with the release. In these cases, you’ll want to block the bad release from moving forward.

The updated deployment process for a simple rollback would look like this:

1. Calculate Deployment Mode
2. Deploy MySQL Container (skip during rollback)
3. Deploy PetClinic web
4. Run Flyway job (skip during rollback)
5. Verify the deployment
6. Notify stakeholders
7. Block Release Progression (only during rollback)

Calculate deployment mode

Calculate Deployment Mode is a community step template created by Octopus Deploy. It compares the release number being deployed with the current release number for the environment. When the release number is greater than the current release number, it is a deployment. When it is less, then it is a rollback. The step templates sets a number of output variables, including ones you can use in variable run conditions.

Skipping database steps

Deploying the MySQL container and running the Flyway job should be skipped in a rollback scenario. For this guide, we’re going to assume that the deployment of the MySQL container wasn’t the cause for the rollback. As previously stated, database rollbacks are out of scope for this guide, so we’ll skip the Flyway Job container as well.

To ensure that both of those steps are not run during a rollback, use the following output variable from the Calculate Deployment Mode step as the Variable Run condition.

#{Octopus.Action[Calculate Deployment Mode].Output.RunOnDeploy}

Block release Progression

Blocking Release Progression is an optional step to add to your rollback process. The Block Release Progression step template uses the API to prevent the rolled back release from progressing.

This step includes the following parameters:

• Octopus Url: #{Octopus.Web.BaseUrl} (default value)
• Octopus API Key: API Key with permissions to block releases
• Release Id to Block: #{Octopus.Release.CurrentForEnvironment.Id} (default value)
• Reason: This can be pulled from a manual intervention step or set to Rolling back to #{Octopus.Release.Number}

This step will only run on a rollback; set the run condition for this step to:

#{Octopus.Action[Calculate Deployment Mode].Output.RunOnRollback}

To unblock that release, go to the release page and click the UNBLOCK button.

Complex rollback process

A feature of Kubernetes is the revision history of the cluster components. The command kubectl rollout history deployment.v1.apps/<deployment-name> lists all deployment revisions.

REVISION  CHANGE-CAUSE
1         <none>
2         <none>
3         <none>

Using this feature, we can create a rollback process that would allow us to roll back quickly.

The new deployment process would look like this:

1. Calculate Deployment Mode
2. Rollback Reason (only during rollback)
3. Deploy MySQL Container (skip during rollback)
4. Deploy PetClinic web
5. Run Flyway job (skip during rollback)
6. Verify the deployment
7. Notify stakeholders
8. Rollback to the previous version for PetClinic Web (only during rollback)
9. Block Release Progression (only during rollback)

Next, we’ll go through the newly added and altered steps:

Rollback reason

This is a Manual Intervention step that prompts the user for the reason they are rolling back. The text entered is stored in an output variable which will be used in the Block Release Progression step further down the process.

Deploy PetClinic Web

The revision history command for Kubernetes showed that there were multiple revisions stored within Kubernetes for deployment. However, it’s not obvious as to which revision belongs to which Octopus release. Adding a kubernetes.io/change-cause annotation to the Deploy PetClinic Web step would add the Octopus Release Number as the change-cause so we could later parse it for which revision to roll back to.

Running kubectl rollout history deployment.v1.apps/<deployment-name> would now show something like this.

REVISION  CHANGE-CAUSE
1         2021.09.23.0
2         2021.09.23.1
3         2021.09.23.2

Rollback to the previous version for Petclinic Web

Using the annotation from the Deploy PetClinic Web, we can use the following script to identify the revision we want to roll back to and perform the rollback using the built-in functionality of Kubernetes. This step uses the Run a Kubectl CLI Script step with the following code.

# Init variables
$k8sRollbackVersion = 0
$rollbackVersion = $OctopusParameters['Octopus.Release.Number']
$namespace = $OctopusParameters['Project.Namespace.Name']
$deploymentName = $OctopusParameters['Project.Petclinic.Deployment.Name']

# Get revision history
Write-Host "Getting deployment $deploymentName revision history ..."
$revisionHistory = (kubectl rollout history deployment.v1.apps/$deploymentName -n $namespace)
$revisionHistory = $revisionHistory.Split("`n")

# Loop through history starting at index 2 (the first couple of lines aren't versions)
Write-Host "Searching revision history for version $rollbackVersion ..."
for ($i = 2; $i -lt $revisionHistory.Count - 1; $i++)
{
    # Split it into two array elements
    $revisionSplit = $revisionHistory[$i].Split(" ", [System.StringSplitOptions]::RemoveEmptyEntries)
    
    # Check version
    if ($revisionSplit[1] -eq $rollbackVersion)
    {
        # Record version index
        Write-Host "Version $rollbackVersion found!"
        $k8sRollbackVersion = $revisionSplit[0]
        
        # Get out of for
        break
    }
}

# Check to see if something was found
if ($k8sRollbackVersion -gt 0)
{
    # Issue rollback
    Write-Host "Rolling Kubernetes deployment $deploymentName to revision $k8sRollbackVersion ..."
    kubectl rollout undo deployment.v1.apps/$deploymentName -n $namespace --to-revision=$k8sRollbackVersion
}
else
{
    Write-Error "Version $rollbackVersion not found in the cluster revision history."
}

Block release progression

The Rollback Reason step captures the reason for the rollback. We can pass the text entered in this step to the Reason field of this step by using the following output variable.

#{Octopus.Action[Rollback reason].Output.Manual.Notes}

Choosing a rollback strategy

It is our recommendation that you start with the simple rollback strategy, moving to the complex if you determine that the simple method doesn’t suit your needs.

Help us continuously improve

Please let us know if you have any feedback about this page.

Send feedback

Page updated on Wednesday, July 31, 2024