Using Variables in Azure Devops Pipelines

Recently I got a chance to work a lot with Azure Devops, specially with various pipelines. I was pleasantly surprised by the breadth and depth of the features provided for various ALM stages and specially for developing build and release pipelines.

I liked the ease of working with YAML based pipelines and how it gets integrated with repository, allowing us to maintain all the versions along with editing from within the code editor. However, one of the most powerful feature I liked was the use of Variables, which can make pipelines really dynamic and help keep many configurable parameters out of developers view.

Types of Variables

In general, variables in Azure Devops can be classified under System Variables, Environment Variables and User Defined Variables.

System Variables: Contains predefined values for the pipeline run, like Build.ArtifactStagingDirectory, Build.BuildID etc. A comprehensive list of System variables can be found in this article.
Environment Variables: These are specific to OS being used and get injected in pipeline in platform specific ways.
User Defined Variables: These are the variables which we set and (re)use in pipelines. This is a vast area – can be written in different ways like macro, template or runtime – and at different levels in a pipeline like root, stage or job levels. This provides the flexibility and power to make the pipeline extremely configurable and reusable.

So, I am going to focus on usage of specifically User Defined Variables within Azure Devops Pipelines in this article.

Declare Variables

There are various ways to declare User Defined Variables in pipelines.

Declare Variables in the pipeline YAML

This is the most straightforward way to declare any variable. Just use the name/value pair under the section variables. This can be declared at root, stage or job levels.

When a variable is defined at the top of a YAML, it will be available to all jobs and stages in the pipeline and is a global variable. Variables at the job level override variables at the root and stage level. Variables at the stage level override variables at the root level.

variables:
 - name: servername
   value: server01
 - name: connectionname
   value: connection01

This is easier to use, as variables can be declared and used within the same YAML pipeline and is visible explicitly. However, this also make it difficult to use with different values as everytime the YAML needs to be changed and checked in!

Declare Variables in Pipeline UI

Now, this option makes it a easier to change values without charging the YAML pipeline everytime. We can declare these variables using the “Variables” button on top right corner of the YAML pipeline. Just provide the same name/value pair and save.

The checkbox “Let users override this value when running this pipeline” enables overwriting the variables value within the YAML pipeline. Like if based on some logic within pipeline, you want to change the value to server02.

Declare variables in Variable Groups

This is my favourite option of all. This not only provides a clear separation between the YAML and variables, but also helps with reusability. One variable group can be used in multiple pipelines. It can not only keep name/value pair but also secret files. Along with these, we can secure the variable groups with granular permissions & Approval checks and restrict them to selected pipelines as well.

In this example, I am using multiple variable groups where some are going to be used in all pipelines, whereas some are specific to specific environments.

And within each variable group, multiple variables can be declared

We can also link a variable group with Azure keyvault and select the secrets to create variables out of them. We can pick n choose which secrets to link. You can see a bit more about this later in the article.

Using Variables in YAML Pipelines

Now that we have covered some options of declaring variables, let’s take a look at how to use them in our YAML pipelines. All we need to do is to refer our variable groups in our pipeline and all those variables across those variable groups will become available to be used throughout the pipeline

trigger:
- master

pool:
  vmImage: 'windows-latest'

variables:
- group: Infra
- group: Infra-SIT
- group: Secrets

Macro Syntax

This is probably the most popular way to using variables. The syntax is simply $(variablename). Variables with macro syntax get processed before a task executes during runtime.

So, how to use these variables in Devops Pipelines to perform various tasks?

Using Variables to Deploy ARM Templates

This example uses the task “ARM Template Deployment” of Azure Devops. In the example below, values in bold are variables to deploy an Azure ARM template.

          - task: AzureResourceManagerTemplateDeployment@3
            displayName: "Deploy Firewall Policies"
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ResourceGroup)'
              location: 'West Europe'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/Firewall/FirewallPolicyDeploy.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/Firewall/FirewallPolicy-Parameters-SIT.json'
              deploymentMode: 'Incremental'

But what if, we want to use variables within ARM template file – FirewallPolicy-Parameters-SIT.json in this example? Yes, it’s true that you can hardcode values in parameters file and create multiple parameters files for different environments like UAT and PROD, but the idea here is to use variables where such values are used in different other places as well. This is how my ARM Parameters file looks like:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "value": "westeurope"
        },
      "resourceGroup": {
        "value": "ReadFromVariable_Infra-SIT_ResourceGroup"
      },
      "firewallPolicyName": {
        "value": "ReadFromVariable_Infra-SIT_FirewallPolicyName"
      }
    }
}

I have just added some values in the parameter here to highlight that the values will be actually read from variables. We can use “overrideParameters” to pass additional parameters in the pipeline, which will overwrite the values in runtime. The syntax is “-<parametername> $(variable)”. I have highlighted the section below.

          - task: AzureResourceManagerTemplateDeployment@3
            displayName: "Deploy Firewall Policies"
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ResourceGroup)'
              location: 'West Europe'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/Firewall/FirewallPolicyDeploy.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/Firewall/FirewallPolicy-Parameters-SIT.json'
              overrideParameters: '-resourceGroup "$(ResourceGroup)" -firewallPolicyName $(FirewallPolicyName)'
              deploymentMode: 'Incremental'

Using Variables with PowerShell Scripts

Executing Powershell/Bash scripts are one of the most used tasks in Azure Devops. So, how to do the same variable replacement in that case? Let’s take a look at the script below. It just creates a Keyvault in Azure.

param (
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string] $AKsResourceGroupName = 'ReadFromVariable_Env_ResourceGroup',

    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string] $keyVault = 'ReadFromVariable_Env_KeyVaultName',

)


Write-Host "--- Create Managed Identity for keyvault to AKS Env---" -ForegroundColor Cyan


az identity create `
  --resource-group $AKsResourceGroupName `
  --name $KeyvaultManagedidentity

Write-Host "--- Complete: Key vault Managed identity for AKS Env ---" -ForegroundColor Green

Now, how to execute this script from within Azure Devops pipeline? We use the task “Azure CLI” of Devops and pass the variables as arguments.

          - task: AzureCLI@2
            displayName: "Create Keyvault"
            inputs:
              azureSubscription: '$(ARMConnection)'
              scriptType: 'ps'
              scriptLocation: 'scriptPath'
              scriptPath: '$(System.ArtifactsDirectory)/AKS-Infra/CreateKeyvault.ps1'
              arguments: '-AKsResourceGroupName $(ResourceGroup) -keyVault $(KeyvaultName)'

Using Variables in JSON/XML Files

So, till now we used variables in ARM json files and PowerShell scripts, but what if you have some XML or text files which you want to use as part of the solution and want to use variables inside those to achieve some runtime replacement of values.

There is an out of the box task “File Transform” that you can use to replace the variables declared inside those files with the corresponding values from the variables under variable groups.

Using Azure Keyvault with Variables

This is one of the recommended way of storing and using secrets in your application, like admin passwords or any other such secrets.

The idea is simple, you create such secrets in Azure key vault. Provide the Azure Devops service connection service principal account permission to Get and List the secrets.

Then create a variable group and link to that Keyvault store, as explained previously in this article. Now, we can simply use the task “Azure Key Vault” to fetch the keys and populate the associated variables in run time.

          - task: AzureKeyVault@1
            displayName: 'Get Secrets from Azure Key Vault'
            inputs:
              azureSubscription: '$(ARMConnection)'
              KeyVaultName: '$(KeyvaultName)'
              SecretsFilter: '*'
              RunAsPreJob: true

Using Replace Token Task

This is an extension that you can install in your Devops organization, which expands the token replacement feature mentioned in previous step. This action supports a number of other file types. You can install this directly from here. You can find the details about this task here. This is also commonly used after getting secrets from Key vault to replace the secrets in corresponding files.

This task can be used like this:

          - task: replacetokens@3
            displayName: "Replace Tokens in Secrets & Specs"
            inputs:
              rootDirectory: '$(System.ArtifactsDirectory)'
              targetFiles: |
                **/*.json
                **/*.txt
              encoding: 'auto'
              writeBOM: true
              actionOnMissing: 'warn'
              keepToken: true
              tokenPrefix: '$('
              tokenSuffix: ')'
              useLegacyPattern: false
              enableTelemetry: false

In the above example, it will look into all the json and txt files in your root directory and if it finds any matching variables in those files, will replace those will the corresponding values from the Key vault or any other variable groups.

Final Thoughts

Of course there is much more to Variables in Azure Devops than I have highlighted here. But the idea was to explain how to make use of variables practically. For detailed understanding, we do have an awesome article from Microsoft.

Hope this helps.

Thanks,
Anupam

Cloud Decoded