Sitecore Production Environment on Azure Kubernetes Services – Part 4 (Putting everything together with Azure Devops)

Finally I found sometime to write the last article of this series – Sitecore Production Environment on Azure Kubernetes Services – Series, in which I will cover combining everything together to automate most of the deployment part with Azure Devops Pipelines.

Till now, we have prepared the Azure Components (check out the Part 1 of this series), External Data Sources (check out Part 2 of this series) and Sitecore installation (check out Part 3 of this series). Now, we’ll create various Azure pipelines to put all these steps together.

Problem Statement

As you can see in pervious articles, there are quite a number of steps involved in correctly setting up a Sitecore production environment on AKS. It would be extremally useful to automate most of it, so that we can (re)create the entire environment within a matter of hours and not days.

What we need to do in Azure Devops:

Prepare Azure Devops Repo
Create Pipeline to Deploy Azure Resources
Create Pipeline to Install Sitecore

How to go about it

Before we could start putting our scripts, JSON and yml files in Azure Devops Repo, we need to make some preparations. There are some resources which are required to be there (like Keyvault) before we move ahead with deploying all other resources. Also, there are some Sitecore pre-requisites (like preparing tls certificates and populating Keyvault) which I have explained in the article – Sitecore Setup on AKS (Part 3) – already.

Initial Preparations

It’s better to use Azure Devops Variables which will help avoid much of the repeated hardcoded values. I had written another article on this topic – Using Variables in Azure Devops.

I have created multiple variable groups to hold variables which will be utilized in different pipeline steps.

Some examples of these variables are Names of your Azure resources, IP ranges, AKS and Sitecore Credentials and Configuration etc.

Prepare Azure Devops Repo

The way I had structured My Devops Repo is really straight forward. Main top level folders – AKS-Infra and Sitecore. You can another folder “Manual Scripts”, that holds all the scripts which need to be executed once, like generating certificates etc. Other folders inside AKS-Infra contain one folder each for each Azure component.

Of course, you can choose to use any other structure you like. In my case, I used it to deploy Sitecore to multiple environments – SIT, UAT and PROD, so most of these folders hold structure like this – One ARM Deploy template File and multiple parameters files, one for each environment. In the next section, you can see how can we use “overrideParameters” tag in Devops pipeline to overwrite some of the parameters in these parameters files at execution time.

For Sitecore, XP1 folder contains all the files from Sitecore and Overlays contains any additional configurations required. I had explained the concept of overlays in my previous article of this series under “Preparing Sitecore Overlays Files” section.

Pipeline to Deploy Azure Resources

At this point, we have all the files required in our report and we can start preparing the pipelines. Since the Keyvault needs to be created and populated before all other resources (to that it can hold all secrets and other credentials), I created a separate pipeline for it.

          - task: AzureResourceManagerTemplateDeployment@3
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ResourceGroup)'
              location: 'West Europe'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/AKS-Infra/KeyVaultDeploy.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/AKS-Infra/KeyVault-Parameters-PROD.json'
              overrideParameters: '-objectId $(ObjectIDForKeyVault) -keyVaultName $(KeyvaultName)'
              deploymentMode: 'Incremental'

In the above example, highlighted ones are variables being read from Azure Devops variable groups.

Once Keyvault gets created, use the manual script from Rob’s excellent article about this topic to populate it. You can also add any other additional secrets there.

Next, we create another pipeline and put all other resources one after other in the same sequence in which they are expected to be created.

Create ACR: I had created in a separate subscription so that it can be accessed from all Sitecore environments. Second step of this imports the Sitecore XP1 images from Sitecore repo to our ACR.

         - task: AzureResourceManagerTemplateDeployment@3
            displayName: 'Create ACR in Production'
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ACRResourceGroup)'
              location: '$(location)'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/AKS-Infra/ACR-Deploy.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/AKS-Infra/ACR-Parameters.json'
              overrideParameters: '-acrName $(ACRName)'
              deploymentMode: 'Incremental'

          - task: AzureCLI@2
            displayName: "Import Sitecore Containers to ACR"
            inputs:
              azureSubscription: '$(ARMConnection)'
              scriptType: 'ps'
              scriptLocation: 'scriptPath'
              scriptPath: '$(System.ArtifactsDirectory)/AKS-Infra/ImportContainerImages.ps1'
              arguments: '-ACRName $(ACRName)'

Create VNET

          - task: AzureResourceManagerTemplateDeployment@3
            displayName: "Create VNET"
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ResourceGroup)'
              location: 'West Europe'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/VNET/VNETDeploy.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/VNET/VNET-Paramters-PROD.json'
              overrideParameters: '-location $(Location) -virtualNetworkName $(VirtualNetworkName) -virtualNetworkAddressPrefixes ["$(VirtualNetworkAddressPrefixes)"] -aksSubnetAddressPrefix "$(AKSSubnetAddressPrefix)" -AzureFirewallSubnetPrefix "$(AzureFirewallSubnetPrefix)" -applicationGatewaySubnetAddressPrefix "$(ApplicationGatewaySubnetAddressPrefix)" -EDSSubnetAddressPrefix $(EDSSubnetPrefix)'
              deploymentMode: 'Incremental'

Create SQL

          - task: AzureResourceManagerTemplateDeployment@3
            displayName: "Create SQL"
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ResourceGroup)'
              location: 'West Europe'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/EDS/SQL/SQLDeploy.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/EDS/SQL/SQL-Parameters-PROD.json'
              overrideParameters: '-location $(Location) -sqlServerLogin $(sitecore-databaseusername) -sqlServerPassword $(sitecore-databasepassword)  -sqlServerName $(sitecore-databaseservername) -elasticPoolName $(sitecore-database-elastic-pool-name)'
              deploymentMode: 'Incremental'

Create Redis

          - task: AzureResourceManagerTemplateDeployment@3
            displayName: "Create Redis"
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ResourceGroup)'
              location: 'West Europe'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/EDS/Redis/RedisDeploy-PROD.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/EDS/Redis/Redis-Parameters-PROD.json'
              overrideParameters: '-location $(Location) -redisCacheName $(redisCacheName) -existingVirtualNetworkName $(VirtualNetworkName)'
              deploymentMode: 'Incremental'

          - task: AzureCLI@2
            displayName: "Update Redis Connection String"
            inputs:
                azureSubscription: '$(ARMConnection)'
                scriptType: 'ps'
                scriptLocation: 'scriptPath'
                scriptPath: '$(System.ArtifactsDirectory)/Scripts/UpdateRedisConnectionString.ps1'
                arguments: '-ResourceGroupName $(ResourceGroup) -RedisCacheName "$(redisCacheName)" -VaultName $(KeyvaultName)'

Create SOLR: As explained in the Part 2 of this series, this creates a 3 VM setup for SOLR.

          - task: AzurePowerShell@5
            displayName: "Create Solr"
            inputs:
              azureSubscription: '$(ARMConnection)'
              ScriptType: 'FilePath'
              ScriptPath: '$(System.ArtifactsDirectory)/EDS/Solr-Prod/deploy.ps1'
              ScriptArguments: '-deploymentId "CEAUESA" -resourceGroupName "$(ResourceGroup)" -location "$(Location)" -templateFile "$(solrDeployTemplate)" -templateParameterFile "$(solrDeployTemplateParameters)" -adminUsername "$(sitecore-solr-admin-username)" -adminPassword "$(sitecore-solr-admin-password)"  -virtualNetworkAddressPrefixes "$(VirtualNetworkAddressPrefixes)"'
              preferredAzurePowerShellVersion: '3.1.0'

Create SOLR Load Balancer

          - task: AzureResourceManagerTemplateDeployment@3
            displayName: "Create Solr Load Balancer"
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ResourceGroup)'
              location: 'West Europe'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/EDS/Solr-Prod/templates/SolrLBDeploy-PROD.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/EDS/Solr-Prod/templates/SolrLB-Parameters-PROD.json'
              deploymentMode: 'Incremental'

Create Application Gateway

          - task: AzureResourceManagerTemplateDeployment@3
            displayName: "Create Application Gateway"
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ResourceGroup)'
              location: 'West Europe'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/AppGw-LW/AppGw-LWDeploy.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/AppGw-LW/AppGw-LW-Paramters-PROD.json'
              overrideParameters: '-location $(Location) -virtualNetworkName $(VirtualNetworkName) -logAnalyticsWorkspaceName $(LogAnalyticsWorkspaceName) -applicationGatewayName $(AppGatewayName) -wafPolicyName $(ApplicationGatewayWAFPolicyName)  -virtualNetworkAddressPrefixes ["$(VirtualNetworkAddressPrefixes)"] -aksSubnetAddressPrefix "$(AKSSubnetAddressPrefix)" -AzureFirewallSubnetPrefix "$(AzureFirewallSubnetPrefix)" -applicationGatewaySubnetAddressPrefix "$(ApplicationGatewaySubnetAddressPrefix)" -EDSSubnetAddressPrefix "$(EDSSubnetPrefix)"'
              deploymentMode: 'Incremental'

Deploy WAF Rules

          - task: AzureResourceManagerTemplateDeployment@3
            displayName: "Deploy WAF Rules"
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ResourceGroup)'
              location: 'West Europe'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/AppGw-LW/WAFRulesDeply.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/AppGw-LW/WAFRules-Parameters-PROD.json'
              overrideParameters: '-wafPolicyName $(ApplicationGatewayWAFPolicyName)'
              deploymentMode: 'Incremental'

Create Azure Firewall

          - task: AzureCLI@2
            displayName: "Deploy and Configure Firewall"
            inputs:
                azureSubscription: '$(ARMConnection)'
                scriptType: 'ps'
                scriptLocation: 'scriptPath'
                scriptPath: '$(System.ArtifactsDirectory)/Firewall/DeployFirewall.ps1'
                arguments: '-location $(Location) -FWResourceGroupName $(ResourceGroup) -VNETName $(VirtualNetworkName) -AKSSubnetName $(AKSSubnetName) -FWName $(FirewallName) -FWPIP $(FirewallIP) -routetable $(RouteTable)'


          - task: AzureResourceManagerTemplateDeployment@3
            displayName: "Deploy Firewall Policies"
            inputs:
              deploymentScope: 'Resource Group'
              azureResourceManagerConnection: '$(ARMConnection)'
              subscriptionId: '$(Subscription)'
              action: 'Create Or Update Resource Group'
              resourceGroupName: '$(ResourceGroup)'
              location: 'West Europe'
              templateLocation: 'Linked artifact'
              csmFile: '$(System.ArtifactsDirectory)/Firewall/FirewallPolicyDeploy.json'
              csmParametersFile: '$(System.ArtifactsDirectory)/Firewall/FirewallPolicy-Parameters-PROD.json'
              overrideParameters: '-resourceGroup "$(ResourceGroup)" -firewallPolicyName $(FirewallPolicyName) '
              deploymentMode: 'Incremental'

Create AKS

          - task: AzureCLI@2
            displayName: "Deploy AKS Core"
            inputs:
                azureSubscription: '$(ARMConnection)'
                scriptType: 'ps'
                scriptLocation: 'scriptPath'
                scriptPath: '$(System.ArtifactsDirectory)/AKS-Core/CreateAKS-UDR.ps1'
                arguments: '-Location $(Location) -ResourceGroupName $(ResourceGroup) -AKSName $(AKSName) -VNETName $(VirtualNetworkName) -AKSSubnetName $(AKSSubnetName) -APPGateWayName $(AppGatewayName) -APPGateWayNameSubnetName $(APPGateWayNameSubnetName) -LogWorkspaceName $(LogAnalyticsWorkspaceName) -KubernetesVersion $(KubernetesVersion) -WindowsAdminUsername $(WindowsAdminUsername) -WindowsAdminPassword $(WindowsAdminPassword) -AKSSystemNodeCount $(AKSSystemNodeCount) -AADProfileAdminGroupObjectIDs $(AADProfileAdminGroupObjectIDs) -ServiceCIDR $(ServiceCIDR) -DNSServiceIP $(DNSServiceIP) -DockerBridgeAddress $(DockerBridgeAddress) -MaxSystemPods $(MaxSystemPods) -AvailabilityZones $(AvailabilityZones) -SystemNodeVMSize $(SystemNodeVMSize) -SystemNodePoolName $(SystemNodePoolName) -SystemNodePoolNodeLabels "$(SystemNodePoolNodeLabels)" -UserNodePoolName1 $(UserNodePoolName1)  -UserNodePoolName2 $(UserNodePoolName2) -UserNodeVMSize $(UserNodeVMSize) -MaxUserPods $(MaxUserPods) -AKSUserNodeCount $(AKSUserNodeCount) -UserNodePoolNodeLabels "$(UserNodePoolNodeLabels)" -OSDiskSize $(OSDiskSize) -SystemNodePoolNodeTags "$(SystemNodePoolNodeTags)" -UserNodePoolNodeTags "$(UserNodePoolNodeTags)" -AKSManagedIdentity $(AKSManagedIdentity) -UserNodePoolNodeLabels2 "$(UserNodePoolNodeLabels2)"'

Update Access Policies

          - task: AzureCLI@2
            displayName: "Update Access Policies"
            inputs:
              azureSubscription: '$(ARMConnection)'
              scriptType: 'ps'
              scriptLocation: 'scriptPath'
              scriptPath: '$(System.ArtifactsDirectory)/Scripts/UpdateAccesspolicies.ps1'
              arguments: '-AKSName $(AKSName) -ResourceGroupName $(ResourceGroup) -keyVault $(KeyvaultName) -KeyvaultManagedidentity $(KeyvaultManagedidentity) -AppGatewayName $(AppGatewayName) -AppGatewayManagedIdentity $(AppGatewayManagedIdentity) -AKSManagedidentity $(AKSManagedidentity)'

Pipeline to Install Sitecore

Once all required Azure Resources are created and configured, we can start deploying Sitecore. Below are the main set of steps which are required. As explain in my previous article, I am using Overlays files for all of the specifications and configurations.

Deploy Secrets

 - task: Kubernetes@1
            displayName: Deploy Secrets Overlays
            inputs:
              connectionType: 'Kubernetes Service Connection'
              kubernetesServiceEndpoint: '$(SitecoreClusterConnection)'
              namespace: '$(Namespace)'
              command: 'apply'
              arguments: '-k $(System.ArtifactsDirectory)/k8s-specs/Overlays/secrets/'
              secretType: 'dockerRegistry'
              containerRegistryType: 'Azure Container Registry'
              versionSpec: '$(KubernetesVersion)'
              outputFormat: 'none'

Deploy Init Specifications: This deploys the Jobs in AKS cluster to initialize SOLR and SQL. We need to wait till the initialization gets completed before moving to next steps.

          - task: Kubernetes@1
            displayName: Deploy Init Specifications
            inputs:
              connectionType: 'Kubernetes Service Connection'
              kubernetesServiceEndpoint: '$(SitecoreClusterConnection)'
              namespace: '$(Namespace)'
              command: 'apply'
              arguments: ' -k $(System.ArtifactsDirectory)/k8s-specs/Overlays/init'
              secretType: 'dockerRegistry'
              containerRegistryType: 'Azure Container Registry'
              versionSpec: '$(KubernetesVersion)'
              outputFormat: 'none'

          - task: Kubernetes@1
            displayName: Wait for Deploying Solar Init Specifications
            inputs:
              connectionType: 'Kubernetes Service Connection'
              kubernetesServiceEndpoint: '$(SitecoreClusterConnection)'
              namespace: '$(Namespace)'
              command: 'wait'
              arguments: '--for=condition=Complete job.batch/solr-init --timeout=900s'
              secretType: 'dockerRegistry'
              containerRegistryType: 'Azure Container Registry'
              versionSpec: '$(KubernetesVersion)'

          - task: Kubernetes@1
            displayName: Wait for Deploying SQL Init Specifications
            inputs:
              connectionType: 'Kubernetes Service Connection'
              kubernetesServiceEndpoint: '$(SitecoreClusterConnection)'
              namespace: '$(Namespace)'
              command: 'wait'
              arguments: '--for=condition=Complete job.batch/mssql-init --timeout=900s'
              secretType: 'dockerRegistry'
              containerRegistryType: 'Azure Container Registry'
              versionSpec: '$(KubernetesVersion)'

Deploy Persistent Volume Claim

        - task: Kubernetes@1
            displayName: Deploy Persistent Volume Claim
            inputs:
              connectionType: 'Kubernetes Service Connection'
              kubernetesServiceEndpoint: '$(SitecoreClusterConnection)'
              namespace: '$(Namespace)'
              command: 'apply'
              arguments: '-f $(System.ArtifactsDirectory)/k8s-specs/XP1/volumes/azurefile'
              secretType: 'dockerRegistry'
              containerRegistryType: 'Azure Container Registry'
              versionSpec: '$(KubernetesVersion)'
              outputFormat: 'none'

Deploy Sitecore Specifications: This steps deploys Sitecore to AKS Cluster.

          - task: Kubernetes@1
            displayName: Deploy Application Specifications
            inputs:
              connectionType: 'Kubernetes Service Connection'
              kubernetesServiceEndpoint: '$(SitecoreClusterConnection)'
              namespace: '$(Namespace)'
              command: 'apply'
              arguments: '-k $(System.ArtifactsDirectory)/k8s-specs/Overlays'
              secretType: 'dockerRegistry'
              containerRegistryType: 'Azure Container Registry'
              versionSpec: '$(KubernetesVersion)'
              outputFormat: 'none'

If everything goes well, you will see all the PODs in running state after about 20-30 mins.

Exposing Sitecore using Azure Application Gateway Ingress Controller

Once you see the pods are up and running, you can expose your site by deploying the Ingress Controller. In my example, I have used Azure Application Gateway Ingress Controller (AGIC), but you can very well use “nginx” as mentioned in the Sitecore official installation document.

I had already shared the YAML construct for the AGIC in my previous article under section “Exposing Sitecore using Azure Application Gateway Ingress Controller”. You can either add this in a pipeline step or execute this manually within the AKS Cluster.

And that’s it. You have now successfully build and deployed your Sitecore in AKS Cluster. You should be able to access your Sitecore application over the internet now 🙂

Feel free to ask me any specific details which I might have skipped.

Enjoy,
Anupam

Cloud Decoded