In this article I want to provide information about some of the features that we can use in Azure pipeline. Azure pipelines have lots of task plugins and enough features to handle most common continuous integration/ continuous delivery (CI/CD) flows. Azure YAML syntax for Azure pipeline is proprietary and can’t be used with other CI/CD platforms.
Introduction
Azure DevOps is a DevOps server offered by Microsoft which helps you to simplify DevOps processes. Azure DevOps provides version control (Azure Repo), project management (Azure Boards), testing (Test Plans) and its CI/CD service (Azure Pipelines).
Azure DevOps allows you to create a pipeline in two different formats. Firstly, classic editor, which is a GUI based format. This format doesn’t require prior knowledge. Secondly, YAML based format, where basic understanding of YAML is required.
In classic editor, you can define the pipeline in the Azure DevOps with a graphical user interface. Using classic editor, you can create, build, and release pipelines in the DevOps portal. The build pipeline will create artifacts that can be used for code deployment in environments like staging and production. Classic editor pipeline is not part of version control, meaning you can’t track changes.
Azure YAML allows you to define your CI/CD strategy as code and the pipeline file is then committed to repo. As a result, you can take advantage of pull requests, history and code review just like other application code. Unlike classic editor, the YAML pipeline file is version control enabled so you can rollback to any previous working version.
The YAML pipeline covers the following points:
- Prerequisites
- Predefined Variables
- Agent Node Pool
- Run Time Parameters
- Trigger one Pipeline from another
- Publish and Consume Artifacts in Pipeline.
- Azure DevOps Secure Library
- Conclusion
- References
1. Prerequisites
- Azure DevOps account
- Basic knowledge about YAML. Find more details about prerequisites here
2. Predefined variables
In Azure DevOps many variables already exist by default. Such variables are called pre-defined or system variables. Predefined variables are automatically set by the system and are read-only. They can represent simple strings and numbers. Find below some of the pre-defined variables more commonly used.
2.1 Agent.BuildDirectory / Pipeline.Workspace
This variable stores local path on agent for running build, both variables have the same values.
Example: /home/pipelineagent/_work/1
2.2 Agent.TempDirectory
A temporary folder that is cleaned after each pipeline job. You can use this variable to store secret files during the pipeline and clean it up after that.
2.3 Build.BuildId / Build.BuildNumber
The ID for the completed build.
2.4 System.DefaultWorkingDirectory
The local path on the agent where you download source code files.
3. Agent Node Pool:
In order to run, build and deploy a pipeline for your application you need at least one agent. When you configure your pipeline, you have two agent options: Azure managed agent and self-hosted agent. An agent can be a virtual machine or a container which acts as a compute engine. These operating systems can be Windows, Ubuntu or Mac OS.
The important point to consider here is the cost. With a self-hosted agent you can use the Azure pipeline for free. Here you are able to provide compute resources and use Azure DevOps pipelines as an orchestrator. A self hosted agent allows you to install software or libraries required to build and deploy a pipeline.
However, if you have not exceeded the Azure pipeline’s quota, you can use the Microsoft-hosted agent without any cost.
- The Azure pipeline’s quota includes: 1 Microsoft hosted agent with 1800 minutes per month for CI/CD and 1 self-hosted job with unlimited minutes per month.
3.1 How to use a Microsoft-hosted agent?
To use a Microsoft-hosted agent in your Azure pipeline, mention the operating system name as shown in the below YAML format:
pool:
vmImage: ubuntu-20.04
3.2 How to use a self-hosted agent?
- Go to Organization Settings.
- Click on Agent Pools.
- Click on Add Pool.
4. Fill the details as shown in the picture below. Then on the next screen click on New Agent.
5. Open New Tab. To configure a new agent we will need a PAT (Personal Access Token). The screenshots below explain how to create a PAT:
Note: Don’t forget to copy the token as soon as you generate it as you will not be able to see it again.
6. Go back to the Agent Pools page download and install the agent following the instructions provided by Microsoft. When the program is running, you will have to provide your PAT and other details to be able to configure it.
Once your agent is configured you can see your agent in the Azure DevOps portal.
4. Run time parameters
Previously, the parameters were passed using the variables section defined in the pipeline. The main issue with this method is that all variables are string, so there is no validation to check whether the correct value has been passed.
For example:
Now we have an option to define the runtime parameters, including the parameters key which takes the boolean parameter value. Parameters should be set at the beginning of a YAML. In the example below, the trigger is set to none so that you can choose the value of installNewrelic. By default, it is configured as false.
When you run the pipeline manually you will be presented with a user interface (shown below) that allows the user to select this runtime parameter before running the pipeline. If you don’t make a selection, the default option false is used and the ‘Install NewRelic’ task will be skipped.
5. Trigger one pipeline from another
In some cases we have pipeline dependency on each other. Here it can be helpful to trigger a pipeline from another pipeline. All you need to do is add the code below your YAML pipeline section.
# this is being defined in org.devops.demo – QA pipeline
trigger: none
resources:
pipelines:
– pipeline: DevBuild
source: org.devops.demo – DEV
branch: develop
trigger:
branches:
– develop
By adding trigger: none the QA pipeline will not trigger on commit and only triggers when DEV pipeline finishes it’s job.
6. Publish and consume artifacts in pipelines
Azure DevOps pipeline artifacts allow you to share files between stages in a pipeline or between different pipelines. Artifacts are outputs of the build process that need to be deployed or consumed by another job.
6.1 Publish or upload artifacts in a pipeline:
steps:
– publish: $(System.DefaultWorkingDirectory)/app
artifact: myapp
Explanation:
– In order to publish an artifact, a path of the file or folder is required.
– An artifact name is optional, but you might want to add it as it enables you to use it anywhere on the pipeline.
6.2 Download Artifacts in the same pipeline.
steps:
– download: current
artifact: myapp
Explanation:
– current means current run of the pipeline.
6.3 Download Artifacts in another pipeline:
Use the below task in the YAML pipeline to consume artifacts from another pipeline.
# this is being defined in org.devops.demo – QA pipeline
steps:
– task: DownloadPipelineArtifact@2
inputs:
source: ‘specific’
project: ‘DevOps Demo’
pipeline: 786 # ID of the pipeline from which you are downloading artifacts
runVersion: ‘latestFromBranch’
preferTriggeringPipeline: true
runBranch: develop # Branch of the artifacts pipeline
artifact: ‘myapp’ # Artifact Name
path: ‘$(Pipeline.Workspace)/myapp
7. Azure DevOps Secure Library:
Sometimes you have credentials that you can’t commit in repo, but you want to use that file and deploy on the application server in a secure way. In this case you can use Secure Library which allows you to store credentials files, and download and upload on the server through the pipeline.
To upload file in Secure Library follow these steps:
Go to Azure DevOps → Click on Pipelines → Library → Click on Secure Files → Click on “+ Secure file”
Example: Here I have uploaded secret.php
This is how to download a file through the pipeline from the Secure Library:
– task: DownloadSecureFile@1
name: mySecretFile # Any name which you can refer further in the pipeline.
displayName: ‘Download Secret File’
inputs:
secureFile: ‘secret.php’
– script: |
cp $(mySecretFile.secureFilePath) $(Pipeline.Workspace)/config
Here,
– Above code will download secret.php at $(Pipeline.Workspace)/config directory.
8. Conclusion:
To summarize, this article discusses various features in Azure DevOps pipelines which provide control on CI/CD; and you can do everything using code. Declarative YAML pipeline adds the cherry on top due to its easy to use syntax and it has a lot of amazing features to be explored (for instance its caching feature to reduce pipeline execution time, custom conditions etc.). I hope this will help you to adopt and use YAML based Azure DevOps pipeline without any hassle.
9. References
- For more details about Azure Pipelines, please visit the official documentation.
- Environment is also one of the features of Azure DevOps Pipeline, Click here to read more.
- Further reading http://www.anup.co.in
