I am a fan of using semantic versioning (a.k.a. SemVer) for data solutions, following the v1.0.0 pattern. It helps in the communication between team members and stakeholders, by limiting ambiguity and misunderstandings related to the version of your solution’s releases. With semantic versioning, the trick is to increment the version according to the changes you have made since the latest release. Manually keeping track of that is not an easy task, especially for small teams, without the capacity to have somebody dedicated to this administration task. I found a way to make this a lot easier, leaning on the Pull Request description! And as a bonus, we will create some nice release notes automatically 🚀.
Using the Pull Request description
My implementation of automatic semantic versioning relies on the data engineer to select the correct type of change in the Pull Request description.
These options of course relate to the Patch/Minor/Major of the SemVer pattern. I think these labels are easier to understand for data engineers because we usually talk about creating a bug fix or feature. Feel free to tweak these labels to your liking, but make sure you update the pipeline code as well.
Required Azure DevOps extensions
This setup relies on the Azure DevOps extensions listed below. Please review them carefully, and only install them in your organization if they pass your internal standards/governance rules regarding extensions.
- GitTools (we the tool GitVersion that is in this library)
- Pull Request Utils
- Generate Release Notes (Crossplatform)
- WIKI Updater Tasks
Add the Pull Request template
To automatically serve these options for every Pull Request, I use a template (check out the Docs to learn more about Pull Request templates). This is the code for my standard Pull Request:
Add this in a .md file in your git repository in a folder called .azuredevops, and it will be automatically picked up by new Pull Requests (Azure DevOps will scan for Markdown files in that folder).
Add the Release notes template
The process for automatically adding a release note in the wiki needs a template file. Please see the Markdown file below. Yes, it looks very weird now, with all those mustache brackets, but no worries. All of those {{variable}} thingies will be automatically replaced.
Add this file in your git repository in a folder called .azuredevops, with the name release-notes-template.md. If you use another name or location, remember to change the values in the YAML pipeline as well (we will add that pipeline later).
Add the GitVersion configuration file
The process for semantic versioning (GitVersion tool) relies on a config file. Please see the file below.
Add this to your git repository in the .azuredevops folder, with the name gitversion.yml. If you use another name or location, remember to change the values in the YAML pipeline as well (we will add that pipeline later).
If you already have a release number for your project, you can set the initial seed value in the first line of the above file. Where it now says next-version: 1.0.
Required Azure DevOps permissions and settings
To be able to make this setup work correctly, we need a couple of permissions and settings.
- Make sure that the Build Service user of your project has the
ContributeandCreate tagpermissions on the git code repository that you work from:
- Make sure that the Build Service user has
Contributepermissions on the git repo of your project wiki. You can manage the Wiki security by clicking on the tripple dotted menu of the wiki:
- Disable the project setting called
Limit job authorization scope to referenced Azure DevOps repositories. See the screenshot below where to find that setting. If it is un-editable, first go to the Organization settings and disable it there, then come back to the project settings and disable it here.
Add the pipeline for Pull Request administration
To pick up the selected type of update from the Pull Request description, I have created the Azure DevOps pipeline shown below (.yml). You will have to add that to your repo in a location of your preference with the name of your preference. I call it pipeline-pullrequest-administration.yml.
This YAML pipeline performs the following tasks:
- Install GitVersion to be used in the last task.
- Retrieve the Pull Request description and store it in a variable to use it in the next task.
- Add a git commit message based on what was selected in the PR description, marking the branch as either a
patch,minorormajorchange. This will be picked up by A) the next task, and by B) in another DevOps YAML pipeline that is triggered right after the merge is completed. - Determine the correct semantic version (note, this is only the version of the feature branch, it does not have to be the same as the final version).
Now that we have this pipeline, let put it to use. First, add a new pipeline in your Azure DevOps project, pointing to the .yml file in your repo. I have called my pipeline Pull Request administration.
Secondly, go to the policies page of your main branch:
Add a Build Validation, based on the pipeline you have just added.
This will start the Pull Request administration pipeline every time we publish a Pull Request 🚀!
Add the pipeline for release administration
We also need a second Azure DevOps YAML pipeline for our release administration. Please see the script below, save it as a pipeline .yml file in your repo, in your preferred location. I use the name pipeline-release-administration.yml.
Please update the repo reference to the correct URL of your git repo of the wiki, not the wiki itself:
This YAML pipeline performs the following tasks:
- Job CalculateVersion
- Install GitVersion to be used in the last task.
- Determine the correct semantic version.
- Update the Build.BuildNumber to use SemVer, as by default it uses FullSemVer (not my preferrence).
- Add git tag for the calculated semantic version (e.g. v2.0.1).
- Job CreateReleaseNotes
- Generates a release notes file
- Publishes the release notes in the project wiki
Add a new pipeline in your Azure DevOps project, pointing to the .yml file in your repo that you have just uploaded (pipeline-release-administration.yml). I have called my pipeline Release administration.
Great, this pipeline will now be automatically triggered for each commit performed on the main or master branch, because of the trigger section in the .yml file! In other words, after each Pull Request is completed, this pipeline will be started 🚀.
Bonus: Tags
After every Pull Request completes, a new Tag will be automatically added, pointing to the (git) version of your code.
Bonus: Release Notes
And, after each Pull Request completes, a note will be automatically prepended in the release notes page in the wiki for the new version. It will have a list of associated Pull Requests (usually just a single request) and the associated work items. And, a download link is added as well. I tried to mimic the standard GitHub release notes pages 😉.
Limitations and considerations
- You can’t use the
Squash mergetype, when completing the Pull Request to your main branch. We need the commit messages of the feature branch (with+semver:) to be transferred to the main branch.
Pingback: Automating Semantic Versioning with Azure DevOps – Curated SQL
It seems that the command $(System.PullRequest.SourceBranch) is no found. I’m currently reporting this issue to Microsoft https://github.com/MicrosoftDocs/azure-devops-docs/issues/11644. However, I would like to know if you have any possible solutions to this.
Hi Alan,
Can it be that you are not running this pipeline as part of a Pull Request ‘build validation’, but stand-alone?
In that case, that command can’t be used!
It seems that it was related to some of the permissions from my branch. However, now that I’m passing this argument:
“`
========================== Starting Command Output ===========================
/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . ‘/home/vsts/work/_temp/bf8dab5c-1ff9-4100-a027-e0d8296e0e5a.ps1’
ParserError: /home/vsts/work/_temp/bf8dab5c-1ff9-4100-a027-e0d8296e0e5a.ps1:9
Line |
9 | $PRdesc = ” git commit -a -m “+semver:major [skip azurepipelines]” — …
| ~
| You must provide a value expression following the ‘+’ operator.
##[error]PowerShell exited with code ‘1’.
“`
Can you verify the double quotes? Sometimes when copy-pasting scripts they are not the regular double quote..
It seems that I’m getting an additional ” at the beginning of git, instead of just passing this as what I set it up.
git commit –allow-empty -a -m “+semver: patch [skip azurepipelines]”
Do you know how to passed the + without having any other issues?
Any idea why I’m getting the following error when running the Release administration pipeline: “GitVersion configuration file not found at /home/vsts/work/1/s/.azuredevops/gitversion.yml”?
Do you have that file in your repository in a folder called “.azuredevops”?
The “Add git commit message for SemVer” powershell task seems to trip over pull request descriptions containing double-quotes. Is it save to change the $PRdesc variable to a Here-String, or is there another solution? Like this:
$PRdesc = @”
$(RetrievePullRequestDescription.PullRequest.DescriptionContent)
“@
how did you resolve this one?
I am getting an error when the Release administration pipeline runs. It fails at “Publish to the wiki” step. The error i get is as follows
##[error]Error: Cloning into ‘/home/vsts/work/1/s\repo’…
fatal: unable to update url base from redirection:
asked for: https://buildagent:***@dev.azure.com/****/api/_wiki/wikis/api.wiki/info/refs?service=git-upload-pack
redirect: https://spsprodweu5.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2Fme
It’s always generate the same version for me witch is 1.1.0 even though I tried with differents branches names. My target branch is named “dev” and my feature branch is named “feature”. Do you have an idea about the possible cause ?
I like the idea of the PR type, but I don’t think the categories represent actual semantic versioning constructs. Semantic versioning is all about the contract, not not the perceived quantum of work. That makes semantic versioning opinion based and not meaningful / rules-based, thus really missing the whole point
Anyway that we can use the same git tag into the docker tag at the time we are pushing the new image to the container registry?
How can I add the same git tag into my docker image inside of my pipeline? Is it possible to use the same git tag repo inside of my container registry?
trying to enable the Pull Request administration and got this error:
Doing git push..
remote: 001f# service=git-receive-pack
remote: 0000000000aaTF401027: You need the Git ‘GenericContribute’ permission to perform this action. Details: identity ‘Build\94740699-97d9-42ba-ad55-a3ee42c061cd’, scope ‘repository’.
remote: TF401027: You need the Git ‘GenericContribute’ permission to perform this action. Details: identity ‘Build\94740699-97d9-42ba-ad55-a3ee42c061cd’, scope ‘repository’.
fatal: unable to access ‘https://labasaservice.visualstudio.com/FFC/_git/FFC_SERVER/’: The requested URL returned error: 403
Done.
##[error]PowerShell exited with code ‘1’.
Finishing: Add git commit message for SemVer
Make sure that the Build Service user of your project has the Contribute and Create tag permissions on the git code repository that you work from.
And, make sure that the Build Service user has Contribute permissions on the git repo of your project wiki. You can manage the Wiki security by clicking on the tripple dotted menu of the wiki.
that’s exactly what i did (Project Collection Build Service Accounts), but from some reason still got this error…
BTW the solution is to taking user ID from the error: ‘94740699-97d9-42ba-ad55-a3ee42c061cd’ and looked for it in the project setting -> repos-> security. and allow it contribute permission
Hey Dave,
from some reason i got a failure message on the step of – ‘Pull Request administration’ pipeline:
– task: gitversion/execute@0
displayName: Determine SemVer
attached the log from this step (with system.debug= true):
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]check path : /agent/_work/_tasks/gitversion/execute_9013cf7f-ee8d-49f4-a39b-db244928d391/0.9.13/lib.json
##[debug]adding resource file: /agent/_work/_tasks/gitversion/execute_9013cf7f-ee8d-49f4-a39b-db244928d391/0.9.13/lib.json
##[debug]system.culture=en-US
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]set DOTNET_CLI_TELEMETRY_OPTOUT=true
##[debug]Processed: ##vso[task.setvariable variable=DOTNET_CLI_TELEMETRY_OPTOUT;isOutput=false;issecret=false;]true
##[debug]set DOTNET_NOLOGO=true
##[debug]Processed: ##vso[task.setvariable variable=DOTNET_NOLOGO;isOutput=false;issecret=false;]true
##[debug]targetPath=undefined
##[debug]useConfigFile=false
##[debug]configFilePath=/agent/_work/1/s
##[debug]updateAssemblyInfo=false
##[debug]updateAssemblyInfoFilename=undefined
##[debug]additionalArguments=undefined
##[debug]Build.SourcesDirectory=/agent/_work/1/s
##[debug]Build.SourcesDirectory=/agent/_work/1/s
Command: dotnet-gitversion /agent/_work/1/s /output json /output buildserver
##[debug]which ‘dotnet-gitversion’
##[debug]not found
##[debug]task result: Failed
##[error]Error: Unable to locate executable file: ‘dotnet-gitversion’. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
##[debug]Processed: ##vso[task.issue type=error;]Error: Unable to locate executable file: ‘dotnet-gitversion’. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
##[debug]Processed: ##vso[task.complete result=Failed;done=true;]Error: Unable to locate executable file: ‘dotnet-gitversion’. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
Finishing: Determine SemVer
any idea why is that ?
hi Dave,
I am getting below error in Generate release notes
Generating the release notes, the are 1 template(s) to process
##[error]Cannot find template file /home/vsts/work/1/s/.azuredevops/release-notes-template.md
##[error]Missing template file
##[error]Missing template file
Hi Dave, first and foremost thanks for writing this awesome blogpost. I am very interested in getting SemVer to work, and your solution seems like the easiest way towards it. So I followed along until the end. Created the MD files, yml files, set the proj setting right, the permissions and created the pipelines based on the files. Now as a test I created a feature branch based on main and added a simple test file. Now, when I create the PR the build starts running. It notices my checked ‘feature’ box in the PR description and PowerShell outputs it as such, so up till this point all fine. Now at the end of the PS script it pushes to the PR’s branch, and that’s where I get an error: “hint: Updates were rejected because the remote contains work that you do
hint: not have locally.” I tried adding a push after the branch checkout but without success. Not sure where this error is coming from. So, any ideas? Thanks in advance for taking the time to answer.
Hi Vincent , did you get a solution into this?
Just wanted to help others after getting this error too.
In my case it was because the pipeline had shallow fetch turned on.
https://github.com/GitTools/GitVersion/issues/3091#issuecomment-1240999129
this link helped me turn it off and then the pipeline worked without a hitch (at least for me)
Hello,
It’s very wird. In my implementation of this solution when the XplatGenerateReleaseNotes@3 generate the release note, it add all the Pull Request (The PRs of the previous release and the PRs of the new releases). So more I create releases, biger is ths list of PRs in the release note. Do you know how to configure that in order to have only the PRs related to the latest build ? Thanks 🙂
Vincent , I tried using the command below and it worked using -f forced push.
git push –set-upstream -f origin $(“$(System.PullRequest.SourceBranch)”.replace(‘refs/heads/’, ”))
I am facing now a problem in next step
“Gitversion could not determine which branch to treat as the development branch (default is ‘develop’) nor release-able branch (default is ‘main’ or ‘master’), either locally or remotely. Ensure the local clone and checkout match the requirements or considering using ‘GitVersion Dynamic Repositories'”
hi Dave,
I am getting below error in calculating version
##[error]Error: GitVersion configuration file not found at /home/vsts/work/1/s/.azuredevops/gitversion.yml
How can i add date for each release notes ? what is the parameter that i should add ? date ? # 🚀 Release v{{buildDetails.buildNumber}} {date} ?
You can add this snippet to your pipeline “pipeline-release-administration”
## Job to create release notes file and publish it to the wiki
– job: CreateReleaseNotes
displayName: Creating release notes
dependsOn: CalculateVersion
steps:
# Generates a release notes file
– task: XplatGenerateReleaseNotes@3
displayName: “Generate release notes”
inputs:
outputfile: ‘$(System.DefaultWorkingDirectory)\releasenotes.md’
templateLocation: File
customHandlebarsExtensionCode: |
module.exports = {date_formatter() {
const data = new Date();
const options = { weekday: ‘long’, year: ‘numeric’, month: ‘long’, day: ‘numeric’ };
const dataFormatada = data.toLocaleDateString(‘eg-US’, options);
}};
templatefile: “.azuredevops/release-notes-template.md”
dumpPayloadToConsole: false
dumpPayloadToFile: false
replaceFile: True
getParentsAndChildren: False
getAllParents: False
getIndirectPullRequests: False
stopOnError: True
considerPartiallySuccessfulReleases: False
In your release-notes-template.md you can look like this
# 🚀 Release v{{buildDetails.buildNumber}} {{date_formatter}}
I got this error
A task is missing. The pipeline references a task called ‘PullRequestDescription’. This usually indicates the task isn’t installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 0, job ‘CalculateVersion’, step ‘RetrievePullRequestDescription’.)
20230228.21