Automatic semantic versioning (bonus: with release notes!)

I am a fan of using semantic versioning (a.k.a. SemVer) for data solutions, following the v1.0.0 pattern. It helps in the communication between team members and stakeholders, by limiting ambiguity and misunderstandings related to the version of your solution’s releases. With semantic versioning, the trick is to increment the version according to the changes you have made since the latest release. Manually keeping track of that is not an easy task, especially for small teams, without the capacity to have somebody dedicated to this administration task. I found a way to make this a lot easier, leaning on the Pull Request description! And as a bonus, we will create some nice release notes automatically 🚀.

Using the Pull Request description

My implementation of automatic semantic versioning relies on the data engineer to select the correct type of change in the Pull Request description.

Typ of update. Fix, Feature or Big feature.

These options of course relate to the Patch/Minor/Major of the SemVer pattern. I think these labels are easier to understand for data engineers because we usually talk about creating a bug fix or feature. Feel free to tweak these labels to your liking, but make sure you update the pipeline code as well.

Required Azure DevOps extensions

This setup relies on the Azure DevOps extensions listed below. Please review them carefully, and only install them in your organization if they pass your internal standards/governance rules regarding extensions.

  1. GitTools (we the tool GitVersion that is in this library)
  2. Pull Request Utils
  3. Generate Release Notes (Crossplatform)
  4. WIKI Updater Tasks

Add the Pull Request template

To automatically serve these options for every Pull Request, I use a template (check out the Docs to learn more about Pull Request templates). This is the code for my standard Pull Request:

Add this in a .md file in your git repository in a folder called .azuredevops, and it will be automatically picked up by new Pull Requests (Azure DevOps will scan for Markdown files in that folder).

Screenshot of the .azuredevops folder in the repo, with the pull request template in it.

Add the Release notes template

The process for automatically adding a release note in the wiki needs a template file. Please see the Markdown file below. Yes, it looks very weird now, with all those mustache brackets, but no worries. All of those {{variable}} thingies will be automatically replaced.

Add this file in your git repository in a folder called .azuredevops, with the name release-notes-template.md. If you use another name or location, remember to change the values in the YAML pipeline as well (we will add that pipeline later).

Add the GitVersion configuration file

The process for semantic versioning (GitVersion tool) relies on a config file. Please see the file below.

Add this to your git repository in the .azuredevops folder, with the name gitversion.yml. If you use another name or location, remember to change the values in the YAML pipeline as well (we will add that pipeline later).

If you already have a release number for your project, you can set the initial seed value in the first line of the above file. Where it now says next-version: 1.0.

Required Azure DevOps permissions and settings

To be able to make this setup work correctly, we need a couple of permissions and settings.

  • Make sure that the Build Service user of your project has the Contribute and Create tag permissions on the git code repository that you work from:
  • Make sure that the Build Service user has Contribute permissions on the git repo of your project wiki. You can manage the Wiki security by clicking on the tripple dotted menu of the wiki:
  • Disable the project setting called Limit job authorization scope to referenced Azure DevOps repositories. See the screenshot below where to find that setting. If it is un-editable, first go to the Organization settings and disable it there, then come back to the project settings and disable it here.

Add the pipeline for Pull Request administration

To pick up the selected type of update from the Pull Request description, I have created the Azure DevOps pipeline shown below (.yml). You will have to add that to your repo in a location of your preference with the name of your preference. I call it pipeline-pullrequest-administration.yml.

This YAML pipeline performs the following tasks:

  • Install GitVersion to be used in the last task.
  • Retrieve the Pull Request description and store it in a variable to use it in the next task.
  • Add a git commit message based on what was selected in the PR description, marking the branch as either a patch, minor or major change. This will be picked up by A) the next task, and by B) in another DevOps YAML pipeline that is triggered right after the merge is completed.
  • Determine the correct semantic version (note, this is only the version of the feature branch, it does not have to be the same as the final version).

Now that we have this pipeline, let put it to use. First, add a new pipeline in your Azure DevOps project, pointing to the .yml file in your repo. I have called my pipeline Pull Request administration.

Secondly, go to the policies page of your main branch:

Add a Build Validation, based on the pipeline you have just added.

This will start the Pull Request administration pipeline every time we publish a Pull Request 🚀!

Add the pipeline for release administration

We also need a second Azure DevOps YAML pipeline for our release administration. Please see the script below, save it as a pipeline .yml file in your repo, in your preferred location. I use the name pipeline-release-administration.yml.

Please update the repo reference to the correct URL of your git repo of the wiki, not the wiki itself:

This YAML pipeline performs the following tasks:

  • Job CalculateVersion
    • Install GitVersion to be used in the last task.
    • Determine the correct semantic version.
    • Update the Build.BuildNumber to use SemVer, as by default it uses FullSemVer (not my preferrence).
    • Add git tag for the calculated semantic version (e.g. v2.0.1).
  • Job CreateReleaseNotes
    • Generates a release notes file
    • Publishes the release notes in the project wiki

Add a new pipeline in your Azure DevOps project, pointing to the .yml file in your repo that you have just uploaded (pipeline-release-administration.yml). I have called my pipeline Release administration.

Great, this pipeline will now be automatically triggered for each commit performed on the main or master branch, because of the trigger section in the .yml file! In other words, after each Pull Request is completed, this pipeline will be started 🚀.

Bonus: Tags

After every Pull Request completes, a new Tag will be automatically added, pointing to the (git) version of your code.

Bonus: Release Notes

And, after each Pull Request completes, a note will be automatically prepended in the release notes page in the wiki for the new version. It will have a list of associated Pull Requests (usually just a single request) and the associated work items. And, a download link is added as well. I tried to mimic the standard GitHub release notes pages 😉.

Limitations and considerations

  • You can’t use the Squash merge type, when completing the Pull Request to your main branch. We need the commit messages of the feature branch (with +semver:) to be transferred to the main branch.
Pass the sauce

This Post Has 26 Comments

    1. Dave Ruijter

      Hi Alan,
      Can it be that you are not running this pipeline as part of a Pull Request ‘build validation’, but stand-alone?
      In that case, that command can’t be used!

      1. alan

        It seems that it was related to some of the permissions from my branch. However, now that I’m passing this argument:

        “`
        ========================== Starting Command Output ===========================
        /usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . ‘/home/vsts/work/_temp/bf8dab5c-1ff9-4100-a027-e0d8296e0e5a.ps1’
        ParserError: /home/vsts/work/_temp/bf8dab5c-1ff9-4100-a027-e0d8296e0e5a.ps1:9
        Line |
        9 | $PRdesc = ” git commit -a -m “+semver:major [skip azurepipelines]” — …
        | ~
        | You must provide a value expression following the ‘+’ operator.

        ##[error]PowerShell exited with code ‘1’.

        “`

        1. Dave Ruijter

          Can you verify the double quotes? Sometimes when copy-pasting scripts they are not the regular double quote..

  1. Alan

    It seems that I’m getting an additional ” at the beginning of git, instead of just passing this as what I set it up.

    git commit –allow-empty -a -m “+semver: patch [skip azurepipelines]”

    Do you know how to passed the + without having any other issues?

  2. Raluca

    Any idea why I’m getting the following error when running the Release administration pipeline: “GitVersion configuration file not found at /home/vsts/work/1/s/.azuredevops/gitversion.yml”?

    1. Dave Ruijter

      Do you have that file in your repository in a folder called “.azuredevops”?

  3. Wout

    The “Add git commit message for SemVer” powershell task seems to trip over pull request descriptions containing double-quotes. Is it save to change the $PRdesc variable to a Here-String, or is there another solution? Like this:
    $PRdesc = @”
    $(RetrievePullRequestDescription.PullRequest.DescriptionContent)
    “@

    1. cocoy

      how did you resolve this one?

  4. Etonde

    I am getting an error when the Release administration pipeline runs. It fails at “Publish to the wiki” step. The error i get is as follows

    ##[error]Error: Cloning into ‘/home/vsts/work/1/s\repo’…
    fatal: unable to update url base from redirection:
    asked for: https://buildagent:***@dev.azure.com/****/api/_wiki/wikis/api.wiki/info/refs?service=git-upload-pack
    redirect: https://spsprodweu5.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2Fme

  5. Samy

    It’s always generate the same version for me witch is 1.1.0 even though I tried with differents branches names. My target branch is named “dev” and my feature branch is named “feature”. Do you have an idea about the possible cause ?

  6. Just a guy

    I like the idea of the PR type, but I don’t think the categories represent actual semantic versioning constructs. Semantic versioning is all about the contract, not not the perceived quantum of work. That makes semantic versioning opinion based and not meaningful / rules-based, thus really missing the whole point

  7. hvaandres

    Anyway that we can use the same git tag into the docker tag at the time we are pushing the new image to the container registry?

  8. alan

    How can I add the same git tag into my docker image inside of my pipeline? Is it possible to use the same git tag repo inside of my container registry?

  9. Yinon Turgeman

    trying to enable the Pull Request administration and got this error:
    Doing git push..
    remote: 001f# service=git-receive-pack
    remote: 0000000000aaTF401027: You need the Git ‘GenericContribute’ permission to perform this action. Details: identity ‘Build\94740699-97d9-42ba-ad55-a3ee42c061cd’, scope ‘repository’.
    remote: TF401027: You need the Git ‘GenericContribute’ permission to perform this action. Details: identity ‘Build\94740699-97d9-42ba-ad55-a3ee42c061cd’, scope ‘repository’.
    fatal: unable to access ‘https://labasaservice.visualstudio.com/FFC/_git/FFC_SERVER/’: The requested URL returned error: 403
    Done.
    ##[error]PowerShell exited with code ‘1’.
    Finishing: Add git commit message for SemVer

    1. Dave Ruijter

      Make sure that the Build Service user of your project has the Contribute and Create tag permissions on the git code repository that you work from.

      And, make sure that the Build Service user has Contribute permissions on the git repo of your project wiki. You can manage the Wiki security by clicking on the tripple dotted menu of the wiki.

  10. Yinon Turgeman

    that’s exactly what i did (Project Collection Build Service Accounts), but from some reason still got this error…

    1. Yinon Turgeman

      BTW the solution is to taking user ID from the error: ‘94740699-97d9-42ba-ad55-a3ee42c061cd’ and looked for it in the project setting -> repos-> security. and allow it contribute permission

  11. Yinon Turgeman

    Hey Dave,
    from some reason i got a failure message on the step of – ‘Pull Request administration’ pipeline:
    – task: gitversion/execute@0
    displayName: Determine SemVer

    attached the log from this step (with system.debug= true):

    ##[debug]Agent.ProxyUrl=undefined
    ##[debug]Agent.CAInfo=undefined
    ##[debug]Agent.ClientCert=undefined
    ##[debug]Agent.SkipCertValidation=undefined
    ##[debug]Agent.ProxyUrl=undefined
    ##[debug]Agent.CAInfo=undefined
    ##[debug]Agent.ClientCert=undefined
    ##[debug]check path : /agent/_work/_tasks/gitversion/execute_9013cf7f-ee8d-49f4-a39b-db244928d391/0.9.13/lib.json
    ##[debug]adding resource file: /agent/_work/_tasks/gitversion/execute_9013cf7f-ee8d-49f4-a39b-db244928d391/0.9.13/lib.json
    ##[debug]system.culture=en-US
    ##[debug]Agent.ProxyUrl=undefined
    ##[debug]Agent.CAInfo=undefined
    ##[debug]Agent.ClientCert=undefined
    ##[debug]Agent.SkipCertValidation=undefined
    ##[debug]set DOTNET_CLI_TELEMETRY_OPTOUT=true
    ##[debug]Processed: ##vso[task.setvariable variable=DOTNET_CLI_TELEMETRY_OPTOUT;isOutput=false;issecret=false;]true
    ##[debug]set DOTNET_NOLOGO=true
    ##[debug]Processed: ##vso[task.setvariable variable=DOTNET_NOLOGO;isOutput=false;issecret=false;]true
    ##[debug]targetPath=undefined
    ##[debug]useConfigFile=false
    ##[debug]configFilePath=/agent/_work/1/s
    ##[debug]updateAssemblyInfo=false
    ##[debug]updateAssemblyInfoFilename=undefined
    ##[debug]additionalArguments=undefined
    ##[debug]Build.SourcesDirectory=/agent/_work/1/s
    ##[debug]Build.SourcesDirectory=/agent/_work/1/s
    Command: dotnet-gitversion /agent/_work/1/s /output json /output buildserver
    ##[debug]which ‘dotnet-gitversion’
    ##[debug]not found
    ##[debug]task result: Failed
    ##[error]Error: Unable to locate executable file: ‘dotnet-gitversion’. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
    ##[debug]Processed: ##vso[task.issue type=error;]Error: Unable to locate executable file: ‘dotnet-gitversion’. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
    ##[debug]Processed: ##vso[task.complete result=Failed;done=true;]Error: Unable to locate executable file: ‘dotnet-gitversion’. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
    Finishing: Determine SemVer

    any idea why is that ?

  12. chetanya gehlot

    hi Dave,

    I am getting below error in Generate release notes

    Generating the release notes, the are 1 template(s) to process
    ##[error]Cannot find template file /home/vsts/work/1/s/.azuredevops/release-notes-template.md
    ##[error]Missing template file
    ##[error]Missing template file

  13. Vincent Verweij

    Hi Dave, first and foremost thanks for writing this awesome blogpost. I am very interested in getting SemVer to work, and your solution seems like the easiest way towards it. So I followed along until the end. Created the MD files, yml files, set the proj setting right, the permissions and created the pipelines based on the files. Now as a test I created a feature branch based on main and added a simple test file. Now, when I create the PR the build starts running. It notices my checked ‘feature’ box in the PR description and PowerShell outputs it as such, so up till this point all fine. Now at the end of the PS script it pushes to the PR’s branch, and that’s where I get an error: “hint: Updates were rejected because the remote contains work that you do
    hint: not have locally.” I tried adding a push after the branch checkout but without success. Not sure where this error is coming from. So, any ideas? Thanks in advance for taking the time to answer.

    1. gagandeep

      Hi Vincent , did you get a solution into this?

  14. Samy

    Hello,
    It’s very wird. In my implementation of this solution when the XplatGenerateReleaseNotes@3 generate the release note, it add all the Pull Request (The PRs of the previous release and the PRs of the new releases). So more I create releases, biger is ths list of PRs in the release note. Do you know how to configure that in order to have only the PRs related to the latest build ? Thanks 🙂

  15. Gagan

    Vincent , I tried using the command below and it worked using -f forced push.
    git push –set-upstream -f origin $(“$(System.PullRequest.SourceBranch)”.replace(‘refs/heads/’, ”))

    I am facing now a problem in next step

    “Gitversion could not determine which branch to treat as the development branch (default is ‘develop’) nor release-able branch (default is ‘main’ or ‘master’), either locally or remotely. Ensure the local clone and checkout match the requirements or considering using ‘GitVersion Dynamic Repositories'”

  16. delpierro

    hi Dave,
    I am getting below error in calculating version
    ##[error]Error: GitVersion configuration file not found at /home/vsts/work/1/s/.azuredevops/gitversion.yml

Leave a Reply