Automatic semantic versioning (bonus: with release notes!)

I am a fan of using semantic versioning (a.k.a. SemVer) for data solutions, following the v1.0.0 pattern. It helps in the communication between team members and stakeholders, by limiting ambiguity and misunderstandings related to the version of your solution’s releases. With semantic versioning, the trick is to increment the version according to the changes you have made since the latest release. Manually keeping track of that is not an easy task, especially for small teams, without the capacity to have somebody dedicated to this administration task. I found a way to make this a lot easier, leaning on the Pull Request description! And as a bonus, we will create some nice release notes automatically 🚀.

Using the Pull Request description

My implementation of automatic semantic versioning relies on the data engineer to select the correct type of change in the Pull Request description.

Typ of update. Fix, Feature or Big feature.

These options of course relate to the Patch/Minor/Major of the SemVer pattern. I think these labels are easier to understand for data engineers because we usually talk about creating a bug fix or feature. Feel free to tweak these labels to your liking, but make sure you update the pipeline code as well.

Required Azure DevOps extensions

This setup relies on the Azure DevOps extensions listed below. Please review them carefully, and only install them in your organization if they pass your internal standards/governance rules regarding extensions.

  1. GitTools (we the tool GitVersion that is in this library)
  2. Pull Request Utils
  3. Generate Release Notes (Crossplatform)
  4. WIKI Updater Tasks

Add the Pull Request template

To automatically serve these options for every Pull Request, I use a template (check out the Docs to learn more about Pull Request templates). This is the code for my standard Pull Request:

Add this in a .md file in your git repository in a folder called .azuredevops, and it will be automatically picked up by new Pull Requests (Azure DevOps will scan for Markdown files in that folder).

Screenshot of the .azuredevops folder in the repo, with the pull request template in it.

Add the Release notes template

The process for automatically adding a release note in the wiki needs a template file. Please see the Markdown file below. Yes, it looks very weird now, with all those mustache brackets, but no worries. All of those {{variable}} thingies will be automatically replaced.

Add this file in your git repository in a folder called .azuredevops, with the name release-notes-template.md. If you use another name or location, remember to change the values in the YAML pipeline as well (we will add that pipeline later).

Add the GitVersion configuration file

The process for semantic versioning (GitVersion tool) relies on a config file. Please see the file below.

Add this to your git repository in the .azuredevops folder, with the name gitversion.yml. If you use another name or location, remember to change the values in the YAML pipeline as well (we will add that pipeline later).

If you already have a release number for your project, you can set the initial seed value in the first line of the above file. Where it now says next-version: 1.0.

Required Azure DevOps permissions and settings

To be able to make this setup work correctly, we need a couple of permissions and settings.

  • Make sure that the Build Service user of your project has the Contribute and Create tag permissions on the git code repository that you work from:
  • Make sure that the Build Service user has Contribute permissions on the git repo of your project wiki. You can manage the Wiki security by clicking on the tripple dotted menu of the wiki:
  • Disable the project setting called Limit job authorization scope to referenced Azure DevOps repositories. See the screenshot below where to find that setting. If it is un-editable, first go to the Organization settings and disable it there, then come back to the project settings and disable it here.

Add the pipeline for Pull Request administration

To pick up the selected type of update from the Pull Request description, I have created the Azure DevOps pipeline shown below (.yml). You will have to add that to your repo in a location of your preference with the name of your preference. I call it pipeline-pullrequest-administration.yml.

This YAML pipeline performs the following tasks:

  • Install GitVersion to be used in the last task.
  • Retrieve the Pull Request description and store it in a variable to use it in the next task.
  • Add a git commit message based on what was selected in the PR description, marking the branch as either a patch, minor or major change. This will be picked up by A) the next task, and by B) in another DevOps YAML pipeline that is triggered right after the merge is completed.
  • Determine the correct semantic version (note, this is only the version of the feature branch, it does not have to be the same as the final version).

Now that we have this pipeline, let put it to use. First, add a new pipeline in your Azure DevOps project, pointing to the .yml file in your repo. I have called my pipeline Pull Request administration.

Secondly, go to the policies page of your main branch:

Add a Build Validation, based on the pipeline you have just added.

This will start the Pull Request administration pipeline every time we publish a Pull Request 🚀!

Add the pipeline for release administration

We also need a second Azure DevOps YAML pipeline for our release administration. Please see the script below, save it as a pipeline .yml file in your repo, in your preferred location. I use the name pipeline-release-administration.yml.

Please update the repo reference to the correct URL of your git repo of the wiki, not the wiki itself:

This YAML pipeline performs the following tasks:

  • Job CalculateVersion
    • Install GitVersion to be used in the last task.
    • Determine the correct semantic version.
    • Update the Build.BuildNumber to use SemVer, as by default it uses FullSemVer (not my preferrence).
    • Add git tag for the calculated semantic version (e.g. v2.0.1).
  • Job CreateReleaseNotes
    • Generates a release notes file
    • Publishes the release notes in the project wiki

Add a new pipeline in your Azure DevOps project, pointing to the .yml file in your repo that you have just uploaded (pipeline-release-administration.yml). I have called my pipeline Release administration.

Great, this pipeline will now be automatically triggered for each commit performed on the main or master branch, because of the trigger section in the .yml file! In other words, after each Pull Request is completed, this pipeline will be started 🚀.

Bonus: Tags

After every Pull Request completes, a new Tag will be automatically added, pointing to the (git) version of your code.

Bonus: Release Notes

And, after each Pull Request completes, a note will be automatically prepended in the release notes page in the wiki for the new version. It will have a list of associated Pull Requests (usually just a single request) and the associated work items. And, a download link is added as well. I tried to mimic the standard GitHub release notes pages 😉.

Limitations and considerations

  • You can’t use the Squash merge type, when completing the Pull Request to your main branch. We need the commit messages of the feature branch (with +semver:) to be transferred to the main branch.
Pass the sauce
Tags from the story
, , ,
More from Dave Ruijter

Mini-series part 2: Metadata-Driven CDM Folder Creation Using Azure Databricks

This is the second blog post in a mini-series about a new...
Read More

1 Comment

Leave a Reply

Your email address will not be published. Required fields are marked *