Efficiently update your local git branches tracking remote ones

Suppose you are working on several shared git repositories.

If you work on a repository everyday, you normally will have your repository up-to-date. However if you only work on a repository once a week or less, next time you visit your local repository, all your branches will probably be behind your origin/upstream repository.

Normally you will update your local repo before starting to work, by doing the following:

$ git fetch
$ git checkout master
$ git pull
$ git checkout develop
$ git pull

This is a bit cumbersome, even with a GUI you normally follow these steps by clicking and using menu commands. If you have more local branches tracking remote feature branches, the procedure will be even more painful.

After a while, you will dread revisiting old projects because of this, and will think it’s better to scrap your local repo and re-clone it again in order to avoid manually updating all your local branches.

But fear not, with a bit of scripting, we can automate this tedious task, and using a nice git fetch trick we will do it without checking out other brances, so the procedure will be lightning fast.

The gist of the script is shown here:

unset ambtrackings
declare -A ambtrackings
while read branch; do
  upstream=$(git rev-parse --abbrev-ref $branch@{upstream} 2>/dev/null)
  if [[ $? == 0 ]]; then
    remote=${upstream%/*}
    remotebranch=${upstream#*/}
    echo $branch tracks $upstream: $remotebranch on remote $remote
    ambtrackings[$remote]="${ambtrackings[$remote]} $remotebranch:$branch"
  else
    echo $branch has no upstream configured
  fi
done < <(git for-each-ref --format='%(refname:short)' refs/heads/*) git checkout --detach for i in "${!ambtrackings[@]}"; do   echo "Updating local branches tracking remote: $i"   echo "> [remotebranch]:[localbranch]: ${ambtrackings[$i]}"
  git fetch $i ${ambtrackings[$i]}
done
git checkout -

You can put this somewhere in your $PATH, or configure it as a custom action if you’re using SourceTree, and you can go directly from here:

gitbranchesbeforeupdate

to here:

gitbranchesafterupdate

by executing the script:

$ updatelocalbranches.sh
develop tracks origin/develop: develop on remote origin

hotfix/PJYS-8-... tracks origin/hotfix/PJYS-8-...: hotfix/PJYS-8-... on remote origin

master tracks origin/master: master on remote origin

Note: checking out 'HEAD'.

You are in 'detached HEAD' state. [...]

HEAD is now at 2401da1... Merge pull request #5 in PRUEB/pruebas-de-stash-y-jira from feature/PJYS-8-automatizar-las-versiones-del-codigo to develop

Updating local branches tracking remote: origin
> [remotebranch]:[localbranch]:  develop:develop hotfix/PJYS-8-...:hotfix/PJYS-8-... master:master

From http://codman2-d.esios.ree.es:7990/bitbucket/scm/prueb/pruebas-de-stash-y-jira
   2401da1..e601e0b  develop    -> develop
   8b99600..6534e9e  hotfix/PJYS-8-automatizar-las-versiones-del-codigo -> hotfix/PJYS-8-automatizar-las-versiones-del-codigo

Your branch is up-to-date with 'origin/develop'.

Previous HEAD position was 2401da1... Merge pull request #5 in PRUEB/pruebas-de-stash-y-jira from feature/PJYS-8-... to develop
Switched to branch 'develop'

The script is a bit verbose right now in order to see what’s going on. But its output may be tweaked/removed at will.

It also supports more than one remote. As long as your trackings are correctly set, it will update them against the right remote.

Using this is a breeze and a joy. Having your branches behind your origin/upstream will no longer be an issue.

Enjoy!

Managing your project version with gradle plugins

In a previous post we saw that we could dynamically construct our version based on the state of our repository.

However if we do not want to go through the hassle of creating such a function ourselves, we could always use one of the many versioning gradle plugins available.

A simple search on the gradle plugins portal will yield several pages of plugins for us to choose from.

I first tried the nebula-release-plugin. The changes to my build.gradle were quite simple: remove my custom calculate function (and its use under allprojects) and simply appy the plugin:

plugins {
   id 'nebula.nebula-release' version '4.0.1'
}

This will give you versioning out of the box depending on your branch, and on several commandline parameters.

The plugin is quite configurable. However it includes steps to push to remote repositories that you might not want to enforce, and it’s not straightforward to remove such steps.

Intrigued, I tried gradle-git, since the nebula plugin is based on it. However it suffers from the same problem as the nebula plugin, it asks for credentials for fetching and pushing to the git repository (I do not necessarily want the plugin to do the pushing for me).

The nebula plugin had some provisions to at least ignore git checks via an undocumented property: release.disableGitChecks, but then it will not tag the commit either (not too problematic).

In the end I went back to the nebula-release-plugin, but configuring ssh keys to avoid annoying credentials propmting. I like that it integrates with normal gradle-java tasks, so I can still run gradlew build and I will get a development version automatically:

ME@MYPC MINGW64 /d/DatosXP/fuentes/pruebas-de-stash-y-jira (develop)
$ ./gradlew build
-------------------------------------------------
Inferred project: javagittest, version: 1.8.0-dev.13+2401da1
Prueba de git y java - version: 1.8.0-dev.13+2401da1
-------------------------------------------------
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:jar
:assemble
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:test UP-TO-DATE
:check UP-TO-DATE
:build

BUILD SUCCESSFUL

Total time: 7.554 secs

If I want to tag a new release I will use one of the tasks provided by the plugin (mind your branches):

ME@MYPC MINGW64 /d/DatosXP/fuentes/pruebas-de-stash-y-jira (release/1.8.x)
$ ./gradlew candidate
-------------------------------------------------
Inferred project: javagittest, version: 1.8.0-rc.1
Prueba de git y java - version: 1.8.0-rc.1
-------------------------------------------------
:releaseCheck
:candidate
:prepare
:compileJava UP-TO-DATE
[...]
:check UP-TO-DATE
:build
:release
Tagging repository as v1.8.0-rc.1
Pushing changes in [refs/heads/release/1.8.x, v1.8.0-rc.1] to origin

BUILD SUCCESSFUL

Total time: 9.989 secs

This will push the new tag to our origin remote, so we need to be careful with it.

When we are happy with the final version (for example after merging into master) we can make the final release, which will be also tagged and pushed:

ME@MYPC MINGW64 /d/DatosXP/fuentes/pruebas-de-stash-y-jira (master)
$ ./gradlew final
-------------------------------------------------
Inferred project: javagittest, version: 1.8.0
Prueba de git y java - version: 1.8.0
-------------------------------------------------
:releaseCheck
:final
:prepare
:compileJava UP-TO-DATE
[...]
:check UP-TO-DATE
:build
:release
Tagging repository as v1.8.0
Pushing changes in [refs/heads/master, v1.8.0] to origin

BUILD SUCCESSFUL

Total time: 9.685 secs

If we want to re-build this version (at a CI server for example) we can use the property -Prelease.useLastTag=true in the gradlew command.

Other useful properties are -Prelease.scope=major or -Prelease.scope=patch if we want the version bump to increase the major or the patch part of the version instead of the default (minor).

[2016/06/01 Update]

These past days I’ve been struggling a little bit trying to use the nebula-release plugin but it really bothers me that the plugin pushes things for me.

Yesterday I found another very promising plugin, greatly configurable, which doesn’t force the push on me: enter axion-release-plugin

After a few hours testing and reading the docs I have come up with my initial configuration which does all I wanted:

  • autoincrementing version
  • adding -SNAPSHOT automatically for not-tagged-commits
  • versioning scheme configurable per branch
  • support for major.minor.patch-rcX
  • access to the version without need for tagging

In case you are curious, my initial config looks like this:

scmVersion {
    tag {
        prefix = ''
    }

    // Incrementing policy
    versionIncrementer 'incrementMinor'  //default
    branchVersionIncrementer = [
       'master': 'incrementPrerelease',
       'release/.*': 'incrementPrerelease',
       'hotfix/.*': 'incrementPrerelease'
    ]

    // Decorators
    versionCreator 'simple'
    branchVersionCreator = [
       'feature/.*': 'versionWithBranch'
    ]

    localOnly = true  //no pushing to remote
}

And I can access the version without building like this:

ME@MYPC MINGW64 /d/DatosXP/fuentes/pruebas-de-stash-y-jira (feature/switch_to_gradle)
$ ./gradlew currentVersion
:currentVersion

Project version: 1.1.0-feature-switch_to_gradle-SNAPSHOT

BUILD SUCCESSFUL

Total time: 8.643 secs

Be sure to visit axion’s page as it has outstanding documentation.

This plugin fits like a glove!

Managing your project version with gradle

In a previous post I explained how we versioned our project poms.

After giving up on dinamically setting the version with maven, I decided to bite the bullet and try to switch a relatively small project to gradle as a proof of concept.

Most of the conversions from maven to gradle were quite straightforward. I had a small bump with izpack, but it all worked out after a couple hours of internet searching.

The interesting part lies with the dynamic calculation of the project version’s based on the state of the git repository.

In our build.gradle we can define the version for the main project and its subprojects with a call to a function:

allprojects{
   version = calculateVersion()
}

In this function we make use of git describe to calculate our version:

//major and minor are mandatory
def evalVersion (strVer) {
   def(v1, v2, v3) = strVer.tokenize('.')
   v1 = v1 ?: 0
   v2 = v2 ?: 0
   return [v1, v2, v3]
}

def calculateVersion() {

        def longVersionName = "git -C ${rootDir} describe --tags --long".execute().text.trim()
        logger.info("longVersionName = $longVersionName")
        def longVersionNameT = longVersionName.tokenize('-')
        def fullVersionTag, milestone, versionBuild, gitSha
        if (longVersionNameT.size() > 3) {
           (fullVersionTag, milestone, versionBuild, gitSha) = longVersionName.tokenize('-')
        } else {
           (fullVersionTag, versionBuild, gitSha) = longVersionName.tokenize('-')
        }

        def(versionMajor, versionMinor, versionPatch) = evalVersion(fullVersionTag)

        def branch = "git -C ${rootDir} symbolic-ref --short -q HEAD".execute().text.trim()

        //if there is no tag...
        if ((versionBuild != "0") && (branch != 'master')) versionMinor = versionMinor.toInteger() +1

        def versionPatchStr = versionPatch ? "." + versionPatch : ""
        def versionBuildStr = (versionBuild == "0") ? "" : "-g"+versionBuild

        def versionName =""

        // ok for develop, feature and master
        versionName = "$versionMajor.${versionMinor}${versionPatchStr}${versionBuildStr}"

        def fbmatch = branch =~ /feature\/(.+)/
        if (fbmatch) {
            def lista = fbmatch[0][1].tokenize('-')
            versionName += "-FB_" +lista[0] + "-" + lista[1]
        }

        if (branch != 'master') {
            versionName += "-SNAPSHOT"
        }

        def rbmatch = branch =~ /release\/(.+)/
        if (rbmatch) {
            (versionMajor, versionMinor, versionPatch) = evalVersion(rbmatch[0][1])
            versionPatchStr = versionPatch ? "." + versionPatch : ""
            versionName = "$versionMajor.${versionMinor}${versionPatchStr}-RC${versionBuild}"
        }

        def hbmatch = branch =~ /hotfix\/(.+)/
        if (hbmatch) {
            (versionMajor, versionMinor, versionPatch) = evalVersion(hbmatch[0][1])
            versionPatchStr = versionPatch ? "." + versionPatch : "";
            versionName = "$versionMajor.${versionMinor}${versionPatchStr}-HF${versionBuild}"
        }

        return versionName
}

The idea behind this function is to calculate dynamically the version, based on which git branch we’re on and what’s in the closest git tag to our current commit:

  • The previous git tag should be in the format: tagMajor.tagMinor[.tagPatch]
  • “-gG”: from git describe, G is the number of commits until the last tag. If G=0, it will not be part of the version name.
  • develop branch: tagMajor.tagMinor+1[.tagPatch][-gG]-SNAPSHOT -> 1.1.0-SNAPSHOT or 1.1.0-g8-SNAPSHOT
  • feature/XXXX-N-… branch: tagMajor.tagMinor+1[.tagPatch][-gG]-FB_XXXX-N-SNAPSHOT -> 1.1-g2-FB_KESL-2-SNAPSHOT
  • release/x.y.z branch: x.y.z-RC[G]
  • hotfix/x.y.z branch: x.y.z-HF[G]
  • master branch: tagMajor.tagMinor[.tagPatch][-gG]

As you can see, you can complicate the version calculation as much as you want. This is only an example I used a couple years back.

I’ve seen recently a reference to nebula-release-plugin, and I’d like to try it out in order to remove my custom version calculation. I will post about my experience with it soon.

Managing your pom version under git

Most of our java projects are compiled with maven. If your VCS supports decent branching (like git), it is only natural that the version of your poms will need to be updated as your project evolves and new development versions, release candidates and final versions are created.

Doing these changes manually is not recommended for complex projects (where more than one pom is involved). Thankfully maven has some handy plugins to, at least partially, automate some of the tasks.

In particular, jgitflow-maven-plugin has been a life-saver. The only caveat for me is that too many commits for changing the poms are created, thus I only use this plugin for creating the release branch, not for finalizing it.

That said, I’m not very satisfied with all the steps we have to take in order to increment the versions of the different components.

Currently the list of steps to take when we want to release a new version includes:

  • Doing a git-flow release start with jgitflow:
    $ mvn external.atlassian.jgitflow:jgitflow-maven-plugin:release-start
  • Final reviews of the documentation, potentially adding some commits to the release branch

Since I don’t like jgitflow-maven-plugin’s extra commits when merging back to develop, I go another route:

  • Manually remove -SNAPSHOT from versions:
    $ mvn org.eclipse.tycho:tycho-versions-plugin:0.21.0:set-version -DnewVersion=1.0-m9 -Dartifacts=xxx-core-parent,xxx-core,xxx-utils
    $ git commit -a -m "[manually] updating poms with non-snapshot versions"
    
  • Finalize the release with AVH git flow (configuring git to use my local pom merge driver)

However, even though with the last two points we reduce the number of extra commits, all this still leaves the project’s history riddled with commits for bumping and changing the pom versions:

extracommitspom

Yes, only two, but in my mind that’s two too many.

In an upcoming post I will detail how we removed these extra commits, by moving to a completely different build system.