DevOps | 7 Steps to Automate a Delivery Pipeline

The 7 DevOps Interactions that Automate Software Delivery

No other website provides a detailed guide on how to automate your entire software delivery pipeline.

DevOps is what Google, Amazon and Twitter use to deliver working software once, (sometimes twice) a day. It is also known as Continuous Delivery – and the point is to automate the software delivery pipeline – from soup to nuts.

DevOps – Machine 2 Machine Interactions

An event triggers each interaction – one player contacts another – and something changes hands. There will be a request and a response, one player produces and the other consumes. So it begs the question

  1. Players – Who talks to who?
  2. Event – What triggers the conversation?
  3. Producer/Consumer – What changes hands?
  4. Value – Why Bother? What do we get?
  5. How To – How to Set it up – Step by Step

Your job is to configure the pipeline so that the actors can talk to who they need, when they need.

DevOps – What Do Humans Do? – Human beings write the software and unit tests, and they create artifacts like images and web page templates. We keep it simple to help you get your DevOps off the ground.

We then detail how to configure code style checks, prevent coupling between components, clamp down on complexity, set the bar for test coverage and test quality and branch out to include integration and system tests. Also human beings can step in to give the final go-ahead for a release to production.

The 7 DevOps Interactions

This table joins up the DevOps skeleton. You have 7 interactions to configure. Do this, and you will have a business grade, automated, software delivery pipeline.

# Interaction Player 1 Player 2 Description
7 Publishing Software CI Server (Jenkins) Repository Manager (Nexus) The assets are published to either the releases or snapshots libraries.

1. IDE to Repository Manager (Read) – IntelliJ to Nexus

The IDE goes to the library of published artifacts (our Nexus Repository) and requests the dependent and build helper artifacts. To get this interaction working we need to configure IntelliJ to make the request from the Nexus Repository.

2. CI Server to Repository Manager (Publish) – Jenkins to Nexus

After building and testing the artifacts, the Jenkins Continuous Integration server should publish the artifacts into one of two places.

  1. the Nexus snapshots repository
  2. the Nexus releases repository

The version number decides which repository – snapshots or releases. There is one catch.

It’s in the Louvre, baby

Once you publish - it's done. You cannot unpublish nor change it.

When an author publishes a book, she cannot correct even the teeniest typo. The book at the published version is set in stone. Major or minor amendments can only be published with an incremented version number.

It’s in the Louvre baby – it’s finished

A Louvre security guard spots a dishevelled, unkempt man dabbing at a masterpiece – paintbrush in one hand, palette in the other.

“Hey, stop that!” – bellowed the security guard, pulling the man away.

“Wait – You don’t understand” – came the response – “I am the artist and this is my work”.

“I’m applying some finishing touches – he added.

The security guard shook his head saying

     It’s in the Louvre baby, it’s finished.

When you publish an artifact – it’s published. No one, not even you can change it.

Nexus will not let you re-publish an artifact it already possesses with a given version number. The Jenkins job run with mvn deploy will work the first time, but the second time it will fail.

This rule does not apply to snapshot versions – you can deploy them and they will overwrite each other if needs be.

The 2 Parts of Releasing Software into Production

There is a push phase and a pull phase when you “release” into production.

  1. publish (push) – publishing a version of your software to a (your) repository
  2. upgrade (pull) – draw down an updated web application (or service) into production

The correct terminology is publishing into your repository for step 1 and upgrading the production system for step 2.

If you don’t run a live (production system) – your “releases” are “push releases”.

Publish (Push) Releases

If your team writes components (like log4j or perhaps a financial calculator) – you do not need to perform “pull” releases. Whenever the next version is ready, you push the usually jar archives into a repository. That’s it. Sooner or later your customers will get round to using the new archive in their dependency tree.

Upgrade (Pull) Relases

When you run live (production) systems – you will likely do both pull and push releases. Most enterprises run and maintain a subset of the below application services. In turn “releasing” software to each area involves pulling.

To release (upgrade) to a

  1. web application you pull a WAR file (version) from your repository
  2. scheduled job runner – you pull a JAR (or JARs) from your repository
  3. web service endpoint – you pull down the service descriptors and jars
  4. J2EE enterprise application – you pull an EAR from your repository
  5. queue processing engine – you pull down Jars or EARs from your repository
  6. static website – you pull html, images, CSS and JavaScript files from your repository
  7. production database – you pull in create, update table and data change scripts

When running production systems you will likely perform BOTH push and pull releasing.

Let’s discover the exact steps that take place when you release software using an automated DevOps (continuous delivery) methodology.

How to Publish Software Push Release

What do you do when you are ready to perform an automated push release. Here are the steps – you (the human being) just triggers the process. Nightly builds (SNAPHOT) releases occur without human involvement – not so when pushing production quality artifacts into your Nexus, Artifactory or Archiva repository.

  1. Step 1 » – you update your POM version numbers with the mvn versions command
  2. Step 2 » – you check in your software into your SCM (version control) tool
  3. Step 3 » – your Continuous Integration server (watching the SCM) trigger mvn deploy
  4. Step 4 » – after the (Jenkins) CI server runs the tests it notices that the version number does not include “SNAPSHOT”
  5. Step 5 » – omitting SNAPSHOT tells the maven deploy command to push the artifacts into your RELEASES repository
  6. Step 6 » – Jenkins stamps a SNAPSHOT version number so that subsequent check-ins deploy to the SNAPSHOTs repository
  7. Step 7 » – that’s it! It can send out an e-mail informing everyone of the upgraded artifacts.

The Maven Releases Plugin is over-complex and convoluted, hence we advise using a strategy similar to this one for your push releases.

Let’s break down the above high level release tactics so that you know exactly how to setup a DevOps automated release function.

Step 1 – Updating POM Versions with mvn versions

The Maven Deploy command has a built-in rule that examines the version number looking for the word SNAPSHOT. It’s existence causes a deploy (push) into the snapshots repository. It’s absence triggers a deployment into the live “Releases” repository.

Pre-Conditions Before Upgrading the Verson

Check your POM (or POMs) for the distributionManagement section.
Check your Maven settings file for the repository and servers definitions.

How to Update the POM with a release versions.

Do this at the root POM and every POM will be updated together in a multi-module project. Most multi-module projects run the same version number for every module (and the parent POM).

Change directory to where your pom.xml file lives.

  1. 1. svn update – to get your working copy up to date.
  2. 2. mvn versions:set -DnewVersion=0.1.6 -DgenerateBackupPoms=false – your new version will be 0.1.6
  3. 3. svn commit -m "Publishing component from version 0.1.5 to 0.1.6" – your new version will be 0.1.6

You can script the above commands into 1 if you’d like. They are the same for any maven project – nothing custom! It’s much simpler than employing the Maven Release Plugin which over-complicates and convolutes the essence of publishing software.

Addendum

Run this command at the parent module level and it will change all the version numbers and create backups of every pom.xml file it changes.

mvn versions:set -DnewVersion=0.0.3

After the CI Server deploys at night (nightly build) – you should run the versions command again and the next version number with SNAPSHOT like this.

mvn versions:set -DnewVersion=0.0.4-SNAPSHOT

And when you are ready to release again – you set the new version to 0.0.4.

Check the Nexus repository to view the released versions of your software components.

Use generateBackupPoms=false to dispense with the backup poms (which duplicate the work of your version control tool).

mvn versions:set -DnewVersion=0.0.3 -DgenerateBackupPoms=false

Either that or run mvn versions:commit which deletes the backup poms.

Jenkins Script to Auto-Reset Snapshot Revision

After a successful publish build – the next “regular” build would fail because the published artifacts cannot be rewritten (it’s in the Louvre, baby)!

This script added as a Jenkins “post-build action” will reset back to a SNAPSHOT version number.


#!/bin/bash +x

## ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ##
#  Function [scmProjectRevision]
## ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ##
#  Determine the SCM [revision] (after checkin) that last changed the project
#  specified by the subversion URL (environment variable set by Jenkins).
## ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ##
function scmProjectRevision
{
    local leftSideStringMarker="Last Changed Rev"
    local rightSideStringMarker="Last Changed Date"

    local repositoryInfo=`svn info $SVN_URL`
    local strSpaceColonSpaceNum=${repositoryInfo#*$leftSideStringMarker}
    local strStartingWithNumber=${strSpaceColonSpaceNum%%$rightSideStringMarker*}
    local revisionNumberWithDirtySpace=${strStartingWithNumber#*:}

    local scmRevisionNo=`echo $revisionNumberWithDirtySpace`
    echo "$scmRevisionNo"
    return 0
}

echo "[BBW] ----"
echo "[BBW] ------------------------------------------------------------------------ ##"
echo "[BBW] ----  Build Business Websites - Resetting Snapshot Version  ------------ ##"
echo "[BBW] ------------------------------------------------------------------------ ##"
echo "[BBW] ----"
echo "[BBW] ----  Success building $JOB_NAME at revision [$SVN_REVISION] # [ $BUILD_ID ]."
echo "[BBW] ----  Resetting to SNAPSHOT build @ workspace location below]."
echo "[BBW] ----  Workspace = $WORKSPACE"
echo "[BBW] ----"
echo "[BBW] ------------------------------------------------------------------------ ##"
echo "[BBW] ----"

# --------------------------------------------------------------------------------------- ##
revisionLastChanged=`scmProjectRevision`
revisionFromJenkins=$SVN_REVISION
mvn versions:set -DnewVersion=0.0-$revisionLastChanged-SNAPSHOT -DgenerateBackupPoms=false
# --------------------------------------------------------------------------------------- ##

echo "[BBW] ----"
echo "[BBW] ------------------------------------------------------------------------ ##"
echo "[BBW] ----  The Snapshot Version is now 0.0-$revisionLastChanged-SNAPSHOT  --- ##"
echo "[BBW] ------------------------------------------------------------------------ ##"
echo "[BBW] ----"
echo "[BBW] ----  Will now attempt to check in the updated pom.xml files."
echo "[BBW] ----  Checkin destination will be to subversion url below."
echo "[BBW] ----  Url = $SVN_URL"
echo "[BBW] ----"

# --------------------------------------------------------------------------------------- ##
svn commit -m "Jenkins SNAPSHOT version reset. Revision [$SVN_REVISION] ID [ $BUILD_ID ]."
svn update
postCheckinRevision=`scmProjectRevision`
revisionsDifference=$((postCheckinRevision - revisionLastChanged))
# --------------------------------------------------------------------------------------- ##

echo "[BBW] ----"
echo "[BBW] ------------------------------------------------------------------------ ##"
echo "[BBW] ----  Last revision touching project = [ $revisionLastChanged ]."
echo "[BBW] ----  Jenkins (Environment) Revision = [ $revisionFromJenkins ]."
echo "[BBW] ----  Post Version Check-in Revision = [ $postCheckinRevision ]."
echo "[BBW] ----  Revision increment step amount = [ $revisionsDifference ]."
echo "[BBW] ----"

if [ $revisionsDifference -gt 0 ] ; then
    echo "[BBW] ------------------------------------------------------------------------ ##"
    echo "[BBW] --- SUCCESSFUL - Project version has been successfully changed."
    echo "[BBW] ------------------------------------------------------------------------ ##"
    echo "[BBW] ----"
    exit 0
else
    echo "[BBW] ----"
    echo "[BBW] ------------------------------------------------------------------------ ##"
    echo "[BBW] ---- ERROR - At least 1 difference ( not $revisionsDifference ) expected."
    echo "[BBW] ------------------------------------------------------------------------ ##"
    echo "[BBW] ----"
    exit 42
fi

Note that the line #!/bin/bash +x switches off the Jenkins command echo for the whole script.

There are 2 important lines above. The first uses maven to reset the version without generating backup pom.xml files (as they are versioned).

The second line commit’s the pom.xml changes so that the next build will read up a snapshot version and will publish to the snapshot repository (not the releases repository).

Jenkins Change to SNAPSHOT Version Script Output

The script running below after the Jenkins build changes the project version to a SNAPSHOT.

from version 0.0.6 to 0.0-566-SNAPSHOT

01:49:36 [INFO] ------------------------------------------------------------------------
01:49:36 [INFO] BUILD SUCCESSFUL
01:49:36 [INFO] ------------------------------------------------------------------------
01:49:36 [INFO] Total time: 5 seconds
01:49:36 [INFO] Finished at: Wed Aug 17 01:49:36 BST 2016
01:49:36 [INFO] Final Memory: 23M/56M
01:49:36 [INFO] ------------------------------------------------------------------------
01:49:36 [PublishTools] $ /bin/bash +x /tmp/hudson4028876218977096937.sh
01:49:36 [BBW] ----
01:49:36 [BBW] ------------------------------------------------------------------------ ##
01:49:36 [BBW] ----  Build Business Websites - Resetting Snapshot Version  ------------ ##
01:49:36 [BBW] ------------------------------------------------------------------------ ##
01:49:36 [BBW] ----
01:49:36 [BBW] ----  Success building PublishTools at revision [566] # [ 42 ].
01:49:36 [BBW] ----  Resetting to SNAPSHOT build @ workspace location below].
01:49:36 [BBW] ----  Workspace = /var/lib/jenkins/workspace/PublishTools
01:49:36 [BBW] ----
01:49:36 [BBW] ------------------------------------------------------------------------ ##
01:49:36 [BBW] ----
01:49:37 [INFO] Scanning for projects...
01:49:38 [INFO] Searching repository for plugin with prefix: 'versions'.
01:49:38 [INFO] ------------------------------------------------------------------------
01:49:38 [INFO] Building The Utility Factory for Enterprise Software
01:49:38 [INFO]    task-segment: [versions:set] (aggregator-style)
01:49:38 [INFO] ------------------------------------------------------------------------
01:49:41 [INFO] [versions:set {execution: default-cli}]
01:49:41 [INFO] Searching for local aggregator root...
01:49:41 [INFO] Local aggregation root: /var/lib/jenkins/workspace/PublishTools
01:49:41 [INFO] Processing change of uk.co.apollo13:factory:0.0.6 -> 0.0-566-SNAPSHOT
01:49:41 [INFO] Processing uk.co.apollo13:factory
01:49:41 [INFO]     Updating project uk.co.apollo13:factory
01:49:41 [INFO]         from version 0.0.6 to 0.0-566-SNAPSHOT
01:49:41 [INFO] 
01:49:41 [INFO] ------------------------------------------------------------------------
01:49:41 [INFO] BUILD SUCCESSFUL
01:49:41 [INFO] ------------------------------------------------------------------------
01:49:41 [INFO] Total time: 4 seconds
01:49:41 [INFO] Finished at: Wed Aug 17 01:49:41 BST 2016
01:49:42 [INFO] Final Memory: 22M/54M
01:49:42 [INFO] ------------------------------------------------------------------------
01:49:42 [BBW] ----
01:49:42 [BBW] ------------------------------------------------------------------------ ##
01:49:42 [BBW] ----  The Snapshot Version is now 0.0-566-SNAPSHOT  --- ##
01:49:42 [BBW] ------------------------------------------------------------------------ ##
01:49:42 [BBW] ----
01:49:42 [BBW] ----  Will now attempt to check in the updated pom.xml files.
01:49:42 [BBW] ----  Checkin destination will be to subversion url below.
01:49:42 [BBW] ----  Url = https://www.build-business-websites.co.uk/assets/apollo13.co.uk/factory
01:49:42 [BBW] ----
01:49:42 Sending        pom.xml
01:49:42 Transmitting file data .
01:49:42 Committed revision 567.
01:49:42 Updating '.':
01:49:42 At revision 567.
01:49:42 [BBW] ----
01:49:42 [BBW] ------------------------------------------------------------------------ ##
01:49:42 [BBW] ----  Last revision touching project = [ 566 ].
01:49:42 [BBW] ----  Jenkins (Environment) Revision = [ 566 ].
01:49:42 [BBW] ----  Post Version Check-in Revision = [ 567 ].
01:49:42 [BBW] ----  Revision increment step amount = [ 1 ].
01:49:42 [BBW] ----
01:49:42 [BBW] ------------------------------------------------------------------------ ##
01:49:42 [BBW] --- SUCCESSFUL - Project version has been successfully changed.
01:49:42 [BBW] ------------------------------------------------------------------------ ##
01:49:42 [BBW] ----
01:49:42 Finished: SUCCESS

Handling Jenkins Failure to Authenticate on Subversion Commit?

If Jenkins fails to authenticate on commit there is win-win way to handle it. Even when you add the credentials Jenkins can still complain.

The Jenkins Subversion Authentication failed (E215004)

23:35:33 svn: E215004: Authentication failed and interactive prompting is disabled; see the --force-interactive option
23:35:33 svn: E215004: Commit failed (details follow):
23:35:33 svn: E215004: No more credentials or we tried too many times.
23:35:33 Authentication failed

Also Jenkins security is poor so giving Jenkins the password to your repository in plain text is not a thought to relish.

And the last thing you need is to install yet another plugin that may destabilize your build system.

There is another way. A win-win on all accounts.

Pre-Authenticate Subversion Access for Jenkins

Go to your command line and become Jenkins with sudo su - Jenkins

Create a folder in the Jenkins home and go to it.

svn checkout https://www.build-business-websites.co.uk/assets/middleware

Make a wee change in your new working copy.

touch wee-file.txt

Now check-in your wee change

svn commit -m "setting up Jenkins credentials on the server side"

You will be prompted for a username and password. Enter them. Say Yes. That’s it!

Now back in the UI – Jenkins will now talk to Subversion – and it does not hold the credentials.

The Jenkins Working Copy Format Too Old Issue

Don’t upgrade your working copy. It is fine. All you need to do is to go into the Manage Jenkins area ( Subversion settings ) and change one box.

Change the Subversion version from 1.4 to 1.8 in the Jenkins – Manage Jenkins – Subversion.

Did You Know – Archiva Documentation

The Archiva complete reference for both users and administrators (pdf format)

The Archiva repository manager pdf reference documentation

Archiva enables those that depend on your software, to access the released archive files straight from your own repository. This includes your own software built in different components. It enables you to scale and manage your software output and dependencies.

Leave a Reply

Your email address will not be published. Required fields are marked *