Engeenering

Go CD – Continuous delivery through pipelines

Nenad Božić

|

Jul 09, 2021

To compete in today’s IT market, you must be truly agile, you must listen to your customers, and deliver features promptly.

To support business development and marketing in their lean strategies we, as developers, must leverage fast deliveries and deployments and test automation. Continuous Delivery makes it possible to continuously adapt software in line with user feedback, shifts in the market, and changes in business strategy. Testing, support, development, and operations work together as one delivery team to automate and streamline the build, test, and release process.

There are a lot of quality tools out there. For a long time, we used Jenkins as the most widespread CD tool, with a great community and a lot of plugins and integrations with other tools. What we lacked was a natural pipeline flow and good visualization. We also lacked some more advanced features like pipeline dependencies, conditional triggering jobs from many pipelines, templating, etc. We needed to look elsewhere and we decided to go with Go CD, a product by ThoughtWorks which became open-source in 2014. It is a Java/Ruby on Rails advanced continuous integration and release management system, according to their website. The major reason why we chose it was that they modeled pipelines as first-class citizens and that, in our opinion, it used the right abstraction for the delivery pipeline. But let us start from the beginning.

GoCD overview

At the highest level, Go consists of two main components, the Go Server and multiple Go Agents. The system works on a pull model where agents periodically poll the server for work.

goCD-architecture.png

The main flow of Go goes through a couple of following stages:

  1. The user adds a pipeline with material
  2. MDU (material update sub-system) notices a new commit
  3. The scheduler schedules a build
  4. Go agents poll for work and get assignments
  5. The agent does the work

Let’s talk about the main building blocks of Go. As stated before, the main abstract is a pipeline which is the highest unit of work with its inputs and outputs. The input object of one pipeline is called a material and it can be either a version control resource (Git, Gerrit, Subversion, Mercurial) or an output from another pipeline. The output of a pipeline is called an artifact. Since there is one server and multiple agents, there is no guarantee that the whole pipeline will be performed by the same Go agent. Artifacts are copied to the Go server and picked up by agents who require them for their jobs.

Each pipeline consists of one or more stages, where each stage has one or more jobs and each job has one or more tasks. Granularity to this level of detail is done because of parallelism. Inside pipeline stages are sequential and can be triggered automatically on success or manually. Within each stage, jobs are parallel. The outcome of a stage is considered a failure if at least one job fails. Again, tasks within each job are also sequential.

After installing the Go server and client there is no need for an extensive configuration. However, it is recommended to create a separate partition on a computer’s hard disk for Go server artifacts (artifacts can grow over time and problems may occur). In server configuration, there is also an admin tab for URL configuration. We needed to get feedback on failing builds, so we integrated Go with LDAP so each user of Go had an email and could subscribe to build information based on preferred filters. Here is a link that explains the authentication process.

It is worth mentioning that Go CD has a powerful API for power users where the entire configuration can be performed via REST. It has great documentation with examples, JSON requests, and responses. Here is a link to Go CD API documentation.

Pipeline dependencies

Go supports pipeline dependencies. Artifacts defined in upstream dependencies can be accessed by downstream dependencies. A downstream pipeline can be configured either to be triggered automatically (for example for building on a development environment) or manually (for example for building on a production environment).

Multiple pipeline dependency is called fan-in and it ensures that pipeline is triggered only when all upstream dependencies finish. Upstream dependencies consist of other pipelines or version control which make them powerful. If you have a client-server application and have functional tests on the client’s side, which depend on the server being updated, you can make a client functional test pipeline that will trigger on commit on the client and successful build and deployment of server-side.

goCD-fanIn.png

The additional challenge here is a diamond-like dependency, where it is not enough for both upstream dependencies to finish but to have the right versions. The following diagram depicts that problem. Here, configuration is really important, C1 must be set as material for both C2 and C3, and C2 and C3 are materials for the pipeline Package. The package will auto-trigger when both C2 and C3 go green with the same version of the code.

dieamond-problem.png

Pipeline templates

The template engine is a great effort and time saver. Each pipeline can be promoted to the template and, based on that template, other pipelines can be built with a few clicks. We used this extensively for deployment pipelines. Usually, there are multiple environments (development, stage, UAT, production) and the deployment process is the same with only a few parameters which are different. You can create one deployment pipeline and test it. When you make sure it works, extract the template out of it and clone the deployment pipelines for other environments. Differences can usually be covered with a couple of parameters, which can be created upon pipeline creation.

Conclusion

In the introductory part, we mentioned that pipelines are modeled in Go CD as first-class citizens. In Jenkins, you can order a row of boxes and let the flow go through each one of them until it finishes. Each box here in Jenkins is equivalent to each task in Go CD. Moreover, in Go CD, each box is a pipeline itself with its stages, jobs, and tasks.

Go CD is a fairly new player in the automation world with a refreshing UI and a couple of nice concepts. The community is still growing but it is responsive.

We had a couple of problems which we posted on StackOverflow and usually got answers pretty quickly. We are using the Gerrit/Github plugin to notify Github PR on failed or passed build which is being actively developed. New releases are pushed frequently. Documentation is great, especially API documentation. It’s our pleasure to use such a great UI and a couple of nice advanced features. You can model your pipeline in a great variety of ways. Some features are missing but we are all about open source so, in the future, we will try to help this project and start contributing.

Stay up to date!

Stay at the AI frontier. Explore, learn, and subscribe for the latest in tech trends and advancements!