introducing mazzle - a pipelines as code tool

This is a prototype. YMMV

mazzle is a command line tool to coordinate large complicated environments that are built from multiple devops tools. mazzle is kind of a task runner and its web GUI is modelled to appear like a pipelined continuous integration server.

See this project on Github

infrastructure as code and pipelines as code

Write self-descriptive pipelines in dot syntax renderable by graphviz and executable by this tool. mazzle uses Graphviz dot file syntax for its configuration. In mazzle, you specify the order and dependencies of pipeline execution via flow of data between components. Data passes between components via environment variables.

parallel build execution

mazzle knows what parts of your environment infrastructure can run together concurrently and in parallel due to its configuration being a graph file. Here is an example graph and GUI screenshot.


This pipeline is fairly complicated environment that brings up two worker nodes with Ansible and provisions Hashicorp Vault with Terraform. Notice how some components such as dns, security, vault volume can all begin running at the same time as they are independent.


pipeline view


log history

environments view



devops-pipeline is for deterministically creating computer environments. An example environment is one that could use AWS, Terraform, Packer, shell scripts, Ansible, docker, Chef Inspec for testing. devops-pipeline allows you to chain together tools for running on your developer workstation. devops-pipeline models the flow of data between tools and uses environment variables to pass along data. devops-pipeline is meant to be used after each change whereby it runs validations, unit tests, smoke tests and deployments tests. devops-pipeline is meant to be ran after each and every change to your infrastructure code. You should be able to change any two parts of your infrastructure - at any layer - and test those changes together.

how it works

devops-pipeline understands .dot files and interprets them to work out what must be done. All devops-pipeline does is walk the directories mentioned by your dot file and run a script inside each folder, named after the lifecycle stage. This is where you define what should happen for that stage of the pipeline. This is what gives devops-pipeline the flexibility to run any devops tool.

Your code is separated by directory by each tool. Like a monorepository, you divide your code by tool, so you have a directory for ansible code, a directory for terraform code. Devops-pipeline loops through these directories and runs a lifecycle command that you have in each code directory.

For example, you have the following code directories:


Devops-pipeline will cd to these directories and run a lifecycle command, which could be any of the following items in bold:

idiom - provision SSH workers at the beginning of your pipeline

Unlike Jenkins and gocd, worker nodes are considered to be part of your pipeline. An idiom in devops-pipeline is that your early stages in your pipeline is provisioning worker nodes. These worker nodes run the remainder of the build. You can replace --workers with --discover-workers-from-output <output name> where output name is the name of an ouput from your machine provisioning component that contains a list of server hostnames or IP addresses that you can SSH onto.

Here is an example of ansible provisioning EC2 instances and installing dependencies on worker nodes, then running packer to build an AMI and launching that AMI with terraform.

The at symbol @ at the beginning of a component reference means that this component builds on the master node.

digraph G {
	"@ansible/machines" -> "@ansible/provision-workers"-> "packer/source-ami" -> "terraform/appservers";

idiom - building development workstations

An idiom is that developer workstations are provisioned by devops-pipeline which are your workstations you use for development.

digraph G {
	"@vagrant/devbox" -> "@ansible/workers" -> "@ansible/workers-provision";

Example - building and using an AMI


digraph G {
   "packer/ubuntu" -> "terraform/appserver";

In the above pipeline packer/ubuntu is a component that that uses packer to create machine images on AWS with Java installed. packer/ubuntu outputs an AMI ID. terraform/appserver is another component that needs this AMI ID to bring up a new instance running that AMI.

Example - building and releasing a Java app


digraph G {
  "ansible/machines" -> "gradle/app" -> "ansible/deploy" -> "ansible/release";

ansible/machines is a component that provisions machines running java. gradle/app is a component that builds from source a Java app. One of gradle/app’s outputs is a path to an artifact; a set of jar files.


  1. Ubuntu

Install ansible

sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

Install terraform with plugins

git clone
cd devops-pipeline
./  # adds devops-pipeline to your path via /etc/profile.d/
sudo apt-get install python3-pip
pip3 install -r requirements.txt
cd devops_pipeline/web ; npm install ; npm build

Create a GPG key

gpg --full-generate-key
  1. Logout and log back in
git clone
cd devops-pipeline-starter
  1. Update ~/.aws/env to have the following. Upload a keypair to AWS using the AWS console.
  1. Take a look around the demo project, which brings up 7 nodes in AWS. See to see how it fits together.

  2. Update common.tfvars to include your key name, key path, IP address. Bring up demo project (this will cost money)

devops-pipeline home \
           --file \
	   --gui \
	   --discover-workers-from-output workers \
	   --workers-key <path to ssh private key> \
	   --workers-user ubuntu \
	   --keys <gpg key email> \
	   --rebuild 'packer/source-ami*' 'packer/authenticated-ami*' 'packer/vault-ami*' 'packer/ubuntu-java*'

Go to http://localhost:5000. Click switch to environment under home.

SSH workers

You don’t always want to run builds on the master node (where you run devops-pipeline from) You can specify a list of hosts to run builds on remote servers via SSH.

devops-pipeline --file \
    --gui \
    --workers node1 node2 \
    --workers-key ~/.ssh/worker-ssh-key \
    --workers-user ubuntu

If you’re provisioning worker nodes as part of your pipeline, which is what we recommend, you can output the machine hostnames as an output and use --discover-workers-from-output output-name.

passing data along a pipeline

Each life cycle command shell script is called with the environment variables:

Your lifecycle script must take this environment variables and write to files at their locations. The OUTPUT_PATH is an absolute path to a file that you should write outputs as a JSON file. The EXIT_CODE_PATH is where you should write an exit code of the lifecycle command. If it is 0 then the build is considered to be successful.

ARTIFACT_PATH is a path that you should save an archive of parts of your working directory to be required from another pipeline.

Each time devops-pipeline runs a pipeline, it passes outputs of all upstream variables to to your pipeline command as environment variables.

the builds directory


The builds directory is where logs and other outputs go:

depending on artifacts

To depend on an artifact, such as an archived workspace from a previous build (see ARTIFACT_PATH above) create a file called require in your root of a provider directory. It should contain the name of the pipeline to pull artifacts from and the words latest or latestSuccessful. Latest means the last created artifact and latestSuccessful means the last artifact that was a succeeding build.

For example, the following pulls in gradle/app



if [ -z $ENV ] ; then
  echo "need to provide environment name"
  exit 1

echo $COMPONENT >&2

if [ -z $COMPONENT ] ; then
  echo "need to provide component name"
  exit 1

if [[ "${COMPONENT}" == "deploy" ]]; then
  echo "gradle/app latestSuccessful"

why devops-pipeline