Henry S. Coelho

Continuous integration and deployment of microservices in a monorepo with Kubernetes and CircleCI

I have been reading a lot about DevOps; more specifically, continuous integration, continuous deployment, and project layout. I also heard a lot about monorepos for microservices - apparently they are much easier to manage than separate repositories for every microservice - but how can I join these three things together? This is what I am exploring, and I am going to share my results here.

First, what exactly am I looking for? Well, these are my constraints:

  1. The solution should be applicable to most languages, but it has to work for Go
  2. I want to have both Continuous Integration and Deployment
  3. I must be able to deploy on Kubernetes
  4. Shared code must live in a dedicated directory, not inside one of the microservices
  5. The CD script must only deploy microservices that were actually changed. I don't want to deploy all microservices all the time
  6. The CI script must be able to test all microservices
  7. The solution must work with CircleCI

I will leave a link to the final result at the end of this post.

Here is what we are going to do:

  1. Come up with a simple directory structure, so we can know where to put the microservices and shared code
  2. Create some shared, boilerplate code for the microservices and put it in the shared code package
  3. Create a simple microservice using the shared code
  4. Create test and deployment scripts for the microservice
  5. Clone the microservice, just so we will have more than one
  6. Create the scripts to test and deploy only the microservices we need
  7. Create the configuration for CI/CD with CircleCI

1- Directory structure

I will start by making a basic directory structure. This project will be called go-microservice and will live inside my GOPATH (~/go/src/github.com/hscasn/go-microservice):

.
├── apps     # I will put my microservices here
├── Makefile # Instructions to build, clean, deploy, etc
├── pkg      # Shared code goes here
└── scripts  # Shell scripts

Very simple structure, but it is all we need.

Some people may prefer naming the apps directory services or something like that, which makes sense. However, if one of the microservices is an MVC app, you will probably have another services directory inside. I want to avoid confusion, so I'll call it apps. The microservices will be put directly inside that package, like this:

├── apps
│   ├── app1 # Microservice 1
│   ├── app2 # Microservice 2
│   ...

And then every microservice can have their own internal, scripts, and so on.

2- Shared code

I already have some some code that can be shared among other microservices, so I will put it inside the pkg directory. The idea behind this package is to group common utilities to reduce the amount of duplicated code. Here is the directory structure of pkg now:

├── pkg
│   ├── api
│   │   ├── api.go
│   │   ├── api_test.go
│   │   ├── health
│   │   │   ├── health.go
│   │   │   └── health_test.go
│   │   ├── ready
│   │   │   ├── ready.go
│   │   │   └── ready_test.go
│   │   └── settings
│   │       ├── loglevel
│   │       │   ├── loglevel.go
│   │       │   └── loglevel_test.go
│   │       ├── settings.go
│   │       └── settings_test.go
│   ├── apiresponse
│   │   ├── apiresponse.go
│   │   └── apiresponse_test.go
│   ├── env
│   │   ├── env.go
│   │   └── env_test.go
│   ├── health
│   │   ├── health.go
│   │   └── health_test.go
│   ├── log
│   │   ├── log.go
│   │   └── log_test.go
│   ├── server
│   │   ├── server.go
│   │   └── server_test.go
│   └── testingtools
│       ├── httprequest.go
│       └── httprequest_test.go

This package contains logic to create a server with an API and Health endpoints for Kubernetes. The implementation doesn't really matter (if you want to see it, take a look at the link on the bottom of the page), the only thing to notice is that I am able to create a server with a healthcheck just by doing this:

func main() {
    // Executed when the server shuts down
    onClose := func() {
        log.Infof("Server %s is shutting down\n", config.Name)
    }

    // Array of services that satisfy .Ping() interface. Ping tells us if the
    // service is healthy or should be killed
    healthChecks := health.Checks{
        "dummyworker1": dummyservice.New(),
    }

    // Starting the microservice by giving it a logger instance, the
    // healthchecks, the port, and a hook to execute when closing
    srv := server.Create(log, healthChecks, 8000, onClose)
    srv.Start()
}

That's all I need to do to create a microservice!

One thing that is important to mention about these shared packages: if anything in this pkg directory changes, all microservices must be redeployed, since all of them will be using one thing or another from it. Keep this in mind for later.

Before we move on, we should also create a few scripts to test the packages.

I will start by creating a script to run additional images on Docker for the tests (such as a Redis instance), so I can just call it with with the image that I want and it will take care of starting/stopping it. This script will live inside the shared scripts directory:

# startcontainer.sh
NAME=${1}
RUN_PARAMS=${2}

function isNotRunning() {
  echo $(docker ps --format "{{.Names}}" | grep -E "^${NAME}$" | wc -l)
}

if [ $(isNotRunning) -eq 0 ]; then
  for CID in $(docker ps -a --format "{{.Names}}" | awk '$1 == "'${NAME}'" {print $1}'); do
    docker stop ${CID}
    docker rm ${CID}
  done
  CMD="docker run -d --name ${NAME} ${RUN_PARAMS}"
  echo "Starting container ${IMAGE} with command '${CMD}'"
  eval ${CMD}
fi

And now we can start populating our Makefile:

# Makefile
# Sets up containers needed for test
test-containers-setup:
    ./scripts/startcontainer.sh redis '-p 6379:6379 redis:4.0'

# Runs the tests on a CI environment. Use if you don't want to set up containers
test-ci:
    go clean --testcache ./pkg/...
    go test -parallel 5 -cover -coverprofile cover.out ./pkg/...

# Sets up the dependencies for tests and run them
test:
    make test-containers-setup
    go clean --testcache ./pkg/...
    go test -parallel 5 -cover -coverprofile cover.out ./pkg/...
    go tool cover -func cover.out

# Removing junk
clean:
    rm -f cover.out

Now we can just run make test to test the shared packages, or make test-ci to test it in the CircleCI environment where no set up is needed.

3- Creating the microservices

I will start by creating one microservice. It is simple, but still complex enough to be a good example.

├── apps
│   ├── app1
│   │   ├── deployments
│   │   │   ├── deployment.yml # Deployment YML for Kubernetes
│   │   │   └── Dockerfile # ...well, the Dockerfile!
│   │   ├── internal
│   │   │   ├── app.go # Main entrypoint
│   │   │   ├── config # App settings
│   │   │   │   ├── config.go
│   │   │   └── dummyservice # A fake service that will fail randomly
│   │   │       └── dummyservice.go
│   │   ├── Makefile # Recipes for cleaning, deploying, building, etc
│   │   └── scripts
│   │       └── deploy.sh # Run to deploy the microservice!

This is important: why would I put my microservice code inside the internal directory? That's because we want to make sure that other microservices will not import code from this one - if the code is shared, it should go inside the pkg directory! If we put our code in the internal directory, Go will prevent other apps from importing it.

Let's start by looking at the Go files:

app.go

// app.go
// This file will instantiate a logger, the config, and then start the server
package main

import (
    "github.com/hscasn/go-microservice/apps/app1/internal/config"
    "github.com/hscasn/go-microservice/apps/app1/internal/dummyservice"
    "github.com/hscasn/go-microservice/pkg/health"
    "github.com/hscasn/go-microservice/pkg/log"
    "github.com/hscasn/go-microservice/pkg/server"
)

func main() {
    config := config.New()
    log := log.New(config.Name, false)

    onClose := func() {
        log.Infof("Server %s is shutting down\n", config.Name)
    }

    s := dummyservice.New()

    healthChecks := health.Checks{
        "dummyworker1": s,
    }

    srv := server.New(log, healthChecks, 8000, onClose)
    srv.Start()
}

config.go

// config.go
// There is only one setting: the name of the microservice,
// which I called a "Framework"
package config

import (
    "github.com/hscasn/go-microservice/pkg/env"
)

// Framework is the top-level configuration struct
type Framework struct {
    Name string
}

// New will recover the environment settings and parse them into a struct
func New() Framework {
    return Framework{
        Name: env.String("FRAMEWORK_NAME"),
    }
}

dummyservice.go

// dummyservice.go
// This is just a mocked servie that will fail the healthcheck randomly
// and cause Kubernetes to restart the pod
package dummyservice

import (
    "math/rand"
)

// Worker represents a dummy service that can be pinged
type Worker struct{}

// Ping will give the status of this dummy worker. It will
// fail randomly just for demo purposes
func (s *Worker) Ping() bool {
    r := rand.Int31() % 10
    return r > 3
}

// New creates a dummy worker
func New() *Worker {
    return &Worker{}
}

That's it! All I need to do is run the app.go file with the FRAMEWORK_NAME environment variable.

4- Test/Deploy scripts for microservices

Deployment

First, let's create the Dockerfile for our Go microservice. This Dockerfile will have a 2-stage build: it will start with a golang container, install some compilation tools, pull the dependencies, and then compile; then, based on this first stage, we will create a new alpine image and just copy the compiled code to use as entrypoint. Our final image will be about 11Mb!

# Dockerfile
# Build stage
FROM golang:latest AS build-env
ARG GOROOTREPO
ARG GOAPPREPO
ENV GOPATH=/go
ADD . ${GOPATH}/src/${GOROOTREPO}
WORKDIR ${GOPATH}/src/${GOROOTREPO}
RUN go get ./...
WORKDIR ${GOPATH}/src/${GOAPPREPO}
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o start ./internal/app.go
WORKDIR /
RUN mv ${GOPATH}/src/${GOAPPREPO}/start ./start

# Final stage
FROM alpine 
WORKDIR /
RUN apk add curl && rm -rf /var/cache/apk
COPY --from=build-env /start /start
ENTRYPOINT ["/start"]
EXPOSE 8000

The build expects two arguments:

Now let's create our Kubernetes deployment YML. Our deployment will also have a dedicated Redis and Horizontal Autoscaler:

# deployment.yml
apiVersion: v1
kind: Service
metadata:
  name: {{frameworkName}}
spec:
  ports:
  - port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    app: {{frameworkName}}
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{frameworkName}}
spec:
  selector:
    matchLabels:
      app: {{frameworkName}}
  replicas: 4
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: {{frameworkName}}
    spec:
      containers:
      - image: {{dockerReg}}/{{frameworkName}}:{{imageVersion}}
        name: {{frameworkName}}
        env:
        - name: FRAMEWORK_NAME
          value: "{{frameworkName}}"
        ports:
        - containerPort: 8000
          name: {{frameworkName}}
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: {{frameworkName}}
spec:
  minReplicas: {{minReplicas}}
  maxReplicas: {{maxReplicas}}
  targetCPUUtilizationPercentage: 80
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: {{frameworkName}}

As you can see, this is not a complete deployment YML: it has some placeholders for variables such as the framework name, number of replicas for the autoscaler, etc. We will populate these placeholders in our deployment script.

Docker? Check. Kubernetes? Check. Now we just need a deployment script:

#!/usr/bin/env bash
# deploy.sh

set -e

# Here we get the commit version. We will use it to tag the Docker image
COMMIT_VERSION=$(git rev-parse HEAD | cut -c1-7)

# Just getting some paths and names
SCRIPTPATH=$(dirname $(readlink -f ${0}))
APPPATH=$(eval "cd ${SCRIPTPATH}/..; pwd")
APPNAME=$(basename $(eval "cd ${APPPATH}; pwd"))
APPREPO=${APPPATH#*go/src/}
ROOTPATH=$(eval "cd ${APPPATH}/../../; pwd")
ROOTREPO=${ROOTPATH#*go/src/}

# Nothing special here. I am sourcing a config file that has
# my docker username (DOCKER_REG)
. ${ROOTPATH}/scripts/config.sh

# Env vars that will be used to populate the Kubernetes deployment:
FRAMEWORK_NAME=${APPNAME}
MIN_REPLICAS=1
MAX_REPLICAS=10
# /Env vars

echo "================================================================================"
echo "  Framework:       ${FRAMEWORK_NAME}"
echo "  Docker register: ${DOCKER_REG}"
echo "  Repository:      ${ROOTREPO}"
echo "  App repository:  ${APPREPO}"
echo "  Commit version:  ${COMMIT_VERSION}"
echo "  Min replicas:    ${MIN_REPLICAS}"
echo "  Max replicas:    ${MAX_REPLICAS}"
echo "================================================================================"
echo "Beginning deploy to cluster(s)."

# Building and pushing image
echo "Building image ${FRAMEWORK_NAME}:${COMMIT_VERSION}"
docker build 
    -f deployments/Dockerfile 
    --build-arg GOROOTREPO="${ROOTREPO}" 
    --build-arg GOAPPREPO="${APPREPO}" 
    -t ${DOCKER_REG}/${FRAMEWORK_NAME}:${COMMIT_VERSION} ${ROOTPATH}

echo "Pushing image ${FRAMEWORK_NAME}:${COMMIT_VERSION} to Docker Hub"
docker push ${DOCKER_REG}/${FRAMEWORK_NAME}:${COMMIT_VERSION}

# Build k8s objects YAML file from template
echo "Building Kubernetes objects YAML file for development from template"
cat ./deployments/deployment.yml 
        | sed s/{{dockerReg}}/${DOCKER_REG}/g 
        | sed s/{{imageVersion}}/${COMMIT_VERSION}/g 
        | sed s/{{frameworkName}}/${FRAMEWORK_NAME}/g 
        | sed s/{{minReplicas}}/${MIN_REPLICAS}/g 
        | sed s/{{maxReplicas}}/${MAX_REPLICAS}/g 
        > ./deployments/_k8s_objects.yml

echo "Deploying to Kubernetes cluster(s)"
kubectl apply -f ./deployments/_k8s_objects.yml

Whenever I run this file, my image gets created and my deployment is applied on Kubernetes!

Testing

Now it's time to create the scripts for testing the microservice. I will start by creating an .env file with the settings to run the tests. This file will live inside the scripts directory:

#.env
FRAMEWORK_NAME=app1

And now we are ready to create the Makefile for the microservice:

# Makefile
# Sourcing the environment variables
envfile = ./scripts/.env
include $(envfile)
export $(shell sed 's/=.*//' $(envfile))

# Sets up containers needed for test
test-containers-setup:
    ../../scripts/startcontainer.sh redis '-p 6379:6379 redis:4.0'

# Runs the tests on a CI environment. Use if you don't want to set up containers
test-ci:
    go clean --testcache ./...
    go test -parallel 5 -cover -coverprofile cover.out ./...

# Sets up the dependencies for tests and run them
test:
    make test-containers-setup
    go clean --testcache ./...
    go test -parallel 5 -cover -coverprofile cover.out ./...
    go tool cover -func cover.out

# Run the application
run:
    go run internal/app.go

# Only build the application
build:
    go build -o start internal/app.go

# Create the container and deploy the application
deploy:
    ./scripts/deploy.sh

# Removing junk
clean:
    rm -f cover.out start deployments/_k8s_objects.yml

5- Making more microservices

After creating a copy of our first microservice, we have now two of them with test and deployment scripts. Here is what the full file tree is looking like:

.
├── apps
│   ├── app1
│   │   ├── deployments
│   │   │   ├── deployment.yml
│   │   │   └── Dockerfile
│   │   ├── internal
│   │   │   ├── app.go
│   │   │   ├── config
│   │   │   │   ├── config.g
│   │   │   └── dummyservice
│   │   │       └── dummyservice.go
│   │   ├── Makefile
│   │   └── scripts
│   │       └── deploy.sh
│   └── app2
│       ├── deployments
│       │   ├── deployment.yml
│       │   └── Dockerfile
│       ├── internal
│       │   ├── app.go
│       │   ├── config
│       │   │   ├── config.go
│       │   └── dummyservice
│       │       └── dummyservice.go
│       ├── Makefile
│       └── scripts
│           └── deploy.sh
├── Makefile
├── pkg
│   ├── api
│   │   ├── api.go
│   │   ├── api_test.go
│   │   ├── health
│   │   │   ├── health.go
│   │   │   └── health_test.go
│   │   ├── ready
│   │   │   ├── ready.go
│   │   │   └── ready_test.go
│   │   └── settings
│   │       ├── loglevel
│   │       │   ├── loglevel.go
│   │       │   └── loglevel_test.go
│   │       ├── settings.go
│   │       └── settings_test.go
│   ├── apiresponse
│   │   ├── apiresponse.go
│   │   └── apiresponse_test.go
│   ├── env
│   │   ├── env.go
│   │   └── env_test.go
│   ├── health
│   │   ├── health.go
│   │   └── health_test.go
│   ├── log
│   │   ├── log.go
│   │   └── log_test.go
│   ├── server
│   │   ├── server.go
│   │   └── server_test.go
│   └── testingtools
│       ├── httprequest.go
│       └── httprequest_test.go
└── scripts
    ├── config.sh
    └── startcontainer.sh

In both app1 and app2 I can run make test to set up the environment and run the test suite, make test-ci to just run the test suite without setting up the environment (like in CircleCI), and make deploy to build and deploy it to my Kubernetes cluster. Now it's time to tie everything together!

6- Scripts to test and deploy only the microservices we need

Now we must create scripts that will go through all microservices and execute commands. Some of these commands must only be executed on microservices that were changed (like build and deployment).

Let's start with a simple script: one that will go through all microservices and run make clean. This we can safely run on every microservice:

# cleanapps.sh
ROOTPATH="$(dirname $(readlink -f ${0}))/.."

for APP in $(ls apps); do
    APPPATH="${ROOTPATH}/apps/${APP}"
    if [ ! -d ${APPPATH} ]; then
        continue
    fi
    cd ${APPPATH}
    make clean
    cd ${ROOTPATH}
done

Next step: let's create a script that will run the tests on every microservice. If we pass the "ci" argument to this script, we will run make test-ci, otherwise we will run make test.

# testapps.sh
ROOTPATH="$(dirname $(readlink -f ${0}))/.."
CI=0
if [ "${1:='-'}" = "ci" ]; then
    CI=1
fi

for APP in $(ls apps); do
    APPPATH="${ROOTPATH}/apps/${APP}"
    if [ ! -d ${APPPATH} ]; then
        continue
    fi
    cd ${APPPATH}
    if [ ${CI} -gt 0 ]; then
        make test-ci
    else
        make test
    fi
    TESTRESULT=$?
    cd ${ROOTPATH}
    if [ ! ${TESTRESULT} -eq 0 ]; then
        (>&2 echo "Tests failed for application ${APPPATH}")
        exit ${TESTRESULT}
    fi
done

Finally, let's create the deployment script. This script must be smarter than the previous one, so we will only build and deploy the microservices that were actually changed in the commit. Here is the idea:

  1. Get the list of all microservices available
  2. If we changed anything in the shared code (the pkg directory), we redeploy all of them
  3. If we did not change the shared code, we check the microservices changed in the last commit and deploy them
# deploychangedapps.sh
ROOTPATH="$(dirname $(readlink -f ${0}))/.."

# Apps (microservices) and shared code changed. These lines look complicated, but
# all they are doing is listing all the changes in the previous commit,
# looking at the directories where they happened, and making a list
# of the microservices/packages changed
MODIFIED_APPS=($(git log --name-only --oneline -1 | sed 1d | grep -e '^apps/' | cut -d "/" -f2 | uniq))
MODIFIED_PKG=($(git log --name-only --oneline -1 | sed 1d | grep -e '^pkg/' | cut -d "/" -f2 | uniq))

# Here we are going through all available microservices. If we changed shared code or we changed it
# on the last commit, we add it to the list of apps to deploy
APPS_TO_DEPLOY=()
for APP in $(ls apps); do
  if [ ${#MODIFIED_PKG[@]} -gt 0 ]; then
    APPS_TO_DEPLOY+=("${APP}")
        continue
  fi
  for MOD_APP in ${MODIFIED_APPS[@]}; do
    if [ ${MOD_APP} = ${APP} ]; then
      APPS_TO_DEPLOY+=("${APP}")
      continue
    fi
  done
done

echo "================================================================================"
echo "Modified apps (${#MODIFIED_APPS[@]}): $(echo ${MODIFIED_APPS[@]} | tr '
' ' ')"
echo "Modified packages (${#MODIFIED_PKG[@]}): $(echo ${MODIFIED_PKG[@]} | tr '
' ' ')"
echo "Apps to deploy (${#APPS_TO_DEPLOY[@]}): $(echo ${APPS_TO_DEPLOY[@]} | tr '
' ' ')"
echo "================================================================================"

# Going through the apps to deploy and running "make deploy"
for APP in ${APPS_TO_DEPLOY[@]}; do
  APPPATH="${ROOTPATH}/apps/${APP}"
  cd ${APPPATH}
  make deploy
  cd ${ROOTPATH}
done

And now we just have to modify the Makefile we made to run the testapp.sh script, the cleanapps.sh script, and include a deploy step that runs the deploychangedapps.sh:

# Makefile
# Sets up containers needed for test
test-containers-setup:
  ./scripts/startcontainer.sh redis '-p 6379:6379 redis:4.0'

# Runs the tests on a CI environment. Use if you don't want to set up containers
test-ci:
  go clean --testcache ./pkg/...
  go test -parallel 5 -cover -coverprofile cover.out ./pkg/...
  ./scripts/testapps.sh ci

# Sets up the dependencies for tests and run them
test:
  make test-containers-setup
  go clean --testcache ./pkg/...
  go test -parallel 5 -cover -coverprofile cover.out ./pkg/...
  ./scripts/testapps.sh
  go tool cover -func cover.out

# Create the containers and deploy the apps modified on the last commit
deploy:
  ./scripts/deploychangedapps.sh

# Removing junk
clean:
  rm -f cover.out
  ./scripts/cleanapps.sh

7- Config for CircleCI

We already got most of the work done. We just need to set up CircleCI to call our make recipes. Here is my config.yml for CircleCI:

version: 2
jobs:

  test:
    working_directory: /go/src/github.com/hscasn/go-microservice
    docker:
      - image: circleci/golang:1.11
      - image: redis:4.0
    steps:
      - checkout
      - run: go get -v ./...
      - run: make test-ci

  deploy:
    working_directory: /go/src/github.com/hscasn/go-microservice
    docker:
      - image: circleci/golang:1.11
    steps:
      - checkout
      - run:
          name: Get dependencies
          command: go get -v ./...
      - run:
          name: Deploy
          command: make deploy

workflows:
  version: 2
  build-deploy:
    jobs:
      - test
      - deploy:
          requires:
              - test
          filters:
            branches:
              only:
                - master

Here I have a workflow that will first test our code by running the test job, and then if we are on the master branch, we will run the deploy job!

One thing that is important to notice there: of course CircleCI does not have access to the Kubernetes cluster by default, so the make deploy script will not work. How the access is given to CircleCI will depend on your case.

If we wanted to deploy on GCP, for example, we would first need to authenticate, setup the Kubernetes context, point to the right context and namespace, and then run the make deploy script. Since this is too off-topic, I will leave these details out of this post.

…and that's it! You can find the end result here.