blog

Continuous integration and deployment of microservices in a monorepo with Kubernetes and CircleCI

I have been reading a lot about DevOps; more specifically, continuous integration, continuous deployment, and project layout. I also heard a lot about monorepos for microservices - apparently they are much easier to manage than separate repositories for every microservice - but how can I join these three things together? This is what I am exploring, and I am going to share my results here. First, what exactly am I looking for? Well, these are my constraints: 1. The solution should be applicable to most languages, but it has to work for **Go** 1. I want to have both Continuous Integration and Deployment 1. I must be able to deploy on **Kubernetes** 1. Shared code must live in a dedicated directory, not inside one of the microservices 1. The CD script must only deploy microservices that were actually changed. I don't want to deploy all microservices all the time 1. The CI script must be able to test all microservices 1. The solution must work with **CircleCI** **I will leave a link to the final result at the end of this post.** Here is what we are going to do: 1. Come up with a simple directory structure, so we can know where to put the microservices and shared code 1. Create some shared, boilerplate code for the microservices and put it in the shared code package 1. Create a simple microservice using the shared code 1. Create test and deployment scripts for the microservice 1. Clone the microservice, just so we will have more than one 1. Create the scripts to test and deploy only the microservices we need 1. Create the configuration for CI/CD with CircleCI # 1- Directory structure I will start by making a basic directory structure. This project will be called `go-microservice` and will live inside my GOPATH (*~/go/src/github.com/hscasn/go-microservice*): ```bash . ├── apps # I will put my microservices here ├── Makefile # Instructions to build, clean, deploy, etc ├── pkg # Shared code goes here └── scripts # Shell scripts ``` Very simple structure, but it is all we need. Some people may prefer naming the `apps` directory `services` or something like that, which makes sense. However, if one of the microservices is an MVC app, you will probably have another `services` directory inside. I want to avoid confusion, so I'll call it `apps`. The microservices will be put directly inside that package, like this: ```bash ├── apps │ ├── app1 # Microservice 1 │ ├── app2 # Microservice 2 │ ... ``` And then every microservice can have their own **internal**, **scripts**, and so on. #2- Shared code I already have some some code that can be shared among other microservices, so I will put it inside the **pkg** directory. The idea behind this package is to group common utilities to reduce the amount of duplicated code. Here is the directory structure of **pkg** now: ```bash ├── pkg │ ├── api │ │ ├── api.go │ │ ├── api_test.go │ │ ├── health │ │ │ ├── health.go │ │ │ └── health_test.go │ │ ├── ready │ │ │ ├── ready.go │ │ │ └── ready_test.go │ │ └── settings │ │ ├── loglevel │ │ │ ├── loglevel.go │ │ │ └── loglevel_test.go │ │ ├── settings.go │ │ └── settings_test.go │ ├── apiresponse │ │ ├── apiresponse.go │ │ └── apiresponse_test.go │ ├── env │ │ ├── env.go │ │ └── env_test.go │ ├── health │ │ ├── health.go │ │ └── health_test.go │ ├── log │ │ ├── log.go │ │ └── log_test.go │ ├── server │ │ ├── server.go │ │ └── server_test.go │ └── testingtools │ ├── httprequest.go │ └── httprequest_test.go ``` This package contains logic to create a server with an API and Health endpoints for Kubernetes. The implementation doesn't really matter (if you want to see it, take a look at the link on the bottom of the page), the only thing to notice is that I am able to create a server with a healthcheck just by doing this: ```go func main() { // Executed when the server shuts down onClose := func() { log.Infof("Server %s is shutting down ", config.Name) } // Array of services that satisfy .Ping() interface. Ping tells us if the // service is healthy or should be killed healthChecks := health.Checks{ "dummyworker1": dummyservice.New(), } // Starting the microservice by giving it a logger instance, the // healthchecks, the port, and a hook to execute when closing srv := server.Create(log, healthChecks, 8000, onClose) srv.Start() } ``` That's all I need to do to create a microservice! One thing that is important to mention about these shared packages: if anything in this **pkg** directory changes, *all microservices must be redeployed*, since all of them will be using one thing or another from it. Keep this in mind for later. Before we move on, we should also create a few scripts to test the packages. I will start by creating a script to run additional images on Docker for the tests (such as a Redis instance), so I can just call it with with the image that I want and it will take care of starting/stopping it. This script will live inside the shared *scripts* directory: ```bash # startcontainer.sh NAME=${1} RUN_PARAMS=${2} function isNotRunning() { echo $(docker ps --format "{{.Names}}" | grep -E "^${NAME}$" | wc -l) } if [ $(isNotRunning) -eq 0 ]; then for CID in $(docker ps -a --format "{{.Names}}" | awk '$1 == "'${NAME}'" {print $1}'); do docker stop ${CID} docker rm ${CID} done CMD="docker run -d --name ${NAME} ${RUN_PARAMS}" echo "Starting container ${IMAGE} with command '${CMD}'" eval ${CMD} fi ``` And now we can start populating our Makefile: ```bash # Makefile # Sets up containers needed for test test-containers-setup: ./scripts/startcontainer.sh redis '-p 6379:6379 redis:4.0' # Runs the tests on a CI environment. Use if you don't want to set up containers test-ci: go clean --testcache ./pkg/... go test -parallel 5 -cover -coverprofile cover.out ./pkg/... # Sets up the dependencies for tests and run them test: make test-containers-setup go clean --testcache ./pkg/... go test -parallel 5 -cover -coverprofile cover.out ./pkg/... go tool cover -func cover.out # Removing junk clean: rm -f cover.out ``` Now we can just run `make test` to test the shared packages, or `make test-ci` to test it in the CircleCI environment where no set up is needed. # 3- Creating the microservices I will start by creating one microservice. It is simple, but still complex enough to be a good example. ```bash ├── apps │ ├── app1 │ │ ├── deployments │ │ │ ├── deployment.yml # Deployment YML for Kubernetes │ │ │ └── Dockerfile # ...well, the Dockerfile! │ │ ├── internal │ │ │ ├── app.go # Main entrypoint │ │ │ ├── config # App settings │ │ │ │ ├── config.go │ │ │ └── dummyservice # A fake service that will fail randomly │ │ │ └── dummyservice.go │ │ ├── Makefile # Recipes for cleaning, deploying, building, etc │ │ └── scripts │ │ └── deploy.sh # Run to deploy the microservice! ``` This is important: why would I put my microservice code inside the `internal` directory? That's because we want to make sure that other microservices will not import code from this one - if the code is shared, it should go inside the **pkg** directory! If we put our code in the `internal` directory, Go will prevent other apps from importing it. Let's start by looking at the Go files: ### app.go ```go // app.go // This file will instantiate a logger, the config, and then start the server package main import ( "github.com/hscasn/go-microservice/apps/app1/internal/config" "github.com/hscasn/go-microservice/apps/app1/internal/dummyservice" "github.com/hscasn/go-microservice/pkg/health" "github.com/hscasn/go-microservice/pkg/log" "github.com/hscasn/go-microservice/pkg/server" ) func main() { config := config.New() log := log.New(config.Name, false) onClose := func() { log.Infof("Server %s is shutting down ", config.Name) } s := dummyservice.New() healthChecks := health.Checks{ "dummyworker1": s, } srv := server.New(log, healthChecks, 8000, onClose) srv.Start() } ``` ### config.go ```go // config.go // There is only one setting: the name of the microservice, // which I called a "Framework" package config import ( "github.com/hscasn/go-microservice/pkg/env" ) // Framework is the top-level configuration struct type Framework struct { Name string } // New will recover the environment settings and parse them into a struct func New() Framework { return Framework{ Name: env.String("FRAMEWORK_NAME"), } } ``` ### dummyservice.go ```go // dummyservice.go // This is just a mocked servie that will fail the healthcheck randomly // and cause Kubernetes to restart the pod package dummyservice import ( "math/rand" ) // Worker represents a dummy service that can be pinged type Worker struct{} // Ping will give the status of this dummy worker. It will // fail randomly just for demo purposes func (s *Worker) Ping() bool { r := rand.Int31() % 10 return r > 3 } // New creates a dummy worker func New() *Worker { return &Worker{} } ``` That's it! All I need to do is run the app.go file with the *FRAMEWORK_NAME* environment variable. # 4- Test/Deploy scripts for microservices ### Deployment First, let's create the Dockerfile for our Go microservice. This Dockerfile will have a 2-stage build: it will start with a **golang** container, install some compilation tools, pull the dependencies, and then compile; then, based on this first stage, we will create a new **alpine** image and just copy the compiled code to use as entrypoint. Our final image will be about 11Mb! ``` # Dockerfile # Build stage FROM golang:latest AS build-env ARG GOROOTREPO ARG GOAPPREPO ENV GOPATH=/go ADD . ${GOPATH}/src/${GOROOTREPO} WORKDIR ${GOPATH}/src/${GOROOTREPO} RUN go get ./... WORKDIR ${GOPATH}/src/${GOAPPREPO} RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o start ./internal/app.go WORKDIR / RUN mv ${GOPATH}/src/${GOAPPREPO}/start ./start # Final stage FROM alpine WORKDIR / RUN apk add curl && rm -rf /var/cache/apk COPY --from=build-env /start /start ENTRYPOINT ["/start"] EXPOSE 8000 ``` The build expects two arguments: - **GOROOTREPO** has the path of our *repository* inside our GOPATH. For example: *github․com/hscasn/go-microservice* - **GOAPPREPO** has the path of our *microservice* inside our GOPATH. For example: *github․com/hscasn/go-microservice/apps/app1* Now let's create our Kubernetes deployment YML. Our deployment will also have a dedicated Redis and Horizontal Autoscaler: ```yaml # deployment.yml apiVersion: v1 kind: Service metadata: name: {{frameworkName}} spec: ports: - port: 8000 protocol: TCP targetPort: 8000 selector: app: {{frameworkName}} type: NodePort --- apiVersion: apps/v1 kind: Deployment metadata: name: {{frameworkName}} spec: selector: matchLabels: app: {{frameworkName}} replicas: 4 strategy: type: Recreate template: metadata: labels: app: {{frameworkName}} spec: containers: - image: {{dockerReg}}/{{frameworkName}}:{{imageVersion}} name: {{frameworkName}} env: - name: FRAMEWORK_NAME value: "{{frameworkName}}" ports: - containerPort: 8000 name: {{frameworkName}} --- apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: {{frameworkName}} spec: minReplicas: {{minReplicas}} maxReplicas: {{maxReplicas}} targetCPUUtilizationPercentage: 80 scaleTargetRef: apiVersion: apps/v1beta1 kind: Deployment name: {{frameworkName}} ``` As you can see, this is not a complete deployment YML: it has some placeholders for variables such as the framework name, number of replicas for the autoscaler, etc. We will populate these placeholders in our deployment script. Docker? Check. Kubernetes? Check. Now we just need a deployment script: ```bash #!/usr/bin/env bash # deploy.sh set -e # Here we get the commit version. We will use it to tag the Docker image COMMIT_VERSION=$(git rev-parse HEAD | cut -c1-7) # Just getting some paths and names SCRIPTPATH=$(dirname $(readlink -f ${0})) APPPATH=$(eval "cd ${SCRIPTPATH}/..; pwd") APPNAME=$(basename $(eval "cd ${APPPATH}; pwd")) APPREPO=${APPPATH#*go/src/} ROOTPATH=$(eval "cd ${APPPATH}/../../; pwd") ROOTREPO=${ROOTPATH#*go/src/} # Nothing special here. I am sourcing a config file that has # my docker username (DOCKER_REG) . ${ROOTPATH}/scripts/config.sh # Env vars that will be used to populate the Kubernetes deployment: FRAMEWORK_NAME=${APPNAME} MIN_REPLICAS=1 MAX_REPLICAS=10 # /Env vars echo "================================================================================" echo " Framework: ${FRAMEWORK_NAME}" echo " Docker register: ${DOCKER_REG}" echo " Repository: ${ROOTREPO}" echo " App repository: ${APPREPO}" echo " Commit version: ${COMMIT_VERSION}" echo " Min replicas: ${MIN_REPLICAS}" echo " Max replicas: ${MAX_REPLICAS}" echo "================================================================================" echo "Beginning deploy to cluster(s)." # Building and pushing image echo "Building image ${FRAMEWORK_NAME}:${COMMIT_VERSION}" docker build -f deployments/Dockerfile --build-arg GOROOTREPO="${ROOTREPO}" --build-arg GOAPPREPO="${APPREPO}" -t ${DOCKER_REG}/${FRAMEWORK_NAME}:${COMMIT_VERSION} ${ROOTPATH} echo "Pushing image ${FRAMEWORK_NAME}:${COMMIT_VERSION} to Docker Hub" docker push ${DOCKER_REG}/${FRAMEWORK_NAME}:${COMMIT_VERSION} # Build k8s objects YAML file from template echo "Building Kubernetes objects YAML file for development from template" cat ./deployments/deployment.yml | sed s/{{dockerReg}}/${DOCKER_REG}/g | sed s/{{imageVersion}}/${COMMIT_VERSION}/g | sed s/{{frameworkName}}/${FRAMEWORK_NAME}/g | sed s/{{minReplicas}}/${MIN_REPLICAS}/g | sed s/{{maxReplicas}}/${MAX_REPLICAS}/g > ./deployments/_k8s_objects.yml echo "Deploying to Kubernetes cluster(s)" kubectl apply -f ./deployments/_k8s_objects.yml ``` Whenever I run this file, my image gets created and my deployment is applied on Kubernetes! ### Testing Now it's time to create the scripts for testing the microservice. I will start by creating an **.env** file with the settings to run the tests. This file will live inside the *scripts* directory: ```bash #.env FRAMEWORK_NAME=app1 ``` And now we are ready to create the *Makefile* for the microservice: ```bash # Makefile # Sourcing the environment variables envfile = ./scripts/.env include $(envfile) export $(shell sed 's/=.*//' $(envfile)) # Sets up containers needed for test test-containers-setup: ../../scripts/startcontainer.sh redis '-p 6379:6379 redis:4.0' # Runs the tests on a CI environment. Use if you don't want to set up containers test-ci: go clean --testcache ./... go test -parallel 5 -cover -coverprofile cover.out ./... # Sets up the dependencies for tests and run them test: make test-containers-setup go clean --testcache ./... go test -parallel 5 -cover -coverprofile cover.out ./... go tool cover -func cover.out # Run the application run: go run internal/app.go # Only build the application build: go build -o start internal/app.go # Create the container and deploy the application deploy: ./scripts/deploy.sh # Removing junk clean: rm -f cover.out start deployments/_k8s_objects.yml ``` ### 5- Making more microservices After creating a copy of our first microservice, we have now two of them with test and deployment scripts. Here is what the full file tree is looking like: ```bash . ├── apps │ ├── app1 │ │ ├── deployments │ │ │ ├── deployment.yml │ │ │ └── Dockerfile │ │ ├── internal │ │ │ ├── app.go │ │ │ ├── config │ │ │ │ ├── config.g │ │ │ └── dummyservice │ │ │ └── dummyservice.go │ │ ├── Makefile │ │ └── scripts │ │ └── deploy.sh │ └── app2 │ ├── deployments │ │ ├── deployment.yml │ │ └── Dockerfile │ ├── internal │ │ ├── app.go │ │ ├── config │ │ │ ├── config.go │ │ └── dummyservice │ │ └── dummyservice.go │ ├── Makefile │ └── scripts │ └── deploy.sh ├── Makefile ├── pkg │ ├── api │ │ ├── api.go │ │ ├── api_test.go │ │ ├── health │ │ │ ├── health.go │ │ │ └── health_test.go │ │ ├── ready │ │ │ ├── ready.go │ │ │ └── ready_test.go │ │ └── settings │ │ ├── loglevel │ │ │ ├── loglevel.go │ │ │ └── loglevel_test.go │ │ ├── settings.go │ │ └── settings_test.go │ ├── apiresponse │ │ ├── apiresponse.go │ │ └── apiresponse_test.go │ ├── env │ │ ├── env.go │ │ └── env_test.go │ ├── health │ │ ├── health.go │ │ └── health_test.go │ ├── log │ │ ├── log.go │ │ └── log_test.go │ ├── server │ │ ├── server.go │ │ └── server_test.go │ └── testingtools │ ├── httprequest.go │ └── httprequest_test.go └── scripts ├── config.sh └── startcontainer.sh ``` In both *app1* and *app2* I can run `make test` to set up the environment and run the test suite, `make test-ci` to just run the test suite without setting up the environment (like in CircleCI), and `make deploy` to build and deploy it to my Kubernetes cluster. Now it's time to tie everything together! # 6- Scripts to test and deploy only the microservices we need Now we must create scripts that will go through all microservices and execute commands. Some of these commands must only be executed on microservices that were changed (like build and deployment). Let's start with a simple script: one that will go through all microservices and run `make clean`. This we can safely run on every microservice: ```bash # cleanapps.sh ROOTPATH="$(dirname $(readlink -f ${0}))/.." for APP in $(ls apps); do APPPATH="${ROOTPATH}/apps/${APP}" if [ ! -d ${APPPATH} ]; then continue fi cd ${APPPATH} make clean cd ${ROOTPATH} done ``` Next step: let's create a script that will run the tests on every microservice. If we pass the "ci" argument to this script, we will run `make test-ci`, otherwise we will run `make test`. ```bash # testapps.sh ROOTPATH="$(dirname $(readlink -f ${0}))/.." CI=0 if [ "${1:='-'}" = "ci" ]; then CI=1 fi for APP in $(ls apps); do APPPATH="${ROOTPATH}/apps/${APP}" if [ ! -d ${APPPATH} ]; then continue fi cd ${APPPATH} if [ ${CI} -gt 0 ]; then make test-ci else make test fi TESTRESULT=$? cd ${ROOTPATH} if [ ! ${TESTRESULT} -eq 0 ]; then (>&2 echo "Tests failed for application ${APPPATH}") exit ${TESTRESULT} fi done ``` Finally, let's create the deployment script. This script must be smarter than the previous one, so we will only build and deploy the microservices that were actually changed in the commit. Here is the idea: 1. Get the list of all microservices available 1. If we changed anything in the shared code (the **pkg** directory), we redeploy all of them 1. If we did not change the shared code, we check the microservices changed in the last commit and deploy them ```bash # deploychangedapps.sh ROOTPATH="$(dirname $(readlink -f ${0}))/.." # Apps (microservices) and shared code changed. These lines look complicated, but # all they are doing is listing all the changes in the previous commit, # looking at the directories where they happened, and making a list # of the microservices/packages changed MODIFIED_APPS=($(git log --name-only --oneline -1 | sed 1d | grep -e '^apps/' | cut -d "/" -f2 | uniq)) MODIFIED_PKG=($(git log --name-only --oneline -1 | sed 1d | grep -e '^pkg/' | cut -d "/" -f2 | uniq)) # Here we are going through all available microservices. If we changed shared code or we changed it # on the last commit, we add it to the list of apps to deploy APPS_TO_DEPLOY=() for APP in $(ls apps); do if [ ${#MODIFIED_PKG[@]} -gt 0 ]; then APPS_TO_DEPLOY+=("${APP}") continue fi for MOD_APP in ${MODIFIED_APPS[@]}; do if [ ${MOD_APP} = ${APP} ]; then APPS_TO_DEPLOY+=("${APP}") continue fi done done echo "================================================================================" echo "Modified apps (${#MODIFIED_APPS[@]}): $(echo ${MODIFIED_APPS[@]} | tr ' ' ' ')" echo "Modified packages (${#MODIFIED_PKG[@]}): $(echo ${MODIFIED_PKG[@]} | tr ' ' ' ')" echo "Apps to deploy (${#APPS_TO_DEPLOY[@]}): $(echo ${APPS_TO_DEPLOY[@]} | tr ' ' ' ')" echo "================================================================================" # Going through the apps to deploy and running "make deploy" for APP in ${APPS_TO_DEPLOY[@]}; do APPPATH="${ROOTPATH}/apps/${APP}" cd ${APPPATH} make deploy cd ${ROOTPATH} done ``` And now we just have to modify the Makefile we made to run the `testapp.sh` script, the `cleanapps.sh` script, and include a `deploy` step that runs the `deploychangedapps.sh`: ``` # Makefile # Sets up containers needed for test test-containers-setup: ./scripts/startcontainer.sh redis '-p 6379:6379 redis:4.0' # Runs the tests on a CI environment. Use if you don't want to set up containers test-ci: go clean --testcache ./pkg/... go test -parallel 5 -cover -coverprofile cover.out ./pkg/... ./scripts/testapps.sh ci # Sets up the dependencies for tests and run them test: make test-containers-setup go clean --testcache ./pkg/... go test -parallel 5 -cover -coverprofile cover.out ./pkg/... ./scripts/testapps.sh go tool cover -func cover.out # Create the containers and deploy the apps modified on the last commit deploy: ./scripts/deploychangedapps.sh # Removing junk clean: rm -f cover.out ./scripts/cleanapps.sh ``` # 7- Config for CircleCI We already got most of the work done. We just need to set up CircleCI to call our `make` recipes. Here is my `config.yml` for CircleCI: ```yaml version: 2 jobs: test: working_directory: /go/src/github.com/hscasn/go-microservice docker: - image: circleci/golang:1.11 - image: redis:4.0 steps: - checkout - run: go get -v ./... - run: make test-ci deploy: working_directory: /go/src/github.com/hscasn/go-microservice docker: - image: circleci/golang:1.11 steps: - checkout - run: name: Get dependencies command: go get -v ./... - run: name: Deploy command: make deploy workflows: version: 2 build-deploy: jobs: - test - deploy: requires: - test filters: branches: only: - master ``` Here I have a workflow that will first test our code by running the `test` job, and then if we are on the `master` branch, we will run the `deploy` job! One thing that is important to notice there: of course CircleCI does not have access to the Kubernetes cluster by default, so the `make deploy` script will not work. How the access is given to CircleCI will depend on your case. If we wanted to deploy on GCP, for example, we would first need to authenticate, setup the Kubernetes context, point to the right context and namespace, and then run the `make deploy` script. Since this is too off-topic, I will leave these details out of this post. ...and that's it! You can find the end result here.