Make for Microservices Useful conventions for developer workflow and CI/CD 

Hi, I’m Bobby I’m on the Technology Fellow’s team at I dislike accidental complexity [email protected] @bobbycalderwood https://github.com/bobby 

The Problem • Encountered on several recent projects, including https:// tinyurl.com/capital-one-cmdr • How to interactively develop one or more services which require several other services at runtime? • How to test such an application locally? • How to build, deploy services which use a variety of different build and deploy toolchains? • How can developers joining the project make sense of any of the above? 

Partial Solutions Pro Con Language build tools Run tests, manage library dependencies, build executable artifacts from my code Prolific, language-specific, plugins rather than .sh for extension Docker Consistent deployment artifact and runtime target Lang build → docker build, adds another layer, poor dev affordances Docker Compose dev-time service orchestration, chaos testing Bootstrapping support services prior to app start, provision/deploy to prod Pile of scripts Cross-platform/cross- language: .sh is geek lingua-franca deps mgmt, no shared conventions, imperative/ verbose README 

A Better Solution • Make! • It’s been around since 1976 • Many developers familiar with it, projects using it • Speaks .sh, so everyone can use/understand it • Acts as a meta-build tool to orchestrate the aforementioned 

Example Project • 3 µ-services • write_service (Clojure) • stream_processor (Java) • read_service (Java) • 3 supporting services • Kafka • Zookeeper (only used by Kafka) • PostgreSQL 

version: "2" services: write_service: build: ./clojure-leiningen-rest-service image: "419056267649.dkr.ecr.us-east-1.amazonaws.com/write_service" ports: - "3000:3000" stream_processor: build: ./java-dropwizard-rest-service image: "419056267649.dkr.ecr.us-east-1.amazonaws.com/stream_processor" ports: - "5005:5005" environment: JAVA_OPTS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" read_service: build: ./java-dropwizard-rest-service image: "419056267649.dkr.ecr.us-east-1.amazonaws.com/read_service" ports: - "8080:8080" - "5006:5005" environment: JAVA_OPTS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" postgres: image: postgres:9.6-alpine ports: - "5432:5432" zookeeper: image: wurstmeister/zookeeper:3.4.6 kafka: image: wurstmeister/kafka:0.10.2.0 environment: KAFKA_ADVERTISED_PORT: 9092 KAFKA_ADVERTISED_HOST_NAME: kafka KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 ports: - "9092:9092" 

Dockerfile (Java services) FROM java:8-alpine MAINTAINER Bobby Calderwood ENV JAVA_OPTS="" RUN mkdir -p /opt ADD target/java-dropwizard-rest-service-*.jar /opt/app.jar EXPOSE 8080 CMD java ${JAVA_OPTS} -jar /opt/app.jar 

Dockerfile (Clojure service) FROM java:8-alpine MAINTAINER Bobby Calderwood RUN mkdir -p /opt ADD target/clojure-leiningen-rest-service-*-standalone.jar /opt/app.jar EXPOSE 8080 CMD ["java", "-jar", "/opt/app.jar"] 

Workflow Targets • Language artifact targets e.g. example/target/*.jar • check • run • stop • distclean • logs • Service interaction e.g. psql, kafka-console-consumer 

# Clojure project build via Leiningen clojure-leiningen-rest-service/target/clojure-leiningen-rest-service-*-standalone.jar: cd clojure-leiningen-rest-service && lein uberjar # Java project build via Maven java-dropwizard-rest-service/target/java-dropwizard-rest-service-*.jar: cd java-dropwizard-rest-service && mvn package # Java project build via Maven java-dropwizard-stream-processor/target/java-dropwizard-stream-processor-*.jar: cd java-dropwizard-stream-processor && mvn package # Run all tests .PHONY: check check: cd clojure-leiningen-rest-service && lein test cd java-dropwizard-rest-service && mvn test cd java-dropwizard-stream-processor && mvn test 

# Run, bootstrap, migrate, and then display status of all services via docker- compose .PHONY: run run: database-migrate # TEMPORARY HAND-WAVING!!! docker-compose --project-name=makefile_microservices ps # Stop the services (but preserve the images and state) .PHONY: stop stop: docker-compose --project-name=makefile_microservices stop # Reset all docker/docker-compose local state .PHONY: distclean distclean: clean -docker-compose --project-name=makefile_microservices rm -f -v -docker network rm makefile_microservices 

# Tail the docker-compose logs .PHONY: logs logs: docker-compose --project-name=makefile_microservices logs $(LOG_OPTIONS) # Run psql in a docker container, connected to the postgres service .PHONY: psql psql: docker run --network makefile_microservices --rm -it --entrypoint psql $ (POSTGRES_IMAGE) -h postgres -U postgres -d $(DATABASE_NAME) $(args) # Tail KAFKA_TOPIC ("test" by default) via kafka-console-consumer in a docker container .PHONY: kafka-console-consumer kafka-console-consumer: docker run --network makefile_microservices --rm -it --entrypoint /opt/kafka/ bin/kafka-console-consumer.sh $(KAFKA_IMAGE) --bootstrap-server kafka:9092 -- topic $(KAFKA_TOPIC) $(args) 

Utility/Orchestration Targets • _network • _run • _sleep • database-bootstrap, database-migrate 

# Create docker external network used by non-compose-managed containers to connect to compose-managed services .PHONY: _network _network: -docker network create makefile_microservices # Run all services via docker-compose .PHONY: _run _run: _network docker-compose --project-name=makefile_microservices up -d # Run all services via docker-compose .PHONY: _sleep _sleep: sleep $(SLEEP_SECONDS) # Create postgres database via psql .PHONY: database-bootstrap database-bootstrap: all _run _sleep -docker run --network makefile_microservices --rm -it --entrypoint createdb $ (POSTGRES_IMAGE) -h postgres -U postgres $(DATABASE_NAME) # Migrate postgres database via psql .PHONY: database-migrate database-migrate: database-bootstrap -docker run --network makefile_microservices --rm -i --entrypoint psql $ (POSTGRES_IMAGE) -h postgres -U postgres -d $(DATABASE_NAME) < $ (DATABASE_MIGRATION_PATH) 

REPL-driven Development # Stops the write_service, and launches a Clojure REPL instead .PHONY: repl repl: run docker-compose stop write_service cd clojure-leiningen-rest-service && lein repl :headless 

Attaching Debuggers read_service: build: ./java-dropwizard-rest-service image: "419056267649.dkr.ecr.us-east-1.amazonaws.com/read_service" ports: - "8080:8080" - "5005:5005" environment: JAVA_OPTS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" 

Workflow Demo 

CI/CD Targets • all (default) • clean • provision • push • deploy 

default: all # Build all docker images .PHONY: all all: clojure-leiningen-rest-service/target/clojure-leiningen-rest-service-*-standalone.jar java-dropwizard-rest-service/target/java-dropwizard-rest-service-*.jar java-dropwizard- stream-processor/target/java-dropwizard-stream-processor-*.jar docker-compose --project-name=makefile_microservices build # Delete all built artifacts, both language builds and docker images .PHONY: clean clean: -docker-compose --project-name=makefile_microservices down --rmi local -cd clojure-leiningen-rest-service && lein clean -cd java-dropwizard-rest-service && mvn clean -cd java-dropwizard-stream-processor && mvn clean # Build a Kubernetes cluster via kops .PHONY: provision provision: kops create cluster --cloud=aws --node-count=3 --node-size=t2.small --zones $(AWS_ZONES) --master-size=t2.large --master-zones $(AWS_ZONES) --name $(KOPS_CLUSTER_NAME) --state $ (KOPS_STATE_STORE) --ssh-public-key=~/.ssh/id_rsa.pub --yes # Push Docker images to ECR .PHONY: push push: all docker-compose --project-name=makefile_microservices push # Deploy the services to Kubernetes .PHONY: deploy deploy: push kompose up 

CI/CD Demo 

Opportunities for Improvement • Too much .PHONY, creates unnecessary re-work • Could possibly track state of environment via local files? • Makefile, docker-compose.yml, and service builds are somewhat coupled • Top-level file doing a lot, not using Make’s advanced features for loading service-specific rules/targets • Use Docker containers for kops and kompose calls 

Benefits • Consistent, unsurprising developer affordances • Declarative documentation, invocation • Imperative implementation • Consistent, unsurprising CI/CD integration points • Feel grayer around the temples… 

Questions? 

References • https://www.gnu.org/software/make/manual/ • http://david.rothlis.net/large-gnu-make/ • http://sites.e-advies.nl/nonrecursive-make.html • https://github.com/bobby/makefile_microservices