Make for Microservices

http://www.devfestdc.org/sessions/using-makemakefile-for-microservices-developer-workflow-and-cicd/

Developing and delivering microservices can be hard. A given service may depend on many other services at runtime, making it tough to get a tight feedback loop during development. Tools like docker-compose help, but they still leave tasks like database bootstrapping and service initialization as manual work for the user. And what about REPL-driven development, hot code reloading, attaching debuggers, and projects that include several languages and build systems? Similar challenges arise when it comes time to integrate, test, and deploy the service.

What if there was a simple, standard tool available on ~all *nix operating systems that could be used to correctly sequence and orchestrate inter-dependent development, build, and deployment tasks? There is, and it’s been available since 1976: Make!

In this talk, we’ll explore using Make as a meta-build/deploy system to solve the problems described above during the development workflow and within CI/CD pipelines.

Transcript

1. Make for Microservices Useful conventions for developer workflow and CI/CD
2. Hi, I’m Bobby I’m on the Technology Fellow’s team at I dislike accidental complexity bobby.calderwood@capitalone.com @bobbycalderwood https://github.com/bobby
3. The Problem • Encountered on several recent projects, including https:// tinyurl.com/capital-one-cmdr • How to interactively develop one or more services which require several other services at runtime? • How to test such an application locally? • How to build, deploy services which use a variety of different build and deploy toolchains? • How can developers joining the project make sense of any of the above?
4. Partial Solutions Pro Con Language build tools Run tests, manage library dependencies, build executable artifacts from my code Prolific, language-specific, plugins rather than .sh for extension Docker Consistent deployment artifact and runtime target Lang build → docker build, adds another layer, poor dev affordances Docker Compose dev-time service orchestration, chaos testing Bootstrapping support services prior to app start, provision/deploy to prod Pile of scripts Cross-platform/cross- language: .sh is geek lingua-franca deps mgmt, no shared conventions, imperative/ verbose README
5. A Better Solution • Make! • It’s been around since 1976 • Many developers familiar with it, projects using it • Speaks .sh, so everyone can use/understand it • Acts as a meta-build tool to orchestrate the aforementioned
6. Example Project • 3 µ-services • write_service (Clojure) • stream_processor (Java) • read_service (Java) • 3 supporting services • Kafka • Zookeeper (only used by Kafka) • PostgreSQL
7. version: "2" services: write_service: build: ./clojure-leiningen-rest-service image: "419056267649.dkr.ecr.us-east-1.amazonaws.com/write_service" ports: - "3000:3000" stream_processor: build: ./java-dropwizard-rest-service image: "419056267649.dkr.ecr.us-east-1.amazonaws.com/stream_processor" ports: - "5005:5005" environment: JAVA_OPTS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" read_service: build: ./java-dropwizard-rest-service image: "419056267649.dkr.ecr.us-east-1.amazonaws.com/read_service" ports: - "8080:8080" - "5006:5005" environment: JAVA_OPTS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" postgres: image: postgres:9.6-alpine ports: - "5432:5432" zookeeper: image: wurstmeister/zookeeper:3.4.6 kafka: image: wurstmeister/kafka:0.10.2.0 environment: KAFKA_ADVERTISED_PORT: 9092 KAFKA_ADVERTISED_HOST_NAME: kafka KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 ports: - "9092:9092"
8. Dockerfile (Java services) FROM java:8-alpine MAINTAINER Bobby Calderwood <bobby@example.com> ENV JAVA_OPTS="" RUN mkdir -p /opt ADD target/java-dropwizard-rest-service-*.jar /opt/app.jar EXPOSE 8080 CMD java ${JAVA_OPTS} -jar /opt/app.jar
9. Dockerfile (Clojure service) FROM java:8-alpine MAINTAINER Bobby Calderwood <bobby@example.com> RUN mkdir -p /opt ADD target/clojure-leiningen-rest-service-*-standalone.jar /opt/app.jar EXPOSE 8080 CMD ["java", "-jar", "/opt/app.jar"]
10. Workflow Targets • Language artifact targets e.g. example/target/*.jar • check • run • stop • distclean • logs • Service interaction e.g. psql, kafka-console-consumer
11. # Clojure project build via Leiningen clojure-leiningen-rest-service/target/clojure-leiningen-rest-service-*-standalone.jar: cd clojure-leiningen-rest-service && lein uberjar # Java project build via Maven java-dropwizard-rest-service/target/java-dropwizard-rest-service-*.jar: cd java-dropwizard-rest-service && mvn package # Java project build via Maven java-dropwizard-stream-processor/target/java-dropwizard-stream-processor-*.jar: cd java-dropwizard-stream-processor && mvn package # Run all tests .PHONY: check check: cd clojure-leiningen-rest-service && lein test cd java-dropwizard-rest-service && mvn test cd java-dropwizard-stream-processor && mvn test
12. # Run, bootstrap, migrate, and then display status of all services via docker- compose .PHONY: run run: database-migrate # TEMPORARY HAND-WAVING!!! docker-compose --project-name=makefile_microservices ps # Stop the services (but preserve the images and state) .PHONY: stop stop: docker-compose --project-name=makefile_microservices stop # Reset all docker/docker-compose local state .PHONY: distclean distclean: clean -docker-compose --project-name=makefile_microservices rm -f -v -docker network rm makefile_microservices
13. # Tail the docker-compose logs .PHONY: logs logs: docker-compose --project-name=makefile_microservices logs $(LOG_OPTIONS) # Run psql in a docker container, connected to the postgres service .PHONY: psql psql: docker run --network makefile_microservices --rm -it --entrypoint psql $ (POSTGRES_IMAGE) -h postgres -U postgres -d $(DATABASE_NAME) $(args) # Tail KAFKA_TOPIC ("test" by default) via kafka-console-consumer in a docker container .PHONY: kafka-console-consumer kafka-console-consumer: docker run --network makefile_microservices --rm -it --entrypoint /opt/kafka/ bin/kafka-console-consumer.sh $(KAFKA_IMAGE) --bootstrap-server kafka:9092 -- topic $(KAFKA_TOPIC) $(args)
14. Utility/Orchestration Targets • _network • _run • _sleep • database-bootstrap, database-migrate
15. # Create docker external network used by non-compose-managed containers to connect to compose-managed services .PHONY: _network _network: -docker network create makefile_microservices # Run all services via docker-compose .PHONY: _run _run: _network docker-compose --project-name=makefile_microservices up -d # Run all services via docker-compose .PHONY: _sleep _sleep: sleep $(SLEEP_SECONDS) # Create postgres database via psql .PHONY: database-bootstrap database-bootstrap: all _run _sleep -docker run --network makefile_microservices --rm -it --entrypoint createdb $ (POSTGRES_IMAGE) -h postgres -U postgres $(DATABASE_NAME) # Migrate postgres database via psql .PHONY: database-migrate database-migrate: database-bootstrap -docker run --network makefile_microservices --rm -i --entrypoint psql $ (POSTGRES_IMAGE) -h postgres -U postgres -d $(DATABASE_NAME) < $ (DATABASE_MIGRATION_PATH)
16. REPL-driven Development # Stops the write_service, and launches a Clojure REPL instead .PHONY: repl repl: run docker-compose stop write_service cd clojure-leiningen-rest-service && lein repl :headless
17. Attaching Debuggers read_service: build: ./java-dropwizard-rest-service image: "419056267649.dkr.ecr.us-east-1.amazonaws.com/read_service" ports: - "8080:8080" - "5005:5005" environment: JAVA_OPTS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
18. Workflow Demo
19. CI/CD Targets • all (default) • clean • provision • push • deploy
20. default: all # Build all docker images .PHONY: all all: clojure-leiningen-rest-service/target/clojure-leiningen-rest-service-*-standalone.jar java-dropwizard-rest-service/target/java-dropwizard-rest-service-*.jar java-dropwizard- stream-processor/target/java-dropwizard-stream-processor-*.jar docker-compose --project-name=makefile_microservices build # Delete all built artifacts, both language builds and docker images .PHONY: clean clean: -docker-compose --project-name=makefile_microservices down --rmi local -cd clojure-leiningen-rest-service && lein clean -cd java-dropwizard-rest-service && mvn clean -cd java-dropwizard-stream-processor && mvn clean # Build a Kubernetes cluster via kops .PHONY: provision provision: kops create cluster --cloud=aws --node-count=3 --node-size=t2.small --zones $(AWS_ZONES) --master-size=t2.large --master-zones $(AWS_ZONES) --name $(KOPS_CLUSTER_NAME) --state $ (KOPS_STATE_STORE) --ssh-public-key=~/.ssh/id_rsa.pub --yes # Push Docker images to ECR .PHONY: push push: all docker-compose --project-name=makefile_microservices push # Deploy the services to Kubernetes .PHONY: deploy deploy: push kompose up
21. CI/CD Demo
22. Opportunities for Improvement • Too much .PHONY, creates unnecessary re-work • Could possibly track state of environment via local files? • Makefile, docker-compose.yml, and service builds are somewhat coupled • Top-level file doing a lot, not using Make’s advanced features for loading service-specific rules/targets • Use Docker containers for kops and kompose calls
23. Benefits • Consistent, unsurprising developer affordances • Declarative documentation, invocation • Imperative implementation • Consistent, unsurprising CI/CD integration points • Feel grayer around the temples…
24. Questions?
25. References • https://www.gnu.org/software/make/manual/ • http://david.rothlis.net/large-gnu-make/ • http://sites.e-advies.nl/nonrecursive-make.html • https://github.com/bobby/makefile_microservices