Rapid Reproducible Builds

Rapid Reproducible Builds

I'd like to talk to everyone about someone we often neglect. That someone keeps our code flowing into production, runs all those automated tests, does things that we might not even remember to deploy to all our environments.

That someone is our build servers.

They tirelessly work to get our changes to our end users and we do neglect them. We don't treat our build servers like production servers, we hack them, we play with them, we mess up the environments. We may have several agents with different runtimes and gems and patch levels and who knows what.

We can do better.

In this talk I'd like to take you through something I've been working on over the last couple of months: Containerising your build. The tool I'll be using for this is Docker, but it's achievable with any VM or container technology. I'll take you through going from a working build on local to building local inside a container and then to moving that build to a build server, explaining what how and why as I go.

Ebbf05aa4a4b254779f6d95cb4811808?s=128

Dewald Viljoen

March 11, 2016
Tweet

Transcript

  1. R A P I D REPRODUCIBLE BUILDS treating everything like

    production Dewald Viljoen @dewald_v
  2. who am i? 2

  3. HOW'S THIS TALK GOING TO WORK and what are we

    actually talking about? 3
  4. "a server is a server is a server, there really

    are no test servers, build servers, dev servers and production servers... and spoons of course" - @dewald_v 4
  5. DOCKER IN 5 MINUTES a quick and dirty look into

    images, containers and all that lies below 5
  6. DOCKER IN 5 MINUTES - IMAGES 6 NGINX IMAGE UBUNTU

    14.04 ADD NGINX ADD NGINX CONFIG Layer 3 Layer 2 Base (Layer 1)
  7. DOCKER IN 5 MINUTES - CONTAINERS 7 NGINX CONTAINER UBUNTU

    14.04 ADD NGINX ADD NGINX CONFIG Layer 3 Layer 2 Base (Layer 1) CONTAINER R/W FILESYSTEM Layer 4 (Only in the Container)
  8. NGINX-1 CONTAINER R/W NGINX-2 CONTAINER R/W NGINX-3 CONTAINER R/W NGINX

    IMAGE DOCKER IN 5 MINUTES - CONTAINERS AND LAYERS 8
  9. DOCKER IN 5 MINUTES - CONTAINER VS VM 9 Image

    credit: Docker Inc.
  10. what can we use containers for? 10

  11. WORKSTATIONS AND BUILD AGENTS the darkest corners of the mind

    11
  12. what's on a workstation? 12

  13. A TYPICAL WORKSTATION 13 WORKSTATION

  14. A TYPICAL WORKSTATION 14 WORKSTATION Basically everything and anything you

    would need to work on your stack
  15. what's on a build agent? 15

  16. A TYPICAL BUILD AGENT 16 (SUPER) BUILD AGENT

  17. A TYPICAL BUILD AGENT 17 (SUPER) BUILD AGENT Basically everything

    and anything you would need to build your app
  18. what's the alternative? 18

  19. DOCKER-POWERED BUILDS applying images and containers to build pipelines 19

  20. NODEJS BUILD IMAGE THE BUILD IMAGE - NODE + GRUNT

    20 UBUNTU 14.04 ADD NODE 4.2.4 ADD GRUNT ADD PACKAGE.JSON & NPM INSTALL Base Image Build Runtime Build Config & Dependencies Build Tool
  21. MAVEN BUILD IMAGE THE BUILD IMAGE - JAVA + MAVEN

    21 UBUNTU 14.04 ADD JDK 1.8 ADD MAVEN 3.3.3 ADD MAVEN SETTINGS.XML Base Image Build Runtime Build Config Build Tool
  22. THE BUILD CONTAINER 22 MAVEN BUILD IMAGE + = SOURCE

    CODE BUILD ARTIFACT
  23. JAVA-APP-BUILD-1 CONTAINER R/W JAVA-APP-BUILD-3 CONTAINER R/W JAVA-APP-BUILD-2 CONTAINER R/W THE

    BUILD CONTAINER - MULTIPLIED 23 MAVEN BUILD IMAGE
  24. ENVIRONMENTS, ENVIRONMENTS, ENVIRONMENTS... 24 LOCAL MACHINE BUILD AGENT DOCKER REGISTRY

  25. THE BUILD CONTAINER - EVEN MORE MULTIPLIED 25 MAVEN 3

    MAVEN 2 NODEJS RUBY JAVA-APP- BUILD JAVA-APP- BUILD NODE-APP- BUILD RUBY-APP- BUILD BUILD AGENT ARTIFACT ARTIFACT ARTIFACT ARTIFACT
  26. OF PIPELINES AND CONTAINERS how do we use this then?

    26
  27. enterprises build all the things... 27

  28. THE SUPER BUILD AGENT 28 BUILD AGENT • Complex •

    Can be containerised but needs a lot of config • Holds all build tools and configuration for any type of build it can service • Results in high build-contention • Difficult to scale • Pushes artifacts to repositories MAVEN 3 MAVEN 2 JDK 1.5 JDK 1.6 JDK 1.7 RUBY PYTHON GRADLE
  29. THE MICRO BUILD AGENT 29 BUILD AGENT • Very simple

    • Can be easily containerised • Holds build tools and configuration for only the type of build it can service • Easier to scale • Can be managed with configuration as code • Pushes artifacts to repositories MAVEN 3 JDK 1.7
  30. THE CONTAINER BUILD AGENT 30 BUILD AGENT DOCKER • Very

    simple • Can be containerised it self (Docker in Docker) • Doesn't have any build dependencies inside, only docker • Pulls build containers from a registry • Simple to scale • Pushes artifacts to repositories • Basically a micro agent running only Docker
  31. so what does a containerised build pipeline look like? 31

  32. PIPELINING CONTAINERS 32 Application build directly on agent Failed due

    to environmental issues
  33. PIPELINING CONTAINERS 33 Build Container (Maven & JDK on Ubuntu)

    Application build in container
  34. PIPELINING CONTAINERS 34 Build Container (Maven & JDK on Ubuntu)

    Application Builds in containers
  35. THE GO SCRIPT not configuring the build on the server

    35
  36. what happens when the build server gets killed? 36

  37. BUILD CONFIGURATION 37 Backup your configuration Configuration as code Keep

    as little configuration on the server as possible Keep build configuration next to the app it builds
  38. THE GO SCRIPT 38 A Script for your build actions

    Knows the commands to build your app Keep it simple. Bash is a good start. Everything that works on the build server should work UNMODIFIED on the local environment. Build server should just call this script Use environment variables for environment changes
  39. THE GO SCRIPT 39 A Script for your build actions

    Knows the commands to build your app Keep it simple. Bash is a good start. Everything that works on the build server should work UNMODIFIED on the local environment. Build server should just call this script Use environment variables for environment changes
  40. PUTTING IT ALL TOGETHER demo time 40

  41. live demo time... (cross your fingers) 41

  42. DEWALD VILJOEN @dewald_v dewaldv@thoughtworks.com THANKS! QUESTIONS?