Pipeline vom Commit bis zur Docker-Testinstanz

Pipelines vom Commit bis zur Docker-Testinstanz

In 5 Minuten vom einem Entwickler-Commit hin zu einer Testinstanz oder zu einem kompletten Deployment: Das ermöglicht zügiges Testen und schnelle Deployment-Zyklen. Möglich wird dies bspw. mit "Docker Cloud". Allerdings ist "Docker Cloud" darauf ausgelegt, mit öffentlichen Repositories und öffentlich erreichbaren Testsystemen zu arbeiten. Ist dies nicht der Fall, so kann man eine solche Pipeline mit einer Kombination von Open-Source-Tools erreichen:
Mit einer Kombination aus Version Control System, Jenkins, Docker und nginx sowie einigen kleinen Netzwerkkniffen ist es möglich, dass ein Entwickler einen Deployment-Prozess definieren kann und dieser automatisch bei jedem Commit durchlaufen wird, ohne dass ein Administrator oder Netzwerkverwalter eingreifen muss.
Ein Testsystem oder auch ein externe Cloud-Server wird per Container-Image automatisch aktualisiert; alle Änderungen sind stets nachvollziehbar.

Transcript

1. www.onwerk.de Pipeline vom Commit bis zur Docker-Testinstanz Jens Doose, Onwerk GmbH DevOps Rhein Neckar Meetup 23.01.2017 @onwerk OnwerkGmbH Onwerk
2. www.onwerk.de The background Onwerk is a software agency developing individual software multiple customers multiple projects …and of course lots of web projects
3. www.onwerk.de Our goal Fast and easy setup of test systems Same behaviour on every target system consistent and reproducible deployment Project lead can test every new completed feature… …before customer can test every new completed feature Developer should be able to define software environment Docker Cloud not usable due to NDAs
4. www.onwerk.de Solution Docker images for software packaging Set of open source tools for environment and network infrastructure dorie-tools for automated creation of images and deploying  supporting developer
5. www.onwerk.de Private Docker registry Need: Storage for Docker images Problem: Unable to host Docker images in Docker Hub due to NDAs Solution: Using private Docker registry, centralized private repository for Docker images Push, pull like GitHub Running as Docker Container – what else?
6. www.onwerk.de The pipeline in a nutshell Testserver Jenkins-Server jobeins •Developer pushes code to VCS •VCS triggers Jenkins build •Jenkins builds project, runs tests SSH: •copies files •(re)starts container Image loaded, container started Docker Registry Image in registry •builds image •push to registry BAM: Container / web project is accessible for testing
7. www.onwerk.de Developer Step 1/2: Developing Developer writes Dockerfile for deploying software / service …and writes docker-compose file for defining environment (db, …) docker-compose file does not contain the image name / tag!
8. www.onwerk.de Developer Step 2/2: Set up CI / CD DockerBuild.sh: 1) builds Docker image with tag: 20170121_build12_develop 2) pushes image to the registry 3) creates docker-compose.override.yml file with image tag Use „dorie-tools“ in a new build step to do the magic
9. www.onwerk.de Developer Step 2/2: Set up CI / CD sample docker-compose.override.yml created by dorie-tools, used to explicit specify image and tag # # Created by DockerBuild.sh, Dorie Tools revision 45 # version: '2' services: servicenamedemojenkins: image: dockerregistry/imagename:20170122_build17_develop
10. www.onwerk.de Developer Step 2/2: Set up CI / CD DockerPullRun.sh: 1) copies docker-compose.override.yml and other files to server 2) and (re)starts container on testserver Use „dorie-tools“ in a new build step to do the magic Result: service is up and running
11. www.onwerk.de Behind the scene: nginx proxy 1/2 Problem: Web services listening on a port. Multiple services on same machine using same port  clash Solution: Using proxy nginx, handle all requests, choosing upstream server based on FQDN (URL) used for request: * myserviceone.testserver.onwerk.local.de * anotherservicetwo.testserver.onwerk.local.de Sounds good? Yes, but…
12. www.onwerk.de Behind the scene: nginx proxy 2/2 …but: nginx usually requires manual configuration of the virtual hosts Solution: using „nginx-proxy“ by Jason Wilder: Automatic reconfiguring nginx based on container start/stop Listening for special setting VIRTUAL_HOST in docker-compose.yml  Configuration of virtual host name Running as Docker Container – what else? Result: Developer can choose port for web service, automatic reconfiguration of nginx proxy
13. www.onwerk.de Behind the scene: DNS Problem: DNS name defined by developer in docker-compose-file must be known by DNS server Sample: http://demojenkinstwo.testserver.onwerk.local.de Solution: Using wildcard entry on DNS server *.testserver.onwerk.local.de
14. www.onwerk.de Behind the scene: Virtual Host List Problem: What was the DNS name of THAT web project again? Solution: using „docker-vhosts“, that displays a list of all virtual hosts of an nginx-proxy instance. Reconfiguring on container start/stop Running as Docker Container – what else?
15. www.onwerk.de The big picture Testserver Jenkins-Server jobeins •Developer pushes code to VCS •VCS triggers Jenkins build •Jenkins builds project, runs tests nginx-proxy with automatic config •nginx-proxy auto configuration SSH: •copies files + docker-override •(re)starts container with docker-override Image loaded, container started nginx chooses target based on name http://jobeins.herakles.dewerk.local.de with wildcard DNS entry: 192.168.1.18 Tester’s machine Docker Registry Image in registry •builds image + docker-override file •push to registry
16. www.onwerk.de ...and for external servers? Same procedure for developer Working on „default“/„master“ branch Automatic transfer of image to external server (dorie-tools are helping) Manual nginx configuration for security No wildcard DNS entry for security
17. www.onwerk.de ...and for production servers? Same procedure for developer Using the images built by Jenkins on „default“/„master“ branch Automatic or manual transfer of image to external server (dorie-tools are helping) Automatic or manual start/stop of containers Manual configuration of web server (nginx/Apache)
18. www.onwerk.de The benefits Only one additional Jenkins Build step / two shell script calls Continuous deployment for free Deployment first: right from the start Docker: all requirements in one package Same procedure for development and production deployment Testable within minutes
19. www.onwerk.de Demo
20. www.onwerk.de Questions? Follow us: Read about it: http://www.onwerk.de/2016/11/05/with-docker-in-5-minutes-from- developer-to-test-server/ @onwerk OnwerkGmbH Onwerk
21. www.onwerk.de Docker Meetup Mannheim Want to hear more about Docker? See you at the Docker Mannheim Meetup https://www.meetup.com/de-DE/Docker-Mannheim/ 1st meetup: 22.03.2017
22. www.onwerk.de Addendum: Links Private Docker registry: https://docs.docker.com/registry/deploying/ Docker Registry Browser: https://github.com/kwk/docker-registry- frontend hg flow: http://nvie.com/posts/a-successful-git-branching-model/ or https://andy.mehalick.com/2011/12/24/an-introduction-to-hgflow/ nginx-proxy https://github.com/jwilder/nginx-proxy docker-vhosts: https://github.com/texthtml/docker-vhosts dorie-tools: https://github.com/OnwerkGmbH/dorie-tools