Microplugins with Docker

Slide 1

Slide 1 text

Microplugins DOCKER ******************* WITH *******************

Slide 2

Slide 2 text

Freelance « DevOps » JULIEN VEY ERIC BELLEMON ********************************************* at bywan @julienvey @haklop

Slide 3

Slide 3 text

BY NOW, WE THINK YOU KNOW WHAT IS DOCKER WHAT ?

Slide 4

Slide 4 text

DOCKER HOW ? Patterns *******************************************

Slide 5

Slide 5 text

MICROSERVICES

Slide 6

Slide 6 text

A MICROSERVICE IS USUALLY A PERSISTANT SERVICE YOU CAN ACCESS VIA AN API

Slide 7

Slide 7 text

GO GO PYTHON RUBY NODE JAVA JAVA PHP ANY LANGUAGE

Slide 8

Slide 8 text

UBUNTU REDHAT DEBIAN CENTOS FEDORA FEDORA DEBIAN CENTOS ANY DISTRIBUTION

Slide 9

Slide 9 text

AWS GCE COREOS ANY WHERE

Slide 10

Slide 10 text

WHY CONTAINERS AND NOT VMS? CONTAINERS ARE LIGHTWEIGHT SPAWN FAST USE LAYERS

Slide 11

Slide 11 text

DOCKER Tips n’ Tricks ******************************************* HOW ?

Slide 12

Slide 12 text

DOCKER IN DOCKER

Slide 13

Slide 13 text

THE DOCKER MODEL DOCKER DAEMON UNIX SOCKET DOCKER REMOTE API DOCKER CLI creates an which exposes the that can be used by the

Slide 14

Slide 14 text

FIRST APPROACH

Slide 15

Slide 15 text

DOCKER IN DOCKER IN DOCKER …

Slide 16

Slide 16 text

docker run --privileged -t -i jpetazzo/dind SPECIAL DOCKER IMAGE

Slide 17

Slide 17 text

docker run --privileged -t -i jpetazzo/dind PRIVILEGED MODE

Slide 18

Slide 18 text

docker run --privileged -t -i jpetazzo/dind PRIVILEGED MODE

Slide 19

Slide 19 text

IT RAISES SOME PROBLEMS INIT SYSTEM DISK SPACE CUSTOM IMAGE

Slide 20

Slide 20 text

SECOND APPROACH

Slide 21

Slide 21 text

DOCKER IN DOCKER IN DOCKER … DOCKER HOST DOCKER SOCKET

Slide 22

Slide 22 text

docker run -v /var/run/docker.sock:/var/run/docker.sock [your image] DOCKER SOCKET AS A DOCKER VOLUME

Slide 23

Slide 23 text

docker run -v /var/run/docker.sock:/var/run/docker.sock [your image] DOCKER SOCKET AS A DOCKER VOLUME

Slide 24

Slide 24 text

NO PRIVILEGED MODE ONLY ONE DOCKER DAEMON (INSTEAD OF ONE PER CONTAINER)

Slide 25

Slide 25 text

What about ? MICRO PLUGINS *********************************************

Slide 26

Slide 26 text

MICROSERVICE: A SMALL SERVICE RUNNING IN A CONTAINER

Slide 27

Slide 27 text

MICROPLUGIN: A SMALL PLUGIN RUNNING IN A CONTAINER

Slide 28

Slide 28 text

What is a ? Plugin Architecture *********************************************

Slide 29

Slide 29 text

A STANDARD INTERFACE A LOADING MECHANISM A PACKAGING MECHANISM A DISTRIBUTION MECHANISM

Slide 30

Slide 30 text

VAGRANT ******************* WITH ******************* A PLUGIN ARCHITECTURE RUBY EXAMPLE

Slide 31

Slide 31 text

A STANDARD INTERFACE require 'vagrant' module VagrantPlugins module Openstack class Provider < Vagrant.plugin('2', :provider) RUBY CLASSES

Slide 32

Slide 32 text

A LOADING MECHANISM RUBY DYNAMIC LOADING SOURCES INSTALLED IN VAGRANT BUNDLE DIR

Slide 33

Slide 33 text

A PACKAGING MECHANISM RUBY GEM PACKAGE CODE AND DOCUMENTATION DESCRIBE DEPENDENCIES

Slide 34

Slide 34 text

A DISTRIBUTION MECHANISM RUBYGEMS.ORG

Slide 35

Slide 35 text

****************** AND ****************** PROS CONS

Slide 36

Slide 36 text

PLATFORM INDEPENDANT PACKAGING AND DISTRIBUTION ARE NOT SPECIFIC TO VAGRANT PROS

Slide 37

Slide 37 text

RUBY ONLY DEPENDENCY MANAGEMENT CAN BE HARD CONS

Slide 38

Slide 38 text

JENKINS ****************** WITH ****************** A PLUGIN ARCHITECTURE JAVA EXAMPLE

Slide 39

Slide 39 text

A STANDARD INTERFACE package jenkins.plugins.git; public abstract class AbstractGitSCMSource extends SCMSource { protected SCMRevision retrieve(…) protected void retrieve(…) } JAVA INTERFACES & CLASSES

Slide 40

Slide 40 text

A LOADING MECHANISM JAVA CLASSLOADING

Slide 41

Slide 41 text

A PACKAGING MECHANISM JPI/HPI = JENKINS/HUDSON PLUG IN CUSTOM PACKAGING (JPI/HPI)

Slide 42

Slide 42 text

UPDATES.JENKINS-CI.ORG A DISTRIBUTION MECHANISM JENKINS HAS ITS OWN

Slide 43

Slide 43 text

****************** AND ****************** PROS CONS

Slide 44

Slide 44 text

PLATFORM INDEPENDANT COMPILED PROS

Slide 45

Slide 45 text

JVM ONLY DISTRIBUTION CONS SPECIFIC MECHANISM PACKAGING

Slide 46

Slide 46 text

Can we do DOCKER ? a Plugin Architecture *********************** WITH ***********************

Slide 47

Slide 47 text

Docker ****************** WITH ****************** A PLUGIN ARCHITECTURE

Slide 48

Slide 48 text

A STANDARD INTERFACE

Slide 49

Slide 49 text

A STANDARD INTERFACE DOCKER VOLUMES ├ config │ ├ config.json │ ├ key ├ src │ ├ main.go │ ├ main_test.go │ ├ Makefile DOCKER PLUGIN INPUT & OUPUT

Slide 50

Slide 50 text

A STANDARD INTERFACE DOCKER VOLUMES ├ config │ ├ config.json │ ├ key ├ src │ ├ main.go │ ├ main_test.go │ ├ Makefile DOCKER PLUGIN A DOCKER PLUGIN B ├ binaries │ ├ main_darwin_amd64 │ ├ main_linux_x86 INPUT & OUPUT

Slide 51

Slide 51 text

A STANDARD INTERFACE ENVIRONMENT VARIABLES docker run my_plugin -e URL=github.com -e DEPTH=3 INPUT ONLY

Slide 52

Slide 52 text

A STANDARD INTERFACE DOCKER EXIT CODE OUTPUT ONLY docker run my_image echo $?

Slide 53

Slide 53 text

A LOADING MECHANISM

Slide 54

Slide 54 text

A LOADING MECHANISM DOCKER RUN docker run my_plugin

Slide 55

Slide 55 text

A PACKAGING MECHANISM

Slide 56

Slide 56 text

A PACKAGING MECHANISM DOCKER BUILD DOCKER IMAGES DOCKERFILES

Slide 57

Slide 57 text

A DISTRIBUTION MECHANISM

Slide 58

Slide 58 text

A DISTRIBUTION MECHANISM DOCKER HUB HUB.DOCKER.COM FOR PUBLIC PLUGINS

Slide 59

Slide 59 text

A DISTRIBUTION MECHANISM ANY DOCKER REGISTRY FOR PUBLIC AND PRIVATE PLUGINS (OR DOCKER DISTRIBUTION NOW)

Slide 60

Slide 60 text

****************** VS ****************** CLASSIC PLUGIN Docker ****************** VS ****************** ARCHITECTURE

Slide 61

Slide 61 text

NEED TO LEARN ONLY ONE LANGUAGE BUT TIED TO THIS LANGUAGE LANGUAGE AGNOSTIC PROGRAMMING LANGUAGE CLASSIC DOCKER

Slide 62

Slide 62 text

NEED TO BE WRAPPED CAN BE NATIVE LANGUAGE CAN BE CHOSEN ACCORDINGLY NATIVE TOOLS CLASSIC DOCKER

Slide 63

Slide 63 text

VERY SPECIALIZED INTERFACES HARD TO MAINTAIN RETRO-COMPATIBILITY SIMPLE INTERFACES BUT ALSO PERMISSIVE COMPATIBILITY CLASSIC DOCKER

Slide 64

Slide 64 text

MANUALLY IMPLEMENTED DOCKER ECOSYSTEM CAN BE TRICKY WITH VOLUMES SCALING CLASSIC DOCKER

Slide 65

Slide 65 text

GIT PLUGIN ******************* WITH ******************* AN EXAMPLE COMPARISON

Slide 66

Slide 66 text

THE COMPETITION RETRIEVE SOURCE CODE Git URL Git Branch SSH Private key Source Code ?

Slide 67

Slide 67 text

JENKINS

Slide 68

Slide 68 text

JENKINS ~600 LINES OF CODE TO DO ‘GIT CLONE’

Slide 69

Slide 69 text

JENKINS ~600 LINES OF CODE TO DO ‘GIT CLONE’

Slide 70

Slide 70 text

DOCKER ├ private-key ├ out │ ├ main.go │ ├ main_test.go │ ├ Makeﬁle GIT PLUGIN CONTAINER $SCM_URL $SCM_BRANCH

Slide 71

Slide 71 text

DOCKER FROM debian:jessie RUN apt-get update \ && apt-get install -y git ADD run.sh /run.sh RUN echo "IdentityFile /private-key" \ >> /etc/ssh/ssh_config CMD ["/run.sh"] #!/bin/bash git clone "$SCM_URL" --recursive /out cd /out && git checkout "$SCM_BRANCH" Dockerﬁle run.sh

Slide 72

Slide 72 text

NOW THAT WE HAVE 1 PLUGIN, HOW CAN WE ORCHESTRATE MULTIPLE PLUGINS ?

Slide 73

Slide 73 text

BUILD PIPELINE EXAMPLE 1. FETCH SOURCE CODE 2. COMPILE & PACKAGE 3. TEST IT

Slide 74

Slide 74 text

BUILD PIPELINE EXAMPLE ORCHESTRATOR CONTAINER

Slide 75

Slide 75 text

GIT PLUGIN CONTAINER ├ private-key $SCM_URL $SCM_BRANCH BUILD PIPELINE EXAMPLE ORCHESTRATOR CONTAINER docker run git_plugin

Slide 76

Slide 76 text

GIT PLUGIN CONTAINER ├ private-key ├ src │ ├ Main.java │ ├ pom.xml $SCM_URL $SCM_BRANCH BUILD PIPELINE EXAMPLE ORCHESTRATOR CONTAINER docker run git_plugin

Slide 77

Slide 77 text

GIT PLUGIN CONTAINER ├ private-key ├ src │ ├ Main.java │ ├ pom.xml $SCM_URL $SCM_BRANCH BUILD PIPELINE EXAMPLE MAVEN PLUGIN CONTAINER ORCHESTRATOR CONTAINER docker run git_plugin docker run maven_plugin

Slide 78

Slide 78 text

GIT PLUGIN CONTAINER ├ private-key ├ src │ ├ Main.java │ ├ pom.xml $SCM_URL $SCM_BRANCH BUILD PIPELINE EXAMPLE MAVEN PLUGIN CONTAINER ├ output │ ├ main.war ORCHESTRATOR CONTAINER docker run git_plugin docker run maven_plugin

Slide 79

Slide 79 text

GIT PLUGIN CONTAINER ├ private-key ├ src │ ├ Main.java │ ├ pom.xml $SCM_URL $SCM_BRANCH BUILD PIPELINE EXAMPLE MAVEN PLUGIN CONTAINER SELENIUM PLUGIN CONTAINER ├ output │ ├ main.war ORCHESTRATOR CONTAINER docker run git_plugin docker run maven_plugin docker run selenium_plugin echo $?

Slide 80

Slide 80 text

GIT PLUGIN CONTAINER ├ private-key ├ src │ ├ Main.java │ ├ pom.xml $SCM_URL $SCM_BRANCH MERCURIAL INSTEAD OF GIT ? MAVEN PLUGIN CONTAINER SELENIUM PLUGIN CONTAINER ├ output │ ├ main.war ORCHESTRATOR CONTAINER docker run git_plugin docker run maven_plugin docker run selenium_plugin echo $?

Slide 81

Slide 81 text

HG PLUGIN CONTAINER ├ private-key ├ src │ ├ Main.java │ ├ pom.xml $SCM_URL $SCM_BRANCH MERCURIAL INSTEAD OF GIT ? MAVEN PLUGIN CONTAINER SELENIUM PLUGIN CONTAINER ├ output │ ├ main.war ORCHESTRATOR CONTAINER docker run hg_plugin docker run maven_plugin docker run selenium_plugin echo $?

Slide 82

Slide 82 text

HG PLUGIN CONTAINER ├ private-key ├ src │ ├ Main.java │ ├ pom.xml $SCM_URL $SCM_BRANCH MAVEN PLUGIN CONTAINER SELENIUM PLUGIN CONTAINER ├ output │ ├ main.war ORCHESTRATOR CONTAINER docker run hg_plugin docker run maven_plugin docker run selenium_plugin echo $? GRADLE INSTEAD OF MAVEN ?

Slide 83

Slide 83 text

HG PLUGIN CONTAINER ├ private-key ├ src │ ├ Main.java │ ├ pom.xml $SCM_URL $SCM_BRANCH GRADLE INSTEAD OF MAVEN ? GRADLE PLUGIN CONTAINER SELENIUM PLUGIN CONTAINER ├ output │ ├ main.war ORCHESTRATOR CONTAINER docker run hg_plugin docker run gradle_plugin docker run selenium_plugin echo $?

Slide 84

Slide 84 text

HG PLUGIN CONTAINER ├ private-key ├ src │ ├ Main.java │ ├ pom.xml $SCM_URL $SCM_BRANCH ANOTHER BUILD WORKFLOW ? GRADLE PLUGIN CONTAINER SELENIUM PLUGIN CONTAINER ├ output │ ├ main.war ORCHESTRATOR CONTAINER docker run hg_plugin docker run gradle_plugin docker run selenium_plugin echo $?

Slide 85

Slide 85 text

HG PLUGIN CONTAINER ├ private-key ├ src │ ├ Main.java │ ├ pom.xml $SCM_URL $SCM_BRANCH ANOTHER BUILD WORKFLOW ? GRADLE PLUGIN CONTAINER SELENIUM PLUGIN CONTAINER ├ output │ ├ main.war YOUR OWN ORCHESTRATOR CONTAINER docker run hg_plugin docker run gradle_plugin docker run selenium_plugin echo $?

Slide 86

Slide 86 text

Our Experiment BAZOOKa ******************* WITH *******************

Slide 87

Slide 87 text

Our Experiment BAZOOKa ******************* WITH *******************

Slide 88

Slide 88 text

No content

Slide 89

Slide 89 text

CONCLUSION *************************************************

Slide 90

Slide 90 text

DOCKER MICROPLUGINS EASY TO SETUP EXTENSIBLE PERMISSIVE EASY TO DISTRIBUTE LANGUAGE AGNOSTIC

Slide 91

Slide 91 text

QUESTIONS?