Docker ohne Kubernetes Frank Kleine @bovigo Unkonf 13.10.2018 

Docker@B::IT, 2 years later Frank Kleine @bovigo Unkonf 13.10.2018 

WHAT IS B::IT, ACTUALLY? 

B::IT 25 people Identity Management (EIAM, LDAP, etc.) Intranet (InsideNET, ONE) Tools for Collaboration (Wiki, Dev-Jira, Bitbucket) 

A LITTLE B::IT OF HISTORY 

Q4/2015 Department targets for 2016 NoSQL Docker 

Q1/2016 … 

Q2/2016 bit_docker Puppet Environment Toying around in sandbox.lan 

JUNE 21 2016 Yesterday I thought again over the hlt/xenon topic and aggregation on a VM. Couldn’t this be a use case for Docker? Jens (Head of IT Operations Data Services) 

JUNE 2016 Toying around, but more seriously: base images Docker registry in sandbox.lan 

JULY 2016 B::IT Docker Day I B::IT Docker Registry in Infrastructure 

JULY 18 2016 

OCTOBER 2016 

OCTOBER 2016 Kernel panics & incompatibilities between
 Kernel & file systems. https://inside.1and1.org/one/#walls/1112/posts/40582 

NOVEMBER 2016 Stable. But we need to know more. Orchestration! 

A YEAR OF CONTEMPLATION 

2017/18 Kubernetes? Swarm? 

DOCKER SWARM No one ever used this in production. Otherwise the lack of working features can’t be explained. 

KUBERNETES Way too few people in B::IT Expected training curve too steep 

LEARNING CURVE People lost contact with developments So we decided to switch gears 

STRATEGY Switch to containers - learn the basics Evolution, not revolution 

No content

COMPUTESQUAD Group of people interested in further development of B::IT compute platform 

MICROSERVICES(BEFORE MICROSERVICES) 

INSIDENET ~55 single services Mostly PHP Joined via inside.1and1.org domain (proxy) 

APACHE/PHP bitservicebs01…n bitproxy-cluster-bs bitservicebap01…n bitproxy-cluster-bap https://inside.1and1.org/service/ /files/of/service http://0.0.0.0:80/ /files/of/service http://0.0.0.0:80/ 

PROBLEMS Cluster provides one PHP version only Can’t migrate everything at once 

EVOLUTION 

DOCKER bitproxy-cluster-bs bitproxy-cluster-bap https://inside.1and1.org/service/ bitdockerbs01…n bitdockerbap01…n http://service/ http://0.0.0.0:80/ http://service/ http://0.0.0.0:80/ 

APACHE AS LOADBALANCING PROXY 

PROXY ProxySet failonstatus=418 BalancerMember http://bitservicesdockerqabsa01.mw.server.lan:80 retry=10 timeout=2 BalancerMember http://bitservicesdockerqabsa02.mw.server.lan:80 retry=10 timeout=2 ProxyPass /navigation/ "balancer://navigation-proxy/navigation/" 

PROXY One Proxy-Set for each application Failure code != application failure code 

BYREQUESTS => Mon Jun 18 14:24:58 CEST 2018 HTTP/1.1 200 OK Host: bitdocker01:8123 => Mon Jun 18 14:24:59 CEST 2018 HTTP/1.1 200 OK Host: bitdocker02:8123 => Mon Jun 18 14:25:00 CEST 2018 HTTP/1.1 200 OK Host: bitdocker01:8123 => Mon Jun 18 14:25:01 CEST 2018 HTTP/1.1 200 OK Host: bitdocker02:8123 

DOWN: 01 => Mon Jun 18 14:25:51 CEST 2018 HTTP/1.1 200 OK Host: bitdocker01:8123 => Mon Jun 18 14:25:52 CEST 2018 HTTP/1.1 200 OK Host: bitdocker02:8123 => Mon Jun 18 14:25:55 CEST 2018 HTTP/1.1 200 OK Host: bitdocker02:8123 

UP: 01 => Mon Jun 18 14:25:55 CEST 2018 HTTP/1.1 200 OK Host: bitdocker02:8123 => Mon Jun 18 14:25:56 CEST 2018 HTTP/1.1 200 OK Host: bitdocker02:8123 => Mon Jun 18 14:25:57 CEST 2018 HTTP/1.1 200 OK Host: bitdocker02:8123 => Mon Jun 18 14:26:05 CEST 2018 HTTP/1.1 200 OK Host: bitdocker01:8123 => Mon Jun 18 14:26:06 CEST 2018 HTTP/1.1 200 OK Host: bitdocker02:8123 

DOWN: BOTH => Mon Jun 18 14:26:09 CEST 2018 HTTP/1.1 200 OK Host: bitdocker01:8123 => Mon Jun 18 14:26:10 CEST 2018 HTTP/1.1 200 OK Host: bitdocker02:8123 => Mon Jun 18 14:26:11 CEST 2018 HTTP/1.1 503 Service Unavailable => Mon Jun 18 14:26:14 CEST 2018 HTTP/1.1 503 Service Unavailable => Mon Jun 18 14:26:17 CEST 2018 HTTP/1.1 503 Service Unavailable => Mon Jun 18 14:26:18 CEST 2018 HTTP/1.1 200 OK Host: bitdocker01:8123 

KEEPING A SINGLE HOST TOGETHER 

TRÆFIK Instance on each Docker host Listens to Docker backend Routes managed via labels on containers 

TEAPOT Registered in Træfik for path / Always responds 418 I’m a Teapot 

APPLICATION: DOCKER-COMPOSE 

CONTAINER version: "2.0" services: web: image: "bit-registry.1and1.org/bbc/frontend:latest" restart: unless-stopped network_mode: bridge command: [ "-streamLocation", "http://idevplaindockerqsa01.mw.server.lan:8081/hls/", "-streamHost", "idevplaindockerqsa01.mw.server.lan", "-goshHost", "idevplaindockerqsa01.mw.server.lan", "-basepath", "/streams/", "-db", "/secrets/database", "-csrfAuthKey", "/secrets/csrfAuthKey", "-loginURL", "https://stage.inside.1and1.org/signin", "-validateURL", "https://stage.inside.1and1.org/signin/serviceValidate" ] volumes: - /opt/ui/data/bit-docker/credentials/bbc/database:/secrets/database - /opt/ui/data/bit-docker/credentials/bbc/csrfAuthKey:/secrets/csrfAuthKey labels: - "traefik.backend=bbc_frontend" - "traefik.frontend.rule=PathPrefixStrip:/streams" - "traefik.port=8443" - "traefik.enable=true" 

DEPLOYMENT 

DEPLOYMENT tar cf - docker-compose.yml | ssh bitservicedocker deploy group/app 

DELIVERY Takes care of creating the tar file SSHs to all hosts in parallel* 

DELIVERYFILE version: 1.0 application: bit-docker/teapot deployment: parallel environments: qa: cluster: - https://bitbucket.1and1.org/projects/BIT/repos/bit_cluster/raw/bitservicesqa.yml files: - docker-compose-qa.yml prod: cluster: - https://bitbucket.1and1.org/projects/BIT/repos/bit_cluster/raw/bitservicesprod.yml files: - docker-compose.yml 

ROLLOUT On each single server Accepts the tar file, unpacks it Classic blue/green deployment Starts & stops instances w/ docker-compose Zero downtime with deployment mode “parallel” 

FINAL THOUGHTS 

PITFALLS Clean up old images on single hosts! Ensure log-opts: max-size is set when using json-file log-driver 

NEXT Find a good solution for rollbacks Don’t deploy secrets manually 

AND THEN? Is this forever? Probably not. 

NEXT Looking into iCaas Play around with etcd 

THANKS! Frank Kleine @bovigo Unkonf 13.10.2018