Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Pluggable Infrastructure with CI/CD and Docker

Af49604c509b95ea8191c8d67e65fbf1?s=47 Bob Killen
October 15, 2015

Pluggable Infrastructure with CI/CD and Docker

Overview of how the University of Michigan Pathology Department adopted CI/CD and Docker on top of Mesos.


Bob Killen

October 15, 2015

More Decks by Bob Killen

Other Decks in Technology


  1. Pluggable Infrastructure with CI/CD and Docker. Bob Killen Systems Programmer/Analyst

    Senior Department of Pathology University of Michigan Health System Jeffrey Sica Business Systems Analyst Lead Department of Pathology University of Michigan Health System
  2. Disclaimer All opinions spoken by Bob or Jeff are their

    own and not that of the University of Michigan or its affiliates.
  3. A bit of background Central IT vs Pathology Informatics HIPAA

    Concerns Our original infrastructure • Before Docker, there were VMs • Started on Kubernetes • Migrated to Individual Hosts
  4. Challenges we faced • Managing multiple instances of the same

    infrastructure ◦ Public applications must be firewalled from Internal applications (HIPAA) • Rapid prototyping and demoing of applications ◦ Victims of our own success • Large infrastructure, small staff ◦ 40+ independant in house developed applications or services ◦ 1 Designer, 1.5 Developers, 1.5 Sysadmins
  5. Workflow - Outside of Host Cluster Commit and Code Push

    Source Control server notifies Build Server of new commit. Build Server schedules build at new commit. If build successful; tag image and push to registry. Otherwise notify group. After successful build pushed to the registry, trigger cluster update. Notify if problem arises during update. Build
  6. Workflow - In Cluster Cluster is notified of new task.

    Cluster determines hosts with appropriate resource offerings. Hosts pull containers associated with task. Once containers pass healthcheck, front end is notified of host:port Front End updates configuration and reloads with new services. Service is now accessible to public network
  7. The Pillars of a Container Centric Infrastructure

  8. The path we have chosen. Gitlab Jenkins DTR Mesos Docker

  9. Gitlab • Open source • Close-enough mirror to the familiar

    (GitHub) • Low barrier to entry for non-developers • Existing integration into other “pillars” (Jenkins) • Now comes in a handy dandy container ◦ “ docker pull gitlab/gitlab-ce ” ◦ https://hub.docker.com/r/gitlab/gitlab-ce/
  10. Jenkins • Open Source • Time tested solution • Plug-ins

    galore ◦ Docker ◦ Mesos Cluster ◦ Gitlab Webhooks • No real constraints on architecture ◦ No plugin? Use Bash! • Blank slate for any project • Windows Support
  11. Docker Trusted Registry • Security Requirements ◦ 100% on premise.

    ◦ LDAP backed. ◦ RBAC - v0 api available, use and integration capability growing. ▪ https://docs.docker.com/docker-trusted-registry/api/ ◦ One-click upgrade. ◦ User friendly UI. • Support Offerings ◦ 24/7 support REQUIRED for use in a clinical care setting. ◦ Guided install with staff on-site where needed
  12. Mesos • Generic abstraction of resources. Useful to combine varied

    host systems (both physical and virtual). • Caters well to an on prem solution. • Slightly more mature in the cluster solution ecosystem. ◦ Powered by Mesos Users - http://mesos.apache.org/documentation/latest/powered-by-mesos/ • Large framework ecosystem; with lots of active development.
  13. Mesos (Cont.) • https://open.mesosphere.com/frameworks/ Just to name a few….

  14. Mesos (Cont…future) Mesos Windows Epic: https://issues.apache.org/jira/browse/MESOS-3094 Docker + Windows +

    Mesos = Match made in the clouds Screencast Demo: https://www.youtube.com/watch?v=dC2vitIh84M
  15. Docker (CE) • Support Offerings ◦ Touching on this again...24/7

    support is REQUIRED when used in a clinical setting. • Certified stable releases suitable for production use. • Predictable release schedule for upgrade planning. • Orchestration components are swappable...docker engine is not, and is the base requirement for everything.
  16. None
  17. Final Form: Pluggable Infrastructure Pt. 1 • Application container function

    should be independent of infrastructure design. ◦ Container init script + environment or cluster seed script. ▪ container init script variables are static ▪ seed script populates container init script with cluster specific information. ▪ Build server can inject seed script for environment (local dev vs cluster deployment)
  18. Example Init Script $VAR1 and $VAR2 can be set at

    container creation or ‘seeded’ via a script sourced as $ENVIRONMENT_INIT . The supplied script could hit a variety of endpoints for configuration information (consul, etcd, registrator, etc). If nothing is provided, it assumes the default value. This allows for init script to remain static, with only the sourced seed script changing per environment.
  19. Final Form: Pluggable Infrastructure Pt. 2 • Infrastructure components in

    source control; and if at all possible built as containers themselves. ◦ Apply standard ‘infrastructure as code’ methods. ▪ Version Everything ▪ Design tests and integrate with CI/CD
  20. Final Form: Pluggable Infrastructure Pt. 2 (Cont.) • Components that

    may modify host configuration require some level of clean-up. e.g. anything that may require iptables. ◦ trap SIGINT, SIGKILL etc. then perform action. ◦ Does require some form of ‘init’ or process management. - supervisord, tini, monit etc • Use of VRRP / Keepalived and deploying in pairs when services may be bound to a single point of failure (no DNS load balancing etc). ◦ Provides incredibly simple IP failover. ◦ Prerequisites for use: ▪ sysctl - net.ipv4.ip_nonlocal_bind=1 ▪ container run with host networking and cap NET_ADMIN ◦ Added bonus of making it EXTREMELY easy to upgrade and switch services. ▪ If upgrading: push new version to backup, then push to master.
  21. None
  22. None
  23. None
  24. None
  25. None
  26. None
  27. None
  28. Keepalived + HAproxy Demo

  29. Example Keepalived Configuration Master Backup Description • state - MASTER

    or BACKUP - Defines server role in the group. • interface - The interface that keepalived will use for VRRP traffic and monitor by default. • vrrp_unicast_bind - The ip in which vrrp should bind to. • vrrp_unicast_peer - The bind ip of the other node in the group. • virtual_router_id - Unique number between 1-255 that identifies the vrrp group. Multiple instances of vrrp can run at the same time on the same servers. • priority - In scenarios with more than 2 nodes, priority will dictate the failover order with highest priority becoming the master. • advert_int - Healthcheck interval (seconds) • authentication - PASS (cleartext password) or AH (IPSEC Authorization Header). Password and method used by the nodes to validate each other. • virtual_ipaddress - Virtual IP Address to be generated and interface it should be bound to. IPs in this block WILL be tracked by keepalived. Used for private network interface tracking. • virtual_ipaddress_excluded - Virtual IP Address to be generated and interface it should be bound to. IPs in this block will NOT be tracked, but automatically failed over if an event would trigger a failure of IP(s) in the virtual_ipaddress block. Used for public facing IPs. • track_interface - Interface adapters that should have their link state monitored for failover trigger.
  30. None
  31. Future State Challenges • Better Secret Management ◦ Consul /

    Vault injection at runtime and seeded via ENVIRONMENT_INIT. ◦ Mount secure volume • Profiling containers for lxc capabilities + apparmor profiles ◦ https://github.com/jfrazelle/bane • Improve logging with logstash framework ◦ https://github.com/mesos/logstash • Notary integration into build pipeline.
  32. None
  33. Questions? Bob Killen rkillen@med.umich.edu @mrbobbytables github.com/mrbobbytables Jeffrey Sica jsica@med.umich.edu @jeefy