Pluggable Infrastructure with CI/CD and Docker

Overview of how the University of Michigan Pathology Department adopted CI/CD and Docker on top of Mesos.

Transcript

1. Pluggable Infrastructure with CI/CD and Docker. Bob Killen Systems Programmer/Analyst Senior Department of Pathology University of Michigan Health System Jeffrey Sica Business Systems Analyst Lead Department of Pathology University of Michigan Health System
2. Disclaimer All opinions spoken by Bob or Jeff are their own and not that of the University of Michigan or its affiliates.
3. A bit of background Central IT vs Pathology Informatics HIPAA Concerns Our original infrastructure • Before Docker, there were VMs • Started on Kubernetes • Migrated to Individual Hosts
4. Challenges we faced • Managing multiple instances of the same infrastructure ◦ Public applications must be firewalled from Internal applications (HIPAA) • Rapid prototyping and demoing of applications ◦ Victims of our own success • Large infrastructure, small staff ◦ 40+ independant in house developed applications or services ◦ 1 Designer, 1.5 Developers, 1.5 Sysadmins
5. Workflow - Outside of Host Cluster Commit and Code Push Source Control server notifies Build Server of new commit. Build Server schedules build at new commit. If build successful; tag image and push to registry. Otherwise notify group. After successful build pushed to the registry, trigger cluster update. Notify if problem arises during update. Build
6. Workflow - In Cluster Cluster is notified of new task. Cluster determines hosts with appropriate resource offerings. Hosts pull containers associated with task. Once containers pass healthcheck, front end is notified of host:port Front End updates configuration and reloads with new services. Service is now accessible to public network
7. The Pillars of a Container Centric Infrastructure
8. The path we have chosen. Gitlab Jenkins DTR Mesos Docker
9. Gitlab • Open source • Close-enough mirror to the familiar (GitHub) • Low barrier to entry for non-developers • Existing integration into other “pillars” (Jenkins) • Now comes in a handy dandy container ◦ “ docker pull gitlab/gitlab-ce ” ◦ https://hub.docker.com/r/gitlab/gitlab-ce/
10. Jenkins • Open Source • Time tested solution • Plug-ins galore ◦ Docker ◦ Mesos Cluster ◦ Gitlab Webhooks • No real constraints on architecture ◦ No plugin? Use Bash! • Blank slate for any project • Windows Support
11. Docker Trusted Registry • Security Requirements ◦ 100% on premise. ◦ LDAP backed. ◦ RBAC - v0 api available, use and integration capability growing. ▪ https://docs.docker.com/docker-trusted-registry/api/ ◦ One-click upgrade. ◦ User friendly UI. • Support Offerings ◦ 24/7 support REQUIRED for use in a clinical care setting. ◦ Guided install with staff on-site where needed
12. Mesos • Generic abstraction of resources. Useful to combine varied host systems (both physical and virtual). • Caters well to an on prem solution. • Slightly more mature in the cluster solution ecosystem. ◦ Powered by Mesos Users - http://mesos.apache.org/documentation/latest/powered-by-mesos/ • Large framework ecosystem; with lots of active development.
13. Mesos (Cont.) • https://open.mesosphere.com/frameworks/ Just to name a few….
14. Mesos (Cont…future) Mesos Windows Epic: https://issues.apache.org/jira/browse/MESOS-3094 Docker + Windows + Mesos = Match made in the clouds Screencast Demo: https://www.youtube.com/watch?v=dC2vitIh84M
15. Docker (CE) • Support Offerings ◦ Touching on this again...24/7 support is REQUIRED when used in a clinical setting. • Certified stable releases suitable for production use. • Predictable release schedule for upgrade planning. • Orchestration components are swappable...docker engine is not, and is the base requirement for everything.
17. Final Form: Pluggable Infrastructure Pt. 1 • Application container function should be independent of infrastructure design. ◦ Container init script + environment or cluster seed script. ▪ container init script variables are static ▪ seed script populates container init script with cluster specific information. ▪ Build server can inject seed script for environment (local dev vs cluster deployment)
18. Example Init Script $VAR1 and $VAR2 can be set at container creation or ‘seeded’ via a script sourced as $ENVIRONMENT_INIT . The supplied script could hit a variety of endpoints for configuration information (consul, etcd, registrator, etc). If nothing is provided, it assumes the default value. This allows for init script to remain static, with only the sourced seed script changing per environment.
19. Final Form: Pluggable Infrastructure Pt. 2 • Infrastructure components in source control; and if at all possible built as containers themselves. ◦ Apply standard ‘infrastructure as code’ methods. ▪ Version Everything ▪ Design tests and integrate with CI/CD
20. Final Form: Pluggable Infrastructure Pt. 2 (Cont.) • Components that may modify host configuration require some level of clean-up. e.g. anything that may require iptables. ◦ trap SIGINT, SIGKILL etc. then perform action. ◦ Does require some form of ‘init’ or process management. - supervisord, tini, monit etc • Use of VRRP / Keepalived and deploying in pairs when services may be bound to a single point of failure (no DNS load balancing etc). ◦ Provides incredibly simple IP failover. ◦ Prerequisites for use: ▪ sysctl - net.ipv4.ip_nonlocal_bind=1 ▪ container run with host networking and cap NET_ADMIN ◦ Added bonus of making it EXTREMELY easy to upgrade and switch services. ▪ If upgrading: push new version to backup, then push to master.
28. Keepalived + HAproxy Demo
29. Example Keepalived Configuration Master Backup Description • state - MASTER or BACKUP - Defines server role in the group. • interface - The interface that keepalived will use for VRRP traffic and monitor by default. • vrrp_unicast_bind - The ip in which vrrp should bind to. • vrrp_unicast_peer - The bind ip of the other node in the group. • virtual_router_id - Unique number between 1-255 that identifies the vrrp group. Multiple instances of vrrp can run at the same time on the same servers. • priority - In scenarios with more than 2 nodes, priority will dictate the failover order with highest priority becoming the master. • advert_int - Healthcheck interval (seconds) • authentication - PASS (cleartext password) or AH (IPSEC Authorization Header). Password and method used by the nodes to validate each other. • virtual_ipaddress - Virtual IP Address to be generated and interface it should be bound to. IPs in this block WILL be tracked by keepalived. Used for private network interface tracking. • virtual_ipaddress_excluded - Virtual IP Address to be generated and interface it should be bound to. IPs in this block will NOT be tracked, but automatically failed over if an event would trigger a failure of IP(s) in the virtual_ipaddress block. Used for public facing IPs. • track_interface - Interface adapters that should have their link state monitored for failover trigger.
31. Future State Challenges • Better Secret Management ◦ Consul / Vault injection at runtime and seeded via ENVIRONMENT_INIT. ◦ Mount secure volume • Profiling containers for lxc capabilities + apparmor profiles ◦ https://github.com/jfrazelle/bane • Improve logging with logstash framework ◦ https://github.com/mesos/logstash • Notary integration into build pipeline.
33. Questions? Bob Killen rkillen@med.umich.edu @mrbobbytables github.com/mrbobbytables Jeffrey Sica jsica@med.umich.edu @jeefy github.com/jeefy