Chef-LXC - Building and deploying custom containers

Linux containers (LXC) along with Chef for building and deploying custom solutions, environments.

1. Chef and LXC building & deploying custom containers
2. About Me ? • System Administrator at PagerDuty. Bioinformatics post graduate, data mining, application development, system automation • Exploring life like properties in computer systems • FOSS person (recent contributions in ruby-lxc, lxc-chef, chef-metal-lxc, chef, chefspec, serverspec-lxc, graphios, community cookbooks etc)
3. Agenda • Part A: Introduction to containers & LXC • Part B: Using LXC with ruby & chef • Part C: Deployment scenarios (chef + LXC)
4. Part A. Introduction to containers & LXC
5. Container • Operating system virtualization • Instead of a hypervisor(like kvm, vmware) or paravisor (like Xen) your OS itself gives a VM like interface.
6. Containers on other platforms • Operating system virtualization (container) • Zones on solaris(2005) • OpenVZ (patched linux kernel, 2005) • WPARs on AIX, SRP on HP-AUX(2007)
7. LXC • LXC – user space tools to build/run containers on linux kernel (> 3.11) • Uses 3 key kernel features: • Namespaces • cgroup • capabilities
8. LXC - CLI ranjib@automator:~ $ lxc-ls --fancy NAME STATE IPV4 IPV6 AUTOSTART ----------------------------------------- chef-serf STOPPED - - NO father STOPPED - - NO sensu STOPPED - - NO serf STOPPED - - NO trusty STOPPED - - NO ranjib@automator:~ $
9. LXC – Create using CLI ranjib@automator:~ $ lxc-create -n meetup -t download –- \ -d ubuntu -r trusty -a amd64 Using image from local cache Unpacking the rootfs --- You just created an Ubuntu container (release=trusty, arch=amd64, variant=default) The default username/password is: ubuntu / ubuntu To gain root privileges, please use sudo. ranjib@automator:~ $
10. LXC – Start/Stop ranjib@automator:~ $ lxc-start -n meetup -d ranjib@automator:~ $ lxc-ls --fancy NAME STATE IPV4 IPV6 AUTOSTART ----------------------------------------------- chef-serf STOPPED - - NO father STOPPED - - NO meetup RUNNING 10.0.3.171 - NO sensu STOPPED - - NO serf STOPPED - - NO trusty STOPPED - - NO ranjib@automator:~ $
11. LXC - Features • Unprivileged containers • Decoupled rootfs vs container creation • Rootfs customization via templates • Cgroup customization via config file and api • Bindings • Hooks • 1.1 aiming for CRIU support
12. Part B. Using LXC with ruby & chef
13. Ruby and LXC integration • Ruby-lxc binding is 1.0 • Native and covers entire liblxc API • Hosted in the same github repo as LXC • CLI wrapper based ruby bindings are also available (will limit `attach` usage)
14. Ruby-LXC Examples c = LXC::Container.new('foo') c.create('ubuntu') # create with ubuntu template c.start Create and start a container
15. Ruby-LXC Examples Run arbitrary command inside a container c.attach do # going inside container LXC.run_command('ifconfig eth0') end
16. Ruby-LXC Examples Destroy a container c.stop c.destroy
17. Ruby-LXC Examples Change memory limits of a container c = LXC::Container.new('foo') c.cgroup_item('memory.limit_in_bytes') c.set_cgroup_item('memory.limit_in_bytes','10000000') c.save_config
18. Ruby-LXC Examples Destroy a container c.stop c.destroy
19. Chef-LXC Integration LXC cookbook - Uses LXC CLI - Containers are bootstrapped exactly as chef nodes include_recipe 'lxc' lxc_container 'my_container' do action :create run_list ['role[base]'] chef_enabled true end
20. Chef-LXC Integration Chef-metal project - Provides uniform interface for machine as a chef resource - Has LXC driver (alongside vagrant, fog etc) - Treats containers exactly as chef node require 'chef_metal_lxc/lxc_provisioner' with_provisioner ChefMetalLXC::LXCProvisioner.new machine 'mario' do recipe 'postgresql' recipe 'mydb' tag 'mydb_master' end
21. Chef-LXC Integration Chef-lxc gem - Uses liblxc attach method to execute chef resources inside container - Does not install chef inside the container. lxc "web" do template "ubuntu" recipe do package "apache2" service "apache2" do action [:start, :enable] end end action [:create, :start] end
22. Chef and LXC integration • Will be under heavy development since LXC and ruby-LXC is stable now. - unprivileged containers required usernamespace (kernel > 3.11). - ubuntu 14.04 is recommended as host OS.
23. Chef and LXC integration • A common store for pre-baked containers worth exploring • similar to images.linxucontainers.org .. may be one day... lxc-create -n foo -t chef –- \ -d ubuntu -r trusty -a amd64 –-chef-role 'db'
24. Part C. Deployment scenarios (chef & LXC)
25. Deployment scenarios - 1 • One container per host (M → 1Ct) • Independent host/container update strategy • Separation of concerns/dependencies • Common patterns -> reusable components
26. 1 host–1 container (private network) Host Container chef logstash sensu IPtables network Data Volume
27. 1 host–1 container (public network) Host Container chef logstash sensu eth1/ENI network Data Volume
28. Why? • Keep db binary dependencies isolated from host binaries • Dont let an old software dictate your host OS choice. • We can use cgroups to control cpu footprint of new service during changes like. • Db upgrade • App Releases
29. Deployment scenarios -2 • N similar containers per host (1M → NCt) • proxy on host • Complex load balancing options • Frozen containers
30. 1 host–N similar containers Host Container chef logstash sensu HAproxy network Container Container Container Container Container
31. Why? • Using chef on host makes dynamic haproxy configs trivial • Employ blue/green, canary, A/B deployment techniques • Portable, fast deployments. Free cgroup accountring as metrics.
32. Deployment scenarios -3 • Different types containers per host (1M → N different Cts) • Generic case of the earlier pattern • Opportunity to use nested containers
33. 1 host–N containers of different type Host DB container chef logstash sensu HAproxy network DB container DB container App container App container App container
34. Nesting for grouping similar containers Host DB container chef logstash sensu network DB container App Cluster Container HAproxy App container App container App container DB Cluster Container IPtables
35. Why? • Test bed for environment wide automation • Perfect for dev environments • Explore network (iptables for partition, tc for delay) , disk io, cpu throttling effects on clusters etc.
36. Deployment scenarios – many more • Cross platforms (i386, x86_64/amd64, arm) • Build android apps as from jenkins easily • Any arbitrary code evaluation (CI – SaaS) • Rapid deployments – Frozen containers
37. Resources • Wikipedia article on OS virtualization • LXC main website • Stephane Graber's blog series on LXC • Rami Rosen's presentation on LXC • Ruby-LXC • chef-lxc, chef-metal, lxc-cookbook
38. Do It! @RanjibDey ranjib@pagerduty.com