Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chef-LXC - Building and deploying custom containers

Chef-LXC - Building and deploying custom containers

Linux containers (LXC) along with Chef for building and deploying custom solutions, environments.


Ranjib Dey

June 26, 2014


  1. Chef and LXC building & deploying custom containers

  2. About Me ? • System Administrator at PagerDuty. Bioinformatics post

    graduate, data mining, application development, system automation • Exploring life like properties in computer systems • FOSS person (recent contributions in ruby-lxc, lxc-chef, chef-metal-lxc, chef, chefspec, serverspec-lxc, graphios, community cookbooks etc)
  3. Agenda • Part A: Introduction to containers & LXC •

    Part B: Using LXC with ruby & chef • Part C: Deployment scenarios (chef + LXC)
  4. Part A. Introduction to containers & LXC

  5. Container • Operating system virtualization • Instead of a hypervisor(like

    kvm, vmware) or paravisor (like Xen) your OS itself gives a VM like interface.
  6. Containers on other platforms • Operating system virtualization (container) •

    Zones on solaris(2005) • OpenVZ (patched linux kernel, 2005) • WPARs on AIX, SRP on HP-AUX(2007)
  7. LXC • LXC – user space tools to build/run containers

    on linux kernel (> 3.11) • Uses 3 key kernel features: • Namespaces • cgroup • capabilities
  8. LXC - CLI ranjib@automator:~ $ lxc-ls --fancy NAME STATE IPV4

    IPV6 AUTOSTART ----------------------------------------- chef-serf STOPPED - - NO father STOPPED - - NO sensu STOPPED - - NO serf STOPPED - - NO trusty STOPPED - - NO ranjib@automator:~ $
  9. LXC – Create using CLI ranjib@automator:~ $ lxc-create -n meetup

    -t download –- \ -d ubuntu -r trusty -a amd64 Using image from local cache Unpacking the rootfs --- You just created an Ubuntu container (release=trusty, arch=amd64, variant=default) The default username/password is: ubuntu / ubuntu To gain root privileges, please use sudo. ranjib@automator:~ $
  10. LXC – Start/Stop ranjib@automator:~ $ lxc-start -n meetup -d ranjib@automator:~

    $ lxc-ls --fancy NAME STATE IPV4 IPV6 AUTOSTART ----------------------------------------------- chef-serf STOPPED - - NO father STOPPED - - NO meetup RUNNING - NO sensu STOPPED - - NO serf STOPPED - - NO trusty STOPPED - - NO ranjib@automator:~ $
  11. LXC - Features • Unprivileged containers • Decoupled rootfs vs

    container creation • Rootfs customization via templates • Cgroup customization via config file and api • Bindings • Hooks • 1.1 aiming for CRIU support
  12. Part B. Using LXC with ruby & chef

  13. Ruby and LXC integration • Ruby-lxc binding is 1.0 •

    Native and covers entire liblxc API • Hosted in the same github repo as LXC • CLI wrapper based ruby bindings are also available (will limit `attach` usage)
  14. Ruby-LXC Examples c = LXC::Container.new('foo') c.create('ubuntu') # create with ubuntu

    template c.start Create and start a container
  15. Ruby-LXC Examples Run arbitrary command inside a container c.attach do

    # going inside container LXC.run_command('ifconfig eth0') end
  16. Ruby-LXC Examples Destroy a container c.stop c.destroy

  17. Ruby-LXC Examples Change memory limits of a container c =

    LXC::Container.new('foo') c.cgroup_item('memory.limit_in_bytes') c.set_cgroup_item('memory.limit_in_bytes','10000000') c.save_config
  18. Ruby-LXC Examples Destroy a container c.stop c.destroy

  19. Chef-LXC Integration LXC cookbook - Uses LXC CLI - Containers

    are bootstrapped exactly as chef nodes include_recipe 'lxc' lxc_container 'my_container' do action :create run_list ['role[base]'] chef_enabled true end
  20. Chef-LXC Integration Chef-metal project - Provides uniform interface for machine

    as a chef resource - Has LXC driver (alongside vagrant, fog etc) - Treats containers exactly as chef node require 'chef_metal_lxc/lxc_provisioner' with_provisioner ChefMetalLXC::LXCProvisioner.new machine 'mario' do recipe 'postgresql' recipe 'mydb' tag 'mydb_master' end
  21. Chef-LXC Integration Chef-lxc gem - Uses liblxc attach method to

    execute chef resources inside container - Does not install chef inside the container. lxc "web" do template "ubuntu" recipe do package "apache2" service "apache2" do action [:start, :enable] end end action [:create, :start] end
  22. Chef and LXC integration • Will be under heavy development

    since LXC and ruby-LXC is stable now. - unprivileged containers required usernamespace (kernel > 3.11). - ubuntu 14.04 is recommended as host OS.
  23. Chef and LXC integration • A common store for pre-baked

    containers worth exploring • similar to images.linxucontainers.org .. may be one day... lxc-create -n foo -t chef –- \ -d ubuntu -r trusty -a amd64 –-chef-role 'db'
  24. Part C. Deployment scenarios (chef & LXC)

  25. Deployment scenarios - 1 • One container per host (M

    → 1Ct) • Independent host/container update strategy • Separation of concerns/dependencies • Common patterns -> reusable components
  26. 1 host–1 container (private network) Host Container chef logstash sensu

    IPtables network Data Volume
  27. 1 host–1 container (public network) Host Container chef logstash sensu

    eth1/ENI network Data Volume
  28. Why? • Keep db binary dependencies isolated from host binaries

    • Dont let an old software dictate your host OS choice. • We can use cgroups to control cpu footprint of new service during changes like. • Db upgrade • App Releases
  29. Deployment scenarios -2 • N similar containers per host (1M

    → NCt) • proxy on host • Complex load balancing options • Frozen containers
  30. 1 host–N similar containers Host Container chef logstash sensu HAproxy

    network Container Container Container Container Container
  31. Why? • Using chef on host makes dynamic haproxy configs

    trivial • Employ blue/green, canary, A/B deployment techniques • Portable, fast deployments. Free cgroup accountring as metrics.
  32. Deployment scenarios -3 • Different types containers per host (1M

    → N different Cts) • Generic case of the earlier pattern • Opportunity to use nested containers
  33. 1 host–N containers of different type Host DB container chef

    logstash sensu HAproxy network DB container DB container App container App container App container
  34. Nesting for grouping similar containers Host DB container chef logstash

    sensu network DB container App Cluster Container HAproxy App container App container App container DB Cluster Container IPtables
  35. Why? • Test bed for environment wide automation • Perfect

    for dev environments • Explore network (iptables for partition, tc for delay) , disk io, cpu throttling effects on clusters etc.
  36. Deployment scenarios – many more • Cross platforms (i386, x86_64/amd64,

    arm) • Build android apps as from jenkins easily • Any arbitrary code evaluation (CI – SaaS) • Rapid deployments – Frozen containers
  37. Resources • Wikipedia article on OS virtualization • LXC main

    website • Stephane Graber's blog series on LXC • Rami Rosen's presentation on LXC • Ruby-LXC • chef-lxc, chef-metal, lxc-cookbook
  38. Do It! @RanjibDey ranjib@pagerduty.com