Solaris/illumos zones in 3 parts Joshua Timberman @jtimberman Tuesday, August 20, 13 

A SHIP SHIPPING SHIP SHIPPING SHIPPING SHIPS Tuesday, August 20, 13 It's a container joke, bear with me... 

Tuesday, August 20, 13 

% whoami ‣ System Administrator ‣ Opscode ‣ I like Chef ‣ And Hugs ‣ And Beer ‣ And CrossFit #opslife #opscode #opschef #hugops #beerops #crossfit Tuesday, August 20, 13 For those that don't know, this is a GitHub Identicon. I don't know what it means, maybe it's a J and a T. Or, maybe it's a hash tag that fell down. 

Sit right back and let's hear a tale... (part 1) Tuesday, August 20, 13 I could just talk about zones, but then, you could just go read the documentation, and my blog post about them. Instead, let's start off with my story of how I got here. 

Tuesday, August 20, 13 So, I used to work at IBM, as in, up Diagonal Highway. I was in EBusiness hosting, part of Global Services. We ran whatever customers wanted. 

Tuesday, August 20, 13 I was a Solaris administrator, because we had customers that wanted to run that. We had Solaris 2.5.1 through 8 across various customer environments, sometimes in the same customer environment. 

Tuesday, August 20, 13 This is a Sun E450. I worked with a lot of these. Mostly, running Solaris 8. We had other hardware too. There was no virtualization anywhere here, except on the few E10k and 15k systems. 

Tuesday, August 20, 13 Usually in nice grey and purple racks. Except the E450's which were "installed" on shelves. Eventually we had rack mounted systems all proper, but by that time... 

Tuesday, August 20, 13 Eventually though, I got tired of the nonsense of Solaris's archaic ways. I was actually hired at IBM for my knowledge of Linux, and I moved over to the Linux team in our group. I worked here for a couple years, before finally leaving IBM and going to work for the SANS Institute as a Linux administrator. 

Tuesday, August 20, 13 I then left SANS to work at a consulting company that automated startup infrastructures and was building a new product. This is the company that became Opscode. The product was Chef :-). 

Tuesday, August 20, 13 We worked with a bunch of companies that were early adopters of EC2. At the time, EC2 instances were only Linux. They're virtual machines, running on top of Xen with a great API and low operational expense. This isn't a talk about EC2 though. We also worked with companies that were doing Xen/KVM based virtualization. 

Tuesday, August 20, 13 Virtualization in general is awesome, and it allows you to get more out of hardware resources. Supposedly? Consolidation is where it's at, I guess. Of course, mainframe people will tell you they've been virtualizing since the 70's. 

Virtual Machines ‣ Lead to a proliferation of systems management complexity ‣ Hardware abstraction ‣ Resource intensive ‣ Image management ‣ Plethora of technologies Tuesday, August 20, 13 The hypervisor in a VM environment provides a full hardware abstraction. This means you have to have enough memory, CPU, disk space per VM you wish to run, making it resource intensive. Then there's the problem of image management. Finally, there's a plethora of technologies, VMware, KVM, LPARs, Domains, depending on the platform(s) you're using. But this isn't a talk about virtualization, per se... 

Tuesday, August 20, 13 VMs are a heavyweight solution for virtualization. There exists lighter weight alternatives, in "container" technologies. Such as LXC/Cgroups on Linux Cue question, "how are zones better than lxc" - let's talk at The Bar :) 

Containers ‣ Kernel-level virtualization ‣ No HW abstraction ‣ Plethora of technologies Unix/Linux: Good ol' "chroot" BSD: Jails Linux: LXC/cgroup, OpenVZ Solaris/illumos: Zones (aka Containers) Tuesday, August 20, 13 By kernel-level virtualization, we get performance benefits. Launching containers is extremely fast, and they're lightweight. Generally because there's no hardware abstraction, they're more simple that VM technologies There's a plethora of technologies, usually OS-specific: jails, openvz/ lxc/cgroups, and finally, solaris zones (containers) 

An History Lesson (part 2) http://www.flickr.com/photos/10159247@N04/8593167569/ Tuesday, August 20, 13 This is a brief history of Solaris, OpenSolaris, and illumos. For the best background, listen to this talk from Bryan Cantrill: http://smartos.org/2011/12/15/fork-yeah-the-rise-and- development-of-illumos-2/ 

& Containers/Zones Tuesday, August 20, 13 So let's talk about Solaris zones. First of all... Solaris 10 brought a bunch of new toys to the yard! 

Solaris Containers / Zones ‣ Introduced as Solaris Containers in 5.10 (January 2005) ‣ Nuances and pedantry aside, Containers ~ Zones Tuesday, August 20, 13 

How did I get here? ‣ OpenSolaris Since this is really just a story about me... Tuesday, August 20, 13 I installed and used OpenSolaris for ohai, chef resources/provider testing. Then later on I went back looking for it, and ... wat? 

What happened to OpenSolaris? ‣ OpenSolaris ‣ Oracle Solaris http://smartos.org/2011/12/15/fork-yeah-the-rise-and-development-of-illumos-2/ Tuesday, August 20, 13 Except, when Oracle bought Sun, they silently killed the OpenSolaris project. Seriously go listen to Bryan's talk. It's a great lesson in open source project and community governance, and why it is vitally important to be a good steward to your community. 

Okay, I can't afford Oracle Solaris ‣ OpenSolaris ‣ Oracle Solaris ‣ Joyent SmartOS Tuesday, August 20, 13 Along the way, Joyent released SmartOS. This is the hypervisor OS that Joyent uses to build their public cloud offering. It has particular hardware requirements, and until recently, wasn't easy to run in a VM. http://cuddletech.com/blog/?p=821 

Well, I can't run SmartOS... ‣ OpenSolaris ‣ Oracle Solaris ‣ Joyent SmartOS ‣ OmniTI OmniOS Tuesday, August 20, 13 In April last year, OmniTI announced OmniOS, their illumos distribution. It's intended to be installed on real hardware. Well, maybe not the beloved E450 :). 

illumos is... ‣ Fork of OpenSolaris ‣ Free/Open source ‣ http://illumos.org ‣ Includes all the goods: zones, zfs, smf, dtrace, crossbow ‣ Where all the innovation for technology from Solaris is happening Tuesday, August 20, 13 Again, watch Bryan's talk if you want to know the background on all this. 

Why OmniOS? ‣ "Couldn't" run SmartOS, didn't really look at others ‣ OmniOS is a server-focused minimal installation ‣ OmniOS uses IPS, supports SVR4 ‣ OmniTI provides an OmniOS Vagrant box Tuesday, August 20, 13 OmniOS appeals to me for the "stable base platform" aspect of the minimal installation. I like that it strives for compatibility with older Solaris platforms, such as supporting the SVR4 package system. I also love that they make a Vagrant box, which means getting started is a Vagrantfile + "vagrant up" 

Why Zones? ‣ Kernel-level virtualization ‣ Integrated with other Solaris/illumos technologies zfs, dtrace, crossbow ‣ Can't break out of a zone* There are other container technologies, why zones? * at least, I haven't found a reference to it being possible Tuesday, August 20, 13 If you know of research, blog posts, papers, or anything that proves that one can break out of a non-global zone into the global zone, I'd love to hear it. I've heard that it is possible to break out of KVM, Xen, LXC, Jails, but I also don't have references handy. Please email me, [email protected] if you have any. 

Tell me about these zones of which you speak (part 3) http://www.flickr.com/photos/schoffer/144670634 Tuesday, August 20, 13 

The Environment ‣ Consumer-grade hardware (my old gaming PC) Dual-core 2.3GHz CPU 4G memory OS disk (128G) Data disk (500G) 2x GigE NICs Tuesday, August 20, 13 This is just some baseline information about the hardware I'm running all this on. It's useful to note the disks and the NICs, and that this is a pretty "minimally spec'ed" machine (in comparison to what you can get for the money now - this computer is 6 years old now) 

The Commands ‣ format - disk partitioning (info gathering) ‣ zpool - configure storage pools ‣ dladm - administer data links (network interfaces) ‣ zonecfg - set up zone configuration ‣ zoneadm - adminster zones Tuesday, August 20, 13 These are commands that we'll be using, they're all specific to Solaris/ illumos. Well, except format, but hey :). The man pages are really good, and contain everything you'll need to know about the sub-commands and options that I use. Also, all the documentation from Solaris 10 release era, 2005, is still relevant and totally works, available from Oracle's site. 

Hardware: Disks root@menthe:~# format < /dev/null Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c3t0d0 /pci@0,0/pci1043,cb84@d/disk@0,0 1. c3t1d0 /pci@0,0/pci1043,cb84@d/disk@1,0 Specify disk (enter its number): Tuesday, August 20, 13 The format command is used for partitioning disks. It is also about the only reasonable command available to list the actual device names of the disks in the system. It's an interactive command, unless you give it stdin. Is there a command I'm missing to find the cXtXdX devices instead of this? 

ZFS, the last filesystem you'll need ‣ ZFS was introduced with Solaris 10 ‣ Copy on write filesystem ‣ Lightweight snapshots ‣ Volume management built in ‣ Enterprise-grade storage - built for data reliability Tuesday, August 20, 13 This isn't a talk about ZFS, so here are some highlights about ZFS. Maybe I'll come back for one another time :). 

Create a zpool on the "data" disk # zpool create zones c3t1d0 Tuesday, August 20, 13 It is a best practice to have a zpool set aside for zones. A zpool is a collection of disk devices on which you build ZFS filesystems. 

Our zpools root@menthe:~# zpool list NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT rpool 139G 4.45G 135G - 3% 1.00x ONLINE - zones 464G 1.83G 462G - 0% 1.00x ONLINE - Tuesday, August 20, 13 

Unrelated to the examples... root@menthe:~# zfs list /zones/* NAME USED AVAIL REFER MOUNTPOINT zones/base 877M 455G 33K /zones/base zones/fpm 129M 455G 34K /zones/fpm zones/nginx0 783M 455G 34K /zones/nginx0 Tuesday, August 20, 13 This is what I have running before I started making the slides, just to show an example. base, fpm, nginx0 are all zones I was playing with. 

Hardware: NICs root@menthe:~# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE nge0 Ethernet up 1000 full nge0 nge1 Ethernet up 1000 full nge1 Tuesday, August 20, 13 We need to know the physical device names of the network interfaces. I'm already using nge0 for the global zone. 

What is dladm? ‣ OpenSolaris introduced "Crossbow," an all new network stack ‣ dladm is for managing data-links, or network interfaces physical virtual ‣ Other commands: ipadm, if_mpadm Tuesday, August 20, 13 

Create a VNIC # dladm create-vnic -l nge1 vnicdemo0 Tuesday, August 20, 13 Create a new Virtual Network Interface for the zone, associated with a physical ethernet link, nge1. The name of the vnic must end in a number. 

What is a VNIC? ‣ Virtual Network Interface ‣ Each Zone should have its own ‣ Zones cannot see the physical links, only the VNIC Tuesday, August 20, 13 

Zones can't see physical interfaces root@demo:~# dladm show-phys root@demo:~# dladm show-vnic LINK OVER SPEED MACADDRESS MACADDRTYPE VID vnicdemo0 ? 1000 2:8:20:f6:38:6f random 0 Tuesday, August 20, 13 

Creating and Running a Zone ‣ Two commands are used for creating and running zones on Solaris/illumos zonecfg zoneadm Tuesday, August 20, 13 

zonecfg(1) ‣ zonecfg is an interactive command-shell ‣ zonecfg can also import a config file Tuesday, August 20, 13 

demo.conf - zonecfg(1) create -b set zonepath=/zones/demo set brand=ipkg set autoboot=false set ip-type=exclusive add net set physical=vnicdemo0 end Tuesday, August 20, 13 Create a blank config. zonepath is where the zone's filesystem is. Set the brand to use. Brands are operating environments for non-global zones. ipkg is the default zone brand on OmniOS. Whether the zone should be booted automatically at system (global zone) boot. Give the zone its own IP stack, not sharing with the global zone host. This allows it to be on a separate network, and IIRC was required for DHCP Add a new network resource. The physical attribute is the name of the network device created on the global zone (host). 'end' is the end of the 'add net' block 

Configure the new zone # zonecfg demo -f demo.conf Tuesday, August 20, 13 

Install the OS in the new zone # zoneadm -z demo install Tuesday, August 20, 13 This will install the operating system packages into the new zone under the specified zonepath on our storage zpool, zones. 

Copy nsswitch, resolv configuration cp /etc/nsswitch.dns \ /zones/demo/root/etc/nsswitch.conf cp /etc/resolv.conf /zones/demo/root/etc Tuesday, August 20, 13 We want to have name resolution use DNS, so copy the nsswitch.dns file to the new zone, even though it's 2013 and we've used DNS since dinosaurs roamed the earth. Presuming that the /etc/resolv.conf on the global zone is the one we want to use, we copy that to the zone's filesystem, too. This is a huge advantage for zones over image-based VM, as we can actually drop off required configuration before we start the zone, with a image-based VM, we'd have to rebuild an entirely new image. 

Setting up DHCP for the zone # demo.ipadm.conf _ifname=vnicdemo0;_family=2; _ifname=vnicdemo0;_family=26; _ifname=vnicdemo0;_aobjname=vnicdemo0/v4;_dhcp=-1,no; Tuesday, August 20, 13 Write this configuration to a file, demo.ipadm.conf 

Copy the file to the zone cp demo.ipadm.conf /zones/demo/root/etc/ipadm/ipadm.conf Tuesday, August 20, 13 

Boot the Zone # zoneadm -z demo boot Tuesday, August 20, 13 Time to actually boot the zone up so we can use it! 

Log into the Zone root@menthe:~# zlogin demo [Connected to zone 'demo' pts/2] Last login: Sun Aug 18 20:36:28 on pts/2 OmniOS 5.11 omnios-8d266aa 2013.05.04 root@demo:~# logout [Connection to zone 'demo' pts/2 closed] root@menthe:~# Tuesday, August 20, 13 

Use Case (part bonus) http://www.flickr.com/photos/postbear/6843265855 Tuesday, August 20, 13 

Use Case: Cloning Zones for Chef Nodes ‣ Zones can be cloned from the global zone ‣ Install Chef, then clone the zone ‣ Use this zone as a baseline for new zones Tuesday, August 20, 13 

Creating a "chefbase" zone dladm create-vnic -l nge1 vnicchefbase0 zonecfg -z chefbase -f chefbase.conf zoneadm -z chefbase install # copy nsswitch, resolv, ipadm to zone... zoneadm -z chefbase boot zlogin chefbase \ 'curl -L https://www.opscode.com/chef/install.sh | bash' zoneadm -z chefbase halt https://github.com/jtimberman/zone-scripts/blob/master/mkchefbase.sh Tuesday, August 20, 13 This is an excerpt from the script in the zone-scripts repository I created. 

List the Zones root@menthe:~# zoneadm list -vi ID NAME STATUS PATH BRAND IP 0 global running / ipkg shared 16 demo running /zones/demo ipkg excl - chefbase installed /zones/chefbase ipkg excl Tuesday, August 20, 13 

Clone the "chefbase" zone zonecfg -z chefbase export > newzone.conf zonecfg -z newzone -f newzone.conf zoneadm -z newzone clone chefbase # copy nsswitch, resolv, ipadm to newzone zoneadm -z newzone boot mkdir -p /zones/newzone/root/etc/chef # complete chef configuration... https://github.com/jtimberman/zone-scripts/blob/master/mkchefbase.sh Tuesday, August 20, 13 

Completing the Chef Configuration ‣ Chef comes in two "flavors": ‣ Solo (no server) ‣ Client/Server (uses a Chef Server) ‣ We'll assume a Chef Server I'll use Opscode's Enterprise Chef, Hosted Tuesday, August 20, 13 

Configure and run Chef cp ./.chef/validation.pem /zones/newzone/root/etc/chef cat > /zones/$newzone/root/etc/chef/client.rb <

Running Chef on the New Zone # zlogin newzone /opt/chef/bin/chef-client [2013-08-18T21:16:27+00:00] INFO: Forking chef instance to converge... [2013-08-18T21:16:27+00:00] INFO: *** Chef 11.6.0 *** [2013-08-18T21:16:29+00:00] INFO: Client key /etc/chef/client.pem is not present - registering [2013-08-18T21:16:30+00:00] INFO: Run List is [] [2013-08-18T21:16:30+00:00] INFO: Run List expands to [] [2013-08-18T21:16:30+00:00] INFO: Starting Chef Run for newzone [2013-08-18T21:16:30+00:00] INFO: Running start handlers [2013-08-18T21:16:30+00:00] INFO: Start handlers complete. [2013-08-18T21:16:31+00:00] INFO: Loading cookbooks [] [2013-08-18T21:16:31+00:00] WARN: Node newzone has an empty run list. [2013-08-18T21:16:31+00:00] INFO: Chef Run complete in 1.472431742 seconds [2013-08-18T21:16:31+00:00] INFO: Running report handlers [2013-08-18T21:16:31+00:00] INFO: Report handlers complete Tuesday, August 20, 13 

Of course, that's not all... ‣ I used Chef as an example ‣ Perhaps you use another tool/system, or Chef Solo ‣ Perform the "initial setup" after the zone is cloned... Tuesday, August 20, 13 

Think about the possibilities ‣ Containerization ate the shipping world, so to speak It's starting to eat the application delivery world ‣ An application isn't just a package anymore It's an entire environment (look at Java init scripts) ‣ Automation isn't about just installing packages It's about integrating application environments together Tuesday, August 20, 13 

Projects to watch carefully ‣ illumos ‣ docker ‣ coreos ‣ project fifo Tuesday, August 20, 13 

Further resources ‣ http://omnios.omniti.com ‣ http://illumos.org ‣ http://bit.ly/17x8e9j - my blog post ‣ https://github.com/jtimberman/zone-scripts ‣ Chef Cookbooks: zone, zfs, zpool, smf, rbac Tuesday, August 20, 13 

Thank you! Questions? Joshua Timberman @jtimberman http://www.flickr.com/photos/oberazzi/318947873/ Tuesday, August 20, 13