James McPherson: Why Python rocks Solaris

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Why Python Rocks Solaris Kiwi PyCon 2014 James C. McPherson Solaris Modernization Team, Solaris Core Technologies September 2014

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

5 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Why? • Fast to: – Design – Prototype – Test • Testing frameworks are easily available • Careful, sensibly designed language and standard library

Slide 6

Slide 6 text

6 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Why Else? • Lots of open source software uses Python • Python DTrace provider • Support for Python stack extraction in debugging tools – pstack(1m) – mdb(1) (Solaris Modular Debugger)

Slide 7

Slide 7 text

Slide 8

Slide 8 text

8 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Where? • Boot Environment management • Packaging (Image Packaging System) • Installation technologies • OpenStack • Sundry utilities and applications • Source code management (we use Mercurial) • Internal build tools

Slide 9

Slide 9 text

Slide 10

Slide 10 text

10 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Boot Environment management • Unlimited boot environments – Powered by ZFS – No need to add new disks or volumes Always safe OS updates, with integrated rollback • Boot Environments managed with beadm(1m) – Uses python bindings to libzfs – Simple to create, list, activate and destroy Active BE Old BE Updated BE Active BE New BE

Slide 11

Slide 11 text

11 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Image Packaging System (1) • Introduced with Solaris 11 (replaces SysV Packages) • About 94% Python (222k LoC), 4% C – C implementations of highly optimised functions for satisfiability, ELF section manipulation, some system calls and system configuration – Python code includes unit tests • Provides complete software lifecycle management (installation, upgrade/update and removal) • Provides consistent behaviour through dependency tracking, group packages and incorporations • With SysV packages (Solaris 10 and earlier) we applied patches • With IPS we merely install new versions of packages

Slide 12

Slide 12 text

12 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Image Packaging System (2) ● Networked package repositories ● Cryptographically verified ● Complete set of tools ● Package, version, and distribute your internal applications ● Easy to pilot and automate ● Best practice is the default practice ● Can’t stray off track Lightning fast updates at massive scale

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

15 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Installation Technologies Careful design enables maximum code reuse and minimises implementation overheads: get your images to disk faster • All our installation technologies are built on top of IPS: – DC (Distro Constructor) for customised distributions and golden images – AI (Automated Installer) for hands-free installation using host- specific or generic criteria – Unified Archives (new in Solaris 11.2) for system cloning, recovery images and p2v/v2p conversions [OVF-compliant OVA file with non-traditional stream-based payload]

Slide 16

Slide 16 text

16 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Utilities and Applications • ddu – Device Driver Utility identifies devices in your system and which drivers they require • pyzfs – python binding for the zfs comand • RAD – Remote Administration Daemon Provides: client-side bindings in Python (and C and java) secured remote visibility of system statistics (kstats) Allows for programmatic and secured local and remote administration of zones, LDOMs, bare metal and services. • Python bindings for several other system libraries to allow management of services and common data structures

Slide 17

Slide 17 text

17 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Full Distribution of OpenStack in Oracle Solaris 11.2 Horizon Cloud Management Zones & Kernel Zones Nova Compute Virtualization Elastic Virtual Switch Neutron Cloud Networking ZFS File System Cinder/Swift Cloud Storage Unified Archives Glance Image Deployment General Availability 31 July 2014

Slide 18

Slide 18 text

18 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 OpenStack • Python from top to bottom • Oracle Solaris implementation of OpenStack leverages our built-in technologies: – Kernel Zones – ZFS – IPS – Unified Archives – Elastic Virtual Switch (SDN) • Used for our internal Solaris development cloud

Slide 19

Slide 19 text

19 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 OpenStack – pkg(1M) installation to current zone limoncello: # time pkg install cloud/openstack Packages to install: 201 Services to change: 1 Create boot environment: No Create backup boot environment: No DOWNLOAD PKGS FILES XFER (MB) SPEED Completed 201/201 20909/20909 55.0/55.0 150k/s PHASE ITEMS Installing new actions 24564/24564 Updating package state database Done Updating package cache 0/0 Updating image state Done Creating fast lookup database Done Reading search index Done Building new search index 1673/1673 real 10m4.991s user 2m58.065s sys 0m14.751s

Slide 20

Slide 20 text

20 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 OpenStack – a Unified Archive KZ installation Create a new Kernel Zone: # zonecfg -z OpenStack-KZ create -t SYSsolaris-kz Add two virtual interfaces, and set ram allocation to 4Gb: # zonecfg -z OpenStack-KZ "select anet id=0; add mac; set mac- address=auto; end; add mac; set mac-address=random; end; end; select capped-memory; set physical=4G; end;" Install and boot: # zoneadm -z OpenStack-KZ install \ -a /scratch/UARs/sol-11_2-openstack-x86.uar # zoneadm -z OpenStack-KZ boot # zlogin -C OpenStack-KZ Download the OpenStack UAR from http://www.oracle.com/technetwork/server-storage/solaris11/downloads

Slide 21

Slide 21 text

21 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Source Code Management and Internal build tools • We use Mercurial as our SCM of choice – Solaris OS/Networking consolidation (“the core”) is approx 14.8M LoC for each of Solaris 11 and later releases 97% C, 2% Python, 1% Java – Gate staff make extensive use of in-process Mercurial hooks for repository management • Interface to our bug tracking db is Python, with cx_Oracle and CherryPy • Solaris kernel and basic userland consolidation is built using a python marshalling script • Several post-build checks are written in Python • Packaging and delivery to Release Engineering is done with python

Slide 22

Slide 22 text

Slide 23

Slide 23 text

23 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 self.finalize() • Oracle Solaris uses Python from top to bottom • Enables us to innovate, design and iterate faster • Enables you to easily integrate Solaris systems in to your environments (we're actively hiring Python programmers to work on Solaris)

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

26 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Selected references Oracle Solaris 11.2 Documentation Library • Packaging and Delivering Software With the Image Packaging System in Oracle® Solaris 11.2 • Oracle Solaris 11 How-To Articles • OpenStack Cloud Management • Using Unified Archives for System Recovery and Cloning in Oracle Solaris 11.2 • SATisfaction (on the pkg(1m) SAT solver) • Manpages: – pkg(1m), pkg(5) – pkg.depotd(1M), pkgrecv(1M), pkgsend(1M) – distro_const(1M), dc_manifest(4) – archiveadm(1M) – zoneadm(1M)