James McPherson: Why Python rocks Solaris

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
James McPherson:
Why Python rocks Solaris
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2014 - Saturday, 13 Sep 2014 - Track 1
http://kiwi.pycon.org/

**Audience level**

Novice

**Description**

A number of organisations have realised that the relational model for databases is insufficient for certain scenarios. Graph databases are good at modelling relationships. This talk will introduce you to Neo4j (a graph database engine) with Python. How to set it up, how to use it and some questions you can answer with it.

**Abstract**

This talk will introduce you to Neo4j (a graph database engine) with Python. How to set it up, how to use it and some questions you can answer with it.

You don’t need any previous experience with Neo4j or NoSQL databases and very little experience with Python.

We'll cover: Introduction to Graphs Setting up your toolbox Neo4j and Cypher A simple real-world example use case with Neo4j

**YouTube**

https://www.youtube.com/watch?v=eakvNuYjwAk

Transcript

1. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
2. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Why Python Rocks Solaris Kiwi PyCon 2014 James C. McPherson Solaris Modernization Team, Solaris Core Technologies September 2014
3. 3 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Contents Why? Where? How? self.finalize() 1 2 3 4
4. 4 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Why?
5. 5 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Why? • Fast to: – Design – Prototype – Test • Testing frameworks are easily available • Careful, sensibly designed language and standard library
6. 6 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Why Else? • Lots of open source software uses Python • Python DTrace provider • Support for Python stack extraction in debugging tools – pstack(1m) – mdb(1) (Solaris Modular Debugger)
7. 7 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Where?
8. 8 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Where? • Boot Environment management • Packaging (Image Packaging System) • Installation technologies • OpenStack • Sundry utilities and applications • Source code management (we use Mercurial) • Internal build tools
9. 9 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 How?
10. 10 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Boot Environment management • Unlimited boot environments – Powered by ZFS – No need to add new disks or volumes Always safe OS updates, with integrated rollback • Boot Environments managed with beadm(1m) – Uses python bindings to libzfs – Simple to create, list, activate and destroy Active BE Old BE Updated BE Active BE New BE
11. 11 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Image Packaging System (1) • Introduced with Solaris 11 (replaces SysV Packages) • About 94% Python (222k LoC), 4% C – C implementations of highly optimised functions for satisfiability, ELF section manipulation, some system calls and system configuration – Python code includes unit tests • Provides complete software lifecycle management (installation, upgrade/update and removal) • Provides consistent behaviour through dependency tracking, group packages and incorporations • With SysV packages (Solaris 10 and earlier) we applied patches • With IPS we merely install new versions of packages
12. 12 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Image Packaging System (2) • Networked package repositories • Cryptographically verified • Complete set of tools • Package, version, and distribute your internal applications • Easy to pilot and automate • Best practice is the default practice • Can’t stray off track Lightning fast updates at massive scale
13. 13 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Simplifying Deployment On a Large Scale
14. 14 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Simplifying Deployment even more
15. 15 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Installation Technologies Careful design enables maximum code reuse and minimises implementation overheads: get your images to disk faster • All our installation technologies are built on top of IPS: – DC (Distro Constructor) for customised distributions and golden images – AI (Automated Installer) for hands-free installation using host- specific or generic criteria – Unified Archives (new in Solaris 11.2) for system cloning, recovery images and p2v/v2p conversions [OVF-compliant OVA file with non-traditional stream-based payload]
16. 16 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Utilities and Applications • ddu – Device Driver Utility identifies devices in your system and which drivers they require • pyzfs – python binding for the zfs comand • RAD – Remote Administration Daemon Provides: client-side bindings in Python (and C and java) secured remote visibility of system statistics (kstats) Allows for programmatic and secured local and remote administration of zones, LDOMs, bare metal and services. • Python bindings for several other system libraries to allow management of services and common data structures
17. 17 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Full Distribution of OpenStack in Oracle Solaris 11.2 Horizon Cloud Management Zones & Kernel Zones Nova Compute Virtualization Elastic Virtual Switch Neutron Cloud Networking ZFS File System Cinder/Swift Cloud Storage Unified Archives Glance Image Deployment General Availability 31 July 2014
18. 18 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 OpenStack • Python from top to bottom • Oracle Solaris implementation of OpenStack leverages our built-in technologies: – Kernel Zones – ZFS – IPS – Unified Archives – Elastic Virtual Switch (SDN) • Used for our internal Solaris development cloud
19. 19 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 OpenStack – pkg(1M) installation to current zone limoncello: # time pkg install cloud/openstack Packages to install: 201 Services to change: 1 Create boot environment: No Create backup boot environment: No DOWNLOAD PKGS FILES XFER (MB) SPEED Completed 201/201 20909/20909 55.0/55.0 150k/s PHASE ITEMS Installing new actions 24564/24564 Updating package state database Done Updating package cache 0/0 Updating image state Done Creating fast lookup database Done Reading search index Done Building new search index 1673/1673 real 10m4.991s user 2m58.065s sys 0m14.751s
20. 20 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 OpenStack – a Unified Archive KZ installation Create a new Kernel Zone: # zonecfg -z OpenStack-KZ create -t SYSsolaris-kz Add two virtual interfaces, and set ram allocation to 4Gb: # zonecfg -z OpenStack-KZ "select anet id=0; add mac; set mac- address=auto; end; add mac; set mac-address=random; end; end; select capped-memory; set physical=4G; end;" Install and boot: # zoneadm -z OpenStack-KZ install \ -a /scratch/UARs/sol-11_2-openstack-x86.uar # zoneadm -z OpenStack-KZ boot # zlogin -C OpenStack-KZ Download the OpenStack UAR from http://www.oracle.com/technetwork/server-storage/solaris11/downloads
21. 21 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Source Code Management and Internal build tools • We use Mercurial as our SCM of choice – Solaris OS/Networking consolidation (“the core”) is approx 14.8M LoC for each of Solaris 11 and later releases 97% C, 2% Python, 1% Java – Gate staff make extensive use of in-process Mercurial hooks for repository management • Interface to our bug tracking db is Python, with cx_Oracle and CherryPy • Solaris kernel and basic userland consolidation is built using a python marshalling script • Several post-build checks are written in Python • Packaging and delivery to Release Engineering is done with python
22. 22 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 self.finalize()
23. 23 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 self.finalize() • Oracle Solaris uses Python from top to bottom • Enables us to innovate, design and iterate faster • Enables you to easily integrate Solaris systems in to your environments (we're actively hiring Python programmers to work on Solaris)
24. 24 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 What questions do you have?
25. 25 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Further reading
26. 26 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 Selected references Oracle Solaris 11.2 Documentation Library • Packaging and Delivering Software With the Image Packaging System in Oracle® Solaris 11.2 • Oracle Solaris 11 How-To Articles • OpenStack Cloud Management • Using Unified Archives for System Recovery and Cloning in Oracle Solaris 11.2 • SATisfaction (on the pkg(1m) SAT solver) • Manpages: – pkg(1m), pkg(5) – pkg.depotd(1M), pkgrecv(1M), pkgsend(1M) – distro_const(1M), dc_manifest(4) – archiveadm(1M) – zoneadm(1M)
27. 27 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014
28. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014
29. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Kiwi PyCon September 2014 James C. McPherson James.McPherson@oracle.com
30. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |