The Python Deployment Albatross - PyTennessee 2017

Slide 1

Slide 1 text

The Python Deployment Albatross CINDY SRIDHARAN @COPYCONSTRUCT PYTENNESSEE FEBRUARY 5, 2017 NASHVILLE, TN

Slide 2

Slide 2 text

setup.py

Slide 3

Slide 3 text

What’s our goal?

Slide 4

Slide 4 text

Hermetically sealed, uniform, reproducible Python artifacts

Slide 5

Slide 5 text

Hermetically sealed

Slide 6

Slide 6 text

✓ Isolate pure Python dependencies ✓ Isolate compile time native/non-Python dependencies ✓ Isolate runtime native/non-Python dependencies

Slide 7

Slide 7 text

uniform

Slide 8

Slide 8 text

Output of the build process is platform and architecture agnostic

Slide 9

Slide 9 text

Reproducible

Slide 10

Slide 10 text

A set of software development practices that create a verifiable path from human readable source code to the binary code used by computers.

Slide 11

Slide 11 text

What is Python? python hello_world.py Python – or /usr/bin/python – as your system understands it, is a program called the interpreter

Slide 12

Slide 12 text

How does Python know what to import from where? site.py sys.preﬁx sys.exec_preﬁx

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

WHEELS VIRTUALENV PEX DOCKER CONDA NIX

Slide 16

Slide 16 text

wheels

Slide 17

Slide 17 text

but before wheels there were …

Slide 18

Slide 18 text

eggs-ecutable

Slide 19

Slide 19 text

purely a distribution format wheels

Slide 20

Slide 20 text

no build system needed on target host no C compiler required wheels

Slide 21

Slide 21 text

wheels no arbitrary code execution like sdists Ergo faster installation pip builds and caches wheels by default

Slide 22

Slide 22 text

ergo less tied to a speciﬁc version of Python Creates .pyc ﬁles as a part of the installation wheels

Slide 23

Slide 23 text

manylinux wheels

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

virtualenv

Slide 26

Slide 26 text

helps “isolate Python environments”

Slide 27

Slide 27 text

✓ Isolates per-project pure Python dependencies from one another virtualenv

Slide 28

Slide 28 text

virtualenv ✓ Isolates per-project pure Python dependencies from system Python

Slide 29

Slide 29 text

✓ Isolates header ﬁles and shared libraries *if these are packaged* virtualenv

Slide 30

Slide 30 text

greenlet.h is installed local to the virtualenv

Slide 31

Slide 31 text

… as is greenlet.so

Slide 32

Slide 32 text

Where virtualenv falls short Uses system provided headers and .so ﬁles if not packaged

Slide 33

Slide 33 text

- - relocatable  doesn’t always work

Slide 34

Slide 34 text

dh-virtualenv

Slide 35

Slide 35 text

PEX

Slide 36

Slide 36 text

Any directory with an __init__.py is considered a package Python import quirks

Slide 37

Slide 37 text

__init__.py

Slide 38

Slide 38 text

Any directory with a __main__.py is treated as an executable Python import quirks

Slide 39

Slide 39 text

__main__.py package is now executable

Slide 40

Slide 40 text

python –m package will execute package/__main__.py if it exists Python import quirks

Slide 41

Slide 41 text

Adding #!/usr/bin/env python to the beginning of any module makes it an executable Python executables

Slide 42

Slide 42 text

change permissions of ﬁle

Slide 43

Slide 43 text

Zipfiles A zipﬁle with an __init__.py is considered a package

Slide 44

Slide 44 text

Zipfiles A zipﬁle with a __main__.py is treated as an executable

Slide 45

Slide 45 text

zip ﬁle is now executable

Slide 46

Slide 46 text

✓zip ﬁles don’t start until a magic zip number ✓ can add arbitrary strings at the start of the ﬁle ✓ #!/usr/bin/env python PEX

Slide 47

Slide 47 text

zip files are also used at Facebook

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

pex ﬁle

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Uses system provided headers and .so ﬁles if not packaged PEX

Slide 55

Slide 55 text

not cross-platform by default PEX

Slide 56

Slide 56 text

docker treats packaging as a namespacing problem

Slide 57

Slide 57 text

What does it mean to containerize a Python process?

Slide 58

Slide 58 text

Docker image for Python processes

Slide 59

Slide 59 text

BASE IMAGE DEVELOPMENT HEADERS AND LIBRARIES VIRTUALENV PEX

Slide 60

Slide 60 text

Best practices for building Docker images for Python ✓ small images ✓ always use a virtualenv or pex ✓ single process per container

Slide 61

Slide 61 text

Dockerflow

Slide 62

Slide 62 text

Challenges of containerization

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

The Docker engine is a container runtime Overlay Networking With 1.12 in Swarm mode, it’s also a cluster scheduler Process manager … and much, much more (service discovery, load balancing, TLS ...) All compiled into one gigantic binary running as root

Slide 65

Slide 65 text

Logging Metrics Collection Observability Debugging

Slide 66

Slide 66 text

conda

Slide 67

Slide 67 text

CONDA or PIP?

Slide 68

Slide 68 text

PIP lacks a SAT solver

Slide 69

Slide 69 text

CONDA or WHEELS?

Slide 70

Slide 70 text

CONDA or VIRTUALENV?

Slide 71

Slide 71 text

CONDA or DOCKER?

Slide 72

Slide 72 text

VM ==> DOCKER :: DOCKER ==> CONDA

Slide 73

Slide 73 text

✓ Python or other modules ✓ System-level libraries ✓ Executable programs conda package Can be downloaded from remote channels

Slide 74

Slide 74 text

all build dependencies need to be preinstalled in the build preﬁx tarball ﬁles generated by the build script to produce a package

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

NIX

Slide 77

Slide 77 text

referential transparency

Slide 78

Slide 78 text

An expression is said to be referentially transparent if evaluating it gives the same value for same arguments. Such functions are called pure functions.

Slide 79

Slide 79 text

nix expressions Nix expressions specify how to build nix packages, including, if necessary, their dependencies.

Slide 80

Slide 80 text

different users have different “views” of the system profiles

Slide 81

Slide 81 text

profiles

Slide 82

Slide 82 text

garbage collection any package not in use (no symlinks) by any generation of any proﬁle

Slide 83

Slide 83 text

List of all dependencies, recursively, down to the bare minimum necessary to use that derivation closure

Slide 84

Slide 84 text

channels a URL that points to a place that contains a set of Nix expressions and a manifest

Slide 85

Slide 85 text

A use case for nix

Slide 86

Slide 86 text

✓ Statically linked Objective-C, C and Lua code ✓ Every time there’s a MacOS upgrade, hosts need to be reimaged ✓ Application then needs to be recompiled ✓ A nix closure gets around this Why nix closures?

Slide 87

Slide 87 text

Conclusion ✓ Build wheels ✓ Use a virtualenv (or pex), even with Docker ✓ Build small Docker images ✓ Explore conda/nix only if needed ✓ Good Luck!

Slide 88

Slide 88 text

@copyconstruct