Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Python Deployment Albatross - PyTennessee 2017
Search
Cindy Sridharan
February 05, 2017
Technology
1
490
The Python Deployment Albatross - PyTennessee 2017
Second stab at this talk.
Cindy Sridharan
February 05, 2017
Tweet
Share
More Decks by Cindy Sridharan
See All by Cindy Sridharan
Unmasking netpoll.go
copyconstructor
4
2.4k
Monitoring in the time of Cloud Native
copyconstructor
4
380
Prometheus - A Whirlwind Tour
copyconstructor
11
3.5k
Prometheus at Google NYC Tech Talks Nov 2016
copyconstructor
10
2.3k
Other Decks in Technology
See All in Technology
All you need to know about InnoDB Primary Keys
lefred
0
120
Fintech SREの挑戦 PCI DSS対応をスマートにこなすインフラ戦略/Fintech SRE’s Challenge: Smart Infrastructure Strategies for PCI DSS Compliance
maaaato
0
450
君も受託系GISエンジニアにならないか
sudataka
1
370
日経電子版 x AIエージェントの可能性とAgentic RAGによって提案書生成を行う技術
masahiro_nishimi
1
290
株式会社EventHub・エンジニア採用資料
eventhub
0
4.2k
急成長する企業で作った、エンジニアが輝ける制度/ 20250214 Rinto Ikenoue
shift_evolve
2
880
滅・サービスクラス🔥 / Destruction Service Class
sinsoku
6
1.5k
Larkご案内資料
customercloud
PRO
0
600
自動テストの世界に、この5年間で起きたこと
autifyhq
10
7.1k
Developers Summit 2025 浅野卓也(13-B-7 LegalOn Technologies)
legalontechnologies
PRO
0
150
スタートアップ1人目QAエンジニアが QAチームを立ち上げ、“個”からチーム、 そして“組織”に成長するまで / How to set up QA team at reiwatravel
mii3king
1
1.1k
偶然 × 行動で人生の可能性を広げよう / Serendipity × Action: Discover Your Possibilities
ar_tama
1
740
Featured
See All Featured
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
330
21k
Embracing the Ebb and Flow
colly
84
4.6k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
4
400
Being A Developer After 40
akosma
89
590k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
Done Done
chrislema
182
16k
Building Adaptive Systems
keathley
40
2.4k
Scaling GitHub
holman
459
140k
Typedesign – Prime Four
hannesfritz
40
2.5k
Rebuilding a faster, lazier Slack
samanthasiow
79
8.8k
Navigating Team Friction
lara
183
15k
Building Your Own Lightsaber
phodgson
104
6.2k
Transcript
The Python Deployment Albatross CINDY SRIDHARAN @COPYCONSTRUCT PYTENNESSEE FEBRUARY 5,
2017 NASHVILLE, TN
setup.py
What’s our goal?
Hermetically sealed, uniform, reproducible Python artifacts
Hermetically sealed
✓ Isolate pure Python dependencies ✓ Isolate compile time native/non-Python
dependencies ✓ Isolate runtime native/non-Python dependencies
uniform
Output of the build process is platform and architecture agnostic
Reproducible
A set of software development practices that create a verifiable
path from human readable source code to the binary code used by computers.
What is Python? python hello_world.py Python – or /usr/bin/python –
as your system understands it, is a program called the interpreter
How does Python know what to import from where? site.py
sys.prefix sys.exec_prefix
None
None
WHEELS VIRTUALENV PEX DOCKER CONDA NIX
wheels
but before wheels there were …
eggs-ecutable
purely a distribution format wheels
no build system needed on target host no C compiler
required wheels
wheels no arbitrary code execution like sdists Ergo faster installation
pip builds and caches wheels by default
ergo less tied to a specific version of Python Creates
.pyc files as a part of the installation wheels
manylinux wheels
None
virtualenv
helps “isolate Python environments”
✓ Isolates per-project pure Python dependencies from one another virtualenv
virtualenv ✓ Isolates per-project pure Python dependencies from system Python
✓ Isolates header files and shared libraries *if these are
packaged* virtualenv
greenlet.h is installed local to the virtualenv
… as is greenlet.so
Where virtualenv falls short Uses system provided headers and .so
files if not packaged
- - relocatable doesn’t always work
dh-virtualenv
PEX
Any directory with an __init__.py is considered a package Python
import quirks
__init__.py
Any directory with a __main__.py is treated as an executable
Python import quirks
__main__.py package is now executable
python –m package will execute package/__main__.py if it exists Python
import quirks
Adding #!/usr/bin/env python to the beginning of any module makes
it an executable Python executables
change permissions of file
Zipfiles A zipfile with an __init__.py is considered a package
Zipfiles A zipfile with a __main__.py is treated as an
executable
zip file is now executable
✓zip files don’t start until a magic zip number ✓
can add arbitrary strings at the start of the file ✓ #!/usr/bin/env python PEX
zip files are also used at Facebook
None
pex file
None
None
None
None
Uses system provided headers and .so files if not packaged
PEX
not cross-platform by default PEX
docker treats packaging as a namespacing problem
What does it mean to containerize a Python process?
Docker image for Python processes
BASE IMAGE DEVELOPMENT HEADERS AND LIBRARIES VIRTUALENV PEX
Best practices for building Docker images for Python ✓ small
images ✓ always use a virtualenv or pex ✓ single process per container
Dockerflow
Challenges of containerization
None
The Docker engine is a container runtime Overlay Networking With
1.12 in Swarm mode, it’s also a cluster scheduler Process manager … and much, much more (service discovery, load balancing, TLS ...) All compiled into one gigantic binary running as root
Logging Metrics Collection Observability Debugging
conda
CONDA or PIP?
PIP lacks a SAT solver
CONDA or WHEELS?
CONDA or VIRTUALENV?
CONDA or DOCKER?
VM ==> DOCKER :: DOCKER ==> CONDA
✓ Python or other modules ✓ System-level libraries ✓ Executable
programs conda package Can be downloaded from remote channels
all build dependencies need to be preinstalled in the build
prefix tarball files generated by the build script to produce a package
None
NIX
referential transparency
An expression is said to be referentially transparent if evaluating
it gives the same value for same arguments. Such functions are called pure functions.
nix expressions Nix expressions specify how to build nix packages,
including, if necessary, their dependencies.
different users have different “views” of the system profiles
profiles
garbage collection any package not in use (no symlinks) by
any generation of any profile
List of all dependencies, recursively, down to the bare minimum
necessary to use that derivation closure
channels a URL that points to a place that contains
a set of Nix expressions and a manifest
A use case for nix
✓ Statically linked Objective-C, C and Lua code ✓ Every
time there’s a MacOS upgrade, hosts need to be reimaged ✓ Application then needs to be recompiled ✓ A nix closure gets around this Why nix closures?
Conclusion ✓ Build wheels ✓ Use a virtualenv (or pex),
even with Docker ✓ Build small Docker images ✓ Explore conda/nix only if needed ✓ Good Luck!
@copyconstruct