Core Python Packaging & Deployment Tools Core Python Packaging & Deployment Tools ● “setuptools” to build packages ● “twine” to upload packages to a repository ● “pip” to install packages from a repository ● “virtualenv” or “pyvenv” (python3 -m venv) to isolate dependencies ● “setuptools” to build packages ● “twine” to upload packages to a repository ● “pip” to install packages from a repository ● “virtualenv” or “pyvenv” (python3 -m venv) to isolate dependencies
Linux Considerations (Binary Packages) Linux Considerations (Binary Packages) ● Wheel filenames include platform tags (PEP425) ● Python version, Python ABI, platform ABI (OS) ● Example: numpy-1.14.0-cp36-cp36m-macosx_10_10_x86_64.whl ● Linux presents a very diverse ABI spectrum ● Windows and MacOS have way less variation ● Thus PEP513 – manylinux1_{x86_64,i686} ● Restricted set of kernel / userspace ABIs ● Build Python binary packages in old CentOS images ☺ ● Mostly works, with some quirks (e.g. stripping for packaging) ● Wheel filenames include platform tags (PEP425) ● Python version, Python ABI, platform ABI (OS) ● Example: numpy-1.14.0-cp36-cp36m-macosx_10_10_x86_64.whl ● Linux presents a very diverse ABI spectrum ● Windows and MacOS have way less variation ● Thus PEP513 – manylinux1_{x86_64,i686} ● Restricted set of kernel / userspace ABIs ● Build Python binary packages in old CentOS images ☺ ● Mostly works, with some quirks (e.g. stripping for packaging)
Omnibus Packages · Pros and Cons Omnibus Packages · Pros and Cons ⊕ Dependencies not predetermined (by target OS) ⊕ Multi-platform deployment way easier ⊕ Much simpler release process ⊖ Maintainer is responsible for security ⊖ Possibly more version variation in production ⓘ Security updates might be slower – or faster ⓘ Efficiency often dominates other concerns (esp. with deploying in-house tools) ⊕ Dependencies not predetermined (by target OS) ⊕ Multi-platform deployment way easier ⊕ Much simpler release process ⊖ Maintainer is responsible for security ⊖ Possibly more version variation in production ⓘ Security updates might be slower – or faster ⓘ Efficiency often dominates other concerns (esp. with deploying in-house tools)
Binary Extension Packages Trouble-Shooting Binary Extension Packages Trouble-Shooting If stuff doesn‘t build OOTB… ● “dh_strip” and “dh_shlibdeps” exclusions ● Disable “manylinux1” wheels ● Pass the “--no-binary=:all:” option to pip ● … or add “no-manylinux1” to your build environment ● Increases build times (no wheels, or no extension ones) ● Often requires more build dependencies (“*-dev” packages) If stuff doesn‘t build OOTB… ● “dh_strip” and “dh_shlibdeps” exclusions ● Disable “manylinux1” wheels ● Pass the “--no-binary=:all:” option to pip ● … or add “no-manylinux1” to your build environment ● Increases build times (no wheels, or no extension ones) ● Often requires more build dependencies (“*-dev” packages)
Tool for simplified builds of native packages Written in Ruby Tool for simplified builds of native packages Written in Ruby fpm (Effing Package Management) fpm (Effing Package Management)
fpm Project Goals fpm Project Goals ● Simple way to create packages ● … without the need to memorize too much ● Help deliver software with minimal steps or training ● Easily build platform-native packages ● Do many things with one tool, including: ● Creating packages easily (deb, rpm, freebsd, etc) ● Tweaking existing packages (removing files, changing metadata/dependencies) ● Stripping pre/post/maintainer scripts from packages ● Simple way to create packages ● … without the need to memorize too much ● Help deliver software with minimal steps or training ● Easily build platform-native packages ● Do many things with one tool, including: ● Creating packages easily (deb, rpm, freebsd, etc) ● Tweaking existing packages (removing files, changing metadata/dependencies) ● Stripping pre/post/maintainer scripts from packages
Wrap Python package into Debian package Wrap Python package into Debian package ● Results conform to Debian policies ● Usually more work than omnibus packages ● Build main application or library package ● Hunt down the dependency tree ● If no standard package exists, build it ● Iterate (possibly a lot) ● OS dictates (some) dependency versions ● Results conform to Debian policies ● Usually more work than omnibus packages ● Build main application or library package ● Hunt down the dependency tree ● If no standard package exists, build it ● Iterate (possibly a lot) ● OS dictates (some) dependency versions → →
Motivation & Requirements Motivation & Requirements Mission: Monitor a Single Page App (SPA) in a browser. ● Main goals: ● Observability for SPAs / PWAs that run outside of your data center in the browser (and thus outside of classical instrumentation and monitoring) ● Early detection of problems of the user application (catch/report browser-side exceptions) ● Technical infrastructure to support these goals: ● Collection of raw data like metrics, errors / exceptions, and logging events ● Transformation of that data into different forms, and persist it to different targets as needed ● Correlation of client-side interactions with resulting server-side requests Mission: Monitor a Single Page App (SPA) in a browser. ● Main goals: ● Observability for SPAs / PWAs that run outside of your data center in the browser (and thus outside of classical instrumentation and monitoring) ● Early detection of problems of the user application (catch/report browser-side exceptions) ● Technical infrastructure to support these goals: ● Collection of raw data like metrics, errors / exceptions, and logging events ● Transformation of that data into different forms, and persist it to different targets as needed ● Correlation of client-side interactions with resulting server-side requests
jhermann
chiroruxx
uzulla
gen519
seike460
odasho
nttcom
foursue
6onoda
kubukoz
bun913
asazutaiga
tarao
quramy
chriscoyier
jacobian
cromwellryan
bluesmoon
jrom
eileencodes
paigeccino
searls
iamctodd
philhawksworth