Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOps Karlsruhe Meetup · 2018-02-20

DevOps Karlsruhe Meetup · 2018-02-20

DevOps Tool Bazaar · https://www.meetup.com/devops-karlsruhe/events/246690416/

* Packaging Python Software with dh-virtualenv and fpm
* Diagnosing Client-Side Errors with Sentry.io

Jürgen Hermann

February 20, 2018
Tweet

More Decks by Jürgen Hermann

Other Decks in Technology

Transcript

  1. Architecture Board
    Architecture Board
    Application Monitoring Service
    Application Monitoring Service
    Phase 1 – Metrics Gateway
    Phase 1 – Metrics Gateway
    Jürgen Hermann
    DevOps Karlsruhe Meetup · 2018-02-20
    DevOps Tool Bazaar
    DevOps Tool Bazaar
    dh-virtualenv · fpm · Sentry.io
    dh-virtualenv · fpm · Sentry.io
    jhermann
    [email protected]

    View Slide

  2. Python Packaging Overview
    Python Packaging Overview

    Distribution packages ≠ Python packages

    Source and binary distributions

    sdist (zip, tar.gz, tar.bz)

    bdist_wheel (whl)

    Repositories: PyPI · Artifactory · DevPI

    Distribution packages ≠ Python packages

    Source and binary distributions

    sdist (zip, tar.gz, tar.bz)

    bdist_wheel (whl)

    Repositories: PyPI · Artifactory · DevPI

    View Slide

  3. Core Python Packaging & Deployment Tools
    Core Python Packaging & Deployment Tools

    “setuptools” to build packages

    “twine” to upload packages to a repository

    “pip” to install packages from a repository

    “virtualenv” or “pyvenv” (python3 -m venv)
    to isolate dependencies

    “setuptools” to build packages

    “twine” to upload packages to a repository

    “pip” to install packages from a repository

    “virtualenv” or “pyvenv” (python3 -m venv)
    to isolate dependencies

    View Slide

  4. Linux Considerations (Binary Packages)
    Linux Considerations (Binary Packages)

    Wheel filenames include platform tags (PEP425)

    Python version, Python ABI, platform ABI (OS)

    Example: numpy-1.14.0-cp36-cp36m-macosx_10_10_x86_64.whl

    Linux presents a very diverse ABI spectrum

    Windows and MacOS have way less variation

    Thus PEP513 – manylinux1_{x86_64,i686}

    Restricted set of kernel / userspace ABIs

    Build Python binary packages in old CentOS images ☺

    Mostly works, with some quirks (e.g. stripping for packaging)

    Wheel filenames include platform tags (PEP425)

    Python version, Python ABI, platform ABI (OS)

    Example: numpy-1.14.0-cp36-cp36m-macosx_10_10_x86_64.whl

    Linux presents a very diverse ABI spectrum

    Windows and MacOS have way less variation

    Thus PEP513 – manylinux1_{x86_64,i686}

    Restricted set of kernel / userspace ABIs

    Build Python binary packages in old CentOS images ☺

    Mostly works, with some quirks (e.g. stripping for packaging)

    View Slide

  5. Omnibus
    Omnibus
    Packages
    Packages

    View Slide

  6. Omnibus Packages · Pros and Cons
    Omnibus Packages · Pros and Cons
    ⊕ Dependencies not predetermined (by target OS)
    ⊕ Multi-platform deployment way easier
    ⊕ Much simpler release process
    ⊖ Maintainer is responsible for security
    ⊖ Possibly more version variation in production
    ⓘ Security updates might be slower – or faster
    ⓘ Efficiency often dominates other concerns
    (esp. with deploying in-house tools)
    ⊕ Dependencies not predetermined (by target OS)
    ⊕ Multi-platform deployment way easier
    ⊕ Much simpler release process
    ⊖ Maintainer is responsible for security
    ⊖ Possibly more version variation in production
    ⓘ Security updates might be slower – or faster
    ⓘ Efficiency often dominates other concerns
    (esp. with deploying in-house tools)

    View Slide

  7. Tooling: dh-virtualenv
    Tooling: dh-virtualenv

    Native packaging of Python applications

    Debhelper extension

    Combines debhelper + virtualenv + pip

    Keeps “look & feel” of both Debian and Python tooling

    Minimal additional effort

    DEB Package Virtualenv Wheels (Dependencies)
    ⊃ ⊃

    Native packaging of Python applications

    Debhelper extension

    Combines debhelper + virtualenv + pip

    Keeps “look & feel” of both Debian and Python tooling

    Minimal additional effort

    DEB Package Virtualenv Wheels (Dependencies)
    ⊃ ⊃

    View Slide

  8. Installing dh-virtualenv 1.0
    Installing dh-virtualenv 1.0

    Basic DEB tooling:

    apt-get install build-essential debhelper devscripts equivs

    Stretch: A simple APT install

    Jessie:

    apt-get install -t jessie-backports dh-virtualenv

    Ubuntu LTS (Xenial):

    dpkg -i dh-virtualenv_1.0-1_all.deb # download from zesty

    Basic DEB tooling:

    apt-get install build-essential debhelper devscripts equivs

    Stretch: A simple APT install

    Jessie:

    apt-get install -t jessie-backports dh-virtualenv

    Ubuntu LTS (Xenial):

    dpkg -i dh-virtualenv_1.0-1_all.deb # download from zesty

    View Slide

  9. Example: "sentry.io" Debian Packaging
    Example: "sentry.io" Debian Packaging
    https://github.com/1and1/debianized-sentry

    Package a release from PyPI

    Add commonly used plugins

    Add systemd services

    Add default configuration
    https://github.com/1and1/debianized-sentry

    Package a release from PyPI

    Add commonly used plugins

    Add systemd services

    Add default configuration

    View Slide

  10. $ /opt/venvs/sentry/bin/pip freeze | column
    $ /opt/venvs/sentry/bin/pip freeze | column
    hiredis==0.1.6
    hiredis==0.1.6
    honcho==0.7.1
    honcho==0.7.1
    httplib2==0.10.3
    httplib2==0.10.3
    idna==2.6
    idna==2.6
    ipaddress==1.0.19
    ipaddress==1.0.19
    jmespath==0.9.3
    jmespath==0.9.3
    kafka-python==1.3.5
    kafka-python==1.3.5
    kombu==3.0.35
    kombu==3.0.35
    loremipsum==1.0.5
    loremipsum==1.0.5
    lxml==4.1.1
    lxml==4.1.1
    milksnake==0.1.1
    milksnake==0.1.1
    mistune==0.8.3
    mistune==0.8.3
    mmh3==2.3.1
    mmh3==2.3.1
    mock==1.0.1
    mock==1.0.1
    oauth2==1.9.0.post1
    oauth2==1.9.0.post1
    oauthlib==2.0.6
    oauthlib==2.0.6
    olefile==0.44
    olefile==0.44
    percy==1.0.8
    percy==1.0.8
    petname==2.0
    petname==2.0
    Pillow==4.2.1
    Pillow==4.2.1
    progressbar2==3.10.1
    progressbar2==3.10.1
    psycopg2==2.6.2
    psycopg2==2.6.2
    py==1.5.2
    py==1.5.2
    pycparser==2.18
    pycparser==2.18
    PyJWT==1.5.3
    PyJWT==1.5.3
    pyOpenSSL==17.5.0
    pyOpenSSL==17.5.0
    pytest==3.1.3
    pytest==3.1.3
    pytest-django==2.9.1
    pytest-django==2.9.1
    pytest-html==1.9.0
    pytest-html==1.9.0
    python-dateutil==2.6.1
    python-dateutil==2.6.1
    python-ldap==2.5.2
    python-ldap==2.5.2
    amqp==1.4.9
    amqp==1.4.9
    anyjson==0.3.3
    anyjson==0.3.3
    asn1crypto==0.24.0
    asn1crypto==0.24.0
    BeautifulSoup==3.2.1
    BeautifulSoup==3.2.1
    billiard==3.3.0.23
    billiard==3.3.0.23
    boto3==1.4.5
    boto3==1.4.5
    botocore==1.5.70
    botocore==1.5.70
    cached-property==1.3.1
    cached-property==1.3.1
    celery==3.1.18
    celery==3.1.18
    certifi==2017.11.5
    certifi==2017.11.5
    cffi==1.11.2
    cffi==1.11.2
    chardet==3.0.4
    chardet==3.0.4
    click==6.7
    click==6.7
    contextlib2==0.5.5
    contextlib2==0.5.5
    cryptography==2.1.4
    cryptography==2.1.4
    cssselect==1.0.1
    cssselect==1.0.1
    cssutils==0.9.10
    cssutils==0.9.10
    debianized-sentry==8.22.0.2
    debianized-sentry==8.22.0.2
    Django==1.6.11
    Django==1.6.11
    django-auth-ldap==1.2.16
    django-auth-ldap==1.2.16
    django-crispy-forms==1.4.0
    django-crispy-forms==1.4.0
    django-jsonfield==0.9.13
    django-jsonfield==0.9.13
    django-picklefield==0.3.2
    django-picklefield==0.3.2
    django-sudo==2.1.0
    django-sudo==2.1.0
    django-templatetag-sugar==1.0
    django-templatetag-sugar==1.0
    djangorestframework==2.4.8
    djangorestframework==2.4.8
    docutils==0.14
    docutils==0.14
    email-reply-parser==0.2.0
    email-reply-parser==0.2.0
    enum34==1.1.6
    enum34==1.1.6
    exam==0.10.6
    exam==0.10.6
    futures==3.2.0
    futures==3.2.0
    python-memcached==1.59
    python-memcached==1.59
    python-openid==2.2.5
    python-openid==2.2.5
    python-u2flib-server==4.0.1
    python-u2flib-server==4.0.1
    python-utils==2.2.0
    python-utils==2.2.0
    pytz==2017.3
    pytz==2017.3
    PyYAML==3.11
    PyYAML==3.11
    qrcode==5.3
    qrcode==5.3
    raven==5.33.0
    raven==5.33.0
    rb==1.7
    rb==1.7
    redis==2.10.5
    redis==2.10.5
    redis-py-cluster==1.3.4
    redis-py-cluster==1.3.4
    requests==2.18.4
    requests==2.18.4
    requests-oauthlib==0.8.0
    requests-oauthlib==0.8.0
    s3transfer==0.1.12
    s3transfer==0.1.12
    selenium==3.4.3
    selenium==3.4.3
    sentry==8.22.0
    sentry==8.22.0
    sentry-kafka==1.1
    sentry-kafka==1.1
    sentry-ldap-auth==2.5
    sentry-ldap-auth==2.5
    sentry-plugins==>8.22.0
    sentry-plugins==>8.22.0
    setproctitle==1.1.10
    setproctitle==1.1.10
    simplejson==3.8.2
    simplejson==3.8.2
    six==1.10.0
    six==1.10.0
    sqlparse==0.1.19
    sqlparse==0.1.19
    statsd==3.1
    statsd==3.1
    structlog==16.1.0
    structlog==16.1.0
    symbolic==1.1.5
    symbolic==1.1.5
    toronado==0.0.11
    toronado==0.0.11
    ua-parser==0.7.3
    ua-parser==0.7.3
    unidiff==0.5.4
    unidiff==0.5.4
    urllib3==1.22
    urllib3==1.22
    uWSGI==2.0.15
    uWSGI==2.0.15

    View Slide

  11. Binary Extension Packages Trouble-Shooting
    Binary Extension Packages Trouble-Shooting
    If stuff doesn‘t build OOTB…

    “dh_strip” and “dh_shlibdeps” exclusions

    Disable “manylinux1” wheels

    Pass the “--no-binary=:all:” option to pip

    … or add “no-manylinux1” to your build environment

    Increases build times (no wheels, or no extension ones)

    Often requires more build dependencies (“*-dev” packages)
    If stuff doesn‘t build OOTB…

    “dh_strip” and “dh_shlibdeps” exclusions

    Disable “manylinux1” wheels

    Pass the “--no-binary=:all:” option to pip

    … or add “no-manylinux1” to your build environment

    Increases build times (no wheels, or no extension ones)

    Often requires more build dependencies (“*-dev” packages)

    View Slide

  12. debian/control
    debian/control

    View Slide

  13. debian/rules
    debian/rules

    View Slide

  14. debian/prerm
    debian/prerm

    View Slide

  15. Springerle Project Templates
    Springerle Project Templates
    https://github.com/Springerle
    https://github.com/Springerle

    View Slide

  16. References
    References

    Python Packaging User Guide
    https://packaging.python.org/

    dh-virtualenv
    https://github.com/spotify/dh-virtualenv

    fpm (Effing Package Management)
    https://github.com/jordansissel/fpm

    How We Deploy Python Code (Nylas)
    https://www.nylas.com/blog/packaging-deploying-python/

    Python Packaging User Guide
    https://packaging.python.org/

    dh-virtualenv
    https://github.com/spotify/dh-virtualenv

    fpm (Effing Package Management)
    https://github.com/jordansissel/fpm

    How We Deploy Python Code (Nylas)
    https://www.nylas.com/blog/packaging-deploying-python/

    View Slide

  17. Questions?
    Questions?
    Thank you!
    Thank you!

    View Slide

  18. Tool for simplified builds
    of native packages
    Written in Ruby
    Tool for simplified builds
    of native packages
    Written in Ruby
    fpm (Effing Package Management)
    fpm (Effing Package Management)

    View Slide

  19. fpm Project Goals
    fpm Project Goals

    Simple way to create packages

    … without the need to memorize too much

    Help deliver software with minimal steps or training

    Easily build platform-native packages

    Do many things with one tool, including:

    Creating packages easily (deb, rpm, freebsd, etc)

    Tweaking existing packages (removing files, changing
    metadata/dependencies)

    Stripping pre/post/maintainer scripts from packages

    Simple way to create packages

    … without the need to memorize too much

    Help deliver software with minimal steps or training

    Easily build platform-native packages

    Do many things with one tool, including:

    Creating packages easily (deb, rpm, freebsd, etc)

    Tweaking existing packages (removing files, changing
    metadata/dependencies)

    Stripping pre/post/maintainer scripts from packages

    View Slide

  20. Sources and Targets
    Sources and Targets

    Package types:
    dir · gem · deb · npm · rpm ·
    tar · cpan · pear · empty · puppet · python · osxpkg · solaris ·
    p5p · pkgin · freebsd · apk · zip · sh · virtualenv · pacman ·
    pleaserun

    Sources: dir, gem, npm, python, …

    Targets: deb, apk, osxpkg, rpm, pacman, …

    Package types:
    dir · gem · deb · npm · rpm ·
    tar · cpan · pear · empty · puppet · python · osxpkg · solaris ·
    p5p · pkgin · freebsd · apk · zip · sh · virtualenv · pacman ·
    pleaserun

    Sources: dir, gem, npm, python, …

    Targets: deb, apk, osxpkg, rpm, pacman, …

    View Slide

  21. Wrap Python package into Debian package
    Wrap Python package into Debian package

    Results conform to Debian policies

    Usually more work than omnibus packages

    Build main application or library package

    Hunt down the dependency tree

    If no standard package exists, build it

    Iterate (possibly a lot)

    OS dictates (some)
    dependency versions

    Results conform to Debian policies

    Usually more work than omnibus packages

    Build main application or library package

    Hunt down the dependency tree

    If no standard package exists, build it

    Iterate (possibly a lot)

    OS dictates (some)
    dependency versions


    View Slide

  22. fpm and Python3
    fpm and Python3
    fpm --input-type python \
    --python-bin python3 \
    --python-pip pip3 \
    --python-package-name-prefix python3 \

    ¯\_( ツ )_/¯
    Vote for https://github.com/jordansissel/fpm/issues/1361
    fpm --input-type python \
    --python-bin python3 \
    --python-pip pip3 \
    --python-package-name-prefix python3 \

    ¯\_( ツ )_/¯
    Vote for https://github.com/jordansissel/fpm/issues/1361

    View Slide

  23. Questions?
    Questions?
    Thank you!
    Thank you!

    View Slide

  24. View Slide

  25. Motivation & Requirements
    Motivation & Requirements
    Mission: Monitor a Single Page App (SPA) in a browser.

    Main goals:

    Observability for SPAs / PWAs that run outside of your data center in the browser
    (and thus outside of classical instrumentation and monitoring)

    Early detection of problems of the user application
    (catch/report browser-side exceptions)

    Technical infrastructure to support these goals:

    Collection of raw data like metrics, errors / exceptions, and logging events

    Transformation of that data into different forms,
    and persist it to different targets as needed

    Correlation of client-side interactions with resulting server-side requests
    Mission: Monitor a Single Page App (SPA) in a browser.

    Main goals:

    Observability for SPAs / PWAs that run outside of your data center in the browser
    (and thus outside of classical instrumentation and monitoring)

    Early detection of problems of the user application
    (catch/report browser-side exceptions)

    Technical infrastructure to support these goals:

    Collection of raw data like metrics, errors / exceptions, and logging events

    Transformation of that data into different forms,
    and persist it to different targets as needed

    Correlation of client-side interactions with resulting server-side requests

    View Slide

  26. Solution Strategy – Errors & Exceptions
    Solution Strategy – Errors & Exceptions

    Sentry is a “modern realtime error logging and aggregation platform”

    Specialist system tailored to the problem domain
    (with lots of plugins and extensions)

    Excellent and mature client API libraries, e.g. raven.js

    Supports many client runtime environments
    (→ backend use)

    SaaS + On-Premise · Forum · Optional Paid Support

    Inception 2008 · Python · Django · Redis · PostgreSQL

    17948 commits · 322 releases · 292 contributors · BSD 3-Clause

    Sentry is a “modern realtime error logging and aggregation platform”

    Specialist system tailored to the problem domain
    (with lots of plugins and extensions)

    Excellent and mature client API libraries, e.g. raven.js

    Supports many client runtime environments
    (→ backend use)

    SaaS + On-Premise · Forum · Optional Paid Support

    Inception 2008 · Python · Django · Redis · PostgreSQL

    17948 commits · 322 releases · 292 contributors · BSD 3-Clause

    View Slide

  27. View Slide

  28. Weekly Report Email
    Weekly Report Email

    View Slide

  29. Sentry.io On-Premise Deployment
    Sentry.io On-Premise Deployment

    View Slide

  30. Questions?
    Questions?
    Thank you!
    Thank you!

    View Slide