Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Python Deployment Albatross - PyTennessee 2017

The Python Deployment Albatross - PyTennessee 2017

Second stab at this talk.

1a73d340b0467da58dc2591b3a0edd5a?s=128

Cindy Sridharan

February 05, 2017
Tweet

Transcript

  1. The Python Deployment Albatross CINDY SRIDHARAN @COPYCONSTRUCT PYTENNESSEE FEBRUARY 5,

    2017 NASHVILLE, TN
  2. setup.py

  3. What’s our goal?

  4. Hermetically sealed, uniform, reproducible Python artifacts

  5. Hermetically sealed

  6. ✓ Isolate pure Python dependencies ✓ Isolate compile time native/non-Python

    dependencies ✓ Isolate runtime native/non-Python dependencies
  7. uniform

  8. Output of the build process is platform and architecture agnostic

  9. Reproducible

  10. A set of software development practices that create a verifiable

    path from human readable source code to the binary code used by computers.
  11. What is Python? python hello_world.py Python – or /usr/bin/python –

    as your system understands it, is a program called the interpreter
  12. How does Python know what to import from where? site.py

    sys.prefix sys.exec_prefix
  13. None
  14. None
  15. WHEELS VIRTUALENV PEX DOCKER CONDA NIX

  16. wheels

  17. but before wheels there were …

  18. eggs-ecutable

  19. purely a distribution format wheels

  20. no build system needed on target host no C compiler

    required wheels
  21. wheels no arbitrary code execution like sdists Ergo faster installation

    pip builds and caches wheels by default
  22. ergo less tied to a specific version of Python Creates

    .pyc files as a part of the installation wheels
  23. manylinux wheels

  24. None
  25. virtualenv

  26. helps “isolate Python environments”

  27. ✓ Isolates per-project pure Python dependencies from one another virtualenv

  28. virtualenv ✓ Isolates per-project pure Python dependencies from system Python

  29. ✓ Isolates header files and shared libraries *if these are

    packaged* virtualenv
  30. greenlet.h is installed local to the virtualenv

  31. … as is greenlet.so

  32. Where virtualenv falls short Uses system provided headers and .so

    files if not packaged
  33. - - relocatable
 doesn’t always work

  34. dh-virtualenv

  35. PEX

  36. Any directory with an __init__.py is considered a package Python

    import quirks
  37. __init__.py

  38. Any directory with a __main__.py is treated as an executable

    Python import quirks
  39. __main__.py package is now executable

  40. python –m package will execute package/__main__.py if it exists Python

    import quirks
  41. Adding #!/usr/bin/env python to the beginning of any module makes

    it an executable Python executables
  42. change permissions of file

  43. Zipfiles A zipfile with an __init__.py is considered a package

  44. Zipfiles A zipfile with a __main__.py is treated as an

    executable
  45. zip file is now executable

  46. ✓zip files don’t start until a magic zip number ✓

    can add arbitrary strings at the start of the file ✓ #!/usr/bin/env python PEX
  47. zip files are also used at Facebook

  48. None
  49. pex file

  50. None
  51. None
  52. None
  53. None
  54. Uses system provided headers and .so files if not packaged

    PEX
  55. not cross-platform by default PEX

  56. docker treats packaging as a namespacing problem

  57. What does it mean to containerize a Python process?

  58. Docker image for Python processes

  59. BASE IMAGE DEVELOPMENT HEADERS AND LIBRARIES VIRTUALENV PEX

  60. Best practices for building Docker images for Python ✓ small

    images ✓ always use a virtualenv or pex ✓ single process per container
  61. Dockerflow

  62. Challenges of containerization

  63. None
  64. The Docker engine is a container runtime Overlay Networking With

    1.12 in Swarm mode, it’s also a cluster scheduler Process manager … and much, much more (service discovery, load balancing, TLS ...) All compiled into one gigantic binary running as root
  65. Logging Metrics Collection Observability Debugging

  66. conda

  67. CONDA or PIP?

  68. PIP lacks a SAT solver

  69. CONDA or WHEELS?

  70. CONDA or VIRTUALENV?

  71. CONDA or DOCKER?

  72. VM ==> DOCKER :: DOCKER ==> CONDA

  73. ✓ Python or other modules ✓ System-level libraries ✓ Executable

    programs conda package Can be downloaded from remote channels
  74. all build dependencies need to be preinstalled in the build

    prefix tarball files generated by the build script to produce a package
  75. None
  76. NIX

  77. referential transparency

  78. An expression is said to be referentially transparent if evaluating

    it gives the same value for same arguments. Such functions are called pure functions.
  79. nix expressions Nix expressions specify how to build nix packages,

    including, if necessary, their dependencies.
  80. different users have different “views” of the system profiles

  81. profiles

  82. garbage collection any package not in use (no symlinks) by

    any generation of any profile
  83. List of all dependencies, recursively, down to the bare minimum

    necessary to use that derivation closure
  84. channels a URL that points to a place that contains

    a set of Nix expressions and a manifest
  85. A use case for nix

  86. ✓ Statically linked Objective-C, C and Lua code ✓ Every

    time there’s a MacOS upgrade, hosts need to be reimaged ✓ Application then needs to be recompiled ✓ A nix closure gets around this Why nix closures?
  87. Conclusion ✓ Build wheels ✓ Use a virtualenv (or pex),

    even with Docker ✓ Build small Docker images ✓ Explore conda/nix only if needed ✓ Good Luck!
  88. @copyconstruct