The Python Deployment Albatross - PyTennessee 2017

Transcript

1. The Python Deployment Albatross CINDY SRIDHARAN @COPYCONSTRUCT PYTENNESSEE FEBRUARY 5,

   2017 NASHVILLE, TN

2. setup.py

3. What’s our goal?

4. Hermetically sealed, uniform, reproducible Python artifacts

5. Hermetically sealed

6. ✓ Isolate pure Python dependencies ✓ Isolate compile time native/non-Python

   dependencies ✓ Isolate runtime native/non-Python dependencies

7. uniform

8. Output of the build process is platform and architecture agnostic

9. Reproducible

10. A set of software development practices that create a verifiable

    path from human readable source code to the binary code used by computers.

11. What is Python? python hello_world.py Python – or /usr/bin/python –

    as your system understands it, is a program called the interpreter

12. How does Python know what to import from where? site.py

    sys.prefix sys.exec_prefix
13. None
14. None

15. WHEELS VIRTUALENV PEX DOCKER CONDA NIX

16. wheels

17. but before wheels there were …

18. eggs-ecutable

19. purely a distribution format wheels

20. no build system needed on target host no C compiler

    required wheels

21. wheels no arbitrary code execution like sdists Ergo faster installation

    pip builds and caches wheels by default

22. ergo less tied to a specific version of Python Creates

    .pyc files as a part of the installation wheels

23. manylinux wheels

24. None

25. virtualenv

26. helps “isolate Python environments”

27. ✓ Isolates per-project pure Python dependencies from one another virtualenv

28. virtualenv ✓ Isolates per-project pure Python dependencies from system Python

29. ✓ Isolates header files and shared libraries *if these are

    packaged* virtualenv

30. greenlet.h is installed local to the virtualenv

31. … as is greenlet.so

32. Where virtualenv falls short Uses system provided headers and .so

    files if not packaged

33. - - relocatable
 doesn’t always work

34. dh-virtualenv

35. PEX

36. Any directory with an __init__.py is considered a package Python

    import quirks

37. __init__.py

38. Any directory with a __main__.py is treated as an executable

    Python import quirks

39. __main__.py package is now executable

40. python –m package will execute package/__main__.py if it exists Python

    import quirks

41. Adding #!/usr/bin/env python to the beginning of any module makes

    it an executable Python executables

42. change permissions of file

43. Zipfiles A zipfile with an __init__.py is considered a package

44. Zipfiles A zipfile with a __main__.py is treated as an

    executable

45. zip file is now executable

46. ✓zip files don’t start until a magic zip number ✓

    can add arbitrary strings at the start of the file ✓ #!/usr/bin/env python PEX

47. zip files are also used at Facebook

48. None

49. pex file

50. None
51. None
52. None
53. None

54. Uses system provided headers and .so files if not packaged

    PEX

55. not cross-platform by default PEX

56. docker treats packaging as a namespacing problem

57. What does it mean to containerize a Python process?

58. Docker image for Python processes

59. BASE IMAGE DEVELOPMENT HEADERS AND LIBRARIES VIRTUALENV PEX

60. Best practices for building Docker images for Python ✓ small

    images ✓ always use a virtualenv or pex ✓ single process per container

61. Dockerflow

62. Challenges of containerization

63. None

64. The Docker engine is a container runtime Overlay Networking With

    1.12 in Swarm mode, it’s also a cluster scheduler Process manager … and much, much more (service discovery, load balancing, TLS ...) All compiled into one gigantic binary running as root

65. Logging Metrics Collection Observability Debugging

66. conda

67. CONDA or PIP?

68. PIP lacks a SAT solver

69. CONDA or WHEELS?

70. CONDA or VIRTUALENV?

71. CONDA or DOCKER?

72. VM ==> DOCKER :: DOCKER ==> CONDA

73. ✓ Python or other modules ✓ System-level libraries ✓ Executable

    programs conda package Can be downloaded from remote channels

74. all build dependencies need to be preinstalled in the build

    prefix tarball files generated by the build script to produce a package
75. None

76. NIX

77. referential transparency

78. An expression is said to be referentially transparent if evaluating

    it gives the same value for same arguments. Such functions are called pure functions.

79. nix expressions Nix expressions specify how to build nix packages,

    including, if necessary, their dependencies.

80. different users have different “views” of the system profiles

81. profiles

82. garbage collection any package not in use (no symlinks) by

    any generation of any profile

83. List of all dependencies, recursively, down to the bare minimum

    necessary to use that derivation closure

84. channels a URL that points to a place that contains

    a set of Nix expressions and a manifest

85. A use case for nix

86. ✓ Statically linked Objective-C, C and Lua code ✓ Every

    time there’s a MacOS upgrade, hosts need to be reimaged ✓ Application then needs to be recompiled ✓ A nix closure gets around this Why nix closures?

87. Conclusion ✓ Build wheels ✓ Use a virtualenv (or pex),

    even with Docker ✓ Build small Docker images ✓ Explore conda/nix only if needed ✓ Good Luck!

88. @copyconstruct