Solid Python Deployments for Everybody

Without orientation, deployments of Python applications can be tiresome and even painful. This talk attempts to replace anxiety and pain through informed annoyance.

There are extensive additional notes to this talk at http://hynek.me/talks/python-deployments/ .

Transcript

1. PyCon US, 2013
   Solid Python
   Application
   Deployments For
   Everybody
   Hynek Schlawack
   

   View Slide

2. View Slide

3. @hynek
   http://hynek.me
   http://github.com/hynek
   http://www.variomedia.de
   Hi!
   

   View Slide

4. ?
   

   View Slide

5. AHEAD
   

   View Slide

6. http://ox.cx/d
   The One & Only Link
   

   View Slide

7. OPINIONS
   AHEAD
   

   View Slide

8. PaaS
   Schema Migrations
   

   View Slide

9. View Slide

10. Key
    Concept
    

    View Slide

11. easy
    ≠
    simple
    

    View Slide

12. View Slide

13. “Simplicity is
    prerequisite
    for reliability.”
    — Edsger W. Dijkstra
    

    View Slide

14. “…and security.”
    — Every Credible Security Expert Ever
    

    View Slide

15. Put effort into
    making your
    deployments
    simple.
    

    View Slide

16. View Slide

17. Development
    

    View Slide

18. Development
    

    View Slide

19. View Slide

20. No!
    

    View Slide

21. View Slide

22. “Python 2.4 is not
    supported. It came out
    8 years ago. That's older
    than Youtube. Upgrade.”
    — Kenneth Reitz
    

    View Slide

23. Stable Platform
    Key Infrastructure?
    

    View Slide

24. But Hyyyn ek…
    My boss won’t
    let me!
    

    View Slide

25. Development
    tests!
    

    View Slide

26. View Slide

27. אל
    

    View Slide

28. spotty
    outdated
    loss of control
    System Package
    

    View Slide

29. spotty
    outdated
    loss of control
    System Package
    

    View Slide

30. spotty
    outdated
    loss of control
    System Package
    

    View Slide

31. View Slide

32. Use virtualen
    $ virtualenv venv; . venv/bin/activate
    $ pip install pyramid requests
    $ py.test
    …
    $ pip freeze >requirements.txt
    …
    $ pip install -r requirements.txt
    

    View Slide

33. Pin Dep Hard
    “Django == 1.4.3”
    Don’t rely on SemVer!
    update w/ pip-tools
    

    View Slide

34. But Hyyyn ek…
    SECURITY!
    

    View Slide

35. Security‽
    It’s your Job.
    

    View Slide

36. Package It
    

    View Slide

37. + git
    

    View Slide

38. + git
    Ne!
    

    View Slide

39. Fabric
    

    View Slide

40. build tools
    repetitive
    downloads
    What’ Wrong‽
    

    View Slide

41. View Slide

42. .rpm
    .deb
    .pkg.tgz
    

    View Slide

43. introspection
    CM integration
    versatile
    Native Package ‽
    

    View Slide

44. 1. check out from VCS
    2. create virtualenv
    3. install dependencies
    4. do whatever you want
    5. package result
    6. push to your repo
    

    View Slide

45. Abuse the Pipeline
    run tests
    LESS/SASS/CoffeeScript
    compression
    cache busting
    

    View Slide

46. Packaging is hard!
    But Hyyyn ek…
    

    View Slide

47. fpm
    Nope.
    

    View Slide

48. But Hyyyn ek…
    repo server
    

    View Slide

49. Rep Serve
    dpkg -i
    tar.bz2
    

    View Slide

50. Automate!
    

    View Slide

51. from … import Deployment
    def deb(branch=None):
    deploy = Deployment(
    'whois',
    build_deps=['libpq-dev',],
    run_deps=['libpq5',])
    deploy.prepare_app(
    branch=branch)
    deploy.build_deb()
    

    View Slide

52. Lazy?
    

    View Slide

53. There’ more than
    one way t d it…
    

    View Slide

54. View Slide

55. !ل
    

    View Slide

56. Configuration
    Management
    declarative
    describe the goal
    CM choses the path
    

    View Slide

57. Solution
    prise-oriented features to
    to compare the two
    pet Open
    ource
    Puppet
    Enterprise
    ✔
    ✔ ✔
    

    View Slide

58. prise-oriented features to
    to compare the two
    pet Open
    ource
    Puppet
    Enterprise
    ✔
    ✔ ✔
    Not easy at all.
    Solution
    

    View Slide

59. Why anyway?
    safety/security
    reproducible
    “later”
    

    View Slide

60. safety/security
    reproducible
    “later”
    Why anyway?
    

    View Slide

61. safety/security
    reproducible
    “later”
    Why anyway?
    

    View Slide

62. Kate Heddleston
    This Room:
    2:35 p.m.–3:05 p.m.
    Chef: Automating
    web application
    infrastructure
    

    View Slide

63. Test It in Staging
    

    View Slide

64. r t
    

    View Slide

65. r t
    Nein!
    

    View Slide

66. Just don’t.
    

    View Slide

67. Privileged Port
    drop privileges
    authbind
    

    View Slide

68. But Hyyyn ek…
    Need dat POWER!
    

    View Slide

69. Single Purpose
    Worke
    celery
    rq
    zerorpc
    pb/AMP
    

    View Slide

70. Be Paranoid
    /bin/false
    iptables
    file
    sockets
    REVOKE
    ALL
    SSL
    fail2ban
    

    View Slide

71. /bin/false
    iptables
    file
    sockets
    REVOKE
    ALL
    SSL
    fail2ban
    Be Paranoid
    

    View Slide

72. /bin/false
    iptables
    file
    sockets
    REVOKE
    ALL
    SSL
    fail2ban
    Be Paranoid
    

    View Slide

73. /bin/false
    iptables
    file
    sockets
    REVOKE
    ALL
    SSL
    fail2ban
    Be Paranoid
    

    View Slide

74. /bin/false
    iptables
    file
    sockets
    REVOKE
    ALL
    SSL
    fail2ban
    Be Paranoid
    

    View Slide

75. /bin/false
    iptables
    file
    sockets
    REVOKE
    ALL
    SSL
    fail2ban
    Be Paranoid
    

    View Slide

76. $ ./manage.py runserver ▌
    [0] 0:bash*
    

    View Slide

77. View Slide

78. $ ./manage.py runserver ▌
    [0] 0:bash*
    ᔒ༗!
    

    View Slide

79. It’ Easy!
    upstart
    systemd
    supervisord
    circus
    …
    

    View Slide

80. It’ Easy!
    upstart
    systemd
    supervisord
    circus
    …
    

    View Slide

81. Example: upstart
    $ cat /etc/init/yourapp.conf
    start on static-network-up
    stop on deconfiguring-networking
    respawn
    chdir /path/to/yourapp
    setuid yourapp
    exec /path/to/gunicorn_django settings.py
    $ start yourapp
    

    View Slide

82. + mod_wsgi
    

    View Slide

83. + mod_wsgi
    Нет!
    

    View Slide

84. Disclaime
    Using Apache is
    perfectly fine.
    

    View Slide

85. Iff you decide
    consciously
    for it.
    Disclaime
    

    View Slide

86. mod_wsgi
    

    View Slide

87. mod_wsgi ?
    ?
    

    View Slide

88. +
    g
    or
    

    View Slide

89. +
    g
    or
    Better separation
    of Concerns.
    

    View Slide

90. Easy t Set U : gunicorn
    $ gunicorn_django settings.py
    $ gunicorn_paster settings.ini
    

    View Slide

91. $ cat settings.py
    …
    INSTALLED_APPS = (
    …
    "gunicorn",
    )
    …
    $ manage.py run_gunicorn
    Easy t Set U : gunicorn
    

    View Slide

92. Easy t Set U : nginx
    location / {
    proxy_pass http://127.0.0.1:5000;
    }
    location /static/ {
    root /your/app/public/;
    }
    

    View Slide

93. Graham Dumpleton
    This Room:
    3:15 p.m.–3:45 p.m.
    Making Apache suck less for
    hosting Python web
    applications.
    

    View Slide

94. Deploy!
    

    View Slide

95. Ro back!
    

    View Slide

96. Monito
    

    View Slide

97. Monito
    

    View Slide

98. Measure
    

    View Slide

99. Measure
    statsd
    graphite
    scales
    

    View Slide

100. got 1
     

     View Slide

101. FIN
     http://ox.cx/d
     @hynek http://hynek.me
     http://vrmd.de
     

     View Slide