Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Just Enough Ops for Developers

Just Enough Ops for Developers

Originally presented at DjangoCon US 2022

Most developers don’t want to think about operations (aka Ops, DevOps, etc.) PaaS providers (Heroku, Fly, Render, etc.) do an awesome job of getting your app live on the internet, but there’s a lot to ops beyond just deployment.

This talk will answer the following questions:

* How much CPU and memory should I give my services?
* How do I know if the app is overallocated (costing too much money) or underallocated (slow/overloaded)?
* How can I make my app faster?
* Will autoscaling help me save costs?
* What about serverless?

If you’re a developer and want to run your applications successfully without deep DevOps knowledge, this talk is for you. It will help if you have some basic Django developer experience, but other than that, no specific knowledge is necessary!

Peter Baumgartner

October 18, 2022
Tweet

More Decks by Peter Baumgartner

Other Decks in Technology

Transcript

  1. Peter Baumgartner Just Enough Ops for Developers DjangoCon US 2022

    @ipmb lincolnloop.com
  2. About Me • Founder at Lincoln Loop — lincolnloop.com •

    Co-author of High Performance Django — highperformancedjango.com • Building AppPack — apppack.io
  3. Prepping your project for production Watch my talk at DjangoCon

    2019
  4. Why just enough Ops?

  5. PaaS & Managed Services are really good

  6. Forget about 😴 • System security/hardening • Routing/networking • Secrets

    management • Deployments • Scaling how to, not when to • Process Management (systemd, docker, etc.) • Hardware failures
  7. But! You need to understand the basics • CPU •

    RAM • I/O
  8. None
  9. None
  10. CompSci 101

  11. None
  12. Disk (HDD, SSD) Persistent File Storage

  13. • Disk access is slow • Usually ephemeral Disk (HDD,

    SSD) Persistent File Storage
  14. CPU (processor) Code execution

  15. How many CPUs do I need?

  16. How many CPUs do I need? It depends

  17. 1 request/1 process/1 CPU

  18. https://fastapi.tiangolo.com/async/ Cook = CPU Cashier = Python Process Customer =

    Request
  19. How many CPUs do I need? It depends

  20. How many cooks do I need? It depends

  21. More Users → More CPU Rule of 👍

  22. 2 requests/1 process/1 CPU

  23. 2 requests/2 process/2 CPU

  24. 💸

  25. How do I minimize cost?

  26. How do I maximize CPU usage?

  27. Multiple Processes per CPU How do I maximize CPU usage?

  28. 2 requests/2 processes/1 CPU

  29. 2 requests/2 processes/1 CPU

  30. 2 requests/2 processes/1 CPU

  31. gunicorn \ --workers=3 \ …

  32. Tuning Worker Count Each app is different • Start at

    double your CPU count • You can experiment with adding workers until • Not enough memory • Response times plateau or degrade
  33. Improve Application Performance How do I maximize CPU usage?

  34. 1 CPU / 1 minute 100ms response time = 6000

    requests 1s response time = 60 requests
  35. ⏳Slow responses ⏱More CPU time 💰Higher cost

  36. I/O CompSci 101

  37. I/O (Input/Output) Examples • Reading/writing files • Database queries •

    Object storage (S3) • Search index queries • Third-party APIs
  38. Everything waits for input/output

  39. None
  40. Synchronous I/O is blocking 🤓

  41. 🤓 Asynchronous I/O is non-blocking

  42. Async Django Views since Django 3.1 / ORM since Django

    4.1
  43. Asynchronous programming is hard 🤯

  44. Jeff Atwood “Hardware is cheap, Programmers are expensive

  45. gevent How do I maximize CPU usage?

  46. 🪄 magic How do I maximize CPU usage?

  47. 2 workers with gevent

  48. gunicorn \ --worker-class=gevent \ --worker-connections=50 \ …

  49. Not Enough CPU

  50. None
  51. Backlog

  52. Error Codes • 502 Bad Gateway • 503 Service Unavailable

    • 504 Gateway Timeout
  53. Memory (RAM) Ephemeral cache

  54. • “Warm” code cache • Python objects • File processing/manipulation

    • Network sockets • File descriptors Memory (RAM) Ephemeral cache
  55. • Fast • Limited Memory (RAM) Ephemeral cache

  56. Memory usage for a typical Django app

  57. 128 - 512+ MB Memory usage for a typical Django

    app
  58. Memory usage for a typical Django app 128 - 512+

    MB per process
  59. 👷 4 gunicorn workers 📊 512 MB per process 🟰

    2 GB of memory
  60. Over 100% memory → ☠

  61. Not enough memory? • ⬆ Increase allowed memory • ⬇

    Reduce processes/workers
  62. Memory usage should be stable

  63. Stable Memory Usage

  64. Variable (but Stable) Memory Usage

  65. Django memory tips • Don’t read huge files into a

    string/byte object • Don’t process a huge queryset • Use Model.objects.iterator() • Use .values() to avoid creating a model instance • Use .only() to avoid loading large text fields
  66. Memory Leak

  67. Memory Leak Causes • Bug in C extension (no garbage

    collection) • Leaving file descriptors or network sockets open (use context managers) • Global objects (sometimes accidental)
  68. gunicorn \ --max-requests=10000 \ --max-requests-jitter=500 \ …

  69. Databases

  70. Same rules apply

  71. Remember: disk is slow 🌋 Database should fit in RAM

    Rule of 👍
  72. Scaling

  73. None
  74. Horizontal Vertical

  75. Autoscaling is great 🎉

  76. Autoscaling is great 🎉

  77. …but it’s not magic 🪄

  78. Serverless

  79. 1 request/1 process/1 CPU Serverless

  80. Pricing Per request + Allocated resources per millisecond of response

    time Serverless
  81. • Scaling • CPU Allocation • Application server (workers, gevent,

    etc.) Forget about 😴 Serverless
  82. Worry about 😥 • Budgeting/variable costs • Cold starts •

    Database connections • Limitations (upload size, max duration) • Remote shell access Serverless
  83. Final Thoughts 🧐 • Get to know your application —

    “observability” • CPU usage • Memory usage • Response times • Error rates/uptime
  84. Thanks! 👋 @ipmb lincolnloop.com apppack.io