Web Framework Performance Examples from Django and Rails

Talk from QCon San Francisco. Covers common performance problems and patterns, with examples from Django and Rails. Tips for working web developers and suggestions for framework authors.

Transcript

1. www.flickr.com/photos/mugley/5013931959/ Web Framework Performance Examples from Django and Rails QConSF 9th November 2012 gareth rushgrove | morethanseven.net
2. Me
3. Gareth Rushgrove gareth rushgrove | morethanseven.net
4. Curate devopsweekly.com gareth rushgrove | morethanseven.net
5. Blog at morethanseven.net gareth rushgrove | morethanseven.net
6. Work at UK Government Digital Service Text gareth rushgrove | morethanseven.net
7. http://www.flickr.com/photos/benterrett/6852348725/ I am a Civil Servant gareth rushgrove | morethanseven.net
8. Developer, Operations, Product Guy gareth rushgrove | morethanseven.net
9. http://www.flickr.com/photos/iancarroll/5027441664 Introduction (what’s the problem)
10. gareth rushgrove | morethanseven.net Slow(er) languages...
11. gareth rushgrove | morethanseven.net Slow(er) languages...
12. gareth rushgrove | morethanseven.net ...don’t mean slow applications
13. gareth rushgrove | morethanseven.net ...don’t have to mean slow applications
14. gareth rushgrove | morethanseven.net Frameworks can help
15. gareth rushgrove | morethanseven.net Frameworks can help
16. A real example gareth rushgrove | morethanseven.net
17. gareth rushgrove | morethanseven.net A sample application
18. 49ms gareth rushgrove | morethanseven.net Before
19. 6ms gareth rushgrove | morethanseven.net After
20. gareth rushgrove | morethanseven.net This presentation - Analyze your application - Know your framework - Cache everywhere - Instrument everything - Don’t just think about development
21. http://www.flickr.com/photos/iancarroll/5027441664 Analyze (count everything)
22. gareth rushgrove | morethanseven.net 1. From the browser
23. gareth rushgrove | morethanseven.net YSlow
24. gareth rushgrove | morethanseven.net webpagetest.org
25. gareth rushgrove | morethanseven.net 2. From the code
26. gareth rushgrove | morethanseven.net Django debug toolbar
27. gareth rushgrove | morethanseven.net Django debug toolbar
28. gareth rushgrove | morethanseven.net Django debug toolbar
29. gareth rushgrove | morethanseven.net Django debug toolbar
30. gareth rushgrove | morethanseven.net Profiling middleware
31. gareth rushgrove | morethanseven.net rack-mini-profiler
32. gareth rushgrove | morethanseven.net rack-mini-profiler details
33. gareth rushgrove | morethanseven.net Rack Insight
34. gareth rushgrove | morethanseven.net Rack Insight
35. gareth rushgrove | morethanseven.net Rails footnotes
36. gareth rushgrove | morethanseven.net New Relic development mode
37. gareth rushgrove | morethanseven.net 3. Logs
38. gareth rushgrove | morethanseven.net Request log analyzer
39. gareth rushgrove | morethanseven.net Request log analyzer details ᴺ Mean ᴺ StdDev ᴺ Min ᴺ Max ᴺ 95 %tile ᴺ ᴺ 0.16s ᴺ 0.26s ᴺ 0.01s ᴺ 1.74s ᴺ 0.01s-1.08s ᴺ
40. gareth rushgrove | morethanseven.net Django timelog
41. http://www.flickr.com/photos/iancarroll/5027441664 Know your framework (use the parts you need)
42. gareth rushgrove | morethanseven.net 1. Disable what you don’t need
43. gareth rushgrove | morethanseven.net Don’t import rails require "rails/all" in config/application.rb
44. gareth rushgrove | morethanseven.net Just the bits you need in config/application.rb require "action_controller/railtie" require "rails/test_unit/railtie" require "sprockets/railtie"
45. gareth rushgrove | morethanseven.net Django middleware in settings.py MIDDLEWARE_CLASSES = ( 'django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', )
46. gareth rushgrove | morethanseven.net Django installed apps in settings.py INSTALLED_APPS = ( 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', )
47. gareth rushgrove | morethanseven.net 2. Know your ORM
48. gareth rushgrove | morethanseven.net Django select_related()
49. gareth rushgrove | morethanseven.net N+1 problem queryset = Release.objects.all() 11 SQL Queries
50. gareth rushgrove | morethanseven.net Joins to the rescue queryset = Release.objects.select_related() 1 SQL Query
51. gareth rushgrove | morethanseven.net Active Record Include
52. gareth rushgrove | morethanseven.net N+1 problem Release.all 11 SQL Queries
53. gareth rushgrove | morethanseven.net Joins to the rescue Release.includes(:app).all 1 SQL Query
54. gareth rushgrove | morethanseven.net 3. Asset compilation in HTML <link href="/assets/application.css?body=1" ... <link href="/assets/apps.css?body=1" ... <link href="/assets/bootstrap_and_overrides.css?body=1" ... <link href="/assets/releases.css?body=1" ... <link href="/assets/sample.css?body=1" ... <link href="/assets/scaffolds.css?body=1" ... to this <link href="/assets/application.css" ...
55. gareth rushgrove | morethanseven.net Asset compilation in CSS h1 { padding-top: 40px; } /*! * Bootstrap v2.2.1 * * Copyright 2012 Twitter, Inc * Licensed under the Apache License v2.0 * http://www.apache.org/licenses/LICENSE-2.0 * * Designed and built with all the love in the world @twitter by @mdo and @fat. */ article, aside, details, figcaption, figure, footer, header, hgroup, nav, section { display: block; } to this h1{padding-top:40px} article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{displ ay:block}
56. gareth rushgrove | morethanseven.net Asset compilation configuration # Enable the asset pipeline config.assets.enabled = true # Version of your assets config.assets.version = '1.0' in config/application.rb # compress assets config.assets.compress = true # Don’t expands the lines which load the assets config.assets.debug = false in config/environments/production.rb
57. gareth rushgrove | morethanseven.net 4. Different runtimes
58. gareth rushgrove | morethanseven.net Different runtimes - JRuby
59. gareth rushgrove | morethanseven.net Different runtimes - PyPy
60. gareth rushgrove | morethanseven.net 5. Performance tests require 'test_helper' require 'rails/performance_test_help' class ReleaseTest < ActionDispatch::PerformanceTest self.profile_options = { :runs => 10, :metrics => [:wall_time]} def test_release_index get '/releases' end end in test/performance/releases_test.rb
61. gareth rushgrove | morethanseven.net Rails benchmarker rake test:benchmark Started ReleaseTest#test_release_index (110 ms warmup) wall_time: 6 ms Finished in 0.551815 seconds. 1 tests, 0 assertions, 0 failures, 0 errors, 0 skips
62. http://www.flickr.com/photos/iancarroll/5027441664 Cache (everything)
63. gareth rushgrove | morethanseven.net 1. Built-in caching support
64. gareth rushgrove | morethanseven.net Great documentation 1
65. gareth rushgrove | morethanseven.net Great documentation 2
66. gareth rushgrove | morethanseven.net 2. ORM Caching
67. CACHES = { 'default' : dict( BACKEND = 'johnny.backends.memcached.MemcachedCache' LOCATION = ['127.0.0.1:11211'], JOHNNY_CACHE = True, ) } MIDDLEWARE_CLASSES = ( 'johnny.middleware.LocalStoreClearMiddleware', 'johnny.middleware.QueryCacheMiddleware', ) + MIDDLEWARE_CLASSES in test/performance/releases_test.rb gareth rushgrove | morethanseven.net Johnny Cache
68. gareth rushgrove | morethanseven.net First request 4 Queries
69. gareth rushgrove | morethanseven.net Subsequent requests 0 Queries
70. class Release < ActiveRecord::Base acts_as_cached after_save :expire_cache attr_accessible :app_id belongs_to :app def self.recent includes(:app).al end end gareth rushgrove | morethanseven.net Cache-fu
71. class Release < ActiveRecord::Base acts_as_cached after_save :expire_cache attr_accessible :app_id belongs_to :app def self.recent includes(:app).al end end gareth rushgrove | morethanseven.net Add cache behaviour
72. @releases = Release.cached(:recent) gareth rushgrove | morethanseven.net Get cached content
73. gareth rushgrove | morethanseven.net First request 2 Queries
74. gareth rushgrove | morethanseven.net Subsequent requests 0 Queries
75. gareth rushgrove | morethanseven.net 3. HTTP is your friend http://www.mnot.net/cache_docs/
76. from django.views.decorators.cache import cache_control @cache_control(public=True, max_age=3600) gareth rushgrove | morethanseven.net HTTP headers
77. class ReleasesController < ApplicationController def index expires_in 60.minute, :public => true gareth rushgrove | morethanseven.net HTTP headers in app/controllers/releases_controller.rb
78. INSTALLED_APPS = INSTALLED_APPS + ( 'varnishapp', ) VARNISH_MANAGEMENT_ADDRS = ( 'localhost:6082', ) VARNISH_WATCHED_MODELS = ('app.release',) in settings.py gareth rushgrove | morethanseven.net django-varnish configuration
79. gareth rushgrove | morethanseven.net django-varnish admin
80. http://www.flickr.com/photos/iancarroll/5027441664 Instrument (monitor all the things)
81. MIDDLEWARE_CLASSES = MIDDLEWARE_CLASSES + ( 'django_statsd.middleware.GraphiteRequestTimingMiddleware', 'django_statsd.middleware.GraphiteMiddleware', ) STATSD_PATCHES = [ 'django_statsd.patches.db', 'django_statsd.patches.cache', ] INSTALLED_APPS = INSTALLED_APPS + ( 'django_statsd', ) from settings.py gareth rushgrove | morethanseven.net django-statsd
82. gareth rushgrove | morethanseven.net django-statsd on debug toolbar
83. gareth rushgrove | morethanseven.net django-mmstats MIDDLEWARE_CLASSES = ( 'django_mmstats.middleware.MmStatsMiddleware', ) import stats MMSTATS_CLASS = stats.DjangoStats from settings.py from django_mmstats.base import BaseDjangoStats class DjangoStats(BaseDjangoStats): """Add mmstats fields here, just like Django models!""" from stats.py
84. gareth rushgrove | morethanseven.net mmash
85. gareth rushgrove | morethanseven.net librato-rails
86. http://www.flickr.com/photos/iancarroll/5027441664 Not just in development (measure in production)
87. gareth rushgrove | morethanseven.net 1. Logs (again)
88. gareth rushgrove | morethanseven.net Logster
89. gareth rushgrove | morethanseven.net Lograge
90. Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms) to this GET /jobs/833552.json format=json action=jobs#show status=200 duration=58.33 view=40.43 db=15.26 gareth rushgrove | morethanseven.net Lograge
91. gareth rushgrove | morethanseven.net Logstash
92. LOGRAGE %{WORD:method}%{SPACE}%{DATA}% {SPACE}action=%{WORD:controller}#% {WORD:action}%{SPACE}status=%{INT:status}% {SPACE}duration=%{NUMBER:duration}%{SPACE} view=%{NUMBER:view}(%{SPACE}db=% {NUMBER:db})?%{GREEDYDATA} gareth rushgrove | morethanseven.net Logstash plus lograge
93. LOGRAGE %{WORD:method}%{SPACE}%{DATA}% {SPACE}action=%{WORD:controller}#% {WORD:action}%{SPACE}status=%{INT:status}% {SPACE}duration=%{NUMBER:duration}%{SPACE} view=%{NUMBER:view}(%{SPACE}db=% {NUMBER:db})?%{GREEDYDATA} gareth rushgrove | morethanseven.net Rails performance data
94. output { statsd { host => "localhost" tags => [ "lograge" ] timing => [ "<%= @title %>.%{controller}.%{action}.% {method}.duration", "%{duration}" ] } statsd { host => “localhost” tags => [ "lograge" ] timing => [ "<%= @title %>.%{controller}.%{action}.% {method}.view", "%{view}" ] } } gareth rushgrove | morethanseven.net Output to statsd
95. gareth rushgrove | morethanseven.net 2. Metrics
96. gareth rushgrove | morethanseven.net Ganglia
97. gareth rushgrove | morethanseven.net Graphite
98. gareth rushgrove | morethanseven.net Riemann
99. gareth rushgrove | morethanseven.net New Relic newrelic.com
100. gareth rushgrove | morethanseven.net Librato Metrics metrics.librato.com
101. http://www.flickr.com/photos/iancarroll/5027441664 Conclusions (if all you remember is)
102. gareth rushgrove | morethanseven.net Tooling helps. A lot.
103. gareth rushgrove | morethanseven.net Your framework should have - A debug toolbar - Transparent caching support - Hooks for instrumentation - Configurable logging
104. gareth rushgrove | morethanseven.net If not... build them
105. gareth rushgrove | morethanseven.net Share everything
106. The End
107. Thanks for the amazing photos gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/ www.flickr.com/photos/snugglepup/
108. Questions? gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/