Web Framework Performance Examples from Django and Rails

Web Framework Performance Examples from Django and Rails

Talk from QCon San Francisco. Covers common performance problems and patterns, with examples from Django and Rails. Tips for working web developers and suggestions for framework authors.

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

November 09, 2012
Tweet

Transcript

  1. www.flickr.com/photos/mugley/5013931959/ Web Framework Performance Examples from Django and Rails QConSF

    9th November 2012 gareth rushgrove | morethanseven.net
  2. Me

  3. Gareth Rushgrove gareth rushgrove | morethanseven.net

  4. Curate devopsweekly.com gareth rushgrove | morethanseven.net

  5. Blog at morethanseven.net gareth rushgrove | morethanseven.net

  6. Work at UK Government Digital Service Text gareth rushgrove |

    morethanseven.net
  7. http://www.flickr.com/photos/benterrett/6852348725/ I am a Civil Servant gareth rushgrove | morethanseven.net

  8. Developer, Operations, Product Guy gareth rushgrove | morethanseven.net

  9. http://www.flickr.com/photos/iancarroll/5027441664 Introduction (what’s the problem)

  10. gareth rushgrove | morethanseven.net Slow(er) languages...

  11. gareth rushgrove | morethanseven.net Slow(er) languages...

  12. gareth rushgrove | morethanseven.net ...don’t mean slow applications

  13. gareth rushgrove | morethanseven.net ...don’t have to mean slow applications

  14. gareth rushgrove | morethanseven.net Frameworks can help

  15. gareth rushgrove | morethanseven.net Frameworks can help

  16. A real example gareth rushgrove | morethanseven.net

  17. gareth rushgrove | morethanseven.net A sample application

  18. 49ms gareth rushgrove | morethanseven.net Before

  19. 6ms gareth rushgrove | morethanseven.net After

  20. gareth rushgrove | morethanseven.net This presentation - Analyze your application

    - Know your framework - Cache everywhere - Instrument everything - Don’t just think about development
  21. http://www.flickr.com/photos/iancarroll/5027441664 Analyze (count everything)

  22. gareth rushgrove | morethanseven.net 1. From the browser

  23. gareth rushgrove | morethanseven.net YSlow

  24. gareth rushgrove | morethanseven.net webpagetest.org

  25. gareth rushgrove | morethanseven.net 2. From the code

  26. gareth rushgrove | morethanseven.net Django debug toolbar

  27. gareth rushgrove | morethanseven.net Django debug toolbar

  28. gareth rushgrove | morethanseven.net Django debug toolbar

  29. gareth rushgrove | morethanseven.net Django debug toolbar

  30. gareth rushgrove | morethanseven.net Profiling middleware

  31. gareth rushgrove | morethanseven.net rack-mini-profiler

  32. gareth rushgrove | morethanseven.net rack-mini-profiler details

  33. gareth rushgrove | morethanseven.net Rack Insight

  34. gareth rushgrove | morethanseven.net Rack Insight

  35. gareth rushgrove | morethanseven.net Rails footnotes

  36. gareth rushgrove | morethanseven.net New Relic development mode

  37. gareth rushgrove | morethanseven.net 3. Logs

  38. gareth rushgrove | morethanseven.net Request log analyzer

  39. gareth rushgrove | morethanseven.net Request log analyzer details ᴺ Mean

    ᴺ StdDev ᴺ Min ᴺ Max ᴺ 95 %tile ᴺ ᴺ 0.16s ᴺ 0.26s ᴺ 0.01s ᴺ 1.74s ᴺ 0.01s-1.08s ᴺ
  40. gareth rushgrove | morethanseven.net Django timelog

  41. http://www.flickr.com/photos/iancarroll/5027441664 Know your framework (use the parts you need)

  42. gareth rushgrove | morethanseven.net 1. Disable what you don’t need

  43. gareth rushgrove | morethanseven.net Don’t import rails require "rails/all" in

    config/application.rb
  44. gareth rushgrove | morethanseven.net Just the bits you need in

    config/application.rb require "action_controller/railtie" require "rails/test_unit/railtie" require "sprockets/railtie"
  45. gareth rushgrove | morethanseven.net Django middleware in settings.py MIDDLEWARE_CLASSES =

    ( 'django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', )
  46. gareth rushgrove | morethanseven.net Django installed apps in settings.py INSTALLED_APPS

    = ( 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', )
  47. gareth rushgrove | morethanseven.net 2. Know your ORM

  48. gareth rushgrove | morethanseven.net Django select_related()

  49. gareth rushgrove | morethanseven.net N+1 problem queryset = Release.objects.all() 11

    SQL Queries
  50. gareth rushgrove | morethanseven.net Joins to the rescue queryset =

    Release.objects.select_related() 1 SQL Query
  51. gareth rushgrove | morethanseven.net Active Record Include

  52. gareth rushgrove | morethanseven.net N+1 problem Release.all 11 SQL Queries

  53. gareth rushgrove | morethanseven.net Joins to the rescue Release.includes(:app).all 1

    SQL Query
  54. gareth rushgrove | morethanseven.net 3. Asset compilation in HTML <link

    href="/assets/application.css?body=1" ... <link href="/assets/apps.css?body=1" ... <link href="/assets/bootstrap_and_overrides.css?body=1" ... <link href="/assets/releases.css?body=1" ... <link href="/assets/sample.css?body=1" ... <link href="/assets/scaffolds.css?body=1" ... to this <link href="/assets/application.css" ...
  55. gareth rushgrove | morethanseven.net Asset compilation in CSS h1 {

    padding-top: 40px; } /*! * Bootstrap v2.2.1 * * Copyright 2012 Twitter, Inc * Licensed under the Apache License v2.0 * http://www.apache.org/licenses/LICENSE-2.0 * * Designed and built with all the love in the world @twitter by @mdo and @fat. */ article, aside, details, figcaption, figure, footer, header, hgroup, nav, section { display: block; } to this h1{padding-top:40px} article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{displ ay:block}
  56. gareth rushgrove | morethanseven.net Asset compilation configuration # Enable the

    asset pipeline config.assets.enabled = true # Version of your assets config.assets.version = '1.0' in config/application.rb # compress assets config.assets.compress = true # Don’t expands the lines which load the assets config.assets.debug = false in config/environments/production.rb
  57. gareth rushgrove | morethanseven.net 4. Different runtimes

  58. gareth rushgrove | morethanseven.net Different runtimes - JRuby

  59. gareth rushgrove | morethanseven.net Different runtimes - PyPy

  60. gareth rushgrove | morethanseven.net 5. Performance tests require 'test_helper' require

    'rails/performance_test_help' class ReleaseTest < ActionDispatch::PerformanceTest self.profile_options = { :runs => 10, :metrics => [:wall_time]} def test_release_index get '/releases' end end in test/performance/releases_test.rb
  61. gareth rushgrove | morethanseven.net Rails benchmarker rake test:benchmark Started ReleaseTest#test_release_index

    (110 ms warmup) wall_time: 6 ms Finished in 0.551815 seconds. 1 tests, 0 assertions, 0 failures, 0 errors, 0 skips
  62. http://www.flickr.com/photos/iancarroll/5027441664 Cache (everything)

  63. gareth rushgrove | morethanseven.net 1. Built-in caching support

  64. gareth rushgrove | morethanseven.net Great documentation 1

  65. gareth rushgrove | morethanseven.net Great documentation 2

  66. gareth rushgrove | morethanseven.net 2. ORM Caching

  67. CACHES = { 'default' : dict( BACKEND = 'johnny.backends.memcached.MemcachedCache' LOCATION

    = ['127.0.0.1:11211'], JOHNNY_CACHE = True, ) } MIDDLEWARE_CLASSES = ( 'johnny.middleware.LocalStoreClearMiddleware', 'johnny.middleware.QueryCacheMiddleware', ) + MIDDLEWARE_CLASSES in test/performance/releases_test.rb gareth rushgrove | morethanseven.net Johnny Cache
  68. gareth rushgrove | morethanseven.net First request 4 Queries

  69. gareth rushgrove | morethanseven.net Subsequent requests 0 Queries

  70. class Release < ActiveRecord::Base acts_as_cached after_save :expire_cache attr_accessible :app_id belongs_to

    :app def self.recent includes(:app).al end end gareth rushgrove | morethanseven.net Cache-fu
  71. class Release < ActiveRecord::Base acts_as_cached after_save :expire_cache attr_accessible :app_id belongs_to

    :app def self.recent includes(:app).al end end gareth rushgrove | morethanseven.net Add cache behaviour
  72. @releases = Release.cached(:recent) gareth rushgrove | morethanseven.net Get cached content

  73. gareth rushgrove | morethanseven.net First request 2 Queries

  74. gareth rushgrove | morethanseven.net Subsequent requests 0 Queries

  75. gareth rushgrove | morethanseven.net 3. HTTP is your friend http://www.mnot.net/cache_docs/

  76. from django.views.decorators.cache import cache_control @cache_control(public=True, max_age=3600) gareth rushgrove | morethanseven.net

    HTTP headers
  77. class ReleasesController < ApplicationController def index expires_in 60.minute, :public =>

    true gareth rushgrove | morethanseven.net HTTP headers in app/controllers/releases_controller.rb
  78. INSTALLED_APPS = INSTALLED_APPS + ( 'varnishapp', ) VARNISH_MANAGEMENT_ADDRS = (

    'localhost:6082', ) VARNISH_WATCHED_MODELS = ('app.release',) in settings.py gareth rushgrove | morethanseven.net django-varnish configuration
  79. gareth rushgrove | morethanseven.net django-varnish admin

  80. http://www.flickr.com/photos/iancarroll/5027441664 Instrument (monitor all the things)

  81. MIDDLEWARE_CLASSES = MIDDLEWARE_CLASSES + ( 'django_statsd.middleware.GraphiteRequestTimingMiddleware', 'django_statsd.middleware.GraphiteMiddleware', ) STATSD_PATCHES =

    [ 'django_statsd.patches.db', 'django_statsd.patches.cache', ] INSTALLED_APPS = INSTALLED_APPS + ( 'django_statsd', ) from settings.py gareth rushgrove | morethanseven.net django-statsd
  82. gareth rushgrove | morethanseven.net django-statsd on debug toolbar

  83. gareth rushgrove | morethanseven.net django-mmstats MIDDLEWARE_CLASSES = ( 'django_mmstats.middleware.MmStatsMiddleware', )

    import stats MMSTATS_CLASS = stats.DjangoStats from settings.py from django_mmstats.base import BaseDjangoStats class DjangoStats(BaseDjangoStats): """Add mmstats fields here, just like Django models!""" from stats.py
  84. gareth rushgrove | morethanseven.net mmash

  85. gareth rushgrove | morethanseven.net librato-rails

  86. http://www.flickr.com/photos/iancarroll/5027441664 Not just in development (measure in production)

  87. gareth rushgrove | morethanseven.net 1. Logs (again)

  88. gareth rushgrove | morethanseven.net Logster

  89. gareth rushgrove | morethanseven.net Lograge

  90. Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100 Processing

    by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms) to this GET /jobs/833552.json format=json action=jobs#show status=200 duration=58.33 view=40.43 db=15.26 gareth rushgrove | morethanseven.net Lograge
  91. gareth rushgrove | morethanseven.net Logstash

  92. LOGRAGE %{WORD:method}%{SPACE}%{DATA}% {SPACE}action=%{WORD:controller}#% {WORD:action}%{SPACE}status=%{INT:status}% {SPACE}duration=%{NUMBER:duration}%{SPACE} view=%{NUMBER:view}(%{SPACE}db=% {NUMBER:db})?%{GREEDYDATA} gareth rushgrove |

    morethanseven.net Logstash plus lograge
  93. LOGRAGE %{WORD:method}%{SPACE}%{DATA}% {SPACE}action=%{WORD:controller}#% {WORD:action}%{SPACE}status=%{INT:status}% {SPACE}duration=%{NUMBER:duration}%{SPACE} view=%{NUMBER:view}(%{SPACE}db=% {NUMBER:db})?%{GREEDYDATA} gareth rushgrove |

    morethanseven.net Rails performance data
  94. output { statsd { host => "localhost" tags => [

    "lograge" ] timing => [ "<%= @title %>.%{controller}.%{action}.% {method}.duration", "%{duration}" ] } statsd { host => “localhost” tags => [ "lograge" ] timing => [ "<%= @title %>.%{controller}.%{action}.% {method}.view", "%{view}" ] } } gareth rushgrove | morethanseven.net Output to statsd
  95. gareth rushgrove | morethanseven.net 2. Metrics

  96. gareth rushgrove | morethanseven.net Ganglia

  97. gareth rushgrove | morethanseven.net Graphite

  98. gareth rushgrove | morethanseven.net Riemann

  99. gareth rushgrove | morethanseven.net New Relic newrelic.com

  100. gareth rushgrove | morethanseven.net Librato Metrics metrics.librato.com

  101. http://www.flickr.com/photos/iancarroll/5027441664 Conclusions (if all you remember is)

  102. gareth rushgrove | morethanseven.net Tooling helps. A lot.

  103. gareth rushgrove | morethanseven.net Your framework should have - A

    debug toolbar - Transparent caching support - Hooks for instrumentation - Configurable logging
  104. gareth rushgrove | morethanseven.net If not... build them

  105. gareth rushgrove | morethanseven.net Share everything

  106. The End

  107. Thanks for the amazing photos gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/

    www.flickr.com/photos/snugglepup/
  108. Questions? gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/