Debugging Live Python Web Applications

Slide 1

Slide 1 text

Debugging Live Python Web Applications Amjith Ramanujam / Graham Dumpleton DjangoCon US - Sep 2012 Tuesday, September 4, 12

Slide 2

Slide 2 text

Follow along. http://www.speakerdeck.com/u/amjith Tuesday, September 4, 12

Slide 3

Slide 3 text

Overview • Scenario • Monitoring • Analysis • Live Debugging • ispyd Tuesday, September 4, 12

Slide 4

Slide 4 text

Why Debug? Tuesday, September 4, 12

Slide 5

Slide 5 text

Common Problems • Obvious - Python exception and traceback • Subtle - Mem leak, Thread locking, Data corruption • Performance - Site is slow! • Heisen - Shows up only in production. Tuesday, September 4, 12

Slide 6

Slide 6 text

Developers Tuesday, September 4, 12

Slide 7

Slide 7 text

DEVOPS Tuesday, September 4, 12

Slide 8

Slide 8 text

Things to avoid Tuesday, September 4, 12

Slide 9

Slide 9 text

Things to avoid. • Crashing the whole web site. • Corrupt all your customer data. • Make you customer data visible to everyone. • Lose your company lots of money. Tuesday, September 4, 12

Slide 10

Slide 10 text

Things to avoid Tuesday, September 4, 12

Slide 11

Slide 11 text

Managing risk. • Use software to restrict what you can do. • Script changes and procedures to avoid errors. • Test what you are going to do on a separate system. • Develop and document contingency plans. Tuesday, September 4, 12

Slide 12

Slide 12 text

Monitoring Tuesday, September 4, 12

Slide 13

Slide 13 text

Passive monitoring. • Collection of log ﬁle information. • Collection of details about Python exceptions. • Collection of performance data for the server host. • Collection of performance data for the web server. • Collection of performance data for the web application. Tuesday, September 4, 12

Slide 14

Slide 14 text

Log ﬁle collation and analysis. • Open Source • logstash (http://logstash.net) • graylog2 (http://www.graylog2.org) • Commercial Services • Loggly (http://www.loggly.com) • Splunk (http://www.splunk.com) • LogLogic (http://www/loglogic.com) Tuesday, September 4, 12

Slide 15

Slide 15 text

Recording Python exceptions. • Open Source • Sentry (http://pypi.python.org/pypi/sentry) • Commercial Services • New Relic (http://newrelic.com) Tuesday, September 4, 12

Slide 16

Slide 16 text

Server monitoring. • Open Source • Monit (http://mmonit.com) • Munin (http://munin-monitoring.org) • Cacti (http://www.cacti.net) • Nagios (http://www.nagios.org) • Commercial Services • New Relic (http://newrelic.com) - Free Tuesday, September 4, 12

Slide 17

Slide 17 text

Application performance monitoring. • Commercial Services • New Relic (http://newrelic.com) - Lite (Free), Standard and Pro subscriptions. Tuesday, September 4, 12

Slide 18

Slide 18 text

Your web site is slow. Tuesday, September 4, 12

Slide 19

Slide 19 text

Analysis Tuesday, September 4, 12

Slide 20

Slide 20 text

Web page performance analysis. • Online services. • YSlow (http://developer.yahoo.com/yslow/) • GooglePageSpeed (https://developers.google.com/speed/pagespeed/) • WebPageTest (http://www.webpagetest.org/) • Browser plugins. • YSlow (https://addons.mozilla.org/en-US/ﬁrefox/addon/yslow/) • FireBug (http://getﬁrebug.com/) Tuesday, September 4, 12

Slide 21

Slide 21 text

Application performance analysis. Tuesday, September 4, 12

Slide 22

Slide 22 text

Slow transaction summary. Tuesday, September 4, 12

Slide 23

Slide 23 text

Slow transaction details. Tuesday, September 4, 12

Slide 24

Slide 24 text

Unknown consumers of time. ???? ???? Tuesday, September 4, 12

Slide 25

Slide 25 text

Instrumentation via code change. import newrelic.agent @newrelic.agent.function_trace() def open(file, flag=None, mode=0666): ... Tuesday, September 4, 12

Slide 26

Slide 26 text

Instrumentation via conﬁguration. [newrelic] transaction_tracer.function_trace = awesome_dbm:open awesome_dbm:_Database._commit Tuesday, September 4, 12

Slide 27

Slide 27 text

Instrumentation by monkey patching. #config [import-hook:awesome_dbm] enabled = true execute = awesome_dbm_instrumentation:instrument Tuesday, September 4, 12

Slide 28

Slide 28 text

Instrumentation by monkey patching. # awesome_dbm_instrument.py from newrelic.api.function_trace import \ wrap_function_trace def instrument(module): wrap_function_trace(module, 'open') wrap_function_trace(module, '_Database._commit') #config [import-hook:awesome_dbm] enabled = true execute = awesome_dbm_instrumentation:instrument Tuesday, September 4, 12

Slide 29

Slide 29 text

Profiling tools. • Thread sampling. • plop (http://tech.dropbox.com/?p=272) • statprof (http://pypi.python.org/pypi/statprof/) • Full profiling. • cprofile (http://docs.python.org/library/profile.html) • pytrace (http://pypi.python.org/pypi/pytrace) Tuesday, September 4, 12

Slide 30

Slide 30 text

Manual metric collection. • Open Source • metrology - http://metrology.readthedocs.org/en/latest/index.html • mmstats - https://github.com/schmichael/mmstats • pymetrics - https://github.com/jgardner1/Python-Metrics • django-app-metrics - http://pypi.python.org/pypi/django-app-metrics • django-statsd - http://django-statsd.readthedocs.org/en/latest/ Tuesday, September 4, 12

Slide 31

Slide 31 text

Interacting via the browser. • Open Source • Paste Error Middleware • Django-debug-toolbar • Paste Debugger • Flask Debugger Tuesday, September 4, 12

Slide 32

Slide 32 text

Live Debugging (Dark Art) Tuesday, September 4, 12

Slide 33

Slide 33 text

Application backdoors. import logging.config logging.config.fileConfig("logging.conf") backdoor = logging.config.listen() backdoor.start() Tuesday, September 4, 12

Slide 34

Slide 34 text

Interactive access. • Embedded interpreter prompt • eventlet.backdoor - http://eventlet.net/doc/modules/backdoor.html • guppy.heapy.Console - http://guppy-pe.sourceforge.net • twisted.manhole - http://www.lothar.com/tech/twisted/manhole.xhtml Tuesday, September 4, 12

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Introducing ispyd. • Download site. • https://github.com/GrahamDumpleton/wsgi-shell • Aims of the package. • Provide a generic framework for implementing an interactive console. • The commands you can run are targeted at a speciﬁc purpose. • Plugin based so can control what is available and also extendable. • Remotely accessible and execution of commands scriptable. Tuesday, September 4, 12

Slide 37

Slide 37 text

Connecting to processes. $ ispy ispyd.ini (ispyd) servers 1: (1, '/tmp/ispyd-14905.sock') 2: (1, '/tmp/ispyd-14906.sock') 3: (1, '/tmp/ispyd-14907.sock') (ispyd) connect 1 (ispyd:ll345) plugins ['debugger', 'process', 'python', 'wsgi'] Tuesday, September 4, 12

Slide 38

Slide 38 text

DEMO Tuesday, September 4, 12

Slide 39

Slide 39 text

Executing commands. (ispyd:ll345) shell process (process:ll345) help Documented commands (type help ): ======================================== cwd egid euid exit gid help pid prompt uid (process:ll345) cwd /Users/graham Tuesday, September 4, 12

Slide 40

Slide 40 text

Power users. (ispyd:ll345) shell python (python:ll345) console Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type "help", "copyright", "credits" or "license" for more information. (EmbeddedConsole) >>> import os >>> os.getcwd() '/Users/graham' >>> exit() Tuesday, September 4, 12

Slide 41

Slide 41 text

Post-mortem debugging. (ispyd:ll345) shell debugger (debugger:11345) insert __main__:function (debugger:11345) tracebacks {'__main__:function': } (debugger:11345) debug __main__:function > /Users/graham/wsgi.py(15)function() -> raise RuntimeError('xxx') (Pdb) dir() [] (Pdb) __file__ 'wsgi.py' Tuesday, September 4, 12

Slide 42

Slide 42 text

Active requests. (ispyd:ll345) shell requests (debugger:11345) requests ==== 707 ==== thread_id = 140735076232384 start_time = Mon Apr 9 21:49:54 2012 duration = 0.013629 seconds CONTENT_LENGTH = '' ... File: "wsgi.py", line 25, in application.run(host='0.0.0.0', port=port) ... Tuesday, September 4, 12

Slide 43

Slide 43 text

Extending what is monitored. (ispyd:ll345) shell newrelic (newrelic:ll345) function_trace awesome_dbm:open (newrelic:ll345) function_trace amesome_dbm:_Database._commit Tuesday, September 4, 12

Slide 44

Slide 44 text

Creating plugins. import psutil class Shell(object): name = 'psutil' def do_num_cpus(self, line): print >> self.stdout, psutil.NUM_CPUS def do_cpu_times(self, line): print >> self.stdout, psutil.cpu_times() def do_virtual_memory(self, line): print >> self.stdout, psutil.virtual_memory() def do_swap_memory(self, line): print >> self.stdout, psutil.swap_memory() Tuesday, September 4, 12

Slide 45

Slide 45 text

Ideas for third party plugins. • Memory. • Process memory usage. • Statistics on objects in use (heapy). • State of the garbage collector. • Profiling. • Initiate sampled profiling for selected functions. • Django. • Current configuration. • Details of loaded applications. • Details of registered middleware. • Details of template libraries. • Testing URLs against URL resolver. • Statistics on cache usage. Tuesday, September 4, 12

Slide 46

Slide 46 text

What am I trying to say? • Use monitoring so you know when problems arise. • One tool alone is not going to provide everything. • Use complimentary tools to get a full picture. • Build in mechanisms that allow deeper debugging. • Treat debugging like any other deﬁned process. Tuesday, September 4, 12

Slide 47

Slide 47 text

New Relic 30 Day Free Pro Trail http://newrelic.com/30 [email protected] @GrahamDumpleton [email protected] @amjithr Tuesday, September 4, 12