Tracing Error Like A Boss

Transcript

1. Tracing Error Like A Boss (with Sentry and Docker) A presentation by Isman Firmansyah
2. $ whoami a backend engineer // on GitHub // on Twitter iromli @iromli
3. Once Upon A Time Based on true story.
4. Life's Good New app is finished — developed, tested, and deployed to production without errors.
5. Time Goes On ... and On More features, dependencies, typos Missed testcases, sloppy deploys
6. Life's (not) Good My app is broken; it was running fine few days ago.
7. We Tend To Do Been there, done that.
8. The Naive Way 1. SSH to remote server 2. Finding the needle in a haystack (a.k.a. log file) 3. Update the code in dev env and re-deploy
9. The Lazy Way 1. Wait for email sent by app for each error (with full traceback) 2. Update the code in dev env and re-deploy
10. Unpleasant Moments
11. We Need Sane Approach At least i need it!
12. Error Tracing For Humans™ 1. Wait for email sent by app for unique error (with less traceback) 2. See the full traceback in web UI 3. Update the code in dev env and re-deploy
13. Sentry http://getsentry.com/ Sentry is a realtime, platform-agnostic error logging and aggregation platform.
14. Why Sentry It's opensource — transparent and active development Battle-tested at Trusted by notable shops ( , , etc) Host your own or pay the service Errors are groupped, email sent per error group Nice UI Disqus Pinterest Instagram
15. Client Libraries
16. Sentry Plugins ... and many more https://github.com/getsentry/sentry-github https://github.com/gisce/sentry-irc https://github.com/disqus/sentry-graphite https://github.com/ecarreras/sentry-whatsapp
17. Realtime Events & Aggregation
18. Email
19. Traceback
20. Case Closed? Well, depends on your next decision.
21. The Easiest Way Paid Sentry Service Case is closed ... but then the next sections are pointless
22. The D.I.Y. Way Host Your Own Sentry Instance What could be wrong? You don't familiar with Sentry/Django/Python You might have Rock Lee's syndromes
23. Life's Good pt. 2 Installing Sentry on my local Ubuntu machine
24. Life's (not) Good pt. 2 Sentry doesn't work on my CentOS production server
25. You Have Dev-Prod Parity Problem TL;DR http://12factor.net/dev-prod-parity A state where dev != production environment.
26. Docker http://docker.io/ From its official documentation: docker, the Linux Container Runtime, runs Unix processes with strong guarantees of isolation across servers. Your software runs repeatably everywhere because its Container includes any dependencies.
27. Docker: From My Own Perspective Strong foundation — built by people who run DotCloud A Linux container with steroids: lightweight batteries-included — git-like experience, registry, etc. opensource community awesomeness — 7696 GH stars, 1006 forks A handy tool to tackle dev-prod parity PaaS
28. Learn Docker The Hard Way Official documentation Guidebook Videos http://docs.docker.io/ https://github.com/kencochrane/docker-guidebook http://www.youtube.com/results? search_query=docker+dotcloud
29. Talk Coverage Simulation of a broken app Use Sentry to trace the errors Use Docker container to ship Sentry No email integration (leave it as practice for others)
30. Goal(s) At the end of this talk — hopefully — you'll know how to: Trace and fix errors as soon as possible — thanks to Sentry Deploy Sentry to elsewhere painlessly — thanks to Docker Get enough sleep
31. Bring It On!
32. "Hello World" App # Taken from ``dockerfiles/flask-demo/app.py``. # This snippet is running under Docker container. @app.route("/") def hello(): return "Hello World from Flask" @app.route("/about") def about(): a = 1 / 0 # NOQA return "Hello World from Flask, again?" @app.route("/help") def info(): func_a() return "Hello World from Flask, SRSLY?"
33. App Dockerfile # Python env RUN apt-get install -y python-pip RUN pip install flask RUN pip install raven RUN pip install blinker EXPOSE 5000 ENV SENTRY_PROTO tcp ENV SENTRY_AUTH None ENV SENTRY_PROJECT 0 ADD app.py /opt/app.py CMD ["python", "/opt/app.py"]
34. Basic Sentry Stack Requirements: Datastore, e.g. MySQL, Postgres Sentry instance (dooh)
35. MySQL Dockerfile # boring instruction is omitted RUN apt-get -y install mysql-server # make the server listen to 0.0.0.0 RUN sed -i -e"s/^bind-address\s*=\s*127.0.0.1/bind-address = 0.0.0.0/" /etc/mysql/my.cnf ENV DB_USER admin ENV DB_PASSWD password ADD init-mysql.sh /opt/init-mysql.sh EXPOSE 3306 CMD ["/bin/bash", "/opt/init-mysql.sh"]
36. MySQL & Docker # step 1: build the container sudo docker build -rm -t $USER/$MYSQLREPO . # step 2: run the container sudo docker run -v /opt/mysql:/var/lib/mysql -e DB_USER=admin -e DB_PASSWD=password \ -name mysql $USER/$MYSQLREPO # step 3: create a database for Sentry mysql -u $DB_USER -p -h $CONTAINER_IP
37. Sentry Dockerfile # boring instruction is omitted RUN apt-get install python-dev -y RUN apt-get install python-pip -y RUN easy_install -U distribute RUN pip install sentry RUN pip install gevent RUN pip install eventlet RUN pip install django-bcrypt RUN apt-get install libmysqlclient-dev -y RUN pip install MySQL-python ADD sentry.conf.py /opt/sentry.conf.py EXPOSE 9000/tcp 9001/udp ENTRYPOINT ["sentry", "--config=/opt/sentry.conf.py"] CMD ["help"]
38. Sentry Configuration Pt. 1 DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': os.environ.get("SENTRY_DB_NAME", "sentry"), 'USER': os.environ.get("SENTRY_DB_USER", "admin"), 'PASSWORD': os.environ.get("SENTRY_DB_PASS", "password"), 'HOST': os.environ.get("DB_PORT_3306_TCP_ADDR", "localhost"), 'PORT': os.environ.get("DB_PORT_3306_TCP_PORT", 3306), } }
39. Sentry Configuration Pt. 2 SENTRY_URL_PREFIX = os.environ.get( "SENTRY_URL_PREFIX", "http://0.0.0.0:9000") SENTRY_WEB_HOST = '0.0.0.0' SENTRY_WEB_PORT = 9000 SENTRY_WEB_OPTIONS = { 'workers': 3, # the number of gunicorn workers 'limit_request_line': 0, # required for raven-js 'secure_scheme_headers': {'X-FORWARDED-PROTO': 'https'}, "worker_class": "gevent", } SENTRY_UDP_HOST = "0.0.0.0" SENTRY_UDP_PORT = 9001 # for demo only ALLOWED_HOSTS = ["*"]
40. Sentry & Docker # step 1: build the container sudo docker build -rm -t $USER/$SENTRYREPO . # step 2.1: run Sentry service sudo docker run -p 9000:9000 -name sentry -link mysql:db $USER/$SENTRYREPO start http # (optional) step 2.2: create first user if there's no existing user yet sudo docker run -i -t -link mysql:db $USER/$SENTRYREPO createsuperuser
41. Linked Containers # MySQL container sudo docker run -v /opt/mysql:/var/lib/mysql -e DB_USER=admin -e DB_PASSWD=password \ -name mysql $USER/$MYSQLREPO # Sentry container sudo docker run -p 9000:9000 -name sentry -link mysql:db $USER/$SENTRYREPO start http # App container sudo docker run -p 5000:5000 -link sentry:sentry \ -e SENTRY_PROTO=tcp -e SENTRY_PROJECT=2 -e SENTRY_AUTH=$AUTH \ $USER/$APPREPO
42. Advanced Setup Recommended for high-traffic Sentry instance: for background jobs for caching for buffers Rate-limit using Celery Memcache Redis Cyclops
43. Recap Tracing errors using Sentry is relaxing Installing & deploying Sentry elsewhere is trivial — thanks to Docker Advanced Sentry & Docker usage might be your next adventure
44. References http://docs.docker.io/en/latest/ http://sentry.readthedocs.org/en/latest/ http://txt.fliglio.com/2013/11/creating-a-mysql-docker-container/ https://github.com/grue/docker-sentry http://flask.pocoo.org/docs/
45. Questions?
46. Thanks For Listening on GitHub // on Twitter iromli @iromli