Tracing Error Like A Boss

Slide 1

Slide 1 text

Tracing Error Like A Boss (with Sentry and Docker) A presentation by Isman Firmansyah

Slide 2

Slide 2 text

$ whoami a backend engineer // on GitHub // on Twitter iromli @iromli

Slide 3

Slide 3 text

Once Upon A Time Based on true story.

Slide 4

Slide 4 text

Life's Good New app is finished — developed, tested, and deployed to production without errors.

Slide 5

Slide 5 text

Time Goes On ... and On More features, dependencies, typos Missed testcases, sloppy deploys

Slide 6

Slide 6 text

Life's (not) Good My app is broken; it was running fine few days ago.

Slide 7

Slide 7 text

We Tend To Do Been there, done that.

Slide 8

Slide 8 text

The Naive Way 1. SSH to remote server 2. Finding the needle in a haystack (a.k.a. log file) 3. Update the code in dev env and re-deploy

Slide 9

Slide 9 text

The Lazy Way 1. Wait for email sent by app for each error (with full traceback) 2. Update the code in dev env and re-deploy

Slide 10

Slide 10 text

Unpleasant Moments

Slide 11

Slide 11 text

We Need Sane Approach At least i need it!

Slide 12

Slide 12 text

Error Tracing For Humans™ 1. Wait for email sent by app for unique error (with less traceback) 2. See the full traceback in web UI 3. Update the code in dev env and re-deploy

Slide 13

Slide 13 text

Sentry http://getsentry.com/ Sentry is a realtime, platform-agnostic error logging and aggregation platform.

Slide 14

Slide 14 text

Why Sentry It's opensource — transparent and active development Battle-tested at Trusted by notable shops ( , , etc) Host your own or pay the service Errors are groupped, email sent per error group Nice UI Disqus Pinterest Instagram

Slide 15

Slide 15 text

Client Libraries

Slide 16

Slide 16 text

Sentry Plugins ... and many more https://github.com/getsentry/sentry-github https://github.com/gisce/sentry-irc https://github.com/disqus/sentry-graphite https://github.com/ecarreras/sentry-whatsapp

Slide 17

Slide 17 text

Realtime Events & Aggregation

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Traceback

Slide 20

Slide 20 text

Case Closed? Well, depends on your next decision.

Slide 21

Slide 21 text

The Easiest Way Paid Sentry Service Case is closed ... but then the next sections are pointless

Slide 22

Slide 22 text

The D.I.Y. Way Host Your Own Sentry Instance What could be wrong? You don't familiar with Sentry/Django/Python You might have Rock Lee's syndromes

Slide 23

Slide 23 text

Life's Good pt. 2 Installing Sentry on my local Ubuntu machine

Slide 24

Slide 24 text

Life's (not) Good pt. 2 Sentry doesn't work on my CentOS production server

Slide 25

Slide 25 text

You Have Dev-Prod Parity Problem TL;DR http://12factor.net/dev-prod-parity A state where dev != production environment.

Slide 26

Slide 26 text

Docker http://docker.io/ From its official documentation: docker, the Linux Container Runtime, runs Unix processes with strong guarantees of isolation across servers. Your software runs repeatably everywhere because its Container includes any dependencies.

Slide 27

Slide 27 text

Docker: From My Own Perspective Strong foundation — built by people who run DotCloud A Linux container with steroids: lightweight batteries-included — git-like experience, registry, etc. opensource community awesomeness — 7696 GH stars, 1006 forks A handy tool to tackle dev-prod parity PaaS

Slide 28

Slide 28 text

Learn Docker The Hard Way Official documentation Guidebook Videos http://docs.docker.io/ https://github.com/kencochrane/docker-guidebook http://www.youtube.com/results? search_query=docker+dotcloud

Slide 29

Slide 29 text

Talk Coverage Simulation of a broken app Use Sentry to trace the errors Use Docker container to ship Sentry No email integration (leave it as practice for others)

Slide 30

Slide 30 text

Goal(s) At the end of this talk — hopefully — you'll know how to: Trace and fix errors as soon as possible — thanks to Sentry Deploy Sentry to elsewhere painlessly — thanks to Docker Get enough sleep

Slide 31

Slide 31 text

Bring It On!

Slide 32

Slide 32 text

"Hello World" App # Taken from ``dockerfiles/flask-demo/app.py``. # This snippet is running under Docker container. @app.route("/") def hello(): return "Hello World from Flask" @app.route("/about") def about(): a = 1 / 0 # NOQA return "Hello World from Flask, again?" @app.route("/help") def info(): func_a() return "Hello World from Flask, SRSLY?"

Slide 33

Slide 33 text

App Dockerfile # Python env RUN apt-get install -y python-pip RUN pip install flask RUN pip install raven RUN pip install blinker EXPOSE 5000 ENV SENTRY_PROTO tcp ENV SENTRY_AUTH None ENV SENTRY_PROJECT 0 ADD app.py /opt/app.py CMD ["python", "/opt/app.py"]

Slide 34

Slide 34 text

Basic Sentry Stack Requirements: Datastore, e.g. MySQL, Postgres Sentry instance (dooh)

Slide 35

Slide 35 text

MySQL Dockerfile # boring instruction is omitted RUN apt-get -y install mysql-server # make the server listen to 0.0.0.0 RUN sed -i -e"s/^bind-address\s*=\s*127.0.0.1/bind-address = 0.0.0.0/" /etc/mysql/my.cnf ENV DB_USER admin ENV DB_PASSWD password ADD init-mysql.sh /opt/init-mysql.sh EXPOSE 3306 CMD ["/bin/bash", "/opt/init-mysql.sh"]

Slide 36

Slide 36 text

MySQL & Docker # step 1: build the container sudo docker build -rm -t $USER/$MYSQLREPO . # step 2: run the container sudo docker run -v /opt/mysql:/var/lib/mysql -e DB_USER=admin -e DB_PASSWD=password \ -name mysql $USER/$MYSQLREPO # step 3: create a database for Sentry mysql -u $DB_USER -p -h $CONTAINER_IP

Slide 37

Slide 37 text

Sentry Dockerfile # boring instruction is omitted RUN apt-get install python-dev -y RUN apt-get install python-pip -y RUN easy_install -U distribute RUN pip install sentry RUN pip install gevent RUN pip install eventlet RUN pip install django-bcrypt RUN apt-get install libmysqlclient-dev -y RUN pip install MySQL-python ADD sentry.conf.py /opt/sentry.conf.py EXPOSE 9000/tcp 9001/udp ENTRYPOINT ["sentry", "--config=/opt/sentry.conf.py"] CMD ["help"]

Slide 38

Slide 38 text

Sentry Configuration Pt. 1 DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': os.environ.get("SENTRY_DB_NAME", "sentry"), 'USER': os.environ.get("SENTRY_DB_USER", "admin"), 'PASSWORD': os.environ.get("SENTRY_DB_PASS", "password"), 'HOST': os.environ.get("DB_PORT_3306_TCP_ADDR", "localhost"), 'PORT': os.environ.get("DB_PORT_3306_TCP_PORT", 3306), } }

Slide 39

Slide 39 text

Sentry Configuration Pt. 2 SENTRY_URL_PREFIX = os.environ.get( "SENTRY_URL_PREFIX", "http://0.0.0.0:9000") SENTRY_WEB_HOST = '0.0.0.0' SENTRY_WEB_PORT = 9000 SENTRY_WEB_OPTIONS = { 'workers': 3, # the number of gunicorn workers 'limit_request_line': 0, # required for raven-js 'secure_scheme_headers': {'X-FORWARDED-PROTO': 'https'}, "worker_class": "gevent", } SENTRY_UDP_HOST = "0.0.0.0" SENTRY_UDP_PORT = 9001 # for demo only ALLOWED_HOSTS = ["*"]

Slide 40

Slide 40 text

Sentry & Docker # step 1: build the container sudo docker build -rm -t $USER/$SENTRYREPO . # step 2.1: run Sentry service sudo docker run -p 9000:9000 -name sentry -link mysql:db $USER/$SENTRYREPO start http # (optional) step 2.2: create first user if there's no existing user yet sudo docker run -i -t -link mysql:db $USER/$SENTRYREPO createsuperuser

Slide 41

Slide 41 text

Linked Containers # MySQL container sudo docker run -v /opt/mysql:/var/lib/mysql -e DB_USER=admin -e DB_PASSWD=password \ -name mysql $USER/$MYSQLREPO # Sentry container sudo docker run -p 9000:9000 -name sentry -link mysql:db $USER/$SENTRYREPO start http # App container sudo docker run -p 5000:5000 -link sentry:sentry \ -e SENTRY_PROTO=tcp -e SENTRY_PROJECT=2 -e SENTRY_AUTH=$AUTH \ $USER/$APPREPO

Slide 42

Slide 42 text

Advanced Setup Recommended for high-traffic Sentry instance: for background jobs for caching for buffers Rate-limit using Celery Memcache Redis Cyclops

Slide 43

Slide 43 text

Recap Tracing errors using Sentry is relaxing Installing & deploying Sentry elsewhere is trivial — thanks to Docker Advanced Sentry & Docker usage might be your next adventure

Slide 44

Slide 44 text

References http://docs.docker.io/en/latest/ http://sentry.readthedocs.org/en/latest/ http://txt.fliglio.com/2013/11/creating-a-mysql-docker-container/ https://github.com/grue/docker-sentry http://flask.pocoo.org/docs/