Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's wrong with your app?

Keiko Oda
September 18, 2014

What's wrong with your app?

When you start seeing the problem for your production app, what would you do? How would you look into that problem? As a support engineer, I've seen many Rails apps that are having various problems. This talk will cover what you can do when you see the problem, and what you should start doing right now to prevent the problem from happening, based on a real experience.

Keiko Oda

September 18, 2014
Tweet

More Decks by Keiko Oda

Other Decks in Technology

Transcript

  1. What’s wrong
    with your app?
    Keiko Oda (@keiko713)
    Heroku
    Technical Support Engineer

    View full-size slide

  2. About me
    Keiko Oda (@keiko713)
    Joined Heroku April 2013
    Technical support engineer
    San Francisco / Kanazawa

    View full-size slide

  3. We’re the Bento sponsor!
    Heroku book in 日本語
    Advertisem
    ent
    RubyKaigi中はジュンク堂RubyKaigi支店にて
    お求めいただけます!
    サイン会もやるらしい
    Japanese

    View full-size slide

  4. Say “Hi” to Herokai!
    (Herokai = Heroku employee)

    View full-size slide

  5. Matz
    Rubyのパパ兼
    キーノートのプロ
    papa of Ruby
    pro keynote speaker
    Nobu
    パッチモンスター
    patch monster
    Koichi
    燃えるバイトコード
    VMマスター
    burning bytecode
    VM master

    View full-size slide

  6. Richard
    Heroku x Rubyといえば
    この人
    Andy
    とっても陽気なTAM
    very social TAM
    Ayumu
    Heroku Japan teamの
    保護者
    (guardian, chaperon)

    View full-size slide

  7. Caio
    Herokuで4年働いています
    Ryan
    三姉妹のパパ
    Brett
    日本語上手なぶーちゃん

    View full-size slide

  8. MY APP IS DOWN!

    View full-size slide

  9. Heroku H12
    request timeout error

    View full-size slide

  10. Cause 1: slow requests
    Most common cause of H12
    Go to your New Relic + logging add-on to
    find the cause, do whatever you need to
    do
    * Just you have bad code?
    * Maybe external service problem?
    * Could be database problem?

    View full-size slide

  11. Cause 2: traffic spike
    Sudden traffic spike can cause the
    downtime
    Is your app ready for the sudden traffic
    spike?

    View full-size slide

  12. Cause 3: memory quota
    exceed
    If you use more memory than what you
    can use, it will make your app super slow
    Temporary solution: Restarting the app
    Memory usage increased from Ruby 2.1
    Your Unicorn worker # is appropriate?
    Could be memory leak - monitor your
    memory usage using tools

    View full-size slide

  13. Start doing right meow!

    View full-size slide

  14. Do You Have Logging?
    Logging is very important
    Log tells you a lot
    Logging add-ons available at https://
    addons.heroku.com/#logging
    Heroku only stores 1500 lines of log

    View full-size slide

  15. Monitor your app!
    New Relic is a good tool to start with
    https://addons.heroku.com/newrelic

    View full-size slide

  16. Monitor your app!
    Availability monitoring
    (e.g. New Relic, Pingdom)
    Error monitoring
    (e.g. Rollbar, Honeybadger)

    View full-size slide

  17. Monitor your app!
    Troubleshoot your database slowness
    with Expensive Queries
    https://devcenter.heroku.com/articles/expensive-
    queries

    View full-size slide

  18. Monitor your app!
    dashboard.heroku.com

    View full-size slide

  19. Choose the right server
    The app can be bad really easiliy depends
    on the server
    Use the server that can handle requests
    concurrently (e.g. Unicorn, Puma)

    View full-size slide

  20. Choose the right server
    example: bugs.ruby-lang.org
    http://www.hsbt.org/diary/20140125.html
    thin passenger

    View full-size slide

  21. Proper timeout setting
    Unicorn timeout: 15s or less
    => prevent long-running requests from causing a
    cascading request queueing buildup
    rack-timeout: 10s or less
    => rack-timeout to be triggered before the Unicorn
    timeout, raise the error
    Address the long-running actions and
    optimize the response time under 500ms

    View full-size slide

  22. More tips? Ask Herokai!
     Rubyならまかせて!
    日本語でOK
    経験豊富なサポート
    エンジニア/TAM

    View full-size slide

  23. Thank you!
    ご清聴ありがとうございました  :)

    View full-size slide