What's wrong with your app?

When you start seeing the problem for your production app, what would you do? How would you look into that problem? As a support engineer, I've seen many Rails apps that are having various problems. This talk will cover what you can do when you see the problem, and what you should start doing right now to prevent the problem from happening, based on a real experience.

Transcript

1. What’s wrong with your app? Keiko Oda (@keiko713) Heroku Technical Support Engineer
2. About me Keiko Oda (@keiko713) Joined Heroku April 2013 Technical support engineer San Francisco / Kanazawa
3. We’re the Bento sponsor! Heroku book in 日本語 Advertisem ent RubyKaigi中はジュンク堂RubyKaigi支店にて お求めいただけます！ サイン会もやるらしい Japanese
4. Say “Hi” to Herokai! (Herokai = Heroku employee)
5. Matz Rubyのパパ兼 キーノートのプロ papa of Ruby pro keynote speaker Nobu パッチモンスター patch monster Koichi 燃えるバイトコード VMマスター burning bytecode VM master
6. Richard Heroku x Rubyといえば この人 Andy とっても陽気なTAM very social TAM Ayumu Heroku Japan teamの 保護者 (guardian, chaperon)
7. Caio Herokuで4年働いています Ryan 三姉妹のパパ Brett 日本語上手なぶーちゃん
8. MY APP IS DOWN!
9. Heroku H12 request timeout error
10. Cause 1: slow requests Most common cause of H12 Go to your New Relic + logging add-on to find the cause, do whatever you need to do * Just you have bad code? * Maybe external service problem? * Could be database problem?
11. Cause 2: traffic spike Sudden traffic spike can cause the downtime Is your app ready for the sudden traffic spike?
12. Cause 3: memory quota exceed If you use more memory than what you can use, it will make your app super slow Temporary solution: Restarting the app Memory usage increased from Ruby 2.1 Your Unicorn worker # is appropriate? Could be memory leak - monitor your memory usage using tools
13. Start doing right meow!
14. Do You Have Logging? Logging is very important Log tells you a lot Logging add-ons available at https:// addons.heroku.com/#logging Heroku only stores 1500 lines of log
15. Monitor your app! New Relic is a good tool to start with https://addons.heroku.com/newrelic
16. Monitor your app! Availability monitoring (e.g. New Relic, Pingdom) Error monitoring (e.g. Rollbar, Honeybadger)
17. Monitor your app! Troubleshoot your database slowness with Expensive Queries https://devcenter.heroku.com/articles/expensive- queries
18. Monitor your app! dashboard.heroku.com
19. Choose the right server The app can be bad really easiliy depends on the server Use the server that can handle requests concurrently (e.g. Unicorn, Puma)
20. Choose the right server example: bugs.ruby-lang.org http://www.hsbt.org/diary/20140125.html thin passenger
21. Proper timeout setting Unicorn timeout: 15s or less => prevent long-running requests from causing a cascading request queueing buildup rack-timeout: 10s or less => rack-timeout to be triggered before the Unicorn timeout, raise the error Address the long-running actions and optimize the response time under 500ms
22. More tips? Ask Herokai! 　Rubyならまかせて！ 日本語でOK 経験豊富なサポート エンジニア/TAM
23. Thank you! ご清聴ありがとうございました  :)