Scaling Cloud Servers for Seasonal Traffic – ClearTax

I gave this talk at Nasscom's Annual Technology Conference in New Delhi on 12th December, 2014.

ClearTax (YC S14) helps individuals e-File their Income Tax Returns on the web and mobile. The demand curve for our services is very seasonal – with a huge spike in traffic close to 31st July (the main tax filing deadline in India), and at the end of each quarter.

This talk focuses on the key learnings of ClearTax: how to scale a large service in the cloud with a very small startup team.

Feel free to reach out to me if you have any questions / suggestions! I'm _anks on twitter.

PS: ClearTax is hiring engineers & designers. For more info, visit https://cleartax.in/meta/careers

Transcript

1. Scaling Cloud Servers for Seasonal Traffic Ankit Solanki https://cleartax.in
2. or… What happens when you go Viral (Don’t Panic)
3. About ClearTax
4. The Spike
5. The Spike Most startups have to deal with it ClearTax — E-Filing Deadlines Flipkart — Big Billion Sale Amazon — Black Friday Netflix — House of Cards release
6. The Spike It’s not always predictable (think Slashdot-effect) Can be 10-100x your usual traffic Probably will not last
7. The Spike These are usually new users to your product You don’t want to lose them
8. Be Prepared Monitoring New Relic, Librato, Pingdom, PagerDuty many other tools Horizontal Scaling
9. Monitoring Set up a dashboard
10. Monitoring: Set up Notifications! Learn about outages as they happen (and) Learn about usage spikes as they happen
11. Scaling Your Server Scale-Up ⬍ X X X X X Scale-Out ⬌ X X X X X X X X X X X X X X X X X X X X X X X X X X X X
12. Horizontal Scaling Spoiler alert: It’s painful if you hadn’t planed for it
13. Horizontal Scaling Checklist ✔ Don’t rely on local filesystem Store static assets in S3 or equivalent
14. Horizontal Scaling Checklist ✔ Don’t rely on local memory No in-memory sessions. Use redis or equivalent
15. Horizontal Scaling Checklist ✔ One-Click Deployments Rely on platforms PAAS – Heroku-like platforms Or something provided by your Cloud Provider (Or, roll your own – Docker, Chef, Puppet, Ansible)
16. Recommendation for Startups Go with a PAAS when you’re young Build the product, don’t do DevOps when you don’t need to!
17. This was Scaling 101 But, be prepared for things to break when the spike hits
18. What ClearTax did in July Added a new sub-system for background tasks Infrastructure for bringing up web and background workers, on-demand Logging, monitoring sub-systems put in place
20. Growing Pains In first week of July, noticed slow response times Bottleneck identified: Database
21. Down the rabbithole we went…
22. Database From initial analysis, problem was due to locks and contention Did a thorough code review Wrote tooling to visualize execution times
24. Database Ended up identifying some potential locks Saved 20% execution time (according to our tests)
25. Reality Check We were still as slow as before
26. 60% Load on DB, everything is fine
27. 70% Load: Everything is broken!
28. Final Solution Upgrading database infrastructure to use larger machines with more memory
29. Learnings Look at your resource utilization! Don’t push resources to their limits
30. Learnings Distributed systems are hard Find bottlenecks as soon as possible Real world usage is different from load tests
31. Another Rule Anything not monitored will break (at the worst possible time)
32. Disks will fill up. Machines will crash. Database will run out of space.
33. Think about building a resilient system
34. Monitor more than just hardware
35. Devops should monitor more than just hardware With sudden traffic — Your payment gateway may go down Your Email provider may start throttling you
36. Key Takeaways
37. Have the basics in place Monitor everything! Things will fail – this is expected Attempt fixes by upgrading infrastructure first
38. Thank you Questions? PS: ClearTax is hiring. Send an email to careers+engg@cleartax.in