Ticketea, scaling a highly transactional business

A talk about things that we've done to scale ticketea demanding growing business given by @maraujop at Mad scalability

Transcript

1. SCALING A HIGHLY
   TRANSACTIONAL BUSINESS
   

   View Slide

2. 2
   @maraujop @ticketeaeng
   

   View Slide

3. Who are we?
   3
   @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco
   @imanolcg @RafaRM20 @iamcarlosedo
   @andrea_mgr
   @Mc_Arena_pr
   @javitxudedios @Maquert @gnufede @willyfrog_ @ShideShugo
   

   View Slide

4. Some numbers, peak hour stats
   4
   2014
   6582 tickets
   2015
   21287 tickets
   

   View Slide

5. FRONTEND HISTORY
   

   View Slide

6. “Microservices are about scaling the
   number of engineers not the number of
   requests”
   Jay Kreps
   (Co-Creator of kafka)
   

   View Slide

7. BALANCE YOUR SYSTEM
   

   View Slide

8. A little bit of history
   8
   

   View Slide

9. frontend Django
   9
   

   View Slide

10. Pros
    10
    •5 coders worked at the company
    •Fast to build, easy to test and deploy
    •Simple stack
    PROS
    

    View Slide

11. Cons
    11
    •Workers blocked on API connections
    •Errors in backoffice could crash the site
    •Slow API means nothing loads
    CONS
    

    View Slide

12. View Slide

13. Afrodita sync system
    13
    

    View Slide

14. Afrodita connection system
    14
    

    View Slide

15. use js
    

    View Slide

16. 2014
    (350 frontends)
    100x
    Django
    Frontend
    

    View Slide

17. 2015
    (20 frontends)
    3.5x
    Afrodita
    

    View Slide

18. SYSTEMS
    

    View Slide

19. View Slide

20. CLOUD
    PROVIDER
    TICKETEA
    

    View Slide

21. Why the cloud?
    21
    •High availability despite of load-spikes
    •Scale from 3 servers to 400 in minutes
    •Awesome managed services
    

    View Slide

22. Why the cloud?
    22
    •Easier to be fault tolerant
    •They have great uptime
    •We build AMIs on deploys and use autoscaling
    •Preheating
    

    View Slide

23. BACKEND
    

    View Slide

24. MY LEGACY CODE IS SLOW
    

    View Slide

25. Scaling
    25
    MY LEGACY CODE IS SLOW
    •Scale In: Bigger machine
    •Scale Out: More machines behind a load balancer
    

    View Slide

26. View Slide

27. Things you can do
    27
    •Optimize where your real bottleneck is
    •Cache more things
    •Speed up your database usage
    •Denormalize your database
    •Execute less code
    •Speed up your code
    

    View Slide

28. “The central enemy of reliability is
    complexity”
    Andy Bechtolsheim
    (Co-founder Sun Microsystems)
    

    View Slide

29. Execute less code
    29
    •Each time your workers are executing your
    code they are not able to attend new requests
    •Move everything not essential to background
    tasks, specially I/O bound jobs
    •Watch your timeouts
    

    View Slide

30. Database
    30
    •Relational databases are “almost” never the
    problem
    •Check your slowlog (create indexes)
    •Optimize slow or recurrent queries
    •Use read replicas (careful with replication lag)
    

    View Slide

31. View Slide

32. Database contingency
    32
    

    View Slide

33. Database contingency
    33
    

    View Slide

34. Database contingency
    34
    •Wrapping everything in a transaction is not cost effective
    •Deadlocks are tricky to spot
    •Caching data in this situation can be counter-productive
    

    View Slide

35. Database contingency
    35
    •Shard counters
    •Use another technology, like REDIS (consistency)
    

    View Slide

36. Stop unnecessary background jobs
    

    View Slide

37. Find more about us
    37
    http://www.infoq.com/articles/monolith-to-multilith
    https://speakerdeck.com/ticketeaeng
    https://twitter.com/ticketeaeng
    http://engineering.ticketea.com
    

    View Slide

38. THANKS
    

    View Slide