Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ticketea, scaling a highly transactional business

Ticketea, scaling a highly transactional business

A talk about things that we've done to scale ticketea demanding growing business given by @maraujop at Mad scalability

00927a856336d961bdc7028722fe5897?s=128

Ticketea Engineering

March 22, 2016
Tweet

Transcript

  1. SCALING A HIGHLY TRANSACTIONAL BUSINESS

  2. 2 @maraujop @ticketeaeng

  3. Who are we? 3 @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @imanolcg

    @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ @ShideShugo
  4. Some numbers, peak hour stats 4 2014 6582 tickets 2015

    21287 tickets
  5. FRONTEND HISTORY

  6. “Microservices are about scaling the number of engineers not the

    number of requests” Jay Kreps (Co-Creator of kafka)
  7. BALANCE YOUR SYSTEM

  8. A little bit of history 8

  9. frontend Django 9

  10. Pros 10 •5 coders worked at the company •Fast to

    build, easy to test and deploy •Simple stack PROS
  11. Cons 11 •Workers blocked on API connections •Errors in backoffice

    could crash the site •Slow API means nothing loads CONS
  12. None
  13. Afrodita sync system 13

  14. Afrodita connection system 14

  15. use js

  16. 2014 (350 frontends) 100x Django Frontend

  17. 2015 (20 frontends) 3.5x Afrodita

  18. SYSTEMS

  19. None
  20. CLOUD PROVIDER TICKETEA

  21. Why the cloud? 21 •High availability despite of load-spikes •Scale

    from 3 servers to 400 in minutes •Awesome managed services
  22. Why the cloud? 22 •Easier to be fault tolerant •They

    have great uptime •We build AMIs on deploys and use autoscaling •Preheating
  23. BACKEND

  24. MY LEGACY CODE IS SLOW

  25. Scaling 25 MY LEGACY CODE IS SLOW •Scale In: Bigger

    machine •Scale Out: More machines behind a load balancer
  26. None
  27. Things you can do 27 •Optimize where your real bottleneck

    is •Cache more things •Speed up your database usage •Denormalize your database •Execute less code •Speed up your code
  28. “The central enemy of reliability is complexity” Andy Bechtolsheim (Co-founder

    Sun Microsystems)
  29. Execute less code 29 •Each time your workers are executing

    your code they are not able to attend new requests •Move everything not essential to background tasks, specially I/O bound jobs •Watch your timeouts
  30. Database 30 •Relational databases are “almost” never the problem •Check

    your slowlog (create indexes) •Optimize slow or recurrent queries •Use read replicas (careful with replication lag)
  31. None
  32. Database contingency 32

  33. Database contingency 33

  34. Database contingency 34 •Wrapping everything in a transaction is not

    cost effective •Deadlocks are tricky to spot •Caching data in this situation can be counter-productive
  35. Database contingency 35 •Shard counters •Use another technology, like REDIS

    (consistency)
  36. Stop unnecessary background jobs

  37. Find more about us 37 http://www.infoq.com/articles/monolith-to-multilith https://speakerdeck.com/ticketeaeng https://twitter.com/ticketeaeng http://engineering.ticketea.com

  38. THANKS