Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ticketea, scaling a highly transactional business

Ticketea, scaling a highly transactional business

A talk about things that we've done to scale ticketea demanding growing business given by @maraujop at Mad scalability

Ticketea Engineering

March 22, 2016
Tweet

More Decks by Ticketea Engineering

Other Decks in Programming

Transcript

  1. SCALING A HIGHLY
    TRANSACTIONAL BUSINESS

    View Slide

  2. 2
    @maraujop @ticketeaeng

    View Slide

  3. Who are we?
    3
    @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco
    @imanolcg @RafaRM20 @iamcarlosedo
    @andrea_mgr
    @Mc_Arena_pr
    @javitxudedios @Maquert @gnufede @willyfrog_ @ShideShugo

    View Slide

  4. Some numbers, peak hour stats
    4
    2014
    6582 tickets
    2015
    21287 tickets

    View Slide

  5. FRONTEND HISTORY

    View Slide

  6. “Microservices are about scaling the
    number of engineers not the number of
    requests”
    Jay Kreps
    (Co-Creator of kafka)

    View Slide

  7. BALANCE YOUR SYSTEM

    View Slide

  8. A little bit of history
    8

    View Slide

  9. frontend Django
    9

    View Slide

  10. Pros
    10
    •5 coders worked at the company
    •Fast to build, easy to test and deploy
    •Simple stack
    PROS

    View Slide

  11. Cons
    11
    •Workers blocked on API connections
    •Errors in backoffice could crash the site
    •Slow API means nothing loads
    CONS

    View Slide

  12. View Slide

  13. Afrodita sync system
    13

    View Slide

  14. Afrodita connection system
    14

    View Slide

  15. use js

    View Slide

  16. 2014
    (350 frontends)
    100x
    Django
    Frontend

    View Slide

  17. 2015
    (20 frontends)
    3.5x
    Afrodita

    View Slide

  18. SYSTEMS

    View Slide

  19. View Slide

  20. CLOUD
    PROVIDER
    TICKETEA

    View Slide

  21. Why the cloud?
    21
    •High availability despite of load-spikes
    •Scale from 3 servers to 400 in minutes
    •Awesome managed services

    View Slide

  22. Why the cloud?
    22
    •Easier to be fault tolerant
    •They have great uptime
    •We build AMIs on deploys and use autoscaling
    •Preheating

    View Slide

  23. BACKEND

    View Slide

  24. MY LEGACY CODE IS SLOW

    View Slide

  25. Scaling
    25
    MY LEGACY CODE IS SLOW
    •Scale In: Bigger machine
    •Scale Out: More machines behind a load balancer

    View Slide

  26. View Slide

  27. Things you can do
    27
    •Optimize where your real bottleneck is
    •Cache more things
    •Speed up your database usage
    •Denormalize your database
    •Execute less code
    •Speed up your code

    View Slide

  28. “The central enemy of reliability is
    complexity”
    Andy Bechtolsheim
    (Co-founder Sun Microsystems)

    View Slide

  29. Execute less code
    29
    •Each time your workers are executing your
    code they are not able to attend new requests
    •Move everything not essential to background
    tasks, specially I/O bound jobs
    •Watch your timeouts

    View Slide

  30. Database
    30
    •Relational databases are “almost” never the
    problem
    •Check your slowlog (create indexes)
    •Optimize slow or recurrent queries
    •Use read replicas (careful with replication lag)

    View Slide

  31. View Slide

  32. Database contingency
    32

    View Slide

  33. Database contingency
    33

    View Slide

  34. Database contingency
    34
    •Wrapping everything in a transaction is not cost effective
    •Deadlocks are tricky to spot
    •Caching data in this situation can be counter-productive

    View Slide

  35. Database contingency
    35
    •Shard counters
    •Use another technology, like REDIS (consistency)

    View Slide

  36. Stop unnecessary background jobs

    View Slide

  37. Find more about us
    37
    http://www.infoq.com/articles/monolith-to-multilith
    https://speakerdeck.com/ticketeaeng
    https://twitter.com/ticketeaeng
    http://engineering.ticketea.com

    View Slide

  38. THANKS

    View Slide