Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud and Devops at ticketea

Cloud and Devops at ticketea

Talk at Devops Azure Day Madrid 2015 about devops and cloud

Ticketea Engineering

November 16, 2015
Tweet

More Decks by Ticketea Engineering

Other Decks in Programming

Transcript

  1. Cloud and Devops
    at ticketea
    Jose Ignacio Galarza, @igalarzab
    CTO

    View Slide

  2. create tech solutions to simplify
    and enrich the way events are
    managed and attended

    View Slide

  3. ticketea
    Ecosystem

    View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. ticketea
    Who we are?
    @esanchezm @JavierHdez3 @maraujop @igalarzab
    @sullymorland @patoroco edu
    @imanolcg
    @RafaRM20 @iamcarlosedo
    @andrea_mgr
    @Mc_Arena_pr
    @javitxudedios @Maquert @gnufede @willyfrog_

    View Slide

  8. ticketea
    Team
    ✤ 12 developers
    ✤ 3 designers
    ✤ 1 QA

    View Slide

  9. cloud

    View Slide

  10. View Slide

  11. ticketea
    Why the cloud?
    ✤ Offer high-availability despite of load-spikes.
    ✤ Scale from 3 servers to 350 in minutes.
    ✤ Access to awesome services provided by your IaSS.
    ✤ Be more cost efficient.

    View Slide

  12. ticketea
    Why the cloud?
    ✤ Easier to be fault tolerant.
    ✤ They have a huge team improving always their service.
    ✤ Incredible uptime (some of them with 99.999999999%).

    View Slide

  13. ticketea
    Why the cloud?
    IaaS to
    PaaS

    View Slide

  14. Cloud
    provider
    Ticketea

    View Slide

  15. ticketea
    Pitfalls
    ✤ Everything can fail… everything. Have a good plan B.
    ✤ A plan B isn't a plan B, until is tested.
    ✤ Pre-heat everything before big events.
    ✤ Read about your IaSS in forums/email groups.
    ✤ Try to distribute your machines to avoid thunderbolts ;)

    View Slide

  16. architecture

    View Slide

  17. ticketea
    server

    View Slide

  18. ticketea
    Our architecture

    View Slide

  19. ticketea
    Our architecture

    View Slide

  20. ticketea
    Our architecture

    View Slide

  21. ticketea
    Our architecture

    View Slide

  22. ticketea
    Our architecture

    View Slide

  23. ticketea
    Our architecture

    View Slide

  24. ticketea
    servers

    View Slide

  25. ticketea
    The multilith

    View Slide

  26. ticketea
    Distributed systems
    ✤ Granular failing.
    ✤ Avoid single point of failure.
    ✤ Easy code maintainability
    ✤ Choose the best tech for each project.
    ✤ Scale the team easier.

    View Slide

  27. ticketea
    Distributed systems
    ✤ Hard to deploy.
    ✤ Hard to see the whole picture.
    ✤ More complexity: easy to fail.
    ✤ When shit happens: you need a plan B.

    View Slide

  28. don’t panic, we’re devops!

    View Slide

  29. development

    View Slide

  30. ticketea
    Team
    ✤ 12 developers
    ✤ 3 designers
    ✤ 1 QA
    ✤ 0 sysadmin

    View Slide

  31. ticketea
    Team
    no specific devops team

    View Slide

  32. understand your stack

    View Slide

  33. ticketea
    Development cycle

    View Slide

  34. ticketea
    Development keys
    ✤ Code reviews in every pull request.
    ✤ Continuous integration with a strong test suite.
    ✤ Cook your own receipts.
    ✤ One-click deployment.
    ✤ “Continuous” delivery. (not always)

    View Slide

  35. and… easy rollbacks
    ;)

    View Slide

  36. ticketea
    Deploying code
    ✤ We’ve changed between a few models, but finally:
    1. Create a complete new environment in prod machines.
    2. Deploy your code there.
    3. Extract one machine from the load-balancer.
    4. Create a golden-image

    View Slide

  37. ticketea
    Deploying code
    ✤ Atomic deploys.
    ✤ 0s-downtime deploys.
    ✤ Easy rollback to the last N versions.
    ✤ Money-cost friendly ;)
    ✤ Insanely fast to scale machines when needed.

    View Slide

  38. ticketea
    The Rat ‘devops’ race
    API
    Heracles
    Odin
    Frontend
    Chef
    Puppet
    Ansible
    Thor
    Caronte
    Ansible

    View Slide

  39. View Slide

  40. ticketea
    Production vs Development environments
    ✤ Using the same ansible roles
    ✤ Meta-roles:
    roles/
    | ——django/
    | ————tasks/main.yml
    | ——api/
    | ————tasks/main.yml
    |—— tkt-api/
    |———— meta/main.yml
    api.yml
    development.yml

    View Slide

  41. ticketea
    Automating common procedures
    ✤ fab -f heracles.py stop_consuming_messages:queue
    ✤ fab -f frontend.py maintenance_on
    ✤ fab -f sg.py open_sg
    ✤ fab -f api.py fast_rollback:version

    View Slide

  42. metrics

    View Slide

  43. ticketea
    Provisioning
    ✤ Ansible
    ✤ Fabric

    View Slide

  44. ticketea
    Error reporting
    ✤ sentry
    ✤ fabric (fka. crashlytics)

    View Slide

  45. ticketea
    Graphing
    ✤ graphite
    ✤ grafana

    View Slide

  46. ticketea
    Logging and alarming
    ✤ rsyslog
    ✤ graylog
    ✤ cabot
    ✤ slack/email/x connected through a queue

    View Slide

  47. Thanks. ;-)
    Jose Ignacio Galarza, @igalarzab
    CTO

    View Slide