Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud and Devops at ticketea

Cloud and Devops at ticketea

Talk at Devops Azure Day Madrid 2015 about devops and cloud

Ticketea Engineering

November 16, 2015
Tweet

More Decks by Ticketea Engineering

Other Decks in Programming

Transcript

  1. Cloud and Devops
    at ticketea
    Jose Ignacio Galarza, @igalarzab
    CTO

    View full-size slide

  2. create tech solutions to simplify
    and enrich the way events are
    managed and attended

    View full-size slide

  3. ticketea
    Ecosystem

    View full-size slide

  4. ticketea
    Who we are?
    @esanchezm @JavierHdez3 @maraujop @igalarzab
    @sullymorland @patoroco edu
    @imanolcg
    @RafaRM20 @iamcarlosedo
    @andrea_mgr
    @Mc_Arena_pr
    @javitxudedios @Maquert @gnufede @willyfrog_

    View full-size slide

  5. ticketea
    Team
    ✤ 12 developers
    ✤ 3 designers
    ✤ 1 QA

    View full-size slide

  6. ticketea
    Why the cloud?
    ✤ Offer high-availability despite of load-spikes.
    ✤ Scale from 3 servers to 350 in minutes.
    ✤ Access to awesome services provided by your IaSS.
    ✤ Be more cost efficient.

    View full-size slide

  7. ticketea
    Why the cloud?
    ✤ Easier to be fault tolerant.
    ✤ They have a huge team improving always their service.
    ✤ Incredible uptime (some of them with 99.999999999%).

    View full-size slide

  8. ticketea
    Why the cloud?
    IaaS to
    PaaS

    View full-size slide

  9. Cloud
    provider
    Ticketea

    View full-size slide

  10. ticketea
    Pitfalls
    ✤ Everything can fail… everything. Have a good plan B.
    ✤ A plan B isn't a plan B, until is tested.
    ✤ Pre-heat everything before big events.
    ✤ Read about your IaSS in forums/email groups.
    ✤ Try to distribute your machines to avoid thunderbolts ;)

    View full-size slide

  11. architecture

    View full-size slide

  12. ticketea
    server

    View full-size slide

  13. ticketea
    Our architecture

    View full-size slide

  14. ticketea
    Our architecture

    View full-size slide

  15. ticketea
    Our architecture

    View full-size slide

  16. ticketea
    Our architecture

    View full-size slide

  17. ticketea
    Our architecture

    View full-size slide

  18. ticketea
    Our architecture

    View full-size slide

  19. ticketea
    servers

    View full-size slide

  20. ticketea
    The multilith

    View full-size slide

  21. ticketea
    Distributed systems
    ✤ Granular failing.
    ✤ Avoid single point of failure.
    ✤ Easy code maintainability
    ✤ Choose the best tech for each project.
    ✤ Scale the team easier.

    View full-size slide

  22. ticketea
    Distributed systems
    ✤ Hard to deploy.
    ✤ Hard to see the whole picture.
    ✤ More complexity: easy to fail.
    ✤ When shit happens: you need a plan B.

    View full-size slide

  23. don’t panic, we’re devops!

    View full-size slide

  24. ticketea
    Team
    ✤ 12 developers
    ✤ 3 designers
    ✤ 1 QA
    ✤ 0 sysadmin

    View full-size slide

  25. ticketea
    Team
    no specific devops team

    View full-size slide

  26. understand your stack

    View full-size slide

  27. ticketea
    Development cycle

    View full-size slide

  28. ticketea
    Development keys
    ✤ Code reviews in every pull request.
    ✤ Continuous integration with a strong test suite.
    ✤ Cook your own receipts.
    ✤ One-click deployment.
    ✤ “Continuous” delivery. (not always)

    View full-size slide

  29. and… easy rollbacks
    ;)

    View full-size slide

  30. ticketea
    Deploying code
    ✤ We’ve changed between a few models, but finally:
    1. Create a complete new environment in prod machines.
    2. Deploy your code there.
    3. Extract one machine from the load-balancer.
    4. Create a golden-image

    View full-size slide

  31. ticketea
    Deploying code
    ✤ Atomic deploys.
    ✤ 0s-downtime deploys.
    ✤ Easy rollback to the last N versions.
    ✤ Money-cost friendly ;)
    ✤ Insanely fast to scale machines when needed.

    View full-size slide

  32. ticketea
    The Rat ‘devops’ race
    API
    Heracles
    Odin
    Frontend
    Chef
    Puppet
    Ansible
    Thor
    Caronte
    Ansible

    View full-size slide

  33. ticketea
    Production vs Development environments
    ✤ Using the same ansible roles
    ✤ Meta-roles:
    roles/
    | ——django/
    | ————tasks/main.yml
    | ——api/
    | ————tasks/main.yml
    |—— tkt-api/
    |———— meta/main.yml
    api.yml
    development.yml

    View full-size slide

  34. ticketea
    Automating common procedures
    ✤ fab -f heracles.py stop_consuming_messages:queue
    ✤ fab -f frontend.py maintenance_on
    ✤ fab -f sg.py open_sg
    ✤ fab -f api.py fast_rollback:version

    View full-size slide

  35. ticketea
    Provisioning
    ✤ Ansible
    ✤ Fabric

    View full-size slide

  36. ticketea
    Error reporting
    ✤ sentry
    ✤ fabric (fka. crashlytics)

    View full-size slide

  37. ticketea
    Graphing
    ✤ graphite
    ✤ grafana

    View full-size slide

  38. ticketea
    Logging and alarming
    ✤ rsyslog
    ✤ graylog
    ✤ cabot
    ✤ slack/email/x connected through a queue

    View full-size slide

  39. Thanks. ;-)
    Jose Ignacio Galarza, @igalarzab
    CTO

    View full-size slide