Desplegando Ticketea

Desplegando Ticketea

A0a96a5403921890df7a9b2c0f0298c1?s=128

Imanol Cea

March 17, 2016
Tweet

Transcript

  1. @ticketeaEng Imanol Cea @imanolcg

  2. ticketea Who we are? @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @imanolcg

    @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ @ShideShugo
  3. ticketea Sysadmins? @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @imanolcg @RafaRM20 @iamcarlosedo

    @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ @ShideShugo X X X X X X X X X X X X X X X
  4. Our systems

  5. None
  6. ticketea Why use AWS? ✤ We don’t have sysadmins. ✤

    Our traffic is spike-oriented. ✤ Auto-managed AWS services are really helpful. ✤ They improve their service continually. ✤ The uptime of some of their services is 99.999999999%
  7. ticketea AWS services EC2 RDS S3 CloudFront EBS ELB ElasticCache

    DynamoDB SES Route53 SNS SQS VPC CloudWatch
  8. Cloud provider Ticketea

  9. 2014 (350 frontends)

  10. The Good, the bad and the ugly with AWS

  11. ticketea EC2 && ELB ✤ Keep an eye on the

    “Events” page of the admin console. ✤ Use always more than one AZ. ✤ We reached the maximum number of instances of EC2! ✤ Pre-heat the ELBs (open a ticket in the support area).
  12. ticketea RDS ✤ Aurora ✤ Vertical auto-scaling and spikes are

    not friends. ✤ The point-in-time-snapshots system is awesome. ✤ Create new replicas is a one-click process.
  13. ticketea SQS ✤ Everything can fail: also 3 AZ of

    one region. ✤ Automate fallbacks to other regions. ✤ Be prepared to have a 2nd plan to synchronise services. ✤ No messages lost, but receive them more than once (maybe).
  14. ticketea VPC ✤ It’s really difficult to migrate from EC2.

    ✤ We’re trying to migrate gradually. ✤ Check ClassicLink to facilitate the process. ✤ Create a VPN to get access to your network. ✤ Use a Route53 internal domain to name your services.
  15. Architecture

  16. ticketea server

  17. ticketea Our architecture

  18. ticketea Our architecture

  19. ticketea Our architecture

  20. ticketea Distributed systems ✤ Granular failing. ✤ Avoid single point

    of failure. ✤ Easy code maintainability ✤ Choose the best tech for each project. ✤ Scale the team easier.
  21. ticketea Distributed systems ✤ Hard to deploy. ✤ Hard to

    see the whole picture. ✤ More complexity: easy to fail. ✤ When shit happens: you need a plan B.
  22. 2014 (350 frontends)

  23. 2015 (20 frontends)

  24. Deploying code

  25. ticketea The Rat ‘devops’ race API Heracles Odin Frontend Chef

    Puppet Ansible Thor Caronte Ansible “The Reborn”
  26. ticketea From puppet to ansible, through chef

  27. ticketea Executing commands ✤ fab -f heracles.py stop_consuming_messages:queue ✤ fab

    -f frontend.py maintenance_on ✤ fab -f sg.py open_sg ✤ fab -f api.py fast_rollback:version
  28. ticketea Development environment ✤ Use the same ansible roles ✤

    Meta-roles: roles/ api/ tasks/main.yml tkt-api/ meta/main.yml api.yml development.yml
  29. Forseti

  30. ticketea Why did we create forseti? ✤ It was 2013

    and there was no UI for Autoscaling in AWS. ✤ Other tools were too overkill for us. ✤ We wanted something small, quick and for CLI. ✤ We wanted to play around with AWS API :)
  31. ticketea How to use it? forseti deploy app

  32. ticketea deploy_and_snapshot deployer 1. Stops any autoscaling process. 2. It

    relies in other tools to deploy your app. (ansible) 3. Choose a random instance to create an AMI. 4. Setup the autoscaling group, policies and alarms.
  33. ticketea Any other thing? ✤ It has other commands: i.

    status ii. regenerate iii. list iv. clean
  34. ticketea Where can you get it? https://github.com/ticketea/forseti

  35. Graphing and logging

  36. ticketea Error reporting ✤ sentry ✤ fabric (fka. crashlytics)

  37. ticketea Graphing ✤ graphite ✤ grafana ✤ CloudWatch

  38. ticketea Logging and alarming ✤ rsyslog ✤ cabot ✤ Slack

    connected through SNS
  39. @ticketeaEng Thanks! @imanolcg