Desplegando Ticketea

Transcript

1. @ticketeaEng
   Imanol Cea
   @imanolcg
   

   View Slide

2. ticketea
   Who we are?
   @JavierHdez3 @maraujop @igalarzab @sullymorland
   @patoroco @imanolcg @RafaRM20
   @iamcarlosedo
   @andrea_mgr
   @Mc_Arena_pr
   @javitxudedios @Maquert
   @gnufede @willyfrog_ @ShideShugo
   

   View Slide

3. ticketea
   Sysadmins?
   @JavierHdez3 @maraujop @igalarzab @sullymorland
   @patoroco @imanolcg @RafaRM20
   @iamcarlosedo
   @andrea_mgr
   @Mc_Arena_pr
   @javitxudedios @Maquert
   @gnufede @willyfrog_ @ShideShugo
   X X X X
   X
   X
   X
   X
   X X X X
   X
   X
   X
   

   View Slide

4. Our systems
   

   View Slide

5. View Slide

6. ticketea
   Why use AWS?
   ✤ We don’t have sysadmins.
   ✤ Our traffic is spike-oriented.
   ✤ Auto-managed AWS services are really helpful.
   ✤ They improve their service continually.
   ✤ The uptime of some of their services is 99.999999999%
   

   View Slide

7. ticketea
   AWS services
   EC2
   RDS
   S3
   CloudFront
   EBS
   ELB
   ElasticCache
   DynamoDB
   SES
   Route53
   SNS
   SQS
   VPC
   CloudWatch
   

   View Slide

8. Cloud
   provider
   Ticketea
   

   View Slide

9. 2014
   (350 frontends)
   

   View Slide

10. The Good, the bad
    and the ugly with
    AWS
    

    View Slide

11. ticketea
    EC2 && ELB
    ✤ Keep an eye on the “Events” page of the admin console.
    ✤ Use always more than one AZ.
    ✤ We reached the maximum number of instances of EC2!
    ✤ Pre-heat the ELBs (open a ticket in the support area).
    

    View Slide

12. ticketea
    RDS
    ✤ Aurora
    ✤ Vertical auto-scaling and spikes are not friends.
    ✤ The point-in-time-snapshots system is awesome.
    ✤ Create new replicas is a one-click process.
    

    View Slide

13. ticketea
    SQS
    ✤ Everything can fail: also 3 AZ of one region.
    ✤ Automate fallbacks to other regions.
    ✤ Be prepared to have a 2nd plan to synchronise services.
    ✤ No messages lost, but receive them more than once (maybe).
    

    View Slide

14. ticketea
    VPC
    ✤ It’s really difficult to migrate from EC2.
    ✤ We’re trying to migrate gradually.
    ✤ Check ClassicLink to facilitate the process.
    ✤ Create a VPN to get access to your network.
    ✤ Use a Route53 internal domain to name your services.
    

    View Slide

15. Architecture
    

    View Slide

16. ticketea
    server
    

    View Slide

17. ticketea
    Our architecture
    

    View Slide

18. ticketea
    Our architecture
    

    View Slide

19. ticketea
    Our architecture
    

    View Slide

20. ticketea
    Distributed systems
    ✤ Granular failing.
    ✤ Avoid single point of failure.
    ✤ Easy code maintainability
    ✤ Choose the best tech for each project.
    ✤ Scale the team easier.
    

    View Slide

21. ticketea
    Distributed systems
    ✤ Hard to deploy.
    ✤ Hard to see the whole picture.
    ✤ More complexity: easy to fail.
    ✤ When shit happens: you need a plan B.
    

    View Slide

22. 2014
    (350 frontends)
    

    View Slide

23. 2015
    (20 frontends)
    

    View Slide

24. Deploying
    code
    

    View Slide

25. ticketea
    The Rat ‘devops’ race
    API
    Heracles
    Odin
    Frontend
    Chef
    Puppet
    Ansible
    Thor
    Caronte
    Ansible
    “The Reborn”
    

    View Slide

26. ticketea
    From puppet to ansible, through chef
    

    View Slide

27. ticketea
    Executing commands
    ✤ fab -f heracles.py stop_consuming_messages:queue
    ✤ fab -f frontend.py maintenance_on
    ✤ fab -f sg.py open_sg
    ✤ fab -f api.py fast_rollback:version
    

    View Slide

28. ticketea
    Development environment
    ✤ Use the same ansible roles
    ✤ Meta-roles:
    roles/
    api/
    tasks/main.yml
    tkt-api/
    meta/main.yml
    api.yml
    development.yml
    

    View Slide

29. Forseti
    

    View Slide

30. ticketea
    Why did we create forseti?
    ✤ It was 2013 and there was no UI for Autoscaling in AWS.
    ✤ Other tools were too overkill for us.
    ✤ We wanted something small, quick and for CLI.
    ✤ We wanted to play around with AWS API :)
    

    View Slide

31. ticketea
    How to use it?
    forseti deploy app
    

    View Slide

32. ticketea
    deploy_and_snapshot deployer
    1. Stops any autoscaling process.
    2. It relies in other tools to deploy your app. (ansible)
    3. Choose a random instance to create an AMI.
    4. Setup the autoscaling group, policies and alarms.
    

    View Slide

33. ticketea
    Any other thing?
    ✤ It has other commands:
    i. status
    ii. regenerate
    iii. list
    iv. clean
    

    View Slide

34. ticketea
    Where can you get it?
    https://github.com/ticketea/forseti
    

    View Slide

35. Graphing
    and
    logging
    

    View Slide

36. ticketea
    Error reporting
    ✤ sentry
    ✤ fabric (fka. crashlytics)
    

    View Slide

37. ticketea
    Graphing
    ✤ graphite
    ✤ grafana
    ✤ CloudWatch
    

    View Slide

38. ticketea
    Logging and alarming
    ✤ rsyslog
    ✤ cabot
    ✤ Slack connected through SNS
    

    View Slide

39. @ticketeaEng
    Thanks!
    @imanolcg
    

    View Slide