Architecture and Devops 2015

Talk at Devops Madrid, June 2015

Transcript

1. architecture && devops
   @ticketeaEng
   

   View Slide

2. ticketea
   Who we are?
   @esanchezm @JavierHdez3 @maraujop @igalarzab
   @sullymorland @patoroco @Ethervoid
   @imanolcg
   @RafaRM20 @iamcarlosedo
   @andrea_mgr
   @Mc_Arena_pr
   @javitxudedios @Maquert @gnufede @willyfrog_
   

   View Slide

3. ticketea
   Designers
   @esanchezm @JavierHdez3 @maraujop @igalarzab
   @sullymorland @patoroco @Ethervoid
   @imanolcg
   @RafaRM20 @iamcarlosedo
   @andrea_mgr
   @Mc_Arena_pr
   @javitxudedios @Maquert @gnufede @willyfrog_
   X X X X
   X
   X
   X
   X X
   X
   X
   X
   X
   

   View Slide

4. ticketea
   Development
   @esanchezm @JavierHdez3 @maraujop @igalarzab
   @sullymorland @patoroco @Ethervoid
   @imanolcg
   @RafaRM20 @iamcarlosedo
   @andrea_mgr
   @Mc_Arena_pr
   @javitxudedios @Maquert @gnufede @willyfrog_
   X
   X X
   X
   

   View Slide

5. ticketea
   QA
   @esanchezm @JavierHdez3 @maraujop @igalarzab
   @sullymorland @patoroco @Ethervoid
   @imanolcg
   @RafaRM20 @iamcarlosedo
   @andrea_mgr
   @Mc_Arena_pr
   @javitxudedios @Maquert @gnufede @willyfrog_
   X X X X
   X
   X
   X
   X
   X X X X
   X
   X
   X
   

   View Slide

6. ticketea
   Sysadmins
   @esanchezm @JavierHdez3 @maraujop @igalarzab
   @sullymorland @patoroco @Ethervoid
   @imanolcg
   @RafaRM20 @iamcarlosedo
   @andrea_mgr
   @Mc_Arena_pr
   @javitxudedios @Maquert @gnufede @willyfrog_
   X X X X
   X
   X
   X
   X
   X X X X
   X
   X
   X
   X
   

   View Slide

7. do your own deploys!
   

   View Slide

8. Our systems
   

   View Slide

9. View Slide

10. ticketea
    Why use AWS?
    ✤ We don’t have sysadmins.
    ✤ Our traffic is spike-oriented.
    ✤ Auto-managed AWS services are really helpful.
    ✤ They do multi-tenancy easy.
    ✤ They improve their service continually.
    ✤ The uptime of some of their services is 99.999999999%
    

    View Slide

11. ticketea
    AWS services
    EC2
    RDS
    S3
    CloudFront
    EBS
    ELB
    ElasticCache
    DynamoDB
    SES
    Route53
    SNS
    SQS
    VPC
    CloudWatch
    

    View Slide

12. I can quit whenever I want
    

    View Slide

13. The Good, the bad
    and the ugly with
    AWS
    

    View Slide

14. ticketea
    EC2 && ELB
    ✤ Keep an eye on the “Events” page of the admin console.
    ✤ Use always more than one AZ.
    ✤ We reached the maximum number of instances of EC2!
    ✤ Pre-heat the ELBs (open a ticket in the support area).
    

    View Slide

15. ticketea
    RDS
    ✤ Vertical auto-scaling and spikes are not friends.
    ✤ The point-in-time-snapshots system is awesome.
    ✤ The MultiAZ allows you tu vertically scale without downtime.
    ✤ Create new replicas is a one-click process.
    

    View Slide

16. ticketea
    SQS
    ✤ Everything can fail: also 3 AZ of one region.
    ✤ Automate fallbacks to other regions.
    ✤ Be prepared to have a 2nd plan to synchronise services.
    ✤ No messages lost, but receive them more than once (maybe).
    

    View Slide

17. ticketea
    SES
    ✤ We’ve been flagged as SPAM because of our neighbours.
    ✤ Be careful with the service limits.
    ✤ Automate fallbacks to other regions.
    ✤ Configure properly your domain (SPF and DKIM).
    ✤ Use SNS to log the bounces/complains rate.
    

    View Slide

18. ticketea
    Route53
    ✤ Check the expiration date of your domains and SSL certs.
    

    View Slide

19. ticketea
    VPC
    ✤ It’s really difficult to migrate from EC2.
    ✤ We’re trying to migrate gradually.
    ✤ Check ClassicLink to facilitate the process.
    ✤ Create a VPN to get access to your network.
    ✤ Use a Route53 internal domain to name your services.
    

    View Slide

20. Don’t panic, we’re devops!
    

    View Slide

21. Graphing
    and
    logging
    

    View Slide

22. ticketea
    Error reporting
    ✤ sentry
    ✤ fabric (fka. crashlytics)
    

    View Slide

23. ticketea
    Graphing
    ✤ graphite
    ✤ grafana
    ✤ CloudWatch
    

    View Slide

24. ticketea
    Logging and alarming
    ✤ rsyslog
    ✤ cabot
    ✤ Slack connected through SNS
    

    View Slide

25. Deploying
    code
    

    View Slide

26. ticketea
    The Rat ‘devops’ race
    API
    Heracles
    Odin
    Frontend
    Chef
    Puppet
    Ansible
    Thor
    Caronte
    Ansible
    “The Reborn”
    

    View Slide

27. View Slide

28. ticketea
    From puppet to ansible, through chef
    

    View Slide

29. ticketea
    Executing commands
    ✤ fab -f heracles.py stop_consuming_messages:queue
    ✤ fab -f frontend.py maintenance_on
    ✤ fab -f sg.py open_sg
    ✤ fab -f api.py fast_rollback:version
    

    View Slide

30. ticketea
    Development environment
    ✤ Use the same ansible roles
    ✤ Meta-roles:
    roles/
    api/
    tasks/main.yml
    tkt-api/
    meta/main.yml
    api.yml
    development.yml
    

    View Slide

31. ticketea
    server
    

    View Slide

32. ticketea
    Our architecture
    

    View Slide

33. ticketea
    Our architecture
    

    View Slide

34. ticketea
    Our architecture
    

    View Slide

35. ticketea
    Our architecture
    

    View Slide

36. ticketea
    Our architecture
    

    View Slide

37. ticketea
    Our architecture
    

    View Slide

38. ticketea
    Our architecture
    

    View Slide

39. ticketea
    offices
    

    View Slide

40. ticketea
    Distributed systems
    ✤ Granular failing.
    ✤ Avoid single point of failure.
    ✤ Easy code maintainability
    ✤ Choose the best tech for each project.
    ✤ Scale the team easier.
    

    View Slide

41. ticketea
    Distributed systems
    ✤ Hard to deploy.
    ✤ Hard to see the whole picture.
    ✤ More complexity: easy to fail.
    ✤ When shit happens: you need a plan B.
    

    View Slide

42. Forseti
    

    View Slide

43. ticketea
    Why did we create forseti?
    ✤ It was 2013 and there was no UI for Autoscaling in AWS.
    ✤ Other tools were too overkill for us.
    ✤ We wanted something small, quick and for CLI.
    ✤ We wanted to play around with AWS API :)
    

    View Slide

44. ticketea
    What’s forseti?
    ✤ An AWS tool.
    ✤ It mainly manages auto-scaling groups.
    ✤ It’s the director of our deploys.
    

    View Slide

45. ticketea
    How to use it?
    forseti deploy app
    

    View Slide

46. ticketea
    deploy_and_snapshot deployer
    1. Stops any autoscaling process.
    2. It relies in other tools to deploy your app.
    3. Choose a random instance to create an AMI.
    4. Setup the autoscaling group, policies and alarms.
    

    View Slide

47. ticketea
    Any other thing?
    ✤ It has other commands:
    i. status
    ii. regenerate
    iii. list
    iv. clean
    

    View Slide

48. ticketea
    Where can you get it?
    https://github.com/ticketea/forseti
    

    View Slide

49. ticketea
    Questions?
    

    View Slide

50. @ticketeaEng
    Thanks!
    

    View Slide