Desplegando Ticketea

Transcript

1. @ticketeaEng Imanol Cea @imanolcg
2. ticketea Who we are? @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @imanolcg @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ @ShideShugo
3. ticketea Sysadmins? @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @imanolcg @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ @ShideShugo X X X X X X X X X X X X X X X
4. Our systems
6. ticketea Why use AWS? ✤ We don’t have sysadmins. ✤ Our traffic is spike-oriented. ✤ Auto-managed AWS services are really helpful. ✤ They improve their service continually. ✤ The uptime of some of their services is 99.999999999%
7. ticketea AWS services EC2 RDS S3 CloudFront EBS ELB ElasticCache DynamoDB SES Route53 SNS SQS VPC CloudWatch
8. Cloud provider Ticketea
9. 2014 (350 frontends)
10. The Good, the bad and the ugly with AWS
11. ticketea EC2 && ELB ✤ Keep an eye on the “Events” page of the admin console. ✤ Use always more than one AZ. ✤ We reached the maximum number of instances of EC2! ✤ Pre-heat the ELBs (open a ticket in the support area).
12. ticketea RDS ✤ Aurora ✤ Vertical auto-scaling and spikes are not friends. ✤ The point-in-time-snapshots system is awesome. ✤ Create new replicas is a one-click process.
13. ticketea SQS ✤ Everything can fail: also 3 AZ of one region. ✤ Automate fallbacks to other regions. ✤ Be prepared to have a 2nd plan to synchronise services. ✤ No messages lost, but receive them more than once (maybe).
14. ticketea VPC ✤ It’s really difficult to migrate from EC2. ✤ We’re trying to migrate gradually. ✤ Check ClassicLink to facilitate the process. ✤ Create a VPN to get access to your network. ✤ Use a Route53 internal domain to name your services.
15. Architecture
16. ticketea server
17. ticketea Our architecture
18. ticketea Our architecture
19. ticketea Our architecture
20. ticketea Distributed systems ✤ Granular failing. ✤ Avoid single point of failure. ✤ Easy code maintainability ✤ Choose the best tech for each project. ✤ Scale the team easier.
21. ticketea Distributed systems ✤ Hard to deploy. ✤ Hard to see the whole picture. ✤ More complexity: easy to fail. ✤ When shit happens: you need a plan B.
22. 2014 (350 frontends)
23. 2015 (20 frontends)
24. Deploying code
25. ticketea The Rat ‘devops’ race API Heracles Odin Frontend Chef Puppet Ansible Thor Caronte Ansible “The Reborn”
26. ticketea From puppet to ansible, through chef
27. ticketea Executing commands ✤ fab -f heracles.py stop_consuming_messages:queue ✤ fab -f frontend.py maintenance_on ✤ fab -f sg.py open_sg ✤ fab -f api.py fast_rollback:version
28. ticketea Development environment ✤ Use the same ansible roles ✤ Meta-roles: roles/ api/ tasks/main.yml tkt-api/ meta/main.yml api.yml development.yml
29. Forseti
30. ticketea Why did we create forseti? ✤ It was 2013 and there was no UI for Autoscaling in AWS. ✤ Other tools were too overkill for us. ✤ We wanted something small, quick and for CLI. ✤ We wanted to play around with AWS API :)
31. ticketea How to use it? forseti deploy app
32. ticketea deploy_and_snapshot deployer 1. Stops any autoscaling process. 2. It relies in other tools to deploy your app. (ansible) 3. Choose a random instance to create an AMI. 4. Setup the autoscaling group, policies and alarms.
33. ticketea Any other thing? ✤ It has other commands: i. status ii. regenerate iii. list iv. clean
34. ticketea Where can you get it? https://github.com/ticketea/forseti
35. Graphing and logging
36. ticketea Error reporting ✤ sentry ✤ fabric (fka. crashlytics)
37. ticketea Graphing ✤ graphite ✤ grafana ✤ CloudWatch
38. ticketea Logging and alarming ✤ rsyslog ✤ cabot ✤ Slack connected through SNS
39. @ticketeaEng Thanks! @imanolcg