Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Architecture and Devops 2015

Architecture and Devops 2015

Talk at Devops Madrid, June 2015

Ticketea Engineering

June 18, 2015

More Decks by Ticketea Engineering

Other Decks in Programming


  1. ticketea Who we are? @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco

    @Ethervoid @imanolcg @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_
  2. ticketea Designers @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg

    @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X X X X X X X X X X
  3. ticketea Development @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg

    @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X
  4. ticketea QA @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg

    @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X X X X X X X X X X X X
  5. ticketea Sysadmins @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg

    @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X X X X X X X X X X X X X
  6. ticketea Why use AWS? ✤ We don’t have sysadmins. ✤

    Our traffic is spike-oriented. ✤ Auto-managed AWS services are really helpful. ✤ They do multi-tenancy easy. ✤ They improve their service continually. ✤ The uptime of some of their services is 99.999999999%
  7. ticketea AWS services EC2 RDS S3 CloudFront EBS ELB ElasticCache

    DynamoDB SES Route53 SNS SQS VPC CloudWatch
  8. ticketea EC2 && ELB ✤ Keep an eye on the

    “Events” page of the admin console. ✤ Use always more than one AZ. ✤ We reached the maximum number of instances of EC2! ✤ Pre-heat the ELBs (open a ticket in the support area).
  9. ticketea RDS ✤ Vertical auto-scaling and spikes are not friends.

    ✤ The point-in-time-snapshots system is awesome. ✤ The MultiAZ allows you tu vertically scale without downtime. ✤ Create new replicas is a one-click process.
  10. ticketea SQS ✤ Everything can fail: also 3 AZ of

    one region. ✤ Automate fallbacks to other regions. ✤ Be prepared to have a 2nd plan to synchronise services. ✤ No messages lost, but receive them more than once (maybe).
  11. ticketea SES ✤ We’ve been flagged as SPAM because of

    our neighbours. ✤ Be careful with the service limits. ✤ Automate fallbacks to other regions. ✤ Configure properly your domain (SPF and DKIM). ✤ Use SNS to log the bounces/complains rate.
  12. ticketea VPC ✤ It’s really difficult to migrate from EC2.

    ✤ We’re trying to migrate gradually. ✤ Check ClassicLink to facilitate the process. ✤ Create a VPN to get access to your network. ✤ Use a Route53 internal domain to name your services.
  13. ticketea The Rat ‘devops’ race API Heracles Odin Frontend Chef

    Puppet Ansible Thor Caronte Ansible “The Reborn”
  14. ticketea Executing commands ✤ fab -f heracles.py stop_consuming_messages:queue ✤ fab

    -f frontend.py maintenance_on ✤ fab -f sg.py open_sg ✤ fab -f api.py fast_rollback:version
  15. ticketea Development environment ✤ Use the same ansible roles ✤

    Meta-roles: roles/ api/ tasks/main.yml tkt-api/ meta/main.yml api.yml development.yml
  16. ticketea Distributed systems ✤ Granular failing. ✤ Avoid single point

    of failure. ✤ Easy code maintainability ✤ Choose the best tech for each project. ✤ Scale the team easier.
  17. ticketea Distributed systems ✤ Hard to deploy. ✤ Hard to

    see the whole picture. ✤ More complexity: easy to fail. ✤ When shit happens: you need a plan B.
  18. ticketea Why did we create forseti? ✤ It was 2013

    and there was no UI for Autoscaling in AWS. ✤ Other tools were too overkill for us. ✤ We wanted something small, quick and for CLI. ✤ We wanted to play around with AWS API :)
  19. ticketea What’s forseti? ✤ An AWS tool. ✤ It mainly

    manages auto-scaling groups. ✤ It’s the director of our deploys.
  20. ticketea deploy_and_snapshot deployer 1. Stops any autoscaling process. 2. It

    relies in other tools to deploy your app. 3. Choose a random instance to create an AMI. 4. Setup the autoscaling group, policies and alarms.
  21. ticketea Any other thing? ✤ It has other commands: i.

    status ii. regenerate iii. list iv. clean