Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Architecture and Devops 2015

Architecture and Devops 2015

Talk at Devops Madrid, June 2015

00927a856336d961bdc7028722fe5897?s=128

Ticketea Engineering

June 18, 2015
Tweet

Transcript

  1. architecture && devops @ticketeaEng

  2. ticketea Who we are? @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco

    @Ethervoid @imanolcg @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_
  3. ticketea Designers @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg

    @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X X X X X X X X X X
  4. ticketea Development @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg

    @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X
  5. ticketea QA @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg

    @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X X X X X X X X X X X X
  6. ticketea Sysadmins @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg

    @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X X X X X X X X X X X X X
  7. do your own deploys!

  8. Our systems

  9. None
  10. ticketea Why use AWS? ✤ We don’t have sysadmins. ✤

    Our traffic is spike-oriented. ✤ Auto-managed AWS services are really helpful. ✤ They do multi-tenancy easy. ✤ They improve their service continually. ✤ The uptime of some of their services is 99.999999999%
  11. ticketea AWS services EC2 RDS S3 CloudFront EBS ELB ElasticCache

    DynamoDB SES Route53 SNS SQS VPC CloudWatch
  12. I can quit whenever I want

  13. The Good, the bad and the ugly with AWS

  14. ticketea EC2 && ELB ✤ Keep an eye on the

    “Events” page of the admin console. ✤ Use always more than one AZ. ✤ We reached the maximum number of instances of EC2! ✤ Pre-heat the ELBs (open a ticket in the support area).
  15. ticketea RDS ✤ Vertical auto-scaling and spikes are not friends.

    ✤ The point-in-time-snapshots system is awesome. ✤ The MultiAZ allows you tu vertically scale without downtime. ✤ Create new replicas is a one-click process.
  16. ticketea SQS ✤ Everything can fail: also 3 AZ of

    one region. ✤ Automate fallbacks to other regions. ✤ Be prepared to have a 2nd plan to synchronise services. ✤ No messages lost, but receive them more than once (maybe).
  17. ticketea SES ✤ We’ve been flagged as SPAM because of

    our neighbours. ✤ Be careful with the service limits. ✤ Automate fallbacks to other regions. ✤ Configure properly your domain (SPF and DKIM). ✤ Use SNS to log the bounces/complains rate.
  18. ticketea Route53 ✤ Check the expiration date of your domains

    and SSL certs.
  19. ticketea VPC ✤ It’s really difficult to migrate from EC2.

    ✤ We’re trying to migrate gradually. ✤ Check ClassicLink to facilitate the process. ✤ Create a VPN to get access to your network. ✤ Use a Route53 internal domain to name your services.
  20. Don’t panic, we’re devops!

  21. Graphing and logging

  22. ticketea Error reporting ✤ sentry ✤ fabric (fka. crashlytics)

  23. ticketea Graphing ✤ graphite ✤ grafana ✤ CloudWatch

  24. ticketea Logging and alarming ✤ rsyslog ✤ cabot ✤ Slack

    connected through SNS
  25. Deploying code

  26. ticketea The Rat ‘devops’ race API Heracles Odin Frontend Chef

    Puppet Ansible Thor Caronte Ansible “The Reborn”
  27. None
  28. ticketea From puppet to ansible, through chef

  29. ticketea Executing commands ✤ fab -f heracles.py stop_consuming_messages:queue ✤ fab

    -f frontend.py maintenance_on ✤ fab -f sg.py open_sg ✤ fab -f api.py fast_rollback:version
  30. ticketea Development environment ✤ Use the same ansible roles ✤

    Meta-roles: roles/ api/ tasks/main.yml tkt-api/ meta/main.yml api.yml development.yml
  31. ticketea server

  32. ticketea Our architecture

  33. ticketea Our architecture

  34. ticketea Our architecture

  35. ticketea Our architecture

  36. ticketea Our architecture

  37. ticketea Our architecture

  38. ticketea Our architecture

  39. ticketea offices

  40. ticketea Distributed systems ✤ Granular failing. ✤ Avoid single point

    of failure. ✤ Easy code maintainability ✤ Choose the best tech for each project. ✤ Scale the team easier.
  41. ticketea Distributed systems ✤ Hard to deploy. ✤ Hard to

    see the whole picture. ✤ More complexity: easy to fail. ✤ When shit happens: you need a plan B.
  42. Forseti

  43. ticketea Why did we create forseti? ✤ It was 2013

    and there was no UI for Autoscaling in AWS. ✤ Other tools were too overkill for us. ✤ We wanted something small, quick and for CLI. ✤ We wanted to play around with AWS API :)
  44. ticketea What’s forseti? ✤ An AWS tool. ✤ It mainly

    manages auto-scaling groups. ✤ It’s the director of our deploys.
  45. ticketea How to use it? forseti deploy app

  46. ticketea deploy_and_snapshot deployer 1. Stops any autoscaling process. 2. It

    relies in other tools to deploy your app. 3. Choose a random instance to create an AMI. 4. Setup the autoscaling group, policies and alarms.
  47. ticketea Any other thing? ✤ It has other commands: i.

    status ii. regenerate iii. list iv. clean
  48. ticketea Where can you get it? https://github.com/ticketea/forseti

  49. ticketea Questions?

  50. @ticketeaEng Thanks!