$30 off During Our Annual Pro Sale. View Details »

Architecture and Devops 2015

Architecture and Devops 2015

Talk at Devops Madrid, June 2015

Ticketea Engineering

June 18, 2015
Tweet

More Decks by Ticketea Engineering

Other Decks in Programming

Transcript

  1. architecture && devops
    @ticketeaEng

    View Slide

  2. ticketea
    Who we are?
    @esanchezm @JavierHdez3 @maraujop @igalarzab
    @sullymorland @patoroco @Ethervoid
    @imanolcg
    @RafaRM20 @iamcarlosedo
    @andrea_mgr
    @Mc_Arena_pr
    @javitxudedios @Maquert @gnufede @willyfrog_

    View Slide

  3. ticketea
    Designers
    @esanchezm @JavierHdez3 @maraujop @igalarzab
    @sullymorland @patoroco @Ethervoid
    @imanolcg
    @RafaRM20 @iamcarlosedo
    @andrea_mgr
    @Mc_Arena_pr
    @javitxudedios @Maquert @gnufede @willyfrog_
    X X X X
    X
    X
    X
    X X
    X
    X
    X
    X

    View Slide

  4. ticketea
    Development
    @esanchezm @JavierHdez3 @maraujop @igalarzab
    @sullymorland @patoroco @Ethervoid
    @imanolcg
    @RafaRM20 @iamcarlosedo
    @andrea_mgr
    @Mc_Arena_pr
    @javitxudedios @Maquert @gnufede @willyfrog_
    X
    X X
    X

    View Slide

  5. ticketea
    QA
    @esanchezm @JavierHdez3 @maraujop @igalarzab
    @sullymorland @patoroco @Ethervoid
    @imanolcg
    @RafaRM20 @iamcarlosedo
    @andrea_mgr
    @Mc_Arena_pr
    @javitxudedios @Maquert @gnufede @willyfrog_
    X X X X
    X
    X
    X
    X
    X X X X
    X
    X
    X

    View Slide

  6. ticketea
    Sysadmins
    @esanchezm @JavierHdez3 @maraujop @igalarzab
    @sullymorland @patoroco @Ethervoid
    @imanolcg
    @RafaRM20 @iamcarlosedo
    @andrea_mgr
    @Mc_Arena_pr
    @javitxudedios @Maquert @gnufede @willyfrog_
    X X X X
    X
    X
    X
    X
    X X X X
    X
    X
    X
    X

    View Slide

  7. do your own deploys!

    View Slide

  8. Our systems

    View Slide

  9. View Slide

  10. ticketea
    Why use AWS?
    ✤ We don’t have sysadmins.
    ✤ Our traffic is spike-oriented.
    ✤ Auto-managed AWS services are really helpful.
    ✤ They do multi-tenancy easy.
    ✤ They improve their service continually.
    ✤ The uptime of some of their services is 99.999999999%

    View Slide

  11. ticketea
    AWS services
    EC2
    RDS
    S3
    CloudFront
    EBS
    ELB
    ElasticCache
    DynamoDB
    SES
    Route53
    SNS
    SQS
    VPC
    CloudWatch

    View Slide

  12. I can quit whenever I want

    View Slide

  13. The Good, the bad
    and the ugly with
    AWS

    View Slide

  14. ticketea
    EC2 && ELB
    ✤ Keep an eye on the “Events” page of the admin console.
    ✤ Use always more than one AZ.
    ✤ We reached the maximum number of instances of EC2!
    ✤ Pre-heat the ELBs (open a ticket in the support area).

    View Slide

  15. ticketea
    RDS
    ✤ Vertical auto-scaling and spikes are not friends.
    ✤ The point-in-time-snapshots system is awesome.
    ✤ The MultiAZ allows you tu vertically scale without downtime.
    ✤ Create new replicas is a one-click process.

    View Slide

  16. ticketea
    SQS
    ✤ Everything can fail: also 3 AZ of one region.
    ✤ Automate fallbacks to other regions.
    ✤ Be prepared to have a 2nd plan to synchronise services.
    ✤ No messages lost, but receive them more than once (maybe).

    View Slide

  17. ticketea
    SES
    ✤ We’ve been flagged as SPAM because of our neighbours.
    ✤ Be careful with the service limits.
    ✤ Automate fallbacks to other regions.
    ✤ Configure properly your domain (SPF and DKIM).
    ✤ Use SNS to log the bounces/complains rate.

    View Slide

  18. ticketea
    Route53
    ✤ Check the expiration date of your domains and SSL certs.

    View Slide

  19. ticketea
    VPC
    ✤ It’s really difficult to migrate from EC2.
    ✤ We’re trying to migrate gradually.
    ✤ Check ClassicLink to facilitate the process.
    ✤ Create a VPN to get access to your network.
    ✤ Use a Route53 internal domain to name your services.

    View Slide

  20. Don’t panic, we’re devops!

    View Slide

  21. Graphing
    and
    logging

    View Slide

  22. ticketea
    Error reporting
    ✤ sentry
    ✤ fabric (fka. crashlytics)

    View Slide

  23. ticketea
    Graphing
    ✤ graphite
    ✤ grafana
    ✤ CloudWatch

    View Slide

  24. ticketea
    Logging and alarming
    ✤ rsyslog
    ✤ cabot
    ✤ Slack connected through SNS

    View Slide

  25. Deploying
    code

    View Slide

  26. ticketea
    The Rat ‘devops’ race
    API
    Heracles
    Odin
    Frontend
    Chef
    Puppet
    Ansible
    Thor
    Caronte
    Ansible
    “The Reborn”

    View Slide

  27. View Slide

  28. ticketea
    From puppet to ansible, through chef

    View Slide

  29. ticketea
    Executing commands
    ✤ fab -f heracles.py stop_consuming_messages:queue
    ✤ fab -f frontend.py maintenance_on
    ✤ fab -f sg.py open_sg
    ✤ fab -f api.py fast_rollback:version

    View Slide

  30. ticketea
    Development environment
    ✤ Use the same ansible roles
    ✤ Meta-roles:
    roles/
    api/
    tasks/main.yml
    tkt-api/
    meta/main.yml
    api.yml
    development.yml

    View Slide

  31. ticketea
    server

    View Slide

  32. ticketea
    Our architecture

    View Slide

  33. ticketea
    Our architecture

    View Slide

  34. ticketea
    Our architecture

    View Slide

  35. ticketea
    Our architecture

    View Slide

  36. ticketea
    Our architecture

    View Slide

  37. ticketea
    Our architecture

    View Slide

  38. ticketea
    Our architecture

    View Slide

  39. ticketea
    offices

    View Slide

  40. ticketea
    Distributed systems
    ✤ Granular failing.
    ✤ Avoid single point of failure.
    ✤ Easy code maintainability
    ✤ Choose the best tech for each project.
    ✤ Scale the team easier.

    View Slide

  41. ticketea
    Distributed systems
    ✤ Hard to deploy.
    ✤ Hard to see the whole picture.
    ✤ More complexity: easy to fail.
    ✤ When shit happens: you need a plan B.

    View Slide

  42. Forseti

    View Slide

  43. ticketea
    Why did we create forseti?
    ✤ It was 2013 and there was no UI for Autoscaling in AWS.
    ✤ Other tools were too overkill for us.
    ✤ We wanted something small, quick and for CLI.
    ✤ We wanted to play around with AWS API :)

    View Slide

  44. ticketea
    What’s forseti?
    ✤ An AWS tool.
    ✤ It mainly manages auto-scaling groups.
    ✤ It’s the director of our deploys.

    View Slide

  45. ticketea
    How to use it?
    forseti deploy app

    View Slide

  46. ticketea
    deploy_and_snapshot deployer
    1. Stops any autoscaling process.
    2. It relies in other tools to deploy your app.
    3. Choose a random instance to create an AMI.
    4. Setup the autoscaling group, policies and alarms.

    View Slide

  47. ticketea
    Any other thing?
    ✤ It has other commands:
    i. status
    ii. regenerate
    iii. list
    iv. clean

    View Slide

  48. ticketea
    Where can you get it?
    https://github.com/ticketea/forseti

    View Slide

  49. ticketea
    Questions?

    View Slide

  50. @ticketeaEng
    Thanks!

    View Slide