Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automation Principles at Helpshift

Automation Principles at Helpshift

Everyone knows automation in operations is necessary and is a key objective of any DevOps practice. But if not started on the right foot, or with the right objective in mind, can often leads to false starts and bad design choices. Building and extending to incorporate new requirements gets increasingly difficult leading to hacks and rewrites.

At Helpshift we follow certain principles and guidelines that have helped us avoid these pitfalls, and allow us to build a stable and extensible automations framework using Ansible.

I will walk through these principles, the thought process behind them, and the results through demos.

Raghu Udiyar

November 23, 2017
Tweet

More Decks by Raghu Udiyar

Other Decks in Programming

Transcript

  1. Automation Principles at
    Helpshift
    Raghu Udiyar
    Production Engineering Manager @ Helpshift

    View Slide

  2. 75k+ RPS 600M+ MAU
    5+ releases /
    day
    500 GiB / day

    View Slide

  3. Ⅰ - Workflows
    High level objective with all its logic in one place
    End to End Automation, avoiding Piecemeal solutions

    View Slide

  4. E.g. Provision
    Instance Workflow
    $ ansible-playbook provision.yml -e “cli_cloud=aws
    cli_cluster_name=mongocl01 cli_count=6
    cli_instance_type=r3.xlarge...
    Single Interface encapsulates all logic of provisioning
    like instance configuration, monitoring, OS configuration, etc

    View Slide

  5. E.g. Release
    Software
    $ release.sh ...
    Release a service while taking care dependencies, service
    configuration, health-checks, etc

    View Slide

  6. E.g. Setup Kafka
    Cluster
    $ ansible-playbook kafka_cluster.yml -e
    'cli_cluster_name=kafkacl01 cli_count=6 cli_zookper=cl_zk…
    ● Provisioning
    ● Configuration
    ● Monitoring
    ● Backups

    View Slide

  7. Demo: Redis Cluster
    Workflow

    View Slide

  8. Ⅱ - Explicit
    Control Flow
    ● Push vs Pull Model : We follow push model
    ○ E.g: AMI or agent based configuration
    ● Avoid Inversion of Control
    ● Simple, Easy to reason and maintain
    ● Better error handling

    View Slide

  9. Ⅲ - Idempotent
    Workflows
    Idempotency at all levels, i.e. at Workflow level
    Easy to reason, and well defined behaviour

    View Slide

  10. Idempotency with
    Redis cluster
    ▪ If cluster already exists, and configured, won't do any
    side-effects if re-run
    ▪ Change cli_slave_count=2 results in unsurprising, and
    well defined behaviour

    View Slide

  11. Ⅳ- Think like a
    Programmer
    ● Implement consistent Interfaces for workflows
    ● Abstract common tasks together (e.g. provisioning)
    ● Encapsulate all logic inside the respective workflows

    View Slide

  12. Redis Cluster
    abstractions
    ● redis_cluster.yml
    ○ provision.yml
    ■ Provision instance, Users, Monitoring, Disks etc
    ○ redis role
    ■ Configuration, redis_type (master / slave)

    View Slide

  13. Ansible +
    Workflows
    Infrastructure, Orchestration and Configuration
    management

    View Slide

  14. Ansible
    Self Service via Jenkins
    ● Dev test machines
    ● Sandboxes
    ● Prod releases
    Systems Automation
    ● Autoscaling
    ● OS configuration
    ● Security
    Backend Services
    Postgresql, Kafka,
    Mongodb, Nginx,
    Haproxy, redis, etc

    View Slide

  15. In Summary
    1. Automate with workflows in mind
    2. Explicit Control Flow
    3. Idempotency all the way
    4. Think like a programmer

    View Slide

  16. Thank You

    View Slide