Automation Principles at Helpshift Raghu Udiyar Production Engineering Manager @ Helpshift 

75k+ RPS 600M+ MAU 5+ releases / day 500 GiB / day 

Ⅰ - Workflows High level objective with all its logic in one place End to End Automation, avoiding Piecemeal solutions 

E.g. Provision Instance Workflow $ ansible-playbook provision.yml -e “cli_cloud=aws cli_cluster_name=mongocl01 cli_count=6 cli_instance_type=r3.xlarge... Single Interface encapsulates all logic of provisioning like instance configuration, monitoring, OS configuration, etc 

E.g. Release Software $ release.sh ... Release a service while taking care dependencies, service configuration, health-checks, etc 

E.g. Setup Kafka Cluster $ ansible-playbook kafka_cluster.yml -e 'cli_cluster_name=kafkacl01 cli_count=6 cli_zookper=cl_zk… ● Provisioning ● Configuration ● Monitoring ● Backups 

Demo: Redis Cluster Workflow 

Ⅱ - Explicit Control Flow ● Push vs Pull Model : We follow push model ○ E.g: AMI or agent based configuration ● Avoid Inversion of Control ● Simple, Easy to reason and maintain ● Better error handling 

Ⅲ - Idempotent Workflows Idempotency at all levels, i.e. at Workflow level Easy to reason, and well defined behaviour 

Idempotency with Redis cluster ▪ If cluster already exists, and configured, won't do any side-effects if re-run ▪ Change cli_slave_count=2 results in unsurprising, and well defined behaviour 

Ⅳ- Think like a Programmer ● Implement consistent Interfaces for workflows ● Abstract common tasks together (e.g. provisioning) ● Encapsulate all logic inside the respective workflows 

Redis Cluster abstractions ● redis_cluster.yml ○ provision.yml ■ Provision instance, Users, Monitoring, Disks etc ○ redis role ■ Configuration, redis_type (master / slave) 

Ansible + Workflows Infrastructure, Orchestration and Configuration management 

Ansible Self Service via Jenkins ● Dev test machines ● Sandboxes ● Prod releases Systems Automation ● Autoscaling ● OS configuration ● Security Backend Services Postgresql, Kafka, Mongodb, Nginx, Haproxy, redis, etc 

In Summary 1. Automate with workflows in mind 2. Explicit Control Flow 3. Idempotency all the way 4. Think like a programmer 

Thank You