How we built a heroku for ourselves

Presentation done at Quarterly Large Scale Production Engineering (LSPE) meet up (https://www.meetup.com/lspe-in/events/qvgqgdyzfbmb/)

1. How we built a heroku for ourselves
2. Senthil V S siliconsenthil jeechudeka Jeechu Deka
3. Before we start…
4. Done microservices?
5. Used AWS?
6. Know to cook?
7. Know to order food?
8. Cook or Buy
9. Agenda The problem cloudlift, our solution Learnings Q & A
10. Simpl Virtual CC for users make money simple, so that people can live well and do amazing things. PG for merchants We enable payment without banks
11. Platform Simpl Merchant servers to API Merchant apps SDKs Merchant apps Sim pl user facing Apps
12. Simpl architecture Services Streaming Platform Data Pipeline Dashboards Analytics ML models Merchant Apps Simpl Website Different tech stacks Scalability requirements Security requirements services
13. So… Four tech stacks (RoR, GoLang, Node, Python) Five developers. No (Dev)Ops Scaling needs are different Security needs are different Centralized logging Centralized config management
14. We wanted a solution, that…. Multiple tech stacks No (Dev)Ops. Easy to use Standard stack across platform Customizable standard
15. Possible solutions • Hand-crafted (?!) servers • Config management tools (chef, puppet etc.) • PaaS (Heroku, now.sh, AWS ElasticBeanstalk etc.)
16. Config management tools • Using Chef/puppet/Terraform etc. • Heavy-weight. Highly customizable leading to complex codebase. • Platform architecture being reflected. • Every change in service communications could require change in DevOps platform
17. PaaS • Easier to get started • No way to customize • Microservices: Internal vs external • Costly
18. PaaS is force-fitting IaaS is hard What did we do…
19. Or something in-between Cook or Buy
20. Build a PaaS on IaaS
21. Cloudformation • Infra as code • AWS specific • Resources defined in template and applied • Idempotent
22. Cloudformation template
23. ECS • ECS: Elastic Container Service • Orchestrator engine to run containers • Runs on cluster of EC2 instances • Exposes API to run/stop/scale containers • Container repository to push images
24. Service Task definition ✓ Network mode ✓ Constraints Container definition ✓ ECR image version ✓ Env vars ✓ Port mapping ✓ Command to run ✓ Logs ✓ Cluster i.e. set of EC2 instances ✓ Number of tasks ✓ High availability deployment ✓ Task placement ✓ Load balancing (ELB/ALB) ✓ Autoscaling ECS terms ✓ECR: EC2 Container registry ✓ELB: Elastic load balancer ✓ALB: Application load balancer ✓Port mapping ✓ Container port <-> Container instance port ✓ Can be dynamic ✓Task placement ✓ AZ Balanced spread ✓ One Task per Host ✓ BinPack ✓Constraints ✓ allow you to filter the instances ✓ using built-in or custom attributes ✓Network mode ✓ Bridge/Host etc. ✓Tasks run from Task definition ✓Containers run from Container definition
25. Demo Creating a service Deploying the service Scaling it Monitoring & logs
26. Cloudlift architecture click library (cli -> Python methods) Boto Python code to AWS api Cloudformation template
27. Cloudlift • Built by two developers at Simpl & in production use • Used in a heroku style. (Yeah. You can call it heroic style ;)) • Fully customizable • Open sourced
28. What happens on deploy_service ? Push local docker 
 image to ECR Ensure S3 has all required config env.sample Docker registry ECR Update the task definition with image version & env vars ECS runs new containers with updated version Stops the old version once new are up ★ The service name is guessed from current dir ★ image name is same as repo name in 
 spinal case without simpl- prefix ★ latest version is picked to push 1 2 3 4 5
29. A solution Multiple tech stacks No (Dev)Ops. Easy to use Standard stack across platform Customizable standard Docker }Cloudformation CLI
30. Show me code
31. Learnings
32. Microservices & DevOps - Keep it lightweight - Treat service creation/removal a frequent thing to do - Empower developers
33. Microservices & DevOps - Build time dependency => library - Runtime dependency => another microservice - Lifecycle dependency => DevOps
34. ECS is not batteries-included. Simple and does the job. Well.
35. Never hand-code Cloudformation template
36. Bake-in monitoring and alerts
37. Treat secrets and other configurations same.
38. 12 factor apps Env var config Port binding Dev/prod parity Logs as event streams
39. Prefer CLI over GUI Building CLI is easier Easier to integrate in CI tools Security
40. CLI: Use defaults for almost everything
41. CLI: Avoid positional params except, readable command options
42. CLI: Help is must Because it’s for humans
43. CLI: Dry-run is must Because humans err
44. CLI: Dry-run is must Because humans err Confirm before running Ability to abort
45. CLI: Log the progress & details Because humans are impatient Log levels As colorful as possible
46. Follow standard unix exit codes Because machines have to understand too
47. Version it Because things improve
48. Q & A
49. siliconsenthil jeechudeka