Building a [micro]services platform on AWS

Gousto @ AWS Meetup May 2016

Transcript

1. Shaun Pearce VP Engineering Building a [mIcro]services platform on AWS
2. About Gousto • An online recipe box service • Customers come to our site, or use our apps and select from 12 meals each week. • They pick the meals they want to cook and say how many people they’re cooking for • We deliver all the ingredients they need in exact proportions with step-by-step recipe cards. • No planning, no supermarkets and no food waste – you just cook (and eat) • We’re a rapidly growing business • With a diverse set of technology requirements. • And it needs to be delivered yesterday…
3. In the beginning… • Been on AWS from the outset • But getting very little value from being on the platform • Developed a 2 tier API driven platform. • Suffered from tight coupling between API and front end • We had built ourselves a 2 tier monolith • We were experiencing a lot of growing pains: • 2 hour deployment process • Difficult to introduce new clients • 150k lines of code in a single repo – onboarding devs was slow • All infrastructure managed through the console – Massive risk • Production deployments happening every 2 weeks at most Web Site (PHP Laravel) Business Logic (PHP Laravel) Users CMS Users Data Scientists Reporting Read Replica Production Database MySQL MySQL Tightly Coupled APIs
4. Build a platform, not a collection of independent services Common Service NGINX . . . Ansible Roles Platform.yml Tests Network Content Security . . . Platform Templates Data & Cache Auto Scaling & ELBs Alarms & Logs . . . Gousto Service Automated Deployment Pipeline CloudFormation Platform templates Platform Deployment Bucket Ansible Roles and Service Templates
5. This creates consistency from the outset • Upfront investment made consistency the path of least resistance • A service is defined as: • A deployment descriptor • Additional Ansible roles • CloudFormation additions and overrides • The source code • This approach bakes in consistency and best practice • Still allows complete customisation where really needed • Language agnostic • Our first 2 [Micro]services were our monoliths! src ansible cloudFormation service.yml roles overrides additions playbook.yml Service Repository
6. From this… …to this
7. Where did AWS really help?
8. I ♥
9. Asynchronous notifications pattern Customer Service Payment Service Delivery Service . . . I’ve changed address Got it, thanks. <address change> 1 3 2 4 SQS Client owned subscriptions SNS
10. Create a platform wide event log Customer Service Activity Log Service Order Service Product Service Recipe Service . . . AWS Lambda Amazon DynamoDB Platform Deployment Bucket SNS Subscribe to all messages Event API Amazon Redshift Data Scientists AWS Lambda Subscribe to subset of messages
11. Where did we underinvest?
12. n services and >1 team = lots of ways to paginate… • We had standardised our infrastructure and deployment pipelines but neglected our APIs • Needed to create a consistent implementation of • Data representation • Error handling • Pagination • Media management • Versioning • … • We used JSON Schema to standardise primitives • Test conformity to standards within your CI pipeline
13. Build documentation into your workflow editor.swagger.io api.yml bootprint-swagger
14. Don’t forget about your async messaging • Consistent Async messaging is as vital as our APIs • Most messages announced CUD events on domain objects (order-created) • Message contents was normally the JSON representation of the object • Accessing common metadata was brittle • So we introduced a message Schema: • Required message senders to extract consistent metadata from the message payload • Helps to prevent model changes causing cascading breaks across services.
15. Insufficient log management & request tracing • CloudWatch and CloudWatch Logs isn't sufficient to support a Microservice platform in production • UI is lacking the functionality needed to locate specific events in a single Log Group • Don’t even think about trying to correlate events across multiple groups… • Why did we use it: • It already there • Integration with Cloudwatch alarms was a simple way to start filtering and monitoring log events • Very easy to get started and well supported in CloudFormation • We’re now exploring alternatives (ELK stack + injecting UUID for each request)
16. AWS has allowed us to move faster So far • We currently have 9 services in production, launching 2 more in the coming months • Upfront investment has driven consistency into our services • Moved from 1 production deployment per sprint to approx. 3 per day • Gone from 150k + lines of code in a repo to around. 3-4k Next • Looking to use Lambda for more of our smaller services • Will be investing in a CLI to make bootstrapping services even quicker 0 5 10 15 20 25 30 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Week 11 Week 12 Week 13 Week 14 Week 15 Week 16 Week 17 Week 18 Week 19 Week 20 Week 21 Week 22 Week 23 Week 24 Week 25 Week 26 Week 27 Week 28 Week 29 Week 30 Week 31 Week 32 Week 33 Weekly Production Deployments core fe platform gateway products auth gifts admin maintenance
17. @GoustoTech techbrunch.gousto.co.uk shaun@gousto.co.uk