Handle your Lambdas - From event-based processing to Continuous Integration / Munich AWS User Group - Mar 17, 2016

Handle your Lambdas - From event-based processing to Continuous Integration / Munich AWS User Group - Mar 17, 2016

Transcript

1. Handle your Lambdas: From event-based processing to Continuous Integration Sergii Khomenko, Data Scientist sergii.khomenko@stylight.com, @lc0d3r Munich AWS User Group - Mar 17 2016
2. Sergii Khomenko 2 Data scientist at one of the biggest fashion communities, Stylight. Data analysis and visualisation hobbyist, working on problems not only in working time but in free time for fun and personal data visualisations. Speaker at Berlin Buzzwords 2014, ApacheCon Europe 2014, Puppet Camp London 2015, Berlin Buzzwords 2015 , Tableau Conference on Tour 2015, Budapest BI Forum 2015, Crunchsconf 2015, FOSDEM 2016, PyData Amsterdam 2016
3. Profitable Leads Stylight provides its partners with high- quality leads enabling partner shops to leverage Stylight as a ROI positive traffic channel. Inspiration Stylight offers shoppable inspiration that makes it easy to know what to buy and how to style it. Branding & Reach Stylight offers a unique opportunity for brands to reach an audience that is actively looking for style online. Shopping Stylight helps users search and shop fashion and lifestyle products smarter across hundreds of shops. 3 Stylight – Make Style Happen Core Target Group Stylight help aspiring women between 18 and 35 to evolve their style through shoppable inspiration.
4. Stylight – acting on a global scale
5. Experienced & Ambitious Team Innovative cross- functional organisation with flat hierarchy builds a 
 unique team spirit. • +200 employees • 40 Engineers/PhDs • 28 years average age • 63% female • 23 nationalities • 0 suits 5
7. A Brief History of Computing 7
9. 9 Charles Babbage's machine - 1832
10. 10 1940 – 1956: First Generation – Vacuum Tubes
11. 11 1956 – 1963: Second Generation – Transistors
12. 12 1964 – 1971: Third Generation – Integrated Circuits
13. 13 1972- … : Fourth Generation – Microprocessors
16. Immutable infrastructure 16 Infrastructure as Code
17. 17 N e e d t o u p g r a d e ? N o p r o b l e m . B u i l d a n e w , u p g r a d e d s y s t e m a n d t h r o w t h e o l d o n e a w a y . N e w a p p r e v i s i o n ? S a m e t h i n g . B u i l d a s e r v e r ( o r i m a g e ) w i t h a n e w r e v i s i o n a n d t h r o w a w a y t h e o l d o n e s .
21. 21 CloudFormation
22. 22 Terraform
24. 24 Terraform Kubernetes and Docker {Swarm, Compose}
25. Serverless architecture 25
32. Installation 32 export GOPATH=$HOME/go export PATH=$PATH:$GOPATH/bin go get github.com/apex/apex/cmd/apex export AWS_REGION=eu-west-1
36. 36 main.go
37. 37 main.py
38. Python dependencies 38 pip install requests -t functions/dependency
39. 39 apex invoke
40. Outcome 40 • all Lambdas in one place with version control • integration tests with real events • proper CI/CD setup
41. 41 CircleCI deployments
42. 42 CircleCI deployments
43. 43 CircleCI deployments
44. 44 Cloud functions
45. Use-case CloudWatch alerting 45
49. Use-case of outlier detection 49
51. 51 custom unification pipeline Departments Business Intelligence internal processes variety of event types and structures
53. The Future is Now 53
54. 54 AWS Lambda in FRA
55. 55 AWS Lambda in FRA
56. www.stylight.com sergii.khomenko@stylight.com @lc0d3r
57. Related links 57 1. The Babbage Engine 2. The Five Generations of Computers 3. IBM System 360 4. https://github.com/cloudtools/troposphere 5. https://www.terraform.io/ 6. (Docker Compose + Docker Swarm) or Kubernetes 7. Google Cloud Functions 8. https://github.com/apex/apex 9. https://github.com/apex/apex/tree/master/_examples/python/functions/dependency 10.Streaming Data Processing with Amazon Kinesis and AWS Lambda
58. 58 USE CODE - AWS20