Building and Scaling serverless log analytics platform

Serverless architectures has been around for past few years and there has been quite a few skepticism surrounding it. Few might argue that it’s just another buzzword for marketing. But serverless architectures offer more than a catchy buzzword. In this talk we will discuss, what is serverless, when to and when not to use them and how can we use Amazon Web Services to implement a real-time, production grade serverless logging pipeline. By the end of the talk, audience will get an introduction to serverless and also get to know how to design, deploy and scale logging infrastructures using the same.

Transcript

1. BUILDING AND SCALING A LOG ANALYTICS PLATFORM 
 - A SERVERLESS APPROACH Narendran
2. A BIT ABOUT ME… Naren Product Engineer Scaling A.I to millions
 @ MadStreetDen Python, Golang, FOSS, Cycling, Travel
3. TWITTER & GITHUB @DudeWhoCode WEBSITE & BLOG www.dudewho.codes
4. OUR DISTRIBUTED MICROSERVICES
5. ARCHITECTING A LOGGING FRAMEWORK
6. BEHAVIOURS OF DIFFERENT MICROSERVICES • Mostly Idle
 • Horizontally scales up • Rapid scale down
 • Surges
7. EXPECTATIONS FROM ANY LOG FRAMEWORK • High Performance • Secure • No data Loss/overwriting • Dynamically Scalable • Highly Available • Independent
8. KNOW YOUR LOGS • Real time Analysis
 • Historical Analysis

9. THE CONVENTIONAL ARCHITECTURE
10. PROBLEMS FACED
11. THE THING WITH ELK • Threadpool queue size in Elastic Search
 • Adding Logstash filters
 • Burning more money, even for idle time
12. WHO WILL MONITOR THE SYSTEM THAT MONITORS OTHER SYSTEMS?
13. WHO WILL MONITOR THE SYSTEM THAT MONITORS OTHER SYSTEMS?
14. WHATS THE COMMON THING THAT ALL THE COMPONENTS HAVE?

15. WHATS THE COMMON THING THAT ALL THE COMPONENTS HAVE?
 Servers, Servers, Servers
16. Servers, Servers, Servers
17. GOING SERVERLESS (buzzword alert!) Serverless computing is a cloud computing execution model in which the cloud provider dynamically manages the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity. - Wikipedia
18. GOING SERVERLESS No server is easier to manage than “no server” - Werner Wogels, CTO, Amazon
19. GOING SERVERLESS
20. SERVERLESS ARCHITECTURE
21. AMAZON WEB SERVICES (AWS)
22. AWS SERVERLESS
23. AWS SERVERLESS
24. AWS SERVERLESS
25. FAAS ARCHITECTURE
26. •The client application from which we will get our logs
27. • Delivers real-time streaming data to other services such as Amazon S3, Elastic Search. • Configurable producers and consumers
28. • Run code without provisioning servers • Event driven, Highly scalable
29. • Store data at massive scale • Storage tier for many serverless applications
30. • Serverless interactive query service • Point to your data source, define the schema, 
 start querying using standard SQL
31. • Cloud-powered visualization tool • Perform ad-hoc analysis and get quick business insights
32. • Process streaming data in real time with standard SQL in real time • Scales automatically to match throughput
33. • Real time log monitoring • Collect and track metrics, log files, trigger alarms
34. MANAGING THE SERVERLESS MODEL 1. Web Console 2. Code (AWS SDK) 3. CLI
35. DEMO
36. YOU CAN WATCH THE FOLLOWING DEMO AS SCREENCAST AT 
 http://bit.ly/serverless-logging
45. SUMMARY • Usual logging architecture: took us weeks and certain level of expertise to implement • Serverless architecture: Up and running in minutes • No maintenance, nearly zero developer interventions after deployment • This Faas architecture can scale infinitely for millions of logs from thousands of microservices
46. SUMMARY • • • • •
48. SLIDES : www.dudewho.codes/talks THANK YOU, NAREN @DudeWhoCode