Building and Scaling serverless log analytics platform

Serverless architectures has been around for past few years and there has been quite a few skepticism surrounding it. Few might argue that it’s just another buzzword for marketing. But serverless architectures offer more than a catchy buzzword. In this talk we will discuss, what is serverless, when to and when not to use them and how can we use Amazon Web Services to implement a real-time, production grade serverless logging pipeline. By the end of the talk, audience will get an introduction to serverless and also get to know how to design, deploy and scale logging infrastructures using the same.

Transcript

1. BUILDING AND SCALING A LOG ANALYTICS
   PLATFORM 

   - A SERVERLESS APPROACH
   Narendran
   

   View Slide

2. A BIT ABOUT ME…
   Naren
   Product Engineer
   Scaling A.I to millions

   @ MadStreetDen
   Python, Golang, FOSS, Cycling,
   Travel
   

   View Slide

3. TWITTER & GITHUB
   @DudeWhoCode
   WEBSITE & BLOG
   www.dudewho.codes
   

   View Slide

4. OUR DISTRIBUTED MICROSERVICES
   

   View Slide

5. ARCHITECTING A LOGGING FRAMEWORK
   

   View Slide

6. BEHAVIOURS OF DIFFERENT
   MICROSERVICES
   • Mostly Idle

   • Horizontally scales up
   • Rapid scale down

   • Surges
   

   View Slide

7. EXPECTATIONS FROM ANY LOG
   FRAMEWORK
   • High Performance
   • Secure
   • No data Loss/overwriting
   • Dynamically Scalable
   • Highly Available
   • Independent
   

   View Slide

8. KNOW YOUR LOGS
   • Real time Analysis

   • Historical Analysis

   

   View Slide

9. THE CONVENTIONAL ARCHITECTURE
   

   View Slide

10. PROBLEMS FACED
    

    View Slide

11. THE THING WITH ELK
    • Threadpool queue size in Elastic Search

    • Adding Logstash filters

    • Burning more money, even for idle
    time
    

    View Slide

12. WHO WILL MONITOR THE SYSTEM THAT
    MONITORS OTHER SYSTEMS?
    

    View Slide

13. WHO WILL MONITOR THE SYSTEM THAT
    MONITORS OTHER SYSTEMS?
    

    View Slide

14. WHATS THE COMMON THING THAT ALL
    THE COMPONENTS HAVE?

    

    View Slide

15. WHATS THE COMMON THING THAT ALL
    THE COMPONENTS HAVE?

    Servers, Servers, Servers
    

    View Slide

16. Servers, Servers, Servers
    

    View Slide

17. GOING SERVERLESS
    (buzzword alert!)
    Serverless computing is a cloud computing execution model in which the cloud provider
    dynamically manages the allocation of machine resources.
    Pricing is based on the actual amount of resources consumed by an application, rather
    than on pre-purchased units of capacity.
    - Wikipedia
    

    View Slide

18. GOING SERVERLESS
    No server is easier to manage than “no server”
    - Werner Wogels, CTO, Amazon
    

    View Slide

19. GOING SERVERLESS
    

    View Slide

20. SERVERLESS ARCHITECTURE
    

    View Slide

21. AMAZON WEB SERVICES
    (AWS)
    

    View Slide

22. AWS SERVERLESS
    

    View Slide

23. AWS SERVERLESS
    

    View Slide

24. AWS SERVERLESS
    

    View Slide

25. FAAS ARCHITECTURE
    

    View Slide

26. •The client application from
    which we will get our logs
    

    View Slide

27. • Delivers real-time streaming data to other
    services such as Amazon S3, Elastic Search.
    • Configurable producers and consumers
    

    View Slide

28. • Run code without provisioning servers
    • Event driven, Highly scalable
    

    View Slide

29. • Store data at massive scale
    • Storage tier for many serverless applications
    

    View Slide

30. • Serverless interactive query service
    • Point to your data source, define the schema, 

    start querying using standard SQL
    

    View Slide

31. • Cloud-powered visualization tool
    • Perform ad-hoc analysis and get quick business
    insights
    

    View Slide

32. • Process streaming data in real time with standard
    SQL in real time
    • Scales automatically to match throughput
    

    View Slide

33. • Real time log monitoring
    • Collect and track metrics, log files, trigger alarms
    

    View Slide

34. MANAGING THE SERVERLESS MODEL
    1. Web Console
    2. Code (AWS SDK)
    3. CLI
    

    View Slide

35. DEMO
    

    View Slide

36. YOU CAN WATCH THE FOLLOWING
    DEMO AS SCREENCAST AT
    

    http://bit.ly/serverless-logging
    

    View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. View Slide

42. View Slide

43. View Slide

44. View Slide

45. SUMMARY
    • Usual logging architecture: took us weeks and certain level of expertise
    to implement
    • Serverless architecture: Up and running in minutes
    • No maintenance, nearly zero developer interventions after deployment
    • This Faas architecture can scale infinitely for millions of logs from
    thousands of microservices
    

    View Slide

46. SUMMARY
    •
    •
    •
    •
    •
    

    View Slide

47. View Slide

48. SLIDES :
    www.dudewho.codes/talks
    THANK YOU,
    NAREN
    @DudeWhoCode
    

    View Slide