Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building and Scaling serverless log analytics platform

Naren
November 07, 2017

Building and Scaling serverless log analytics platform

Serverless architectures has been around for past few years and there has been quite a few skepticism surrounding it. Few might argue that it’s just another buzzword for marketing. But serverless architectures offer more than a catchy buzzword. In this talk we will discuss, what is serverless, when to and when not to use them and how can we use Amazon Web Services to implement a real-time, production grade serverless logging pipeline. By the end of the talk, audience will get an introduction to serverless and also get to know how to design, deploy and scale logging infrastructures using the same.

Naren

November 07, 2017
Tweet

More Decks by Naren

Other Decks in Programming

Transcript

  1. BUILDING AND SCALING A LOG ANALYTICS
    PLATFORM 

    - A SERVERLESS APPROACH
    Narendran

    View Slide

  2. A BIT ABOUT ME…
    Naren
    Product Engineer
    Scaling A.I to millions

    @ MadStreetDen
    Python, Golang, FOSS, Cycling,
    Travel

    View Slide

  3. TWITTER & GITHUB
    @DudeWhoCode
    WEBSITE & BLOG
    www.dudewho.codes

    View Slide

  4. OUR DISTRIBUTED MICROSERVICES

    View Slide

  5. ARCHITECTING A LOGGING FRAMEWORK

    View Slide

  6. BEHAVIOURS OF DIFFERENT
    MICROSERVICES
    • Mostly Idle

    • Horizontally scales up
    • Rapid scale down

    • Surges

    View Slide

  7. EXPECTATIONS FROM ANY LOG
    FRAMEWORK
    • High Performance
    • Secure
    • No data Loss/overwriting
    • Dynamically Scalable
    • Highly Available
    • Independent

    View Slide

  8. KNOW YOUR LOGS
    • Real time Analysis

    • Historical Analysis


    View Slide

  9. THE CONVENTIONAL ARCHITECTURE

    View Slide

  10. PROBLEMS FACED

    View Slide

  11. THE THING WITH ELK
    • Threadpool queue size in Elastic Search

    • Adding Logstash filters

    • Burning more money, even for idle
    time

    View Slide

  12. WHO WILL MONITOR THE SYSTEM THAT
    MONITORS OTHER SYSTEMS?

    View Slide

  13. WHO WILL MONITOR THE SYSTEM THAT
    MONITORS OTHER SYSTEMS?

    View Slide

  14. WHATS THE COMMON THING THAT ALL
    THE COMPONENTS HAVE?


    View Slide

  15. WHATS THE COMMON THING THAT ALL
    THE COMPONENTS HAVE?

    Servers, Servers, Servers

    View Slide

  16. Servers, Servers, Servers

    View Slide

  17. GOING SERVERLESS
    (buzzword alert!)
    Serverless computing is a cloud computing execution model in which the cloud provider
    dynamically manages the allocation of machine resources.
    Pricing is based on the actual amount of resources consumed by an application, rather
    than on pre-purchased units of capacity.
    - Wikipedia

    View Slide

  18. GOING SERVERLESS
    No server is easier to manage than “no server”
    - Werner Wogels, CTO, Amazon

    View Slide

  19. GOING SERVERLESS

    View Slide

  20. SERVERLESS ARCHITECTURE

    View Slide

  21. AMAZON WEB SERVICES
    (AWS)

    View Slide

  22. AWS SERVERLESS

    View Slide

  23. AWS SERVERLESS

    View Slide

  24. AWS SERVERLESS

    View Slide

  25. FAAS ARCHITECTURE

    View Slide

  26. •The client application from
    which we will get our logs

    View Slide

  27. • Delivers real-time streaming data to other
    services such as Amazon S3, Elastic Search.
    • Configurable producers and consumers

    View Slide

  28. • Run code without provisioning servers
    • Event driven, Highly scalable

    View Slide

  29. • Store data at massive scale
    • Storage tier for many serverless applications

    View Slide

  30. • Serverless interactive query service
    • Point to your data source, define the schema, 

    start querying using standard SQL

    View Slide

  31. • Cloud-powered visualization tool
    • Perform ad-hoc analysis and get quick business
    insights

    View Slide

  32. • Process streaming data in real time with standard
    SQL in real time
    • Scales automatically to match throughput

    View Slide

  33. • Real time log monitoring
    • Collect and track metrics, log files, trigger alarms

    View Slide

  34. MANAGING THE SERVERLESS MODEL
    1. Web Console
    2. Code (AWS SDK)
    3. CLI

    View Slide

  35. DEMO

    View Slide

  36. YOU CAN WATCH THE FOLLOWING
    DEMO AS SCREENCAST AT

    http://bit.ly/serverless-logging

    View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. View Slide

  42. View Slide

  43. View Slide

  44. View Slide

  45. SUMMARY
    • Usual logging architecture: took us weeks and certain level of expertise
    to implement
    • Serverless architecture: Up and running in minutes
    • No maintenance, nearly zero developer interventions after deployment
    • This Faas architecture can scale infinitely for millions of logs from
    thousands of microservices

    View Slide

  46. SUMMARY





    View Slide

  47. View Slide

  48. SLIDES :
    www.dudewho.codes/talks
    THANK YOU,
    NAREN
    @DudeWhoCode

    View Slide