Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building and Scaling serverless log analytics platform

November 07, 2017

Building and Scaling serverless log analytics platform

Serverless architectures has been around for past few years and there has been quite a few skepticism surrounding it. Few might argue that it’s just another buzzword for marketing. But serverless architectures offer more than a catchy buzzword. In this talk we will discuss, what is serverless, when to and when not to use them and how can we use Amazon Web Services to implement a real-time, production grade serverless logging pipeline. By the end of the talk, audience will get an introduction to serverless and also get to know how to design, deploy and scale logging infrastructures using the same.


November 07, 2017

More Decks by Naren

Other Decks in Programming


  1. A BIT ABOUT ME… Naren Product Engineer Scaling A.I to

 @ MadStreetDen Python, Golang, FOSS, Cycling, Travel
  2. EXPECTATIONS FROM ANY LOG FRAMEWORK • High Performance • Secure

    • No data Loss/overwriting • Dynamically Scalable • Highly Available • Independent
  3. THE THING WITH ELK • Threadpool queue size in Elastic

 • Adding Logstash filters
 • Burning more money, even for idle time
  4. GOING SERVERLESS (buzzword alert!) Serverless computing is a cloud computing

    execution model in which the cloud provider dynamically manages the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity. - Wikipedia
  5. GOING SERVERLESS No server is easier to manage than “no

    server” - Werner Wogels, CTO, Amazon
  6. • Delivers real-time streaming data to other services such as

    Amazon S3, Elastic Search. • Configurable producers and consumers
  7. • Serverless interactive query service • Point to your data

    source, define the schema, 
 start querying using standard SQL
  8. • Process streaming data in real time with standard SQL

    in real time • Scales automatically to match throughput
  9. SUMMARY • Usual logging architecture: took us weeks and certain

    level of expertise to implement • Serverless architecture: Up and running in minutes • No maintenance, nearly zero developer interventions after deployment • This Faas architecture can scale infinitely for millions of logs from thousands of microservices