Building and Scaling serverless log analytics platform

by Naren

Slide 1

Slide 1 text

BUILDING AND SCALING A LOG ANALYTICS PLATFORM   - A SERVERLESS APPROACH Narendran

Slide 2

Slide 2 text

A BIT ABOUT ME… Naren Product Engineer Scaling A.I to millions  @ MadStreetDen Python, Golang, FOSS, Cycling, Travel

Slide 3

Slide 3 text

TWITTER & GITHUB @DudeWhoCode WEBSITE & BLOG www.dudewho.codes

Slide 4

Slide 4 text

OUR DISTRIBUTED MICROSERVICES

Slide 5

Slide 5 text

ARCHITECTING A LOGGING FRAMEWORK

Slide 6

Slide 6 text

BEHAVIOURS OF DIFFERENT MICROSERVICES • Mostly Idle  • Horizontally scales up • Rapid scale down  • Surges

Slide 7

Slide 7 text

EXPECTATIONS FROM ANY LOG FRAMEWORK • High Performance • Secure • No data Loss/overwriting • Dynamically Scalable • Highly Available • Independent

Slide 8

Slide 8 text

KNOW YOUR LOGS • Real time Analysis  • Historical Analysis 

Slide 9

Slide 9 text

THE CONVENTIONAL ARCHITECTURE

Slide 10

Slide 10 text

PROBLEMS FACED

Slide 11

Slide 11 text

THE THING WITH ELK • Threadpool queue size in Elastic Search  • Adding Logstash ﬁlters  • Burning more money, even for idle time

Slide 12

Slide 12 text

WHO WILL MONITOR THE SYSTEM THAT MONITORS OTHER SYSTEMS?

Slide 13

Slide 13 text

WHO WILL MONITOR THE SYSTEM THAT MONITORS OTHER SYSTEMS?

Slide 14

Slide 14 text

WHATS THE COMMON THING THAT ALL THE COMPONENTS HAVE? 

Slide 15

Slide 15 text

WHATS THE COMMON THING THAT ALL THE COMPONENTS HAVE?  Servers, Servers, Servers

Slide 16

Slide 16 text

Servers, Servers, Servers

Slide 17

Slide 17 text

GOING SERVERLESS (buzzword alert!) Serverless computing is a cloud computing execution model in which the cloud provider dynamically manages the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity. - Wikipedia

Slide 18

Slide 18 text

GOING SERVERLESS No server is easier to manage than “no server” - Werner Wogels, CTO, Amazon

Slide 19

Slide 19 text

GOING SERVERLESS

Slide 20

Slide 20 text

SERVERLESS ARCHITECTURE

Slide 21

Slide 21 text

AMAZON WEB SERVICES (AWS)

Slide 22

Slide 22 text

AWS SERVERLESS

Slide 23

Slide 23 text

AWS SERVERLESS

Slide 24

Slide 24 text

AWS SERVERLESS

Slide 25

Slide 25 text

FAAS ARCHITECTURE

Slide 26

Slide 26 text

•The client application from which we will get our logs

Slide 27

Slide 27 text

• Delivers real-time streaming data to other services such as Amazon S3, Elastic Search. • Conﬁgurable producers and consumers

Slide 28

Slide 28 text

• Run code without provisioning servers • Event driven, Highly scalable

Slide 29

Slide 29 text

• Store data at massive scale • Storage tier for many serverless applications

Slide 30

Slide 30 text

• Serverless interactive query service • Point to your data source, deﬁne the schema,   start querying using standard SQL

Slide 31

Slide 31 text

• Cloud-powered visualization tool • Perform ad-hoc analysis and get quick business insights

Slide 32

Slide 32 text

• Process streaming data in real time with standard SQL in real time • Scales automatically to match throughput

Slide 33

Slide 33 text

• Real time log monitoring • Collect and track metrics, log ﬁles, trigger alarms

Slide 34

Slide 34 text

MANAGING THE SERVERLESS MODEL 1. Web Console 2. Code (AWS SDK) 3. CLI

Slide 35

Slide 35 text

DEMO

Slide 36

Slide 36 text

YOU CAN WATCH THE FOLLOWING DEMO AS SCREENCAST AT   http://bit.ly/serverless-logging

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

SUMMARY • Usual logging architecture: took us weeks and certain level of expertise to implement • Serverless architecture: Up and running in minutes • No maintenance, nearly zero developer interventions after deployment • This Faas architecture can scale inﬁnitely for millions of logs from thousands of microservices