Slide 1
Slide 1 text
BUILDING AND SCALING A LOG ANALYTICS
PLATFORM
- A SERVERLESS APPROACH
Narendran
Slide 2
Slide 2 text
A BIT ABOUT ME…
Naren
Product Engineer
Scaling A.I to millions
@ MadStreetDen
Python, Golang, FOSS, Cycling,
Travel
Slide 3
Slide 3 text
TWITTER & GITHUB
@DudeWhoCode
WEBSITE & BLOG
www.dudewho.codes
Slide 4
Slide 4 text
OUR DISTRIBUTED MICROSERVICES
Slide 5
Slide 5 text
ARCHITECTING A LOGGING FRAMEWORK
Slide 6
Slide 6 text
BEHAVIOURS OF DIFFERENT
MICROSERVICES
• Mostly Idle
• Horizontally scales up
• Rapid scale down
• Surges
Slide 7
Slide 7 text
EXPECTATIONS FROM ANY LOG
FRAMEWORK
• High Performance
• Secure
• No data Loss/overwriting
• Dynamically Scalable
• Highly Available
• Independent
Slide 8
Slide 8 text
KNOW YOUR LOGS
• Real time Analysis
• Historical Analysis
Slide 9
Slide 9 text
THE CONVENTIONAL ARCHITECTURE
Slide 10
Slide 10 text
PROBLEMS FACED
Slide 11
Slide 11 text
THE THING WITH ELK
• Threadpool queue size in Elastic Search
• Adding Logstash filters
• Burning more money, even for idle
time
Slide 12
Slide 12 text
WHO WILL MONITOR THE SYSTEM THAT
MONITORS OTHER SYSTEMS?
Slide 13
Slide 13 text
WHO WILL MONITOR THE SYSTEM THAT
MONITORS OTHER SYSTEMS?
Slide 14
Slide 14 text
WHATS THE COMMON THING THAT ALL
THE COMPONENTS HAVE?
Slide 15
Slide 15 text
WHATS THE COMMON THING THAT ALL
THE COMPONENTS HAVE?
Servers, Servers, Servers
Slide 16
Slide 16 text
Servers, Servers, Servers
Slide 17
Slide 17 text
GOING SERVERLESS
(buzzword alert!)
Serverless computing is a cloud computing execution model in which the cloud provider
dynamically manages the allocation of machine resources.
Pricing is based on the actual amount of resources consumed by an application, rather
than on pre-purchased units of capacity.
- Wikipedia
Slide 18
Slide 18 text
GOING SERVERLESS
No server is easier to manage than “no server”
- Werner Wogels, CTO, Amazon
Slide 19
Slide 19 text
GOING SERVERLESS
Slide 20
Slide 20 text
SERVERLESS ARCHITECTURE
Slide 21
Slide 21 text
AMAZON WEB SERVICES
(AWS)
Slide 22
Slide 22 text
AWS SERVERLESS
Slide 23
Slide 23 text
AWS SERVERLESS
Slide 24
Slide 24 text
AWS SERVERLESS
Slide 25
Slide 25 text
FAAS ARCHITECTURE
Slide 26
Slide 26 text
•The client application from
which we will get our logs
Slide 27
Slide 27 text
• Delivers real-time streaming data to other
services such as Amazon S3, Elastic Search.
• Configurable producers and consumers
Slide 28
Slide 28 text
• Run code without provisioning servers
• Event driven, Highly scalable
Slide 29
Slide 29 text
• Store data at massive scale
• Storage tier for many serverless applications
Slide 30
Slide 30 text
• Serverless interactive query service
• Point to your data source, define the schema,
start querying using standard SQL
Slide 31
Slide 31 text
• Cloud-powered visualization tool
• Perform ad-hoc analysis and get quick business
insights
Slide 32
Slide 32 text
• Process streaming data in real time with standard
SQL in real time
• Scales automatically to match throughput
Slide 33
Slide 33 text
• Real time log monitoring
• Collect and track metrics, log files, trigger alarms
Slide 34
Slide 34 text
MANAGING THE SERVERLESS MODEL
1. Web Console
2. Code (AWS SDK)
3. CLI
Slide 35
Slide 35 text
DEMO
Slide 36
Slide 36 text
YOU CAN WATCH THE FOLLOWING
DEMO AS SCREENCAST AT
http://bit.ly/serverless-logging
Slide 37
Slide 37 text
No content
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
No content
Slide 42
Slide 42 text
No content
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
No content
Slide 45
Slide 45 text
SUMMARY
• Usual logging architecture: took us weeks and certain level of expertise
to implement
• Serverless architecture: Up and running in minutes
• No maintenance, nearly zero developer interventions after deployment
• This Faas architecture can scale infinitely for millions of logs from
thousands of microservices
Slide 46
Slide 46 text
SUMMARY
•
•
•
•
•
Slide 47
Slide 47 text
No content
Slide 48
Slide 48 text
SLIDES :
www.dudewho.codes/talks
THANK YOU,
NAREN
@DudeWhoCode