A bit about me
Naren
Backend/Product
Engineer
Scaling A.I to millions
@ MadStreetDen
python, golang, FOSS,
cycling, travel
Slide 3
Slide 3 text
twitter :
@DudeWhoCode
www.dudewho.codes
Slide 4
Slide 4 text
IoT
(buzzword #1)
Slide 5
Slide 5 text
“things” in Internet of
Things
Slide 6
Slide 6 text
IoT is really really a
BIG data
(buzzword #2)
- 20 billion connected devices currently
- 400 ZetaBytes of data by 2018
ref : cisco
Slide 7
Slide 7 text
sensors in Boeing 787 generates
of data in an hour of flight
40 TB
Slide 8
Slide 8 text
manager : Hey, you are going to work on a new project.
On one fine day …
Slide 9
Slide 9 text
manager : Hey, you are going to work on a new project.
you : what is it about?
On one fine day …
Slide 10
Slide 10 text
manager : Hey, you are going to work on a new project.
you : what is it about?
manager : You are going to do a PoC to get insights from IoT
data
On one fine day …
Slide 11
Slide 11 text
manager : Hey, you are going to work on a new project.
you : what is it about?
manager : You are going to do a PoC to get insights from IoT
data
you : wow, IoT?
you :
On one fine day …
Slide 12
Slide 12 text
manager : The client will sign the contract if we show some
insights within few days.
On one fine day …
Slide 13
Slide 13 text
Crunching the IoT data
Slide 14
Slide 14 text
3 tier architecture
Slide 15
Slide 15 text
Traditional Implementation
Slide 16
Slide 16 text
• Connected devices
• Heart rate sensors
Slide 17
Slide 17 text
• Django server behind load balancer
• Authentication of registered IoT devices
• Scaling when the throughput or sensor count increases
• ElasticSearch and Kibana
• High physical resource and cluster maintenance for high
throughput
Slide 21
Slide 21 text
No content
Slide 22
Slide 22 text
manager : We lost our contract ..!
Slide 23
Slide 23 text
Everything boils down to this …
Slide 24
Slide 24 text
Everything boils down to this …
servers, servers, servers
Slide 25
Slide 25 text
Everything boils down to this …
servers, servers, servers
Slide 26
Slide 26 text
servers, servers, servers
• Operational overhead :
• create and manage machines/VMs
• Patching O.S, web servers
• Not so easy while implementing microservices
• Auto scaling, DevOps
Slide 27
Slide 27 text
Going serverless
(buzzword #3)
No server is easier to manage than “no server”
- Werner Wogels, CTO, Amazon
Slide 28
Slide 28 text
Going serverless
Slide 29
Slide 29 text
• Low operational overhead
• No versioning issues
• Event driven microservices
• Stateless
• Don’t pay for idle time
Why serverless?
Slide 30
Slide 30 text
Lets implement similar architecture
but without servers
Slide 31
Slide 31 text
Amazon Web Services
(AWS)
Slide 32
Slide 32 text
AWS serverless
Slide 33
Slide 33 text
AWS serverless
Slide 34
Slide 34 text
FaaS
Slide 35
Slide 35 text
• Securely connect devices and interact with
cloud applications
• Device gateway, Message broker, Rules engine
Slide 36
Slide 36 text
• Delivers real-time streaming data to other
services such as Amazon S3, Elastic Search.
• Configurable producers and consumers
Slide 37
Slide 37 text
• Run code without provisioning servers
• Event driven, Highly scalable
Slide 38
Slide 38 text
• Store data at massive scale
• Storage tier for many serverless applications
Slide 39
Slide 39 text
• Serverless interactive query service
• Point to your data source,
define the schema,
start querying using standard SQL
Slide 40
Slide 40 text
• cloud-powered visualization tool
• Perform ad-hoc analysis and get quick
business insights
Slide 41
Slide 41 text
Managing our serverless model
1. Web Console
2. Python code (boto3)
Slide 42
Slide 42 text
s3
Slide 43
Slide 43 text
s3 Kinesis Firehose
Slide 44
Slide 44 text
s3 Kinesis Firehose
Athena
Slide 45
Slide 45 text
s3 Kinesis Firehose
Athena QuickSight
Slide 46
Slide 46 text
Boto3
• Amazon Web Services (AWS) SDK for Python
• Object Oriented API as well as low level access
Slide 47
Slide 47 text
DEMO
Slide 48
Slide 48 text
How to do it in code?
Slide 49
Slide 49 text
create s3 bucket
Slide 50
Slide 50 text
create IoT gateway rule
Slide 51
Slide 51 text
create firehose delivery stream
Slide 52
Slide 52 text
athena query : create table
Slide 53
Slide 53 text
Summary
• Usual approach : takes weeks and certain level of expertise
to implement
• Going serverless : Minutes to implement
• AWS services used : AWS IoT, Lambda, Kinesis Firehose,
s3, Athena, Quicksight
• Easy to use AWS web console
• Boto3 : the official AWS python SDK
Slide 54
Slide 54 text
Final thoughts
• Servers :
DevOps, OnCall,
Downtime, Sleepless
nights, Angry people
• Serverless :
Cost efficient, easy to
deploy and maintain, all
heavy lifting is done by
the provider
Slide 55
Slide 55 text
Caveats
Slide 56
Slide 56 text
Caveats
Slide 57
Slide 57 text
Caveats
• Everytime you debug, you need to deploy it in cloud
• Closely packed env, packages/dependencies
version are decided by provider
• Local development is difficult
• Vendor lockin
Slide 58
Slide 58 text
Caveats
•
•
•
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
slides :
dudewho.codes/talks
Thank you
@DudeWhoCode