IoT meets serverless - Speaker Deck

IoT meets serverless

by Naren

Slide 1

Slide 1 text

IoT meets Serverless Narendran

Slide 2

Slide 2 text

A bit about me Naren Backend/Product Engineer Scaling A.I to millions   @ MadStreetDen python, golang, FOSS, cycling, travel

Slide 3

Slide 3 text

twitter :  @DudeWhoCode www.dudewho.codes

Slide 4

Slide 4 text

IoT (buzzword #1)

Slide 5

Slide 5 text

“things” in Internet of Things

Slide 6

Slide 6 text

IoT is really really a BIG data (buzzword #2) - 20 billion connected devices currently - 400 ZetaBytes of data by 2018 ref : cisco

Slide 7

Slide 7 text

sensors in Boeing 787 generates of data in an hour of ﬂight 40 TB

Slide 8

Slide 8 text

manager : Hey, you are going to work on a new project. On one ﬁne day …

Slide 9

Slide 9 text

manager : Hey, you are going to work on a new project. you : what is it about? On one ﬁne day …

Slide 10

Slide 10 text

manager : Hey, you are going to work on a new project. you : what is it about? manager : You are going to do a PoC to get insights from IoT data On one ﬁne day …

Slide 11

Slide 11 text

manager : Hey, you are going to work on a new project. you : what is it about? manager : You are going to do a PoC to get insights from IoT data you : wow, IoT?    you : On one ﬁne day …

Slide 12

Slide 12 text

manager : The client will sign the contract if we show some insights within few days. On one ﬁne day …

Slide 13

Slide 13 text

Crunching the IoT data

Slide 14

Slide 14 text

3 tier architecture

Slide 15

Slide 15 text

Traditional Implementation

Slide 16

Slide 16 text

• Connected devices • Heart rate sensors

Slide 17

Slide 17 text

• Django server behind load balancer • Authentication of registered IoT devices • Scaling when the throughput or sensor count increases

Slide 18

Slide 18 text

• Kafka • Cluster/node maintainance • Producers, Consumers, Nodes, Partitions, ZooKeepers, oﬀsets

Slide 19

Slide 19 text

• Apache Storm • Cluster/node maintainance • Dedicated resources

Slide 20

Slide 20 text

• ElasticSearch and Kibana • High physical resource and cluster maintenance for high throughput

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

manager : We lost our contract ..!

Slide 23

Slide 23 text

Everything boils down to this …

Slide 24

Slide 24 text

Everything boils down to this … servers, servers, servers

Slide 25

Slide 25 text

Everything boils down to this … servers, servers, servers

Slide 26

Slide 26 text

servers, servers, servers • Operational overhead : • create and manage machines/VMs • Patching O.S, web servers • Not so easy while implementing microservices • Auto scaling, DevOps

Slide 27

Slide 27 text

Going serverless (buzzword #3) No server is easier to manage than “no server” - Werner Wogels, CTO, Amazon

Slide 28

Slide 28 text

Going serverless

Slide 29

Slide 29 text

• Low operational overhead • No versioning issues • Event driven microservices • Stateless • Don’t pay for idle time Why serverless?

Slide 30

Slide 30 text

Lets implement similar architecture but without servers

Slide 31

Slide 31 text

Amazon Web Services (AWS)

Slide 32

Slide 32 text

AWS serverless

Slide 33

Slide 33 text

AWS serverless

Slide 34

Slide 34 text

FaaS

Slide 35

Slide 35 text

• Securely connect devices and interact with cloud applications • Device gateway, Message broker, Rules engine

Slide 36

Slide 36 text

• Delivers real-time streaming data to other services such as Amazon S3, Elastic Search. • Conﬁgurable producers and consumers

Slide 37

Slide 37 text

• Run code without provisioning servers • Event driven, Highly scalable

Slide 38

Slide 38 text

• Store data at massive scale • Storage tier for many serverless applications

Slide 39

Slide 39 text

• Serverless interactive query service • Point to your data source,   deﬁne the schema,   start querying using standard SQL

Slide 40

Slide 40 text

• cloud-powered visualization tool • Perform ad-hoc analysis and get quick business insights

Slide 41

Slide 41 text

Managing our serverless model 1. Web Console 2. Python code (boto3)

Slide 42

Slide 42 text

Slide 43

Slide 43 text

s3 Kinesis Firehose

Slide 44

Slide 44 text

s3 Kinesis Firehose Athena

Slide 45

Slide 45 text

s3 Kinesis Firehose Athena QuickSight

Slide 46

Slide 46 text

Boto3 • Amazon Web Services (AWS) SDK for Python • Object Oriented API as well as low level access

Slide 47

Slide 47 text

DEMO

Slide 48

Slide 48 text

How to do it in code?

Slide 49

Slide 49 text

create s3 bucket

Slide 50

Slide 50 text

create IoT gateway rule

Slide 51

Slide 51 text

create ﬁrehose delivery stream

Slide 52

Slide 52 text

athena query : create table

Slide 53

Slide 53 text

Summary • Usual approach : takes weeks and certain level of expertise to implement • Going serverless : Minutes to implement • AWS services used : AWS IoT, Lambda, Kinesis Firehose, s3, Athena, Quicksight • Easy to use AWS web console • Boto3 : the oﬃcial AWS python SDK

Slide 54

Slide 54 text

Final thoughts • Servers : DevOps, OnCall, Downtime, Sleepless nights, Angry people • Serverless : Cost eﬃcient, easy to deploy and maintain, all heavy lifting is done by the provider

Slide 55

Slide 55 text

Caveats

Slide 56

Slide 56 text

Caveats

Slide 57

Slide 57 text

Caveats • Everytime you debug, you need to deploy it in cloud • Closely packed env, packages/dependencies version are decided by provider • Local development is diﬃcult • Vendor lockin