Slide 1
Slide 1 text
AWS Lambda &
Serverless framework
Lessons learned while building a serverless company
26/01/2017
@Podgeypoos79 - @loige
Slide 2
Slide 2 text
Luciano Mammino
● @loige on Twitter
● Lmammino on GitHub
● Blog at loige.co
● Co-author of:
Node.js Design Patterns (Second Edition)
Slide 3
Slide 3 text
Padraig O’Brien (a.k.a “Podge”)
● @podgeypoos79 on Twitter
● Does not have a book (yet…)
● Organiser of Nodeschool Dublin and Dun Laoghaire
● Organiser of Dublin Serverless meetup
● He is writing his own database, unicorndb
Slide 4
Slide 4 text
Agenda
● Planet 9 Energy
● Serverless
● Security
● Quality
● Developer Experience
● Costs
● Lessons learned
Slide 5
Slide 5 text
planet9energy.com
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
17.520 * * 24
Meter readings Customer size Data versions
≈4 billion data pts / year (tot.)
SMALL customer ≈3 mln data pts / year MEDIUM customer ≈20 mln data pts / year BIG customer ≈110 mln data pts / year
( Per customer / year )
Slide 8
Slide 8 text
● Limited number of “Full stack” engineers
● Write & deploy quality code as fast as possible
● Adopt hot and relevant technologies
● No servers... or more to the point, NO CALLS AT 2 AM!
Our requirements
Slide 9
Slide 9 text
Early experiments
Slide 10
Slide 10 text
Current Technology
Slide 11
Slide 11 text
“Serverless”
Meaning
Serverless framework
AWS Lambda
Slide 12
Slide 12 text
What does
“serverless” mean?
Slide 13
Slide 13 text
Why Serverless
framework?
Slide 14
Slide 14 text
Anatomy of Serverless.yml
Serverless.yml header
Slide 15
Slide 15 text
Defining functions and events
Slide 16
Slide 16 text
Why?
Slide 17
Slide 17 text
What is a lambda?
● Function as a service (FAAS)
● Pay for invocation / processing time
● Virtually “infinite” auto-scaling
● Focus on business logic, not on servers
daaaa!
Slide 18
Slide 18 text
Lambda as micro-services
● Events are first-class citizens
● Every lambda scales independently
● Agility (develop features quick and in an isolated fashion)
Classic micro-services concerns:
● Granularity (how to separate features? BDD? Bounded Contexts?)
● Orchestration (dependencies between lambdas, service discovery…)
Slide 19
Slide 19 text
Anatomy of a Lambda in Node.js
Slide 20
Slide 20 text
Some use cases
● REST over HTTP (API Gateway)
● SNS messages
● Schedule/Cron
● DynamoDB data changes
● S3 files changes
● IoT
Slide 21
Slide 21 text
HTTP REQUEST - API Call
POST /hello/Elvis?foo=bar
{
“test”: “body”
}
Slide 22
Slide 22 text
Security
Authentication
Authorization
Sensitive data
Slide 23
Slide 23 text
Authentication
(JWT tokens) Custom
“authorizer lambda”
“Who is the current user?”
Slide 24
Slide 24 text
Users
API 1
Authentication
Authorizer
API 2
API 3
Credentials
API
Request
JWT token
JWT token
Verify credentials
Validate token
& extract userId
User: Podge
Pass: Unicorns
Slide 25
Slide 25 text
Authorization
“Can Podge trade for Account23 ?”
User Action Resource
Slide 26
Slide 26 text
Authorization
● Custom ACL library imported by every lambda
● Built on top of:
○ node-acl
○ Knex.js
● Persistence in Postgres
User Action Resource
Podge trade Account23
Podge changeSettings Account23
Luciano delete *
... ... ...
Slide 27
Slide 27 text
ACL Lib Example
Slide 28
Slide 28 text
Environment variables
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
Quality
Testing
Continuous Integration
Deployment
Slide 31
Slide 31 text
Testing
JEST Node-TAP
Slide 32
Slide 32 text
Unit Testing
● Split business logic into small testable modules
● Use dependency injection for external resources
(DB, Filesystem, etc.)
● Mock dependencies with Sinon
● Aim for 100% coverage
Slide 33
Slide 33 text
● Use child_process.exec to launch “serverless invoke
local”
● Use node-tap to make assertions on the output
● Test environment recreated locally with docker (e.g.
Postgres test DB).
● Cannot always test locally (Redshift, SNS, SQS…)
Functional Testing
Slide 34
Slide 34 text
Functional test with sls invoke local
Slide 35
Slide 35 text
Building the project
Slide 36
Slide 36 text
Babel custom preset (transpile to Node 4.3.2)
Slide 37
Slide 37 text
Build process
JSdoc
Up to 90% code size reduction
Using Webpack 2 “tree-shaking”
Slide 38
Slide 38 text
Continuous Integration
Git-Flow
● Check code style (ESLint)
● Run unit tests
● Build the project
● Run functional tests
● If commit is on “master”:
Create deployable artifact
● Develop features/fixes locally
using git branches
● Push to GitHub
Slide 39
Slide 39 text
Deployments
local test dev qa production
Deploy lambdas and React apps
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
No content
Slide 42
Slide 42 text
Downloading the artifact from CircleCI
Slide 43
Slide 43 text
Deploying the code using Serverless
Slide 44
Slide 44 text
Developer Experience
Local development
Debugging
Monitoring
Slide 45
Slide 45 text
● Develop locally - Invoke local.
● Deploy to AWS.
● Invoke on AWS and stream the logs.
● This is a SLOW feedback loop.
● Lots of requests to improve local dev.
● Plugins are helping to improve this.
Local development
Slide 46
Slide 46 text
Serverless Plugins
Slide 47
Slide 47 text
Serverless Offline
Slide 48
Slide 48 text
Serverless Api Gateway Logs
Slide 49
Slide 49 text
Serverless Mocha
Slide 50
Slide 50 text
Monitoring
Slide 51
Slide 51 text
No content
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
No content
Slide 55
Slide 55 text
No content
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
No content
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
No content
Slide 61
Slide 61 text
No content
Slide 62
Slide 62 text
Cost
The formula
Cost forecasts
Slide 63
Slide 63 text
$ = Time * Memory * Invocation
Slide 64
Slide 64 text
http://serverlesscalc.com
Slide 65
Slide 65 text
Our current Lambda costs
Slide 66
Slide 66 text
Lessons learned
( we found out…)
Slide 67
Slide 67 text
DEBUGGING
Slide 68
Slide 68 text
How do we do debug then...
● console.log …
● Using debug module
● Enabling detailed logs only when needed (per feature)
Slide 69
Slide 69 text
No content
Slide 70
Slide 70 text
Learn Cloudformation
Slide 71
Slide 71 text
No content
Slide 72
Slide 72 text
No content
Slide 73
Slide 73 text
Api Gateway
Custom domain mapping
It is a manual step…
Slide 74
Slide 74 text
With Api Gateway...
You have 10 30 seconds!
Slide 75
Slide 75 text
AWS Lambda limitations
● 512 MB of TMP space
● Lambdas can only execute for 300 seconds
● Request Response size of 6 MB
● Event size of 128 Kb
● Max 50 MB for deployment package
Slide 76
Slide 76 text
AWS Soft limits
http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html
Slide 77
Slide 77 text
Cold starts...
☃
Slide 78
Slide 78 text
https://www.iopipe.com/2016/09/understanding-aws-lambda-coldstarts/
Slide 79
Slide 79 text
serverlessbeer.com
is a real thing!
Well… it’s not, but it’s a good idea!
(read the tutorial)
Slide 80
Slide 80 text
Recap
● Serverless architectures are COOL!
○ Infinite scalability at low cost
○ Managed service
○ Still has some limitations
● Managing a project might be hard but:
○ Technology progress and open source projects (Serverless.com) will make
things easier
Slide 81
Slide 81 text
Planet 9 Energy tech-team
We are
hiring :)
Peter Saŝa Podge Luciano YOU?
Alberto Joe Domagoj
Hugh Gus Ruth
Check out
Loki.js
Slide 82
Slide 82 text
THANK YOU
Planet9energy.com - [email protected]
Special Thanks to @johnbrett_
Use the ,
Luke!