Planet9energy.com is a new electricity company that is building a sophisticated analytics and energy trading platform for the UK market. Since the earliest days of the company we took the unconventional decision to go serverless and finally we are building the product on top of AWS Lambda and the Serverless framework using Node.js. In this talk we will discuss why we took this radical decision, what are the pros and cons of this approach and what are the main issues we faced as a tech team in our design and development experience. We will discuss how normal things like testing and deployment need to be re-thought to work on a serverless fashion but also the benefits of (almost) infinite auto-scalability and the piece of mind of not having to manage hundreds of servers. Finally we will underline how Node.js seems to fit naturally in this scenario and how it makes developing serverless applications extremely convenient.
Thanks to Padraig O'Brien and Luciano Mammino for speaking this month.
Podge @Podgeypoos79 is a software engineer for over 15 years, most of that was spent developing in .NET and SQL Server, designing and building large scale data intensive applications. Lately he has shifted towards open source technologies and is spending most of his time learning Node.js, Scala and cool data tech like Spark, Cassandra. He is also working on a “super-secret” project called UnicornDB, don’t tell anybody!
In his spare time he helps out with organising some meetups like NodeSchool Dublin, NodeSchool Dun Laoghaire and teaching Kanban via Agile Lean Ireland.
Hosted by Intercom, sponsored by Nearform and organised by Node.js Dublin (https://www.meetup.com/Dublin-Node-js-Meetup/events/236870576/)
AWS Lambda &
Lessons learned while building a serverless company
@Podgeypoos79 - @loige
● @loige on Twitter
● Lmammino on GitHub
● Blog at loige.co
● Co-author of:
Node.js Design Patterns (Second Edition)
Padraig O’Brien (a.k.a “Podge”)
● @podgeypoos79 on Twitter
● Does not have a book (yet…)
● Organiser of Nodeschool Dublin and Dun Laoghaire
● Organiser of Dublin Serverless meetup
● He is writing his own database, unicorndb
● Planet 9 Energy
● Developer Experience
● Lessons learned
17.520 * * 24
Meter readings Customer size Data versions
≈4 billion data pts / year (tot.)
SMALL customer ≈3 mln data pts / year MEDIUM customer ≈20 mln data pts / year BIG customer ≈110 mln data pts / year
( Per customer / year )
● Limited number of “Full stack” engineers
● Write & deploy quality code as fast as possible
● Adopt hot and relevant technologies
● No servers... or more to the point, NO CALLS AT 2 AM!
Anatomy of Serverless.yml
Defining functions and events
What is a lambda?
● Function as a service (FAAS)
● Pay for invocation / processing time
● Virtually “infinite” auto-scaling
● Focus on business logic, not on servers
Lambda as micro-services
● Events are first-class citizens
● Every lambda scales independently
● Agility (develop features quick and in an isolated fashion)
Classic micro-services concerns:
● Granularity (how to separate features? BDD? Bounded Contexts?)
● Orchestration (dependencies between lambdas, service discovery…)
Anatomy of a Lambda in Node.js
Some use cases
● REST over HTTP (API Gateway)
● SNS messages
● DynamoDB data changes
● S3 files changes
HTTP REQUEST - API Call
(JWT tokens) Custom
“Who is the current user?”
& extract userId
“Can Podge trade for Account23 ?”
User Action Resource
● Custom ACL library imported by every lambda
● Built on top of:
● Persistence in Postgres
User Action Resource
Podge trade Account23
Podge changeSettings Account23
Luciano delete *
... ... ...
ACL Lib Example
● Split business logic into small testable modules
● Use dependency injection for external resources
(DB, Filesystem, etc.)
● Mock dependencies with Sinon
● Aim for 100% coverage
● Use child_process.exec to launch “serverless invoke
● Use node-tap to make assertions on the output
● Test environment recreated locally with docker (e.g.
Postgres test DB).
● Cannot always test locally (Redshift, SNS, SQS…)
Functional test with sls invoke local
Building the project
Babel custom preset (transpile to Node 4.3.2)
Up to 90% code size reduction
Using Webpack 2 “tree-shaking”
● Check code style (ESLint)
● Run unit tests
● Build the project
● Run functional tests
● If commit is on “master”:
Create deployable artifact
● Develop features/fixes locally
using git branches
● Push to GitHub
local test dev qa production
Deploy lambdas and React apps
Downloading the artifact from CircleCI
Deploying the code using Serverless
● Develop locally - Invoke local.
● Deploy to AWS.
● Invoke on AWS and stream the logs.
● This is a SLOW feedback loop.
● Lots of requests to improve local dev.
● Plugins are helping to improve this.
Serverless Api Gateway Logs
$ = Time * Memory * Invocation
Our current Lambda costs
( we found out…)
How do we do debug then...
● console.log …
● Using debug module
● Enabling detailed logs only when needed (per feature)
Custom domain mapping
It is a manual step…
With Api Gateway...
You have 10 30 seconds!
AWS Lambda limitations
● 512 MB of TMP space
● Lambdas can only execute for 300 seconds
● Request Response size of 6 MB
● Event size of 128 Kb
● Max 50 MB for deployment package
AWS Soft limits
is a real thing!
Well… it’s not, but it’s a good idea!
(read the tutorial)
● Serverless architectures are COOL!
○ Infinite scalability at low cost
○ Managed service
○ Still has some limitations
● Managing a project might be hard but:
○ Technology progress and open source projects (Serverless.com) will make
Planet 9 Energy tech-team
Peter Saŝa Podge Luciano YOU?
Alberto Joe Domagoj
Hugh Gus Ruth
Planet9energy.com - [email protected]
Special Thanks to @johnbrett_
Use the ,