Building a Serverless company with Node.js, React and the Serverless Framework

Interactive slides available at: http://loige.link/jsday2017

Planet9energy.com is a new electricity company building a sophisticated analytics and energy trading platform for the UK market. Since the earliest draft of the platform, we took the unconventional decision to go serverless and build the product on top of AWS Lambda and the Serverless framework using Node.js. In this talk, I want to discuss why we took this radical decision, what are the pros and cons of this approach and what are the main issues we faced as a tech team in our design and development experience. We will discuss how normal things like testing and deployment need to be re-thought to work on a serverless fashion but also the benefits of (almost) infinite self-scalability and the piece of mind of not having to manage hundreds of servers. Finally, we will underline how Node.js seems to fit naturally in this scenario and how it makes developing serverless applications extremely convenient.

Transcript

1. BUILDING A SERVERLESS COMPANY WITH NODE, REACT AND THE SERVERLESS FRAMEWORK MAY 10TH 2017, Verona Luciano Mammino loige.link/jsday2017 1
2. Who is LUCIANO lmammino loige loige.co 2
3. NeBK20JV -20% eBook NpBK15JV -15% Print loige.link/node-book fancy a coupon? 3
4. fullstackbulletin.com built with ⚡serverless 4
5. Agenda What is Serverless History & definition Advantages & costs How it Works Example on AWS Lambda Example with Serverless Framework Serverless at Planet 9 Architecture Security Quality Monitoring / Logging Step Functions 5
6. PART 1 What is Serverless 6
7. 1996 - Let's order few more servers for this rack... 7
8. 2006 - Let's move the infrastructure in "the cloud"... 8
9. 2013 - I can "ship" the new API to any VPS as a "container" 9
10. TODAY - I ain't got no infrastructure, just code "in the cloud" baby! 10
11. loige.link/serverless-commitstrip 11
12. “ The essence of the serverless trend is the absence of the server concept during software development. — Auth0 loige.link/what-is-serverless 12
13. Serverless & Framework Serverless.com 13
14. Focus on business logic, not on infrastructure Virtually “infinite” auto-scaling Pay for invocation / processing time (cheap!) Advantages of serverless 14
15. Is it cheaper to go serverless? ... It depends 15
16. Car analogy Cars are parked 95% of the time ( ) How much do you use the car? loige.link/car-parked-95 Own a car (Bare metal servers) Rent a car (VPS) City car-sharing (Serverless) 16
17. loige.link/serverless-calc Less than $0.40/day for 1 execution/second 17
18. What can we build? Mobile Backends APIs & Microservices Data Processing pipelines ⚡ Webhooks Bots and integrations ⚙ IoT Backends Single page web applications 18
19. The paradigm Event → 19
20. IF ________________________________ THEN ________________________________ "IF this THEN that" model 20
21. Cloud providers IBM OpenWhisk AWS Lambda Azure Functions Google Cloud Functions Auth0 Webtask 21
22. Serverless and JavaScript Frontend Serverless Web hosting is static, but you can build SPAs (React, Angular, Vue, etc.) Backend Node.js is supported by every provider ⚡ Fast startup (as opposed to Java) Use all the modules on NPM Support other languages/dialects (TypeScript, ClojureScript, ESNext...) 22
23. exports.myLambda = function ( event, context, callback ) { // get input from event and context // use callback to return output or errors } Anatomy of a Node.js lambda on AWS 23
24. Let's build a "useful" Hello World API 24
32. API Gateway config has been generated for us... 32
34. Recap 1. Login to AWS Dashboard 2. Create new Lambda from blueprint 3. Configure API Gateway trigger 4. Configure Lambda and Security 5. Write Lambda code 6. Configure Roles & Publish ⚡Ready! No local testing ... Manual process 34
35. Enter the... 35
36. Get serverless framework npm install --global serverless@latest sls --help 36
37. Create a project mkdir helloWorldApi cd helloWorldApi touch handler.js serverless.yml 37
38. // handler.js 'use strict'; exports.helloWorldHandler = (event, context, callback) => { const name = (event.queryStringParameters && event.queryStringParameters.name) ? event.queryStringParameters.name : 'Verona'; const response = { statusCode: 200, headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({message: `Hello ${name}`}) }; callback(null, response); }; 38
39. # serverless.yml service: sls-helloWorldApi provider: name: aws runtime: "nodejs6.10" functions: helloWorld: handler: "handler.helloWorldHandler" events: - http: path: / method: get 39
40. Local test touch event.json { "queryStringParameters": { "name": "Tim Wagner" }, "httpMethod": "GET", "path": "/" } 40
41. Local test 41
42. Deploy the lambda 42
43. Recap 1. Install serverless framework 2. Create handler & serverless config 3. Test locally (optional) 4. Deploy 5. Party hard! 43
44. PART 2 Serverless at Planet 9 44
45. https://planet9energy.com 45
47. a Big Data company E.g. Meter readings per customer/year 2 × 24 × 365 Half Hours × 25 ~ meter reading points × 24 ~ data versions = load™ of data 47
48. Limited number of “Full stack” engineers ⚡ Write & deploy quality code fast Experiment different approaches over different features Adopt hot and relevant technologies Limited number of servers = LIMITED CALLS AT 2 AM! Why Serverless 48
49. Current infrastructure Serverless land Web API & Jobs Messaging 49
50. Serverless Frontend & Backend Cloufront & S3 API Gateway & Lambda planet9energy.io api.planet9energy.io Access-Control-Allow-Origin: https://planet9energy.io CORS HTTP HEADER Custom error page index.html Serverless Web Hosting Serverless APIs 50
51. Security: Authentication "Who is the current user?" JWT Tokens Custom Authorizer Lambda 51
52. JWT Token Authorizer Login user: "Podge" pass: "Unicorns<3" Users DB Check Credentials JWT token Validate token & extract userId API request 52 API 1 API 2 API 3
53. Security: Authorization "Can Podge trade for Account17 ?" User Action Resource 53
54. User Action Resource Podge trade Account17 Podge changeSettings Account17 Luciano delete * ... ... ... Custom ACL module used by every lambda Built on & Persistence in Postgres node-acl knex.js 54
55. import { can } from '@planet9/acl'; export const tradingHandler = async (event, context, callback) => { const user = event.requestContext.userId; const account = event.pathParameters.accountId; const isAuthorized = await can(user, 'trade', account); if (!isAuthorized) { return callback(new Error('Not Authorized')); } // ... business logic } 55
56. Security: Sensitive Data export const handler = (event, context, callback) => { const CONN_STRING = process.env.CONN_STRING; // ... } 56
57. # serverless.yml functions: myLambda: handler: handler.myHandler environment: CONN_STRING: ${env:CONN_STRING} Serverless environment variables 57
58. Split business logic into small testable modules Use dependency injection for external resources (DB, Filesystem, etc.) Mock stuff with Jest Aim for 100% coverage Nyan Cat test runners! Quality: Unit Tests 58
59. Use child_process.exec to launch "sls invoke local" Make assertions on the JSON output Test environment simulated with Docker (Postgres, Cassandra, etc.) Some services are hard to test locally (SNS, SQS, S3) Quality: Functional Tests 59
60. Git-Flow Feature branches Push code GitHub -> CI Quality: Continuous integration CircleCI Lint code (ESlint) Unit tests Build project Functional tests if commit on "master": create deployable artifact 60
61. Development / Test / Deployment 61
62. Debugging 62
63. ... How do we debug then console.log... Using the module Enable detailed logs only when needed (e.g. export DEBUG=tradingCalculations) debug 63
64. Step Functions Coordinate components of distributed applications Orchestrate different AWS Lambdas Visual flows Different execution patterns (sequential, branching, parallel) 64
66. Some lessons learned... Think serverless as microservices " " ❄ ! Be aware of There is still some infrastructure: use proper tools (Cloudformation, Terraform, ...) microfunctions Cold starts soft limits 66
67. Serverless architectures are COOL! Infinite scalability at low cost Managed service Still, has some limitations Managing a project might be hard but: Technology progress and open source (e.g. Serverless.com) will make things easier Recap 67
68. Thanks loige.link/jsday2017 @loige (special thanks to , & ) @Podgeypoos79 @quasi_modal @augeva Feedback joind.in/talk/79fa0 68