Pro Yearly is on sale from $80 to $50! »

AWS Lambda and Serverless framework: lessons learned while building a serverless company

AWS Lambda and Serverless framework: lessons learned while building a serverless company

Planet9energy.com is a new electricity company that is building a sophisticated analytics and energy trading platform for the UK market. Since the earliest days of the company we took the unconventional decision to go serverless and finally we are building the product on top of AWS Lambda and the Serverless framework using Node.js. In this talk we will discuss why we took this radical decision, what are the pros and cons of this approach and what are the main issues we faced as a tech team in our design and development experience. We will discuss how normal things like testing and deployment need to be re-thought to work on a serverless fashion but also the benefits of (almost) infinite auto-scalability and the piece of mind of not having to manage hundreds of servers. Finally we will underline how Node.js seems to fit naturally in this scenario and how it makes developing serverless applications extremely convenient.

Thanks to Padraig O'Brien and Luciano Mammino for speaking this month.

Speakers Bio:

Padraig O'Brien

Podge @Podgeypoos79 is a software engineer for over 15 years, most of that was spent developing in .NET and SQL Server, designing and building large scale data intensive applications. Lately he has shifted towards open source technologies and is spending most of his time learning Node.js, Scala and cool data tech like Spark, Cassandra. He is also working on a “super-secret” project called UnicornDB, don’t tell anybody!

In his spare time he helps out with organising some meetups like NodeSchool Dublin, NodeSchool Dun Laoghaire and teaching Kanban via Agile Lean Ireland.

Luciano Mammino

Luciano @loige is a Software Engineer born in 1987, the same year that the Nintendo released “Super Mario Bros” in Europe, which, “by chance” is his favourite game! His primary passion is code and he is extremely fascinated by the web, smart apps and everything that's creative like music, art and design. He started coding at the age of 12 using his father's old i386 provided only with DOS and the qBasic interpreter.He is a senior software developer at Planet9Energy in Dublin and he loves JavaScript (React/Node.js). He is also the co-author of "Node.js design patterns" 2nd edition (Packt, http://amzn.to/1ZF279B).

Hosted by Intercom, sponsored by Nearform and organised by Node.js Dublin (https://www.meetup.com/Dublin-Node-js-Meetup/events/236870576/)

F3a6662b3cd161c3c2f13604965ed0f2?s=128

Luciano Mammino

January 26, 2017
Tweet

Transcript

  1. AWS Lambda & Serverless framework Lessons learned while building a

    serverless company 26/01/2017 @Podgeypoos79 - @loige
  2. Luciano Mammino • @loige on Twitter • Lmammino on GitHub

    • Blog at loige.co • Co-author of: Node.js Design Patterns (Second Edition)
  3. Padraig O’Brien (a.k.a “Podge”) • @podgeypoos79 on Twitter • Does

    not have a book (yet…) • Organiser of Nodeschool Dublin and Dun Laoghaire • Organiser of Dublin Serverless meetup • He is writing his own database, unicorndb
  4. Agenda • Planet 9 Energy • Serverless • Security •

    Quality • Developer Experience • Costs • Lessons learned
  5. planet9energy.com

  6. None
  7. 17.520 * * 24 Meter readings Customer size Data versions

    ≈4 billion data pts / year (tot.) SMALL customer ≈3 mln data pts / year MEDIUM customer ≈20 mln data pts / year BIG customer ≈110 mln data pts / year ( Per customer / year )
  8. • Limited number of “Full stack” engineers • Write &

    deploy quality code as fast as possible • Adopt hot and relevant technologies • No servers... or more to the point, NO CALLS AT 2 AM! Our requirements
  9. Early experiments

  10. Current Technology

  11. “Serverless” Meaning Serverless framework AWS Lambda

  12. What does “serverless” mean?

  13. Why Serverless framework?

  14. Anatomy of Serverless.yml Serverless.yml header

  15. Defining functions and events

  16. Why?

  17. What is a lambda? • Function as a service (FAAS)

    • Pay for invocation / processing time • Virtually “infinite” auto-scaling • Focus on business logic, not on servers daaaa!
  18. Lambda as micro-services • Events are first-class citizens • Every

    lambda scales independently • Agility (develop features quick and in an isolated fashion) Classic micro-services concerns: • Granularity (how to separate features? BDD? Bounded Contexts?) • Orchestration (dependencies between lambdas, service discovery…)
  19. Anatomy of a Lambda in Node.js

  20. Some use cases • REST over HTTP (API Gateway) •

    SNS messages • Schedule/Cron • DynamoDB data changes • S3 files changes • IoT
  21. HTTP REQUEST - API Call POST /hello/Elvis?foo=bar { “test”: “body”

    }
  22. Security Authentication Authorization Sensitive data

  23. Authentication (JWT tokens) Custom “authorizer lambda” “Who is the current

    user?”
  24. Users API 1 Authentication Authorizer API 2 API 3 Credentials

    API Request JWT token JWT token Verify credentials Validate token & extract userId User: Podge Pass: Unicorns
  25. Authorization “Can Podge trade for Account23 ?” User Action Resource

  26. Authorization • Custom ACL library imported by every lambda •

    Built on top of: ◦ node-acl ◦ Knex.js • Persistence in Postgres User Action Resource Podge trade Account23 Podge changeSettings Account23 Luciano delete * ... ... ...
  27. ACL Lib Example

  28. Environment variables

  29. None
  30. Quality Testing Continuous Integration Deployment

  31. Testing JEST Node-TAP

  32. Unit Testing • Split business logic into small testable modules

    • Use dependency injection for external resources (DB, Filesystem, etc.) • Mock dependencies with Sinon • Aim for 100% coverage
  33. • Use child_process.exec to launch “serverless invoke local” • Use

    node-tap to make assertions on the output • Test environment recreated locally with docker (e.g. Postgres test DB). • Cannot always test locally (Redshift, SNS, SQS…) Functional Testing
  34. Functional test with sls invoke local

  35. Building the project

  36. Babel custom preset (transpile to Node 4.3.2)

  37. Build process JSdoc Up to 90% code size reduction Using

    Webpack 2 “tree-shaking”
  38. Continuous Integration Git-Flow • Check code style (ESLint) • Run

    unit tests • Build the project • Run functional tests • If commit is on “master”: Create deployable artifact • Develop features/fixes locally using git branches • Push to GitHub
  39. Deployments local test dev qa production Deploy lambdas and React

    apps
  40. None
  41. None
  42. Downloading the artifact from CircleCI

  43. Deploying the code using Serverless

  44. Developer Experience Local development Debugging Monitoring

  45. • Develop locally - Invoke local. • Deploy to AWS.

    • Invoke on AWS and stream the logs. • This is a SLOW feedback loop. • Lots of requests to improve local dev. • Plugins are helping to improve this. Local development
  46. Serverless Plugins

  47. Serverless Offline

  48. Serverless Api Gateway Logs

  49. Serverless Mocha

  50. Monitoring

  51. None
  52. None
  53. None
  54. None
  55. None
  56. None
  57. None
  58. None
  59. None
  60. None
  61. None
  62. Cost The formula Cost forecasts

  63. $ = Time * Memory * Invocation

  64. http://serverlesscalc.com

  65. Our current Lambda costs

  66. Lessons learned ( we found out…)

  67. DEBUGGING

  68. How do we do debug then... • console.log … •

    Using debug module • Enabling detailed logs only when needed (per feature)
  69. None
  70. Learn Cloudformation

  71. None
  72. None
  73. Api Gateway Custom domain mapping It is a manual step…

  74. With Api Gateway... You have 10 30 seconds!

  75. AWS Lambda limitations • 512 MB of TMP space •

    Lambdas can only execute for 300 seconds • Request Response size of 6 MB • Event size of 128 Kb • Max 50 MB for deployment package
  76. AWS Soft limits http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html

  77. Cold starts... ☃

  78. https://www.iopipe.com/2016/09/understanding-aws-lambda-coldstarts/

  79. serverlessbeer.com is a real thing! Well… it’s not, but it’s

    a good idea! (read the tutorial)
  80. Recap • Serverless architectures are COOL! ◦ Infinite scalability at

    low cost ◦ Managed service ◦ Still has some limitations • Managing a project might be hard but: ◦ Technology progress and open source projects (Serverless.com) will make things easier
  81. Planet 9 Energy tech-team We are hiring :) Peter Saŝa

    Podge Luciano YOU? Alberto Joe Domagoj Hugh Gus Ruth Check out Loki.js
  82. THANK YOU Planet9energy.com - tech@planet9energy.com Special Thanks to @johnbrett_ Use

    the , Luke!