$30 off During Our Annual Pro Sale. View Details »

AWS Lambda and Serverless framework: lessons learned while building a serverless company

AWS Lambda and Serverless framework: lessons learned while building a serverless company

Planet9energy.com is a new electricity company that is building a sophisticated analytics and energy trading platform for the UK market. Since the earliest days of the company we took the unconventional decision to go serverless and finally we are building the product on top of AWS Lambda and the Serverless framework using Node.js. In this talk we will discuss why we took this radical decision, what are the pros and cons of this approach and what are the main issues we faced as a tech team in our design and development experience. We will discuss how normal things like testing and deployment need to be re-thought to work on a serverless fashion but also the benefits of (almost) infinite auto-scalability and the piece of mind of not having to manage hundreds of servers. Finally we will underline how Node.js seems to fit naturally in this scenario and how it makes developing serverless applications extremely convenient.

Thanks to Padraig O'Brien and Luciano Mammino for speaking this month.

Speakers Bio:

Padraig O'Brien

Podge @Podgeypoos79 is a software engineer for over 15 years, most of that was spent developing in .NET and SQL Server, designing and building large scale data intensive applications. Lately he has shifted towards open source technologies and is spending most of his time learning Node.js, Scala and cool data tech like Spark, Cassandra. He is also working on a “super-secret” project called UnicornDB, don’t tell anybody!

In his spare time he helps out with organising some meetups like NodeSchool Dublin, NodeSchool Dun Laoghaire and teaching Kanban via Agile Lean Ireland.

Luciano Mammino

Luciano @loige is a Software Engineer born in 1987, the same year that the Nintendo released “Super Mario Bros” in Europe, which, “by chance” is his favourite game! His primary passion is code and he is extremely fascinated by the web, smart apps and everything that's creative like music, art and design. He started coding at the age of 12 using his father's old i386 provided only with DOS and the qBasic interpreter.He is a senior software developer at Planet9Energy in Dublin and he loves JavaScript (React/Node.js). He is also the co-author of "Node.js design patterns" 2nd edition (Packt, http://amzn.to/1ZF279B).

Hosted by Intercom, sponsored by Nearform and organised by Node.js Dublin (https://www.meetup.com/Dublin-Node-js-Meetup/events/236870576/)

Luciano Mammino

January 26, 2017
Tweet

More Decks by Luciano Mammino

Other Decks in Technology

Transcript

  1. AWS Lambda &
    Serverless framework
    Lessons learned while building a serverless company
    26/01/2017
    @Podgeypoos79 - @loige

    View Slide

  2. Luciano Mammino
    ● @loige on Twitter
    ● Lmammino on GitHub
    ● Blog at loige.co
    ● Co-author of:
    Node.js Design Patterns (Second Edition)

    View Slide

  3. Padraig O’Brien (a.k.a “Podge”)
    ● @podgeypoos79 on Twitter
    ● Does not have a book (yet…)
    ● Organiser of Nodeschool Dublin and Dun Laoghaire
    ● Organiser of Dublin Serverless meetup
    ● He is writing his own database, unicorndb

    View Slide

  4. Agenda
    ● Planet 9 Energy
    ● Serverless
    ● Security
    ● Quality
    ● Developer Experience
    ● Costs
    ● Lessons learned

    View Slide

  5. planet9energy.com

    View Slide

  6. View Slide

  7. 17.520 * * 24
    Meter readings Customer size Data versions
    ≈4 billion data pts / year (tot.)
    SMALL customer ≈3 mln data pts / year MEDIUM customer ≈20 mln data pts / year BIG customer ≈110 mln data pts / year
    ( Per customer / year )

    View Slide

  8. ● Limited number of “Full stack” engineers
    ● Write & deploy quality code as fast as possible
    ● Adopt hot and relevant technologies
    ● No servers... or more to the point, NO CALLS AT 2 AM!
    Our requirements

    View Slide

  9. Early experiments

    View Slide

  10. Current Technology

    View Slide

  11. “Serverless”
    Meaning
    Serverless framework
    AWS Lambda

    View Slide

  12. What does
    “serverless” mean?

    View Slide

  13. Why Serverless
    framework?

    View Slide

  14. Anatomy of Serverless.yml
    Serverless.yml header

    View Slide

  15. Defining functions and events

    View Slide

  16. Why?

    View Slide

  17. What is a lambda?
    ● Function as a service (FAAS)
    ● Pay for invocation / processing time
    ● Virtually “infinite” auto-scaling
    ● Focus on business logic, not on servers
    daaaa!

    View Slide

  18. Lambda as micro-services
    ● Events are first-class citizens
    ● Every lambda scales independently
    ● Agility (develop features quick and in an isolated fashion)
    Classic micro-services concerns:
    ● Granularity (how to separate features? BDD? Bounded Contexts?)
    ● Orchestration (dependencies between lambdas, service discovery…)

    View Slide

  19. Anatomy of a Lambda in Node.js

    View Slide

  20. Some use cases
    ● REST over HTTP (API Gateway)
    ● SNS messages
    ● Schedule/Cron
    ● DynamoDB data changes
    ● S3 files changes
    ● IoT

    View Slide

  21. HTTP REQUEST - API Call
    POST /hello/Elvis?foo=bar
    {
    “test”: “body”
    }

    View Slide

  22. Security
    Authentication
    Authorization
    Sensitive data

    View Slide

  23. Authentication
    (JWT tokens) Custom
    “authorizer lambda”
    “Who is the current user?”

    View Slide

  24. Users
    API 1
    Authentication
    Authorizer
    API 2
    API 3
    Credentials
    API
    Request
    JWT token
    JWT token
    Verify credentials
    Validate token
    & extract userId
    User: Podge
    Pass: Unicorns

    View Slide

  25. Authorization
    “Can Podge trade for Account23 ?”
    User Action Resource

    View Slide

  26. Authorization
    ● Custom ACL library imported by every lambda
    ● Built on top of:
    ○ node-acl
    ○ Knex.js
    ● Persistence in Postgres
    User Action Resource
    Podge trade Account23
    Podge changeSettings Account23
    Luciano delete *
    ... ... ...

    View Slide

  27. ACL Lib Example

    View Slide

  28. Environment variables

    View Slide

  29. View Slide

  30. Quality
    Testing
    Continuous Integration
    Deployment

    View Slide

  31. Testing
    JEST Node-TAP

    View Slide

  32. Unit Testing
    ● Split business logic into small testable modules
    ● Use dependency injection for external resources
    (DB, Filesystem, etc.)
    ● Mock dependencies with Sinon
    ● Aim for 100% coverage

    View Slide

  33. ● Use child_process.exec to launch “serverless invoke
    local”
    ● Use node-tap to make assertions on the output
    ● Test environment recreated locally with docker (e.g.
    Postgres test DB).
    ● Cannot always test locally (Redshift, SNS, SQS…)
    Functional Testing

    View Slide

  34. Functional test with sls invoke local

    View Slide

  35. Building the project

    View Slide

  36. Babel custom preset (transpile to Node 4.3.2)

    View Slide

  37. Build process
    JSdoc
    Up to 90% code size reduction
    Using Webpack 2 “tree-shaking”

    View Slide

  38. Continuous Integration
    Git-Flow
    ● Check code style (ESLint)
    ● Run unit tests
    ● Build the project
    ● Run functional tests
    ● If commit is on “master”:
    Create deployable artifact
    ● Develop features/fixes locally
    using git branches
    ● Push to GitHub

    View Slide

  39. Deployments
    local test dev qa production
    Deploy lambdas and React apps

    View Slide

  40. View Slide

  41. View Slide

  42. Downloading the artifact from CircleCI

    View Slide

  43. Deploying the code using Serverless

    View Slide

  44. Developer Experience
    Local development
    Debugging
    Monitoring

    View Slide

  45. ● Develop locally - Invoke local.
    ● Deploy to AWS.
    ● Invoke on AWS and stream the logs.
    ● This is a SLOW feedback loop.
    ● Lots of requests to improve local dev.
    ● Plugins are helping to improve this.
    Local development

    View Slide

  46. Serverless Plugins

    View Slide

  47. Serverless Offline

    View Slide

  48. Serverless Api Gateway Logs

    View Slide

  49. Serverless Mocha

    View Slide

  50. Monitoring

    View Slide

  51. View Slide

  52. View Slide

  53. View Slide

  54. View Slide

  55. View Slide

  56. View Slide

  57. View Slide

  58. View Slide

  59. View Slide

  60. View Slide

  61. View Slide

  62. Cost
    The formula
    Cost forecasts

    View Slide

  63. $ = Time * Memory * Invocation

    View Slide

  64. http://serverlesscalc.com

    View Slide

  65. Our current Lambda costs

    View Slide

  66. Lessons learned
    ( we found out…)

    View Slide

  67. DEBUGGING

    View Slide

  68. How do we do debug then...
    ● console.log …
    ● Using debug module
    ● Enabling detailed logs only when needed (per feature)

    View Slide

  69. View Slide

  70. Learn Cloudformation

    View Slide

  71. View Slide

  72. View Slide

  73. Api Gateway
    Custom domain mapping
    It is a manual step…

    View Slide

  74. With Api Gateway...
    You have 10 30 seconds!

    View Slide

  75. AWS Lambda limitations
    ● 512 MB of TMP space
    ● Lambdas can only execute for 300 seconds
    ● Request Response size of 6 MB
    ● Event size of 128 Kb
    ● Max 50 MB for deployment package

    View Slide

  76. AWS Soft limits
    http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html

    View Slide

  77. Cold starts...

    View Slide

  78. https://www.iopipe.com/2016/09/understanding-aws-lambda-coldstarts/

    View Slide

  79. serverlessbeer.com
    is a real thing!
    Well… it’s not, but it’s a good idea!
    (read the tutorial)

    View Slide

  80. Recap
    ● Serverless architectures are COOL!
    ○ Infinite scalability at low cost
    ○ Managed service
    ○ Still has some limitations
    ● Managing a project might be hard but:
    ○ Technology progress and open source projects (Serverless.com) will make
    things easier

    View Slide

  81. Planet 9 Energy tech-team
    We are
    hiring :)
    Peter Saŝa Podge Luciano YOU?
    Alberto Joe Domagoj
    Hugh Gus Ruth
    Check out
    Loki.js

    View Slide

  82. THANK YOU
    Planet9energy.com - [email protected]
    Special Thanks to @johnbrett_
    Use the ,
    Luke!

    View Slide