Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From 1 to 101 Lambda Functions in Production: Evolving a Serverless Architecture - Node Congress 2021

From 1 to 101 Lambda Functions in Production: Evolving a Serverless Architecture - Node Congress 2021

Building a serverless function or an API is easy. However, things get a bit more complicated as your application grows. What works for a few functions often don't work for hundreds of functions. As your application grows, you'll need to evolve your architecture, deployment, monitoring, and many other things. This talk is a case study of an evolution of the serverless startup's architecture. We started with a single Lambda function in early 2018 and evolved our application through multiple stages and architectures. At the moment application uses GraphQL and more than 100 Lambda functions and serves millions of requests. We faced and solved many issues during the last three years, learned many things, and managed to keep our infrastructure costs low.

2663a9ac90cc6ed420e0e1560db57782?s=128

Slobodan Stojanović

February 19, 2021
Tweet

Transcript

  1. None
  2. A story about Vacation Tracker a 100% serverless startup

  3. A story about Vacation Tracker a 100% serverless startup

  4. A story about Vacation Tracker a 100% serverless startup

  5. A story about Vacation Tracker a 100% serverless startup

  6. A story about Vacation Tracker a 100% serverless startup

  7. A story about Vacation Tracker a 100% serverless startup

  8. In 2016 we decided to build a leave tracking system

    a leave tracking system
  9. In 2016 we decided to build a leave tracking system

    To solve our own leave tracking problem
  10. In 2016 we decided to build a leave tracking system

    2017 To solve our own leave tracking problem
  11. In 2016 we decided to build a leave tracking system

    2017 2018 To solve our own leave tracking problem
  12. The idea was simple: - track leave requests and number

    of remaining days - use some SSO, so we do not n!d to remember more passwords - connect to Slack so that we can have the info in Slack - connect to the calendar so we can subscribe to events
  13. We were solving our problem but we were not sure

    if anyone else will use it
  14. - startups - sma" companies - schools - non-profits -

    teams from enterprises - government organizations - churches
  15. None
  16. The number of unique users by December 1st 263623

  17. The dashboard

  18. Slack

  19. Microsoft Teams

  20. The architecture v0.1 A simple serverless bot

  21. None
  22. Why serverless?

  23. From 1 to 101 Lambda Functions in Production Evolving a

    Serverless Architecture
  24. @slobodan_ Before we continue…

  25. Slobodan Stojanović CTO @ Cloud Horizon & CTO @ Vacation

    Tracker co-author of Serverless Applications with Node.js book AWS Serverless Hero https://slobodan.me
  26. Why serverless?

  27. s e r v e r l e s s

  28. s e r v e r l e s s

    low
  29. s e r v e r l e s s

    low xpensive
  30. s e r v e r l e s s

    low xpensive endor lock-in
  31. s e r v e r l e s s

    low xpensive endor lock-in
  32. Serverless because: - Sma" team without devops experience - AUTO

    SCALABLE - Cheap - Fast to build a prototype - Secure
  33. But it wasn't 100% "serverless"

  34. None
  35. None
  36. - Quick and independent deployments - Easy to understand and

    maintain - Easy to onboard new people - Cheap Benefits:
  37. The Cost: $0/month

  38. Adding new features?

  39. None
  40. None
  41. - Independent deployments - Hard to manage - Hard to

    scale - A bottleneck Downsides:
  42. ~ 100 paying teams

  43. The architecture v1.0 A complex serverless application

  44. Infrastructure-as-a-Code

  45. Service Service (micro?) (micro?)

  46. None
  47. ~150 Lambda functions

  48. FIRST MIGRATIONS (FROM AN OLD SERVICE TO A NEW ONE)

  49. Finding a good architecture

  50. - Everything is in IaaC CloudFormation - Replaced Node.js server

    with serverless services - Added TypeScript - Replaced MongoDB with DynamoDB Things we changed:
  51. - independent deployments - Auto-scalable - Almost 100% uptime out-of-the-box

    - Sti" cheap Benefits:
  52. - Storing state not events - Wasting time on less

    important things - Hard to onboard new developers - Many new services Downsides:
  53. - Storing state not events - Wasting time on less

    important things - Hard to onboard new developers - Many new services - Developers don't like YAML :) Downsides:
  54. ~ 600 paying teams

  55. The architecture v2.0 An event driven system

  56. Finding a good architecture v2.0

  57. Command Query Responsibility Segregation (CQRS)

  58. Why CQRS?

  59. Storing the state vs storing the events

  60. - Ana created a location and moved John and Mike

    - Ana assigned Mike as an approver - Ana assigned a leave policy (20 PTO days per year) - John requested a leave, Ana approved - Ro"over, some unused days were transferred to the next year - Ana changed John's working w!k - Mike added some past leaves for John - Alex transferred John to another location with different policy - …
  61. A quiz question: Calculate John's remaining PTO days for 2021

  62. CQRS to the rescue

  63. Mutation Event storage Subscription Query Read-only table EventBridge Business logic

    Business logic AWS AppSync
  64. Mutation Event storage Subscription Query Read-only table EventBridge Business logic

    Business logic AWS AppSync
  65. Mutation Event storage Subscription Query Read-only table EventBridge Business logic

    Business logic AWS AppSync
  66. 112 Lambda functions

  67. - Fu"y Managed GraphQL - Less code - Better control

    - A" benefits from the previous architecture - Monorepo Benefits:
  68. - More new services to learn - Velocity templates Downsides:

  69. Challenges

  70. Onboarding new developers

  71. Current team - 4 developers (a" fu" stack, one new)

    - 1 Marketing - 1 Customer support - 1 Product manager - Some fr!lance support (marketing + design)
  72. One environment per developer (serverless :))

  73. None
  74. Good to start with

  75. Testing

  76. @slobodan_

  77. @slobodan_

  78. @slobodan_

  79. @slobodan_

  80. @slobodan_

  81. @slobodan_

  82. @slobodan_

  83. @slobodan_

  84. @slobodan_

  85. Migrations

  86. @slobodan_

  87. @slobodan_

  88. @slobodan_

  89. @slobodan_ But, how does this look like?

  90. @slobodan_ describe('DynamoDB repository', () => { describe('unit', () => {

    ... }) describe('integration', () => { beforeAll(() => { // Create test DB }) afterAll(() => { // Destroy test DB }) // Tests }) })
  91. @slobodan_ beforeAll(async () => { const params = { ...

    } await dynamoDb.createTable(params).promise() await dynamoDb.waitFor('tableExists', { TableName: tableName }).promise() })
  92. @slobodan_ afterAll(async () => { await dynamoDb.deleteTable({ TableName: tableName }).promise()

    await dynamoDb.waitFor('tableNotExists', { TableName: tableName }).promise() })
  93. Debugging and monitoring

  94. None
  95. None
  96. Cost

  97. Total cost from 2018: ~$7000

  98. $5000 AWS credits ~$2000 paid

  99. The most expensive bug: ~$3200

  100. None
  101. We have many other smaller challenges, but we are happy

    with serverless!
  102. @slobodan_

  103. @slobodan_ Summary

  104. @slobodan_ • Evolve your architecture with your product • Pick

    a good architecture, it helps you to keep your migration and onboarding costs low • Think about onboarding new team members • Hexagonal architecture is a nice fit for serverless apps • CQRS is also A nice fit for serverless apps and excellent fit for Vacation Tracker • Test your integrations (and app in general) • Testing is not enough, you'll need monitoring and error tracking
  105. @slobodan_

  106. @slobodan_ https://slobodan.me/subscribe