Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From 1 to 201 Lambda functions in production: Evolving a serverless startup architecture

From 1 to 201 Lambda functions in production: Evolving a serverless startup architecture

My slides from AWS Community Day Turkey 2023 (https://aws.cloudturkey.io/)

----

Building a serverless function or an API is easy. However, things get a bit more complicated as your application grows. What works for a few functions often doesn't work for hundreds of functions and services. As your application grows, you'll need to evolve your architecture, deployment, monitoring, and tooling.

This talk is a case study of the serverless startup's architecture evolution. We started with a single Lambda function in early 2018 and evolved our application through multiple stages and architectures. Currently, the application uses CQRS with GraphQL and 200 Lambda functions serving millions of requests. We faced and solved many issues during the last four years, learned many things, and managed to keep our infrastructure costs low.

Slobodan Stojanović

May 06, 2023
Tweet

More Decks by Slobodan Stojanović

Other Decks in Programming

Transcript

  1. @slobodan_ Our journey starts with a prototype, continues with an

    MVP, and follows the evolution to a real product.
  2. @slobodan_ A simple idea •Track leave requests and a number

    of remaining PTO days •Use SSO to avoid another username and password to remember •Integrate with Slack to make and approve PTO requests •Show events in a Google calendar
  3. @slobodan_ Slobodan Stojanović CTO and co-founder of the product I

    am talking about co-author of Serverless Apps with Node.js book AWS Serverless Hero JS Belgrade meetup organizer
  4. @slobodan_ Why serverless? •Faster - it was fast to build

    a prototype and an MVP •Less - we outsourced scaling, maintenance, and security •Focused - we focused on the business logic and minimized time spent on everything else •Cheaper - the cost scales with users, and it starts with $0
  5. @slobodan_ Downsides •Independent deployment for each Lambda function •Hard to

    manage a!er we added 10 Lambda functions •Hard to scale, as a critical part wasn't serverless •An obvious bottleneck
  6. @slobodan_ Most significant changes •(Almost) everything was in AWS CloudFormation

    (IaC) •We replaced Node.js server with serverless services •We started using TypeScript instead of JavaScript •We started a migration from MongoDB to DynamoDB
  7. @slobodan_ Benefits •Easier deployments •App was auto-scalable •We had almost

    100% uptime out-of-the-box •Still very cheap (our infrastructure cost was less than $100/ month)
  8. @slobodan_ Downsides •The big flaw in our system design: we

    were storing a state and not events in our database •We were still wasting a lot of time on less important things •Project complexity and the number of new services increased, and it was harder to onboard new developers •Developers don't like YAML
  9. @slobodan_ In the end, we decided to use Event Sourcing

    & Command Query Responsibility Segregation (CQRS)
  10. @slobodan_ A common flow in our app •Ana created a

    new location and moved John and Mike to that location •Ana assigned Mike as an approver •Ana made a leave policy (20 PTO days per year) •John requested leave, and Ana approved it •Brought forward event happened and five unused days are transferred to the next year balance •Ana changed John's working week •Mike added some past leaves for John •Alex moved John to another location with a different policy
  11. @slobodan_ How do we calculate John's remaining PTO days? Events

    and CQRS to the rescue. As a bonus, we got everything we need for the audit logs.
  12. 1. The client sends an API POST request or GraphQL

    mutation 2. The event is stored in the Events table (append-only, no edits) 3. The DynamoDB streams the event to the Lambda function that sends it to the EventBridge event bus 4. EventBridge triggers the specific business logic Lambda function 5. The business logic Lambda stores the "cached" data to one of the read-only DynamoDB tables 6. And then triggers the mutation that sends a "fake" mutation 7. A "fake" mutation triggers the GraphQl subscription to notify the clients 8. App uses an event bus to "route" the response to the user's platform
  13. We decreased the number of Lambda functions to 101! But

    now we have 230 functions in production !
  14. @slobodan_ Benefits •Fully Managed GraphQL •Less code •Better control •All

    benefits from the previous architecture (without the spaghetti part) •Monorepo and shared types
  15. @slobodan_ Downsides •YAML is still very important (but we also

    added CDK for some smaller services) •Many new AWS services to learn •Velocity templates
  16. @slobodan_ Common questions during onboarding •How do I run this

    locally? •How do we debug errors? •What is a DynamoDB single-table design, and why do we store all events in the same table? •What did we just do?
  17. @slobodan_ It's impossible to run the whole serverless application locally

    But if you are learning to be a pilot, you don't start with driving a plane in your backyard !
  18. @slobodan_ You can simulate parts of your application locally Think

    of it as an early version of a "Flight Simulator" It's helpful, but probably far from the complete set of tools to learn how to fly the plane
  19. @slobodan_ What can we do? •A Lambda function is just

    a function - you can run it locally as any other function* •You can use SAM Local or similar tools to run (a simulation of) a Lambda function locally in the Docker container •You can test in the cloud •You can waste your time trying to simulate everything locally
  20. @slobodan_ TypeScript example (business logic) interface IParams<T> { event: T

    parser: (T) => IParsedData repositories: { users: IUserRepository notifications: INotificationRepository } } export async function businessLogic<T>(params: IParams<T>): Promise<void> { const {event, parser, repositories} = params try { const parsedData = parser(event) await repositories.users.invite(parsedData.email) await repositories.notifications.invitationSent() } catch (err) { await repositories.notifications.failure(err) } }
  21. @slobodan_ TypeScript example (business logic) interface IParams<T> { event: T

    parser: (T) => IParsedData repositories: { users: IUserRepository notifications: INotificationRepository } } export async function businessLogic<T>(params: IParams<T>): Promise<void> { const {event, parser, repositories} = params try { const parsedData = parser(event) await repositories.users.invite(parsedData.email) await repositories.notifications.invitationSent() } catch (err) { await repositories.notifications.failure(err) } }
  22. @slobodan_ TypeScript example (business logic) interface IParams<T> { event: T

    parser: (T) => IParsedData repositories: { users: IUserRepository notifications: INotificationRepository } } export async function businessLogic<T>(params: IParams<T>): Promise<void> { const {event, parser, repositories} = params try { const parsedData = parser(event) await repositories.users.invite(parsedData.email) await repositories.notifications.invitationSent() } catch (err) { await repositories.notifications.failure(err) } }
  23. @slobodan_ TypeScript example (business logic) interface IParams<T> { event: T

    parser: (T) => IParsedData repositories: { users: IUserRepository notifications: INotificationRepository } } export async function businessLogic<T>(params: IParams<T>): Promise<void> { const {event, parser, repositories} = params try { const parsedData = parser(event) await repositories.users.invite(parsedData.email) await repositories.notifications.invitationSent() } catch (err) { await repositories.notifications.failure(err) } }
  24. @slobodan_ TypeScript example (business logic) interface IParams<T> { event: T

    parser: (T) => IParsedData repositories: { users: IUserRepository notifications: INotificationRepository } } export async function businessLogic<T>(params: IParams<T>): Promise<void> { const {event, parser, repositories} = params try { const parsedData = parser(event) await repositories.users.invite(parsedData.email) await repositories.notifications.invitationSent() } catch (err) { await repositories.notifications.failure(err) } }
  25. @slobodan_ TypeScript example (business logic) interface IParams<T> { event: T

    parser: (T) => IParsedData repositories: { users: IUserRepository notifications: INotificationRepository } } export async function businessLogic<T>(params: IParams<T>): Promise<void> { const {event, parser, repositories} = params try { const parsedData = parser(event) await repositories.users.invite(parsedData.email) await repositories.notifications.invitationSent() } catch (err) { await repositories.notifications.failure(err) } }
  26. @slobodan_ TypeScript example (business logic) interface IParams<T> { event: T

    parser: (T) => IParsedData repositories: { users: IUserRepository notifications: INotificationRepository } } export async function businessLogic<T>(params: IParams<T>): Promise<void> { const {event, parser, repositories} = params try { const parsedData = parser(event) await repositories.users.invite(parsedData.email) await repositories.notifications.invitationSent() } catch (err) { await repositories.notifications.failure(err) } }
  27. @slobodan_ TypeScript example (business logic) interface IParams<T> { event: T

    parser: (T) => IParsedData repositories: { users: IUserRepository notifications: INotificationRepository } } export async function businessLogic<T>(params: IParams<T>): Promise<void> { const {event, parser, repositories} = params try { const parsedData = parser(event) await repositories.users.invite(parsedData.email) await repositories.notifications.invitationSent() } catch (err) { await repositories.notifications.failure(err) } }
  28. @slobodan_ TypeScript example (handler) export async function handler(event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult>

    { try { const userRepository = new UsersDynamoDBRepository(process.env.USERS_TABLE) const eventBridgeRepository = new EventBridgeRepository(process.env.EVENT_BUS) await businessLogic<APIGatewayProxyEvent>({ event, parser: parseApiGatewayEvent, repositories: { users: userRepository, notifications: eventBridgeRepository } }) await apiGatewaySuccessResponse() } catch(err) { await apiGatewayErrorResponse() } }
  29. @slobodan_ TypeScript example (handler) export async function handler(event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult>

    { try { const userRepository = new UsersDynamoDBRepository(process.env.USERS_TABLE) const eventBridgeRepository = new EventBridgeRepository(process.env.EVENT_BUS) await businessLogic<APIGatewayProxyEvent>({ event, parser: parseApiGatewayEvent, repositories: { users: userRepository, notifications: eventBridgeRepository } }) await apiGatewaySuccessResponse() } catch(err) { await apiGatewayErrorResponse() } }
  30. @slobodan_ TypeScript example (handler) export async function handler(event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult>

    { try { const userRepository = new UsersDynamoDBRepository(process.env.USERS_TABLE) const eventBridgeRepository = new EventBridgeRepository(process.env.EVENT_BUS) await businessLogic<APIGatewayProxyEvent>({ event, parser: parseApiGatewayEvent, repositories: { users: userRepository, notifications: eventBridgeRepository } }) await apiGatewaySuccessResponse() } catch(err) { await apiGatewayErrorResponse() } }
  31. @slobodan_ describe('DynamoDB repository', () => { describe('unit', () => {

    ... }) describe('integration', () => { beforeAll(() => { // Create test DB }) afterAll(() => { // Destroy test DB }) // Tests }) })
  32. @slobodan_ beforeAll(async () => { const params = { ...

    } await dynamoDb.createTable(params).promise() await dynamoDb.waitFor('tableExists', { TableName: tableName }).promise() })
  33. @slobodan_ afterAll(async () => { await dynamoDb.deleteTable({ TableName: tableName }).promise()

    await dynamoDb.waitFor('tableNotExists', { TableName: tableName }).promise() })
  34. @slobodan_ It's hard to run or simulate a serverless app

    locally. Make small trade-o"s to make your app testable, and you'll be able to move fast.
  35. @slobodan_ Accounts are cheap Create one AWS sub-account per environment

    Create an AWS sub-account for each developer But remember to set budget notifications!
  36. @slobodan_ Learn AWS Identity & Access Management (IAM) Apply the

    minimal permissions model for all AWS services i.e., a Lambda function can do one specific operation only
  37. @slobodan_ But what about distributed denial-of-wallet (DDOW) attacks? "All AWS

    customers benefit from the automatic protections of AWS Shield Standard, at no additional charge."
  38. @slobodan_ Do not waste your time or energy building from

    scratch components in the product or commodity phases
  39. @slobodan_ Joe Armstrong “Make it work, then make it beautiful,

    then if you really, really have to, make it fast. 90% of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful!” creator of Erlang programming language