Evolving a serverless startup architecture - ArmadaJS 2022

A story about building a proﬁtable bootstrapped startup using serverless.

@slobodan_ Our journey starts with a prototype, continues with an
MVP, and follows the evolution to a real product.

@slobodan_ We faced many issues and learned a lot along
the way.

Evolving a serverless startup architecture From 1 to 101 Lambda
functions in production

@slobodan_ The problem & the idea

@slobodan_ Everything started in 2016 with a problem in our
other company.

@slobodan_ Everything started in 2017 with an idea, a prototype,
and a landing page.

@slobodan_ Everything really started in 2018 with an MVP.

@slobodan_ A simple idea •Track leave requests and a number
of remaining PTO days •Use SSO to avoid another username and password to remember •Integrate with Slack to make and approve PTO requests •Show events in a Google calendar

@slobodan_ How hard can it be?

@slobodan_ Slobodan Stojanović CTO and co-founder of the product I
am talking about co-author of Serverless Apps with Node.js book AWS Serverless Hero JS Belgrade meetup organizer

@slobodan_ Architecture

@slobodan_ A prototype

@slobodan_ An MVP architecture: a simple serverless bot*

@slobodan_ Why serverless? •Faster - it was fast to build
a prototype and an MVP •Less - we outsourced scaling, maintenance, and security •Focused - we focused on the business logic and minimized time spent on everything else •Cheaper - the cost scales with users, and it starts with $0

@slobodan_ Benefits •Quick and independent deployments •Easy to understand and
maintain •Easy to onboard new people •Cheap

@slobodan_ The cost of the infrastructure: $0,00

@slobodan_ Adding new features

@slobodan_ Downsides •Independent deployment for each Lambda function •Hard to
manage a!er we added 10 Lambda functions •Hard to scale, as a critical part wasn't serverless •An obvious bottleneck

@slobodan_ ~100 paying teams

@slobodan_ The ﬁrst architecture iteration: a complex serverless spaghetti

@slobodan_ The ﬁrst architecture iteration: a complex serverless application

@slobodan_ Infrastructure as code (IaC) & ﬁrst microservices

@slobodan_ Microservices Service Service (micro?) (micro?)

@slobodan_ We had ~150 Lambda functions

@slobodan_ First migrations from an old service to a new
one

@slobodan_ We realized that we'll need to ﬁnd the right
architecture for our problem

@slobodan_ Most significant changes •(Almost) everything was in AWS CloudFormation
(IaC) •We replaced Node.js server with serverless services •We started using TypeScript instead of JavaScript •We started a migration from MongoDB to DynamoDB

@slobodan_ Benefits •Easier deployments •App was auto-scalable •We had almost
100% uptime out-of-the-box •Still very cheap (our infrastructure cost was less than $100/ month)

@slobodan_ Downsides •The big flaw in our system design: we
were storing a state and not events in our database •We were still wasting a lot of time on less important things •Project complexity and the number of new services increased, and it was harder to onboard new developers •Developers don't like YAML

@slobodan_ The second architecture iteration: an event-driven system with microservices

@slobodan_ The quest for an architecture that solves our problem

@slobodan_ In the end, we decided to use Event Sourcing
& Command Query Responsibility Segregation (CQRS)

@slobodan_ Why Event Sourcing and CQRS?

@slobodan_ Storing state vs storing events

@slobodan_ A common flow in our app •Ana created a
new location and moved John and Mike to that location •Ana assigned Mike as an approver •Ana made a leave policy (20 PTO days per year) •John requested leave, and Ana approved it •Brought forward event happened and five unused days are transferred to the next year balance •Ana changed John's working week •Mike added some past leaves for John •Alex moved John to another location with a diﬀerent policy

@slobodan_ How do we calculate John's remaining PTO days? Events
and CQRS to the rescue. As a bonus, we got everything we need for the audit logs.

1. The client sends an API POST request or GraphQL
mutation 2. The event is stored in the Events table (append-only, no edits) 3. The DynamoDB streams the event to the Lambda function that sends it to the EventBridge event bus 4. EventBridge triggers the specific business logic Lambda function 5. The business logic Lambda stores the "cached" data to one of the read-only DynamoDB tables 6. And then triggers the mutation that sends a "fake" mutation 7. A "fake" mutation triggers the GraphQl subscription to notify the clients

1. The client sends an API POST request or GraphQL
mutation 2. The event is stored in the Events table (append-only, no edits) 3. The DynamoDB streams the event to the Lambda function that sends it to the EventBridge event bus 4. EventBridge triggers the specific business logic Lambda function 5. The business logic Lambda stores the "cached" data to one of the read-only DynamoDB tables 6. And then triggers the mutation that sends a "fake" mutation 7. A "fake" mutation triggers the GraphQl subscription to notify the clients 8. App uses an event bus to "route" the response to the user's platform

Client then query the read-only tables directly using GraphQl

Client then query the read-only tables directly using GraphQl Or
using the RESTful API in case of bots

We decreased the number of Lambda functions to 101! But
now we have 197 functions in production.

@slobodan_ Benefits •Fully Managed GraphQL •Less code •Better control •All
benefits from the previous architecture (without the spaghetti part) •Monorepo and shared types

@slobodan_ Downsides •YAML is still very important (but we also
added CDK for some smaller services) •Many new AWS services to learn •Velocity templates

Lambda functions → business logic Other services → transformation and
orchestration

@slobodan_ The cost of the infrastructure: <1% of MRR ~$750

@slobodan_ Architecture should evolve with your product. Don't waste your
time making it perfect for the MVP.

@slobodan_ Development & Testing

@slobodan_ Common questions during onboarding •How do I run this
locally? •How do we debug errors? •What is a DynamoDB single-table design, and why do we store all events in the same table? •What did we just do?

@slobodan_ It's impossible to run the whole serverless application locally
But if you are learning to be a pilot, you don't start with driving a plane in your backyard !

@slobodan_ You can simulate parts of your application locally Think
of it as an early version of a "Flight Simulator" It's helpful, but probably far from the complete set of tools to learn how to fly the plane

@slobodan_ What can we do? •A Lambda function is just
a function - you can run it locally as any other function* •You can use SAM Local or similar tools to run (a simulation of) a Lambda function locally in the Docker container •You can test in the cloud •You can waste your time trying to simulate everything locally

@slobodan_ Or you can write tests

@slobodan_ How do we test a common client-server app?

@slobodan_ How do we test a serverless app?

@slobodan_ Hexagonal architecture

@slobodan_ Anatomy of a Lambda function

@slobodan_ TypeScript example (business logic) interface IParams<T> { event: T
parser: (T) => IParsedData repositories: { users: IUserRepository notifications: INotificationRepository } } export async function businessLogic<T>(params: IParams<T>): Promise<void> { const {event, parser, repositories} = params try { const parsedData = parser(event) await repositories.users.invite(parsedData.email) await repositories.notifications.invitationSent() } catch (err) { await repositories.notifications.failure(err) } }

@slobodan_ TypeScript example (handler) export async function handler(event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult>
{ try { const userRepository = new UsersDynamoDBRepository(process.env.USERS_TABLE) const eventBridgeRepository = new EventBridgeRepository(process.env.EVENT_BUS) await businessLogic<APIGatewayProxyEvent>({ event, parser: parseApiGatewayEvent, repositories: { users: userRepository, notifications: eventBridgeRepository } }) await apiGatewaySuccessResponse() } catch(err) { await apiGatewayErrorResponse() } }

@slobodan_

@slobodan_ Also useful for migrations

@slobodan_

@slobodan_ Integration tests

@slobodan_ Testing in the cloud

@slobodan_ describe('DynamoDB repository', () => { describe('unit', () => {
... }) describe('integration', () => { beforeAll(() => { // Create test DB }) afterAll(() => { // Destroy test DB }) // Tests }) })

@slobodan_ beforeAll(async () => { const params = { ...
} await dynamoDb.createTable(params).promise() await dynamoDb.waitFor('tableExists', { TableName: tableName }).promise() })

@slobodan_ afterAll(async () => { await dynamoDb.deleteTable({ TableName: tableName }).promise()
await dynamoDb.waitFor('tableNotExists', { TableName: tableName }).promise() })

@slobodan_ Or we deploy the full app and run integration
tests

@slobodan_ Deploy where? Serverless environments are often cheap!

@slobodan_ Our environments 12 similar environments total cost: $750/month

@slobodan_ It's hard to run or simulate a serverless app
locally. Make small trade-o"s to make your app testable, and you'll be able to move fast.

@slobodan_ Security

@slobodan_ Accounts are cheap Create one AWS sub-account per environment
Create an AWS sub-account for each developer But remember to set budget notifications!

@slobodan_ Tools for management and easy repeatability CloudFormation AWS Organization
Formation

@slobodan_ Use AWS SSO

@slobodan_ Use AWS IAM Identity Center (Successor to AWS SSO)

@slobodan_ Learn AWS Identity & Access Management (IAM) Apply the
minimal permissions model for all AWS services i.e., a Lambda function can do one specific operation only

@slobodan_ Use WAF to protect your application

@slobodan_ But what about distributed denial-of-service (DDOS) attacks?

@slobodan_ But what about distributed denial-of-wallet (DDOW) attacks? "All AWS
customers benefit from the automatic protections of AWS Shield Standard, at no additional charge."

@slobodan_ You still need to be sure that your code
is secure

@slobodan_ Learn about security best practices for your service provider
and use them from day one.

@slobodan_ Philosophy

@slobodan_ Focus on the important things only, and outsource everything
else

@slobodan_ But how do we know what's important?

@slobodan_ See "Why the Fuss about Serverless?" (https://youtu.be/SPsaqiegOP4)

@slobodan_ Does this apply only to the infrastructure?

@slobodan_ Do not waste your time or energy building from
scratch components in the product or commodity phases

@slobodan_ Focus on your business logic and outsource everything else.

@slobodan_ Joe Armstrong “Make it work, then make it beautiful,
then if you really, really have to, make it fast. 90% of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful!” creator of Erlang programming language

Thank you! twitter: @slobodan_

Evolving a serverless startup architecture - Ar...

Evolving a serverless startup architecture - ArmadaJS 2022

More Decks by Slobodan Stojanović

Other Decks in Programming

Featured

Transcript