Full AWS Serverless

https://www.meetup.com/preview/San-Francisco-Cloud-Mafia/events/242901762

How an entirely serverless architecture can be achieved using a collection of AWS services, show a case study of an application using S3, Cognito, Lambda, DynamoDB, SNS, SQS, SES, IAM and other AWS services working together. Share decisions were made to build a scalable platform with the ability to add features within days.

Transcript

1. .co.il www. Serverless Architecture
2. .co.il www.
3. .co.il www.
4. .co.il www.
5. .co.il www. www.devops.co.il Evgeny Zislis on.fb.me/Ops-IL
6. .co.il www. dvps.me/martinfowler-bliki-serverless
7. .co.il www. cpu memory I/O time keeping customer box Why am I doing repetitive work here? :( I will be rich soon!
8. .co.il www. I'm watching you!
9. .co.il www.
10. .co.il www. Case Study
11. .co.il www.
12. .co.il www.
13. .co.il www. JavaScript / ECMAScript 6 + AWS SDK ◦ Usable in browser, backend, lambda ◦ Plenty of programmers and freelancers ◦ Avoid variability
14. .co.il www. API Gateway Cognito S3 SNS SQS IAM ElastiCache DynamoDB Backend Services
15. .co.il www. S3 Web Applications ◦ Vue.js ◦ S3 as web server ◦ CloudFront CDN
16. .co.il www. Deploy Web Assets ◦ Atlassian BitBucket Pipelines (free, fast) ◦ minimal babel compile ◦ execute transpiled tests ◦ Upload to S3
17. .co.il www. # package.json { ... "scripts": { "build": "babel src -d dist/src && babel test -d dist/test", "build:test": "tap -Rspec --cov \"dist/test/**/*.test.js\"", }, } CI / CD
18. .co.il www. CI / CD # bitbucket-pipelines.yml --- image: kopterio/node-yarn:7.7 pipelines: default: - step: script: - yarn - npm run -s build - npm run -s build:test
19. .co.il www. Authenticate with Service • Cognito • Vue-Auth-Cognito OSS
20. .co.il www.
21. .co.il www.
22. .co.il www.
23. .co.il www. $context $context.identity.cognitoIdentityId $context.requestId docs.aws.amazon.com/apigateway/latest/developerguide/ api-gateway-mapping-template-reference.html
24. .co.il www.
25. .co.il www. aws apigateway put-rest-api \ --rest-api-id 1234123412 \ --mode overwrite \ --body 'file:///path/to/API_Swagger_template.json' \ --region us-west-2 aws apigateway put-rest-api \ --rest-api-id 1234123412 \ --mode merge \ --body 'file:///path/to/API_Swagger_template.json' \ --region us-west-2
26. .co.il www. /: &path_with_cors options: summary: CORS Response responses: 200: description: Stub CORS response headers: &cors_headers Access-Control-Allow-Origin: { type: string } Access-Control-Allow-Methods: { type: string } Access-Control-Allow-Headers: { type: string } schema: $ref: "#/definitions/Empty" x-amazon-apigateway-integration: passthroughBehavior: "never" type: "mock" responses: default: statusCode: 200 responseParameters: &cors_response_params method.response.header.Access-Control-Allow-Methods: "'GET,OPTIONS'" method.response.header.Access-Control-Allow-Headers: "'Content-Type, ... '" method.response.header.Access-Control-Allow-Origin: "'*'" requestTemplates: application/json: '{"statusCode": 200}' none of that nonsense ... docs.aws.amazon.com /apigateway/latest/ developerguide/how-to-cors.html
27. .co.il www. /v1/audit/aws: <<: *path_with_cors get: summary: Get the audit log. security: - kopter.io: [] parameters: - $ref: '#/parameters/authorization' - $ref: '#/parameters/identityId' - $ref: '#/parameters/requestId' responses: 200: description: An array of audit log events headers: <<: *cors_headers schema: $ref: '#/definitions/AuditLogEvents' 520: description: Error response schema: $ref: '#/definitions/Error' x-amazon-apigateway-integration: requestParameters: integration.request.header.authorization: "'Basic $nginx_auth'" integration.request.header.identityId: "context.identity.cognitoIdentityId" integration.request.header.requestId: "context.requestId" responses:
28. .co.il www. Backend API x
29. .co.il www. Kopter AWS account Process Customer AWS Events CloudTrail S3 Customer AWS account SNS SQS ECS DynamoDB Elastic Beanstalk
30. .co.il www.
31. .co.il www. app.post('/', sqsHandler); app.post('/cron', cronHandler); export async function sqsHandler(req, res) { const queueName = req.get('X-Aws-Sqsd-Queue'); try { switch (queueName) { case process.env.AGGREGATOR_QUEUE_NAME: await new Aggregator(req.body).start(); break; case process.env.CLOUD_TRAIL_INGEST_QUEUE_NAME: await new CloudTrailIngest(req.body).start(); break; default: throw new Error(`no worker found with X-Aws-Sqsd-Queue: ${queueName}`); } } catch (error) { log.error(error); res.status(500).send({}); } res.status(200).send({}); }
32. .co.il www. cron.yaml version: 1 cron: - name: "backup-job" url: "/cron/backup" schedule: "0 */12 * * *" - name: "audit" url: "/cron/audit" schedule: "0 23 * * *"
33. .co.il www.
34. .co.il www. EC2 Events Queue kopter workers Describer Cron EC2 Describer Aggregator Cron SNS *Hourly* CRON Aggregator *Hourly* CRON EC2 Pricing *Hourly* CRON EC2 Describer ? ? CloudTrail Ingest sqsd AWS Accounts sqsd Aggregator Queue sqsd Aggregator sqsd EC2 Pricing Audit Log Queue sqsd Audit Logs Frontend Notifications sqsd Frontend Notifications CloudTrail Ingest EC2 Snapshots EC2 Pricing Backend API sqsd Describer Queue sqsd EC2 Events Price Graph Data Audit Logs Frontend Notifications Frontend Notifications Global ? Legend DB Table Queue Worker sqsd kopter workers kopter workers ᵣ function
35. .co.il www. Lambda $$$$$ $$$$ $$$ $$ $
36. .co.il www. nodejs.org/en/blog/uncategorized/profiling-node-js/
37. .co.il www. export class Handler { static async start() { const queueUrl = process.env.AGGREGATOR_QUEUE_URL; const message = await sqsCheck(queueUrl); await new Aggregator(message).start(); await sqsDeleteMessage(queueUrl, message); } } export async function handler(event, context, callback) { try { await Handler.start(); callback(); } catch (error) { callback(error); } }
38. .co.il www. DynamoDB permissions
39. .co.il www. Version: '2012-10-17' Statement: - Sid: AllowAccessToOnlyItemsMatchingUserID Effect: Allow Action: - dynamodb:GetItem - dynamodb:Query Resource: - arn:aws:dynamodb:us-west-2:123456789012:table/GameScores Condition: ForAllValues:StringEquals: dynamodb:LeadingKeys: - "${www.amazon.com:user_id}" dynamodb:Attributes: - UserId - GameTitle - Wins - Losses - TopScore - TopScoreDateTime StringEqualsIfExists: dynamodb:Select: SPECIFIC_ATTRIBUTES
40. .co.il www. Version: '2012-10-17' Statement: - Sid: LimitAccessToSpecificAttributes Effect: Allow Action: - dynamodb:UpdateItem - dynamodb:GetItem - dynamodb:Query - dynamodb:BatchGetItem - dynamodb:Scan Resource: - arn:aws:dynamodb:us-west-2:123456789012:table/GameScores Condition: ForAllValues:StringEquals: dynamodb:Attributes: - UserId - TopScore StringEqualsIfExists: dynamodb:Select: SPECIFIC_ATTRIBUTES dynamodb:ReturnValues: - NONE - UPDATED_OLD - UPDATED_NEW
41. .co.il www. --- Version: '2012-10-17' Statement: - Action: - s3:ListBucket Effect: Allow Resource: - arn:aws:s3:::mybucket Condition: StringLike: s3:prefix: - "${cognito-identity.amazonaws.com:sub}/*" - Action: - s3:GetObject - s3:PutObject Effect: Allow Resource: - arn:aws:s3:::mybucket/${cognito-identity.amazonaws.com:sub}/* S3 permissions
42. .co.il www. DynamoDB scaling aws.amazon.com/blogs/aws/auto-scale-dynamodb-with-dynamic-dynamodb/
43. .co.il www. ElastiCache "FNR4GZ675EW5UXJW" : { "FNR4GZ675EW5UXJW.JRTCKXETXF" : { "offerTermCode" : "JRTCKXETXF", "sku" : "FNR4GZ675EW5UXJW", "effectiveDate" : "2015-10-01T00:00:00Z", "priceDimensions" : { "FNR4GZ675EW5UXJW.JRTCKXETXF.6YS6EN2CT7" : { "rateCode" : "FNR4GZ675EW5UXJW.JRTCKXETXF.6YS6EN2CT7", "rateType" : "Fixed", "description" : "$0.044 per RDS T2 Small Instance hour (or partial hour) running SQL Server EX - LI", "beginRange" : "0", "endRange" : "Inf", "unit" : "Hrs", "pricePerUnit" : { "USD" : "0.0440000000" AWS EC2 Pricing API ~ 100mb aws.amazon.com/blogs/aws/new-aws-price-list-api/
44. .co.il www. Thank you! Join Operations Israel community on on.fb.me/Ops-IL www.devops.co.il
45. .co.il www. Join Operations Israel community on on.fb.me/Ops-IL www.devops.co.il
46. .co.il www.