Open source application and Ecosystem on Serverless Framework / Serverlessconf Tokyo 2017

Transcript

1. Open source application and Ecosystem on Serverless Framework Kenʼichiro Oyama Fusic Co.,Ltd. 2017.11.3 1 Serverlessconf Tokyo 2017
2. Who 2 Serverlessconf Tokyo 2017
3. k1LoW   Kenʼichiro Oyama   @k1LoW   Fusic Co.,Ltd.   Engineer / TechLead / KIBAN-Unit   Creator of awspec   https://github.com/k1LoW/awspec   Favorite Serverless Framework Plugins   serverless-prune-plugin / serverless-s3-remover 3 Serverlessconf Tokyo 2017
4. Agenda   Open source application using Serverless Framework   Serverless Ninja Tips   Ecosystem on Serverless Framework 4 Serverlessconf Tokyo 2017
5. Open source application using Serverless Framework 5 Serverlessconf Tokyo 2017
6. Serverless Architecture and Serverless OSS   Currently, in Japan, there are many case studies of serverless architecture.   I believe, “serverless” becomes not only “the design concept” but also “one of OSS implementation methods”.   Because, Serverless Framework is a very powerful tool. 6 Serverlessconf Tokyo 2017
7. My “Serverless OSS” using Serverless Framework 7 Serverlessconf Tokyo 2017
8. faultline 8 Serverlessconf Tokyo 2017
9. faultline   Error tracking tool on AWS managed services.   Similar SaaS: Airbrake, Bugsnag, Rollbar, etc   https://github.com/faultline 9 Serverlessconf Tokyo 2017
10. faultline architecture 10 Serverlessconf Tokyo 2017
11. In Fusic 11 Serverlessconf Tokyo 2017 13 Projects 3 Languages 68k Errors 1.26GB Raw logs
12. 9< Months 12 Serverlessconf Tokyo 2017
13. 13 Serverlessconf Tokyo 2017 1 Down 0 Recovery ※DynamoDB Capacity ※faultline have NO recovery server
14. utsusemi 14 Serverlessconf Tokyo 2017
15. utsusemi   A tool to generate a “API-ready” “updatable” static website by crawling the original site.   https://github.com/k1LoW/utsusemi 15 Serverlessconf Tokyo 2017
16. utsusemi architecture 16 Serverlessconf Tokyo 2017
17. After all, What is Serverless Framework?   CFn++   Programable CFn   Service as Code for me 17 Serverlessconf Tokyo 2017
18. Application Deployment with Serverless Framework 18 Serverlessconf Tokyo 2017 Cloud Service Resource Middleware Application code App on IaaS Serverless App
19. Application Deployment with Serverless Framework 19 Serverlessconf Tokyo 2017 Cloud Service Resource Middleware Application code App on IaaS Serverless App
20. Application Deployment with Serverless Framework 20 Serverlessconf Tokyo 2017 Cloud Service Resource Middleware Application code Cloud Service Resource Application code App on IaaS Serverless App
21. Application Deployment with Serverless Framework 21 Serverlessconf Tokyo 2017 Cloud Service Resource Middleware Application code Cloud Service Resource Application code App on IaaS Serverless App
22. Abstract as middleware or service using Serverless Framework   Applications built on manged services can abstract as one middleware or one service.   ex. diamond is already one time series DB service.   ex. faultline is already one error tracking service.   Serverless Framework provides `sls deploy` like `make install`   one command bulid ≒ one command deployment 22 Serverlessconf Tokyo 2017
23. faultline architecture 23 Serverlessconf Tokyo 2017
24. faultline as service 24 Serverlessconf Tokyo 2017
25. We can provide open source middleware and service with Serverless Framework !! 25 Serverlessconf Tokyo 2017
26. Serverless⚡Ninja Tips 26 Serverlessconf Tokyo 2017
27. Serverless Ninja Tips   Serverless Architecture is not general architecture.   ex. Managed services (FaaS / Object Storage / NoSQL)   Need new tips   I have ⚡⚡⚡⚡⚡tips.   POST with config   Reversed Timestamp ID   Instant Job Queue   S3 Object Tagging   Env Sync <- New! 27 Serverlessconf Tokyo 2017
28. POST with config   Include settings in API request parameters wherever possible   ex. Slack incoming webhook URL / channel   ex. GitHub Token   Do NOT manage states as much as you can   If it is SECRET, use KMS encryption 28 Serverlessconf Tokyo 2017
29. Reversed Timestamp ID   Implement "SELECT * FROM posts ORDER BY create_at DESC" on S3   (Math.pow(2, 53) - 1) – unixtime   Number.MAX_SAFE_INTEGER – unixtime   /projects/{project}/errors/{message}/ occurrences/{reversedUnixtime}.json 29 Serverlessconf Tokyo 2017
30. Instant Job Queue   When invoking functions to fan-out recursively   ex. Crawling web 30 Serverlessconf Tokyo 2017
31. S3 Object Tagging   Use AWS “S3 Object tagging” for KVS   ex.   Metadata of file ( s3://bucket/path/to/index.html )   ex. Content-Type / Etag   Details of user profile ( s3://bucket/group-id/user-id.jpg )   ex. Name / address / telno   Labels of issue data ( s3://bucket/project-id/issue.json ) 31 Serverlessconf Tokyo 2017
32. Env Sync   Synchronize the environment variable of the deployment environment and the environment variable of the function execution environment.   The Twelve-Factor App 32 Serverlessconf Tokyo 2017  provider:   name: aws   environment: # env of AWS Lambda function   STAGE: ${env:STAGE, ’develop’} # use env of the deployment environment
33. Ecosystem on Serverless Framework 33 Serverlessconf Tokyo 2017
34. Serverless user   Applications built on manged services can abstract as one middleware or one service.   Serverless Framework provides `sls deploy` like `make install`   I think that more people will USE open source “serverless middleware” and “serverless service”.   Like Redmine for Ruby on Rails   But, user does not know Serverless Framework 34 Serverlessconf Tokyo 2017
35. Need ecosystem like a 'Deploy to Heroku' Button 35 Serverlessconf Tokyo 2017
36. like a 'Deploy to Heroku' Button   “Deploy with Serverless” project ⚡⚡⚡   by Rafal Wilinski   https://github.com/RafalWilinski/deploy-with-serverless   “.slsdeploy.yml” project [PoC]   by Me   https://github.com/k1LoW/slsdeploy 36 Serverlessconf Tokyo 2017
37. .slsdeploy.yml   Put .slsdeploy.yml to your sls app GitHub repo   Define environment variables required for .slsdeploy.yml   Make your sls app correspond to “Env Sync” 37  ---  environment:   UTSUSEMI_TARGET_HOST:   description: Crawl target host   type: text   UTSUSEMI_WORKER_PROCESS:   description: Crawl worker process   type: number   default: 2   UTSUSEMI_BUCKET_NAME:
38. DEMO ( If I have time ) 38 Serverlessconf Tokyo 2017
39. User simple deploy ecosystem!! 39 Serverlessconf Tokyo 2017
40. Conclusion   Applications built on manged services can abstract as one middleware or one service.   Serverless Framework provides `sls deploy` like `make install`   If a mechanism that makes it easier to deploy is provided, the user can obtain a “serverless middleware” and “serverless service” more easily.   I would like to see more open source “serverless middleware” and “serverless service” 40 Serverlessconf Tokyo 2017
41. 41 Weʼre Hiring Engineers! > https://fusic.github.io Thank you! Serverlessconf Tokyo 2017