Slide 1
Slide 1 text
Open source application and Ecosystem
on Serverless Framework
Kenʼichiro Oyama
Fusic Co.,Ltd.
2017.11.3
1
Serverlessconf Tokyo 2017
Slide 2
Slide 2 text
Who
2
Serverlessconf Tokyo 2017
Slide 3
Slide 3 text
k1LoW
Kenʼichiro Oyama
@k1LoW
Fusic Co.,Ltd.
Engineer / TechLead / KIBAN-Unit
Creator of awspec
https://github.com/k1LoW/awspec
Favorite Serverless Framework Plugins
serverless-prune-plugin / serverless-s3-remover
3
Serverlessconf Tokyo 2017
Slide 4
Slide 4 text
Agenda
Open source application using Serverless Framework
Serverless Ninja Tips
Ecosystem on Serverless Framework
4
Serverlessconf Tokyo 2017
Slide 5
Slide 5 text
Open source application
using Serverless Framework
5
Serverlessconf Tokyo 2017
Slide 6
Slide 6 text
Serverless Architecture and Serverless OSS
Currently, in Japan, there are many case studies of
serverless architecture.
I believe, “serverless” becomes not only “the design
concept” but also “one of OSS implementation
methods”.
Because, Serverless Framework is a very powerful
tool.
6
Serverlessconf Tokyo 2017
Slide 7
Slide 7 text
My “Serverless OSS”
using Serverless Framework
7
Serverlessconf Tokyo 2017
Slide 8
Slide 8 text
faultline
8
Serverlessconf Tokyo 2017
Slide 9
Slide 9 text
faultline
Error tracking tool on AWS managed services.
Similar SaaS: Airbrake, Bugsnag, Rollbar, etc
https://github.com/faultline
9
Serverlessconf Tokyo 2017
Slide 10
Slide 10 text
faultline architecture
10
Serverlessconf Tokyo 2017
Slide 11
Slide 11 text
In Fusic
11
Serverlessconf Tokyo 2017
13 Projects
3 Languages
68k Errors
1.26GB Raw logs
Slide 12
Slide 12 text
9< Months
12
Serverlessconf Tokyo 2017
Slide 13
Slide 13 text
13
Serverlessconf Tokyo 2017
1 Down
0 Recovery
※DynamoDB Capacity
※faultline have NO recovery server
Slide 14
Slide 14 text
utsusemi
14
Serverlessconf Tokyo 2017
Slide 15
Slide 15 text
utsusemi
A tool to generate a “API-ready” “updatable”
static website by crawling the original site.
https://github.com/k1LoW/utsusemi
15
Serverlessconf Tokyo 2017
Slide 16
Slide 16 text
utsusemi architecture
16
Serverlessconf Tokyo 2017
Slide 17
Slide 17 text
After all, What is Serverless Framework?
CFn++
Programable CFn
Service as Code
for me
17
Serverlessconf Tokyo 2017
Slide 18
Slide 18 text
Application Deployment
with Serverless Framework
18
Serverlessconf Tokyo 2017
Cloud Service
Resource
Middleware
Application code
App on IaaS Serverless App
Slide 19
Slide 19 text
Application Deployment
with Serverless Framework
19
Serverlessconf Tokyo 2017
Cloud Service
Resource
Middleware
Application code
App on IaaS Serverless App
Slide 20
Slide 20 text
Application Deployment
with Serverless Framework
20
Serverlessconf Tokyo 2017
Cloud Service
Resource
Middleware
Application code
Cloud Service
Resource
Application code
App on IaaS Serverless App
Slide 21
Slide 21 text
Application Deployment
with Serverless Framework
21
Serverlessconf Tokyo 2017
Cloud Service
Resource
Middleware
Application code
Cloud Service
Resource
Application code
App on IaaS Serverless App
Slide 22
Slide 22 text
Abstract as middleware or service
using Serverless Framework
Applications built on manged services can abstract
as one middleware or one service.
ex. diamond is already one time series DB service.
ex. faultline is already one error tracking service.
Serverless Framework provides `sls deploy`
like `make install`
one command bulid ≒ one command deployment
22
Serverlessconf Tokyo 2017
Slide 23
Slide 23 text
faultline architecture
23
Serverlessconf Tokyo 2017
Slide 24
Slide 24 text
faultline as service
24
Serverlessconf Tokyo 2017
Slide 25
Slide 25 text
We can provide
open source middleware and service
with Serverless Framework !!
25
Serverlessconf Tokyo 2017
Slide 26
Slide 26 text
Serverless⚡Ninja Tips
26
Serverlessconf Tokyo 2017
Slide 27
Slide 27 text
Serverless Ninja Tips
Serverless Architecture is not general architecture.
ex. Managed services (FaaS / Object Storage / NoSQL)
Need new tips
I have ⚡⚡⚡⚡⚡tips.
POST with config
Reversed Timestamp ID
Instant Job Queue
S3 Object Tagging
Env Sync <- New!
27
Serverlessconf Tokyo 2017
Slide 28
Slide 28 text
POST with config
Include settings in API request parameters
wherever possible
ex. Slack incoming webhook URL / channel
ex. GitHub Token
Do NOT manage states as much as you can
If it is SECRET, use KMS encryption
28
Serverlessconf Tokyo 2017
Slide 29
Slide 29 text
Reversed Timestamp ID
Implement "SELECT * FROM posts ORDER BY
create_at DESC" on S3
(Math.pow(2, 53) - 1) – unixtime
Number.MAX_SAFE_INTEGER – unixtime
/projects/{project}/errors/{message}/
occurrences/{reversedUnixtime}.json
29
Serverlessconf Tokyo 2017
Slide 30
Slide 30 text
Instant Job Queue
When invoking functions to fan-out recursively
ex. Crawling web
30
Serverlessconf Tokyo 2017
Slide 31
Slide 31 text
S3 Object Tagging
Use AWS “S3 Object tagging” for KVS
ex.
Metadata of file ( s3://bucket/path/to/index.html )
ex. Content-Type / Etag
Details of user profile ( s3://bucket/group-id/user-id.jpg )
ex. Name / address / telno
Labels of issue data ( s3://bucket/project-id/issue.json )
31
Serverlessconf Tokyo 2017
Slide 32
Slide 32 text
Env Sync
Synchronize the environment variable of the
deployment environment and the environment
variable of the function execution environment.
The Twelve-Factor App
32
Serverlessconf Tokyo 2017
provider:
name: aws
environment: # env of AWS Lambda function
STAGE: ${env:STAGE, ’develop’} # use env of the deployment environment
Slide 33
Slide 33 text
Ecosystem on Serverless Framework
33
Serverlessconf Tokyo 2017
Slide 34
Slide 34 text
Serverless user
Applications built on manged services can abstract
as one middleware or one service.
Serverless Framework provides `sls deploy` like
`make install`
I think that more people will USE open source
“serverless middleware” and “serverless service”.
Like Redmine for Ruby on Rails
But, user does not know Serverless Framework
34
Serverlessconf Tokyo 2017
Slide 35
Slide 35 text
Need ecosystem
like a 'Deploy to Heroku' Button
35
Serverlessconf Tokyo 2017
Slide 36
Slide 36 text
like a 'Deploy to Heroku' Button
“Deploy with Serverless” project ⚡⚡⚡
by Rafal Wilinski
https://github.com/RafalWilinski/deploy-with-serverless
“.slsdeploy.yml” project [PoC]
by Me
https://github.com/k1LoW/slsdeploy
36
Serverlessconf Tokyo 2017
Slide 37
Slide 37 text
.slsdeploy.yml
Put .slsdeploy.yml to your sls app GitHub repo
Define environment variables required for .slsdeploy.yml
Make your sls app correspond to “Env Sync”
37
---
environment:
UTSUSEMI_TARGET_HOST:
description: Crawl target host
type: text
UTSUSEMI_WORKER_PROCESS:
description: Crawl worker process
type: number
default: 2
UTSUSEMI_BUCKET_NAME:
Slide 38
Slide 38 text
DEMO
( If I have time )
38
Serverlessconf Tokyo 2017
Slide 39
Slide 39 text
User simple deploy ecosystem!!
39
Serverlessconf Tokyo 2017
Slide 40
Slide 40 text
Conclusion
Applications built on manged services can abstract as one
middleware or one service.
Serverless Framework provides `sls deploy` like `make
install`
If a mechanism that makes it easier to deploy is provided, the
user can obtain a “serverless middleware” and “serverless
service” more easily.
I would like to see more open source “serverless middleware”
and “serverless service”
40
Serverlessconf Tokyo 2017
Slide 41
Slide 41 text
41
Weʼre Hiring Engineers!
> https://fusic.github.io
Thank you!
Serverlessconf Tokyo 2017