Open source application and Ecosystem on Serverless Framework / Serverlessconf Tokyo 2017

Transcript

1. Open source application and Ecosystem
   on Serverless Framework
   Kenʼichiro Oyama
   Fusic Co.,Ltd.
   2017.11.3
   1
   Serverlessconf Tokyo 2017
   

   View Slide

2. Who
   2
   Serverlessconf Tokyo 2017
   

   View Slide

3. k1LoW
     Kenʼichiro Oyama
     @k1LoW
     Fusic Co.,Ltd.
     Engineer / TechLead / KIBAN-Unit
     Creator of awspec
     https://github.com/k1LoW/awspec
     Favorite Serverless Framework Plugins
     serverless-prune-plugin / serverless-s3-remover
   3
   Serverlessconf Tokyo 2017
   

   View Slide

4. Agenda
     Open source application using Serverless Framework
     Serverless Ninja Tips
     Ecosystem on Serverless Framework
   4
   Serverlessconf Tokyo 2017
   

   View Slide

5. Open source application
   using Serverless Framework
   5
   Serverlessconf Tokyo 2017
   

   View Slide

6. Serverless Architecture and Serverless OSS
     Currently, in Japan, there are many case studies of
   serverless architecture.
     I believe, “serverless” becomes not only “the design
   concept” but also “one of OSS implementation
   methods”.
     Because, Serverless Framework is a very powerful
   tool.
   6
   Serverlessconf Tokyo 2017
   

   View Slide

7. My “Serverless OSS”
   using Serverless Framework
   7
   Serverlessconf Tokyo 2017
   

   View Slide

8. faultline
   8
   Serverlessconf Tokyo 2017
   

   View Slide

9. faultline
     Error tracking tool on AWS managed services.
     Similar SaaS: Airbrake, Bugsnag, Rollbar, etc
     https://github.com/faultline
   9
   Serverlessconf Tokyo 2017
   

   View Slide

10. faultline architecture
    10
    Serverlessconf Tokyo 2017
    

    View Slide

11. In Fusic
    11
    Serverlessconf Tokyo 2017
    13 Projects
    3 Languages
    68k Errors
    1.26GB Raw logs
    

    View Slide

12. 9< Months
    12
    Serverlessconf Tokyo 2017
    

    View Slide

13. 13
    Serverlessconf Tokyo 2017
    1 Down
    0 Recovery
    ※DynamoDB Capacity
    ※faultline have NO recovery server
    

    View Slide

14. utsusemi
    14
    Serverlessconf Tokyo 2017
    

    View Slide

15. utsusemi
      A tool to generate a “API-ready” “updatable”
    static website by crawling the original site.
      https://github.com/k1LoW/utsusemi
    15
    Serverlessconf Tokyo 2017
    

    View Slide

16. utsusemi architecture
    16
    Serverlessconf Tokyo 2017
    

    View Slide

17. After all, What is Serverless Framework?
      CFn++
      Programable CFn
      Service as Code
    for me
    17
    Serverlessconf Tokyo 2017
    

    View Slide

18. Application Deployment
    with Serverless Framework
    18
    Serverlessconf Tokyo 2017
    Cloud Service
    Resource
    Middleware
    Application code
    App on IaaS Serverless App
    

    View Slide

19. Application Deployment
    with Serverless Framework
    19
    Serverlessconf Tokyo 2017
    Cloud Service
    Resource
    Middleware
    Application code
    App on IaaS Serverless App
    

    View Slide

20. Application Deployment
    with Serverless Framework
    20
    Serverlessconf Tokyo 2017
    Cloud Service
    Resource
    Middleware
    Application code
    Cloud Service
    Resource
    Application code
    App on IaaS Serverless App
    

    View Slide

21. Application Deployment
    with Serverless Framework
    21
    Serverlessconf Tokyo 2017
    Cloud Service
    Resource
    Middleware
    Application code
    Cloud Service
    Resource
    Application code
    App on IaaS Serverless App
    

    View Slide

22. Abstract as middleware or service
    using Serverless Framework
      Applications built on manged services can abstract
    as one middleware or one service.
      ex. diamond is already one time series DB service.
      ex. faultline is already one error tracking service.
      Serverless Framework provides `sls deploy`
    like `make install`
      one command bulid ≒ one command deployment
    22
    Serverlessconf Tokyo 2017
    

    View Slide

23. faultline architecture
    23
    Serverlessconf Tokyo 2017
    

    View Slide

24. faultline as service
    24
    Serverlessconf Tokyo 2017
    

    View Slide

25. We can provide
    open source middleware and service
    with Serverless Framework !!
    25
    Serverlessconf Tokyo 2017
    

    View Slide

26. Serverless⚡Ninja Tips
    26
    Serverlessconf Tokyo 2017
    

    View Slide

27. Serverless Ninja Tips
      Serverless Architecture is not general architecture.
      ex. Managed services (FaaS / Object Storage / NoSQL)
      Need new tips
      I have ⚡⚡⚡⚡⚡tips.
      POST with config
      Reversed Timestamp ID
      Instant Job Queue
      S3 Object Tagging
      Env Sync <- New!
    27
    Serverlessconf Tokyo 2017
    

    View Slide

28. POST with config
      Include settings in API request parameters
    wherever possible
      ex. Slack incoming webhook URL / channel
      ex. GitHub Token
      Do NOT manage states as much as you can
      If it is SECRET, use KMS encryption
    28
    Serverlessconf Tokyo 2017
    

    View Slide

29. Reversed Timestamp ID
      Implement "SELECT * FROM posts ORDER BY
    create_at DESC" on S3
      (Math.pow(2, 53) - 1) – unixtime
      Number.MAX_SAFE_INTEGER – unixtime
      /projects/{project}/errors/{message}/
    occurrences/{reversedUnixtime}.json
    29
    Serverlessconf Tokyo 2017
    

    View Slide

30. Instant Job Queue
      When invoking functions to fan-out recursively
      ex. Crawling web
    30
    Serverlessconf Tokyo 2017
    

    View Slide

31. S3 Object Tagging
      Use AWS “S3 Object tagging” for KVS
      ex.
      Metadata of file ( s3://bucket/path/to/index.html )
      ex. Content-Type / Etag
      Details of user profile ( s3://bucket/group-id/user-id.jpg )
      ex. Name / address / telno
      Labels of issue data ( s3://bucket/project-id/issue.json )
    31
    Serverlessconf Tokyo 2017
    

    View Slide

32. Env Sync
      Synchronize the environment variable of the
    deployment environment and the environment
    variable of the function execution environment.
      The Twelve-Factor App
    32
    Serverlessconf Tokyo 2017
     provider:
      name: aws
      environment: # env of AWS Lambda function
      STAGE: ${env:STAGE, ’develop’} # use env of the deployment environment
    

    View Slide

33. Ecosystem on Serverless Framework
    33
    Serverlessconf Tokyo 2017
    

    View Slide

34. Serverless user
      Applications built on manged services can abstract
    as one middleware or one service.
      Serverless Framework provides `sls deploy` like
    `make install`
      I think that more people will USE open source
    “serverless middleware” and “serverless service”.
      Like Redmine for Ruby on Rails
      But, user does not know Serverless Framework
    34
    Serverlessconf Tokyo 2017
    

    View Slide

35. Need ecosystem
    like a 'Deploy to Heroku' Button
    35
    Serverlessconf Tokyo 2017
    

    View Slide

36. like a 'Deploy to Heroku' Button
      “Deploy with Serverless” project ⚡⚡⚡
      by Rafal Wilinski
      https://github.com/RafalWilinski/deploy-with-serverless
      “.slsdeploy.yml” project [PoC]
      by Me
      https://github.com/k1LoW/slsdeploy
    36
    Serverlessconf Tokyo 2017
    

    View Slide

37. .slsdeploy.yml
      Put .slsdeploy.yml to your sls app GitHub repo
      Define environment variables required for .slsdeploy.yml
      Make your sls app correspond to “Env Sync”
    37
     ---
     environment:
      UTSUSEMI_TARGET_HOST:
      description: Crawl target host
      type: text
      UTSUSEMI_WORKER_PROCESS:
      description: Crawl worker process
      type: number
      default: 2
      UTSUSEMI_BUCKET_NAME:
    

    View Slide

38. DEMO
    ( If I have time )
    38
    Serverlessconf Tokyo 2017
    

    View Slide

39. User simple deploy ecosystem!!
    39
    Serverlessconf Tokyo 2017
    

    View Slide

40. Conclusion
      Applications built on manged services can abstract as one
    middleware or one service.
      Serverless Framework provides `sls deploy` like `make
    install`
      If a mechanism that makes it easier to deploy is provided, the
    user can obtain a “serverless middleware” and “serverless
    service” more easily.
      I would like to see more open source “serverless middleware”
    and “serverless service”
    40
    Serverlessconf Tokyo 2017
    

    View Slide

41. 41
    Weʼre Hiring Engineers!
    > https://fusic.github.io
    Thank you!
    Serverlessconf Tokyo 2017
    

    View Slide