Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud-agnostic Serverless built with GitLab

tnir
January 15, 2020

Cloud-agnostic Serverless built with GitLab

2020-01-15 Tech Play Serverless #2
TECH PLAY SHIBUYA

tnir

January 15, 2020
Tweet

More Decks by tnir

Other Decks in Technology

Transcript

  1. Cloud-agnostic
    Serverless built with
    GitLab
    @tnir - 2020-01-15 - Tech Play Serverless #2
    @TECH PLAY SHIBUYA

    View full-size slide

  2. Disclaimer
    ● すべて個人の見解です。
    ● The views and opinions represented in this presentation
    are personal to the author of each respective talk and do
    not represent the views or opinions of any organization
    unless explicitly stated. All content provided on this
    presentation is for informational purposes only.

    View full-size slide

  3. @tnir (Takuya Noguchi)
    ● DevOps Tech Lead, Japan Digital Design
    ● Core Team, GitLab
    ● GitLab SuperStar 2018/2019, Superhero 2019
    ● Cloud Native Ambassador, CNCF
    ● Co-organizer, Docker Tokyo
    ● Founder & organizer, GitLab Tokyo
    ● OSS contributors of multiple projects
    ○ kubernetes/website, kubernetes-docs-ja, Django / django-ja,
    pandas, Mattermost, etc.

    View full-size slide

  4. DB1 DB2
    API Gateway
    DB3
    Suppose that DB 2 is not either RDS/Aurora (MySQL,
    PostgreSQL etc.) or DynamoDB.
    (cf. RDS Proxy (re:Invent 2019))
    Connection pooling problem in this architecture (if
    with Lambda)

    View full-size slide

  5. 「amazon lambdaの活用事例につい
    て...」(登壇依頼でのテーマ)
    ● AWS Lambda since 2014
    ● 5+ years passed after re:Invent 2014
    ● → Cloud-agnostic (full) DevOps solution (my personal
    mission)

    View full-size slide

  6. Serverless ≠
    Lambda

    View full-size slide

  7. Multi-cloud (cloud-agnostic)
    ● Lambda looks nice!
    ● →Cloud vendor lock-in?
    ○ →e.g.) 2019/8 Service disruption at AWS Tokyo region
    ○ Multiple regions, compliance with laws etc.
    ○ Want to use Google Cloud Spanner (GCP)
    ○ (Full-managed) Cloud Run GA’ed (Jul 2019):
    https://medium.com/google-cloud-jp/cloud-run-ga-fb31378cd0a1
    ● →cloud native
    ○ →Kubernetes?
    ■ →Kubernetes specialist required, hard Ops
    ● →Extract the common code as framework!!!
    2019/8のAWS Tokyo障害: https://aws.amazon.com/jp/message/56489/

    View full-size slide

  8. https://s.cncf.io

    View full-size slide

  9. https://twitter.com/gitlab/status/1217133723818184704

    View full-size slide

  10. GitLab Serverless

    View full-size slide

  11. GitLab Serverless
    ● Released in Jan 2019
    ○ https://www.publickey1.jp/blog/19/gitlab_serverlessgitlab_116knative.html
    ● Status: Alpha (as Jan 2020)
    ● Offerings:
    1. Functions (TriggerMesh (Knative) / OpenFaaS)
    2. Containers (TriggerMesh (Knative))
    3. AWS Lambda (with Serverless Framework)
    ← pre-defined runtime
    ← arbitrary runtime

    View full-size slide

  12. CI pipeline workflow of Lambda
    image: node:latest
    stages:
    - deploy
    production:
    stage: deploy
    before_script:
    - npm config set prefix /usr/local
    - npm install -g serverless
    script:
    - serverless deploy --stage production --verbose
    environment: production
    ↑ Add this to your .gitlab-ci.yml to run CI pipeline workflow.
    https://gitlab.com/gitlab-org/serverless/examples/serverless-framework-js/blob/c4bab3616b0ccea96c88d8a28a1ca934ff55e0f8/.gitlab-ci.yml
    with Serverless Framework

    View full-size slide

  13. CI pipeline workflow of Functions
    include:
    template: Serverless.gitlab-ci.yml
    functions:build:
    extends: .serverless:build:functions
    environment: production
    functions:deploy:
    extends: .serverless:deploy:functions
    environment: production
    ↑ Add this to your .gitlab-ci.yml to run CI pipeline workflow.
    https://gitlab.com/knative-examples/functions/blob/2741e54eb82f882179114590df72cb73074d1c48/.gitlab-ci.yml
    with Knative (thru gitlabktl)
    GitLab / OpenFaaS runtimes

    View full-size slide

  14. CI pipeline workflow of containers
    Include:
    template: Serverless.gitlab-ci.yml
    build:
    extends: .serverless:build:image
    deploy:
    extends: .serverless:deploy:image
    ↑ Add this to your .gitlab-ci.yml to run CI pipeline workflow.
    https://gitlab.com/knative-examples/knative-ruby-app/blob/ece26c9a98eb0c2cafb70c7904c9dbc35f0a0ded/.gitlab-ci.yml
    with Knative (thru gitlabktl)
    Your Dockerfile required

    View full-size slide

  15. Future of GitLab Serverless
    1. Integrated with TriggerMesh’s Knative Lambda Runtime
    to emulate AWS Lambda on anywhere
    2. Deeper Knative integration on GCP Anthos
    3. (… Separate GitLab Serverless from GitLab itself)

    View full-size slide

  16. Crossplane integration
    ● Cloud-agnostic ≒ everything prepared by us
    ● → Use cloud (managed service) with (some) layer
    ● Crossplane (by Upbound)
    ○ Provides multiple-cloud seamless control plane.
    ○ Enables independent CockroachDB, Elasticsearch, MongoDB
    datastore layers (DBaaS) available.
    https://crossplane.io/

    View full-size slide

  17. DevOps of “flexible” serverless
    with Crossplane

    View full-size slide

  18. Progressive delivery
    ● PD = Extension of continuous delivery (CD) with canaries
    ○ by LaunchDarkly (2018) and Jenkins X (2019)
    ● Automated with rolling back when key metrics are not
    fulfilled after deployed
    ● An optional way to easily introduce CD

    View full-size slide

  19. Observability

    View full-size slide

  20. Monitoring at one place

    View full-size slide

  21. Observability: with context
    with Jaeger

    View full-size slide

  22. https://blog.shiftleft.io/the-shiftleft-vision-68114e5f5efd

    View full-size slide

  23. Security - Application Security
    ● Application security on serverless is as important as
    appsec on non-serverless
    ○ 4 of top 6 attacks are web-application securities
    ● Higher responsibility on serverless with shared
    responsibility model
    ● For IAM/Networking, the same as cloud practice
    ● Developer UX (DX): key to widely spread
    ● → Unified & integrated platform is required

    View full-size slide

  24. User voice from around me
    ● “Hard to see the logs!”
    ○ https://gitlab.com/gitlab-org/security-products/tests/java-maven/-/jobs/402560464
    ● “Where is the security dashboard?”

    View full-size slide

  25. Company-wide (app) security
    Will be released on Jan 22, 2020

    View full-size slide

  26. GitLab DAST, same as legacy apps

    View full-size slide

  27. Uncovered area
    ● Security check for IAM, bucket (S3, GCS) policies
    ○ → Use Terraform, Ansible, CFn, or whatever you want
    ● Capacity management
    ○ → “memory”-based optimization required; no practice (to me)
    ● Chaos engineering framework
    ○ → Use general chaos engineering tools

    View full-size slide

  28. まとめ
    ● クラウド非依存サーバレスへの挑戦
    ● DevOpsのチームへのアダプション〜ツールの改善の
    ループを回している(DevOpsツールの継続的改善)
    ● サーバレスも考えることはたくさんなのに、実践からの
    抽象化は難しい
    ● 日本語圏でのGitLabの利用情報が少ないので試して
    共有していただけると嬉しい

    View full-size slide

  29. @tnir
    Takuya Noguchi
    Tw: @tn961ir
    ● https://github.com/tnir
    ● https://tnir.gitlab.io/

    View full-size slide

  30. 懇親会の後半にGitLabグッズをプレゼント(5名)

    View full-size slide