Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud-agnostic Serverless built with GitLab

January 15, 2020

Cloud-agnostic Serverless built with GitLab

2020-01-15 Tech Play Serverless #2


January 15, 2020

More Decks by tnir

Other Decks in Technology


  1. Disclaimer • すべて個人の見解です。 • The views and opinions represented in

    this presentation are personal to the author of each respective talk and do not represent the views or opinions of any organization unless explicitly stated. All content provided on this presentation is for informational purposes only.
  2. @tnir (Takuya Noguchi) • DevOps Tech Lead, Japan Digital Design

    • Core Team, GitLab • GitLab SuperStar 2018/2019, Superhero 2019 • Cloud Native Ambassador, CNCF • Co-organizer, Docker Tokyo • Founder & organizer, GitLab Tokyo • OSS contributors of multiple projects ◦ kubernetes/website, kubernetes-docs-ja, Django / django-ja, pandas, Mattermost, etc.
  3. DB1 DB2 API Gateway DB3 Suppose that DB 2 is

    not either RDS/Aurora (MySQL, PostgreSQL etc.) or DynamoDB. (cf. RDS Proxy (re:Invent 2019)) Connection pooling problem in this architecture (if with Lambda)
  4. 「amazon lambdaの活用事例につい て...」(登壇依頼でのテーマ) • AWS Lambda since 2014 • 5+

    years passed after re:Invent 2014 • → Cloud-agnostic (full) DevOps solution (my personal mission)
  5. Multi-cloud (cloud-agnostic) • Lambda looks nice! • →Cloud vendor lock-in?

    ◦ →e.g.) 2019/8 Service disruption at AWS Tokyo region ◦ Multiple regions, compliance with laws etc. ◦ Want to use Google Cloud Spanner (GCP) ◦ (Full-managed) Cloud Run GA’ed (Jul 2019): https://medium.com/google-cloud-jp/cloud-run-ga-fb31378cd0a1 • →cloud native ◦ →Kubernetes? ▪ →Kubernetes specialist required, hard Ops • →Extract the common code as framework!!! 2019/8のAWS Tokyo障害: https://aws.amazon.com/jp/message/56489/
  6. GitLab Serverless • Released in Jan 2019 ◦ https://www.publickey1.jp/blog/19/gitlab_serverlessgitlab_116knative.html •

    Status: Alpha (as Jan 2020) • Offerings: 1. Functions (TriggerMesh (Knative) / OpenFaaS) 2. Containers (TriggerMesh (Knative)) 3. AWS Lambda (with Serverless Framework) ← pre-defined runtime ← arbitrary runtime
  7. CI pipeline workflow of Lambda image: node:latest stages: - deploy

    production: stage: deploy before_script: - npm config set prefix /usr/local - npm install -g serverless script: - serverless deploy --stage production --verbose environment: production ↑ Add this to your .gitlab-ci.yml to run CI pipeline workflow. https://gitlab.com/gitlab-org/serverless/examples/serverless-framework-js/blob/c4bab3616b0ccea96c88d8a28a1ca934ff55e0f8/.gitlab-ci.yml with Serverless Framework
  8. CI pipeline workflow of Functions include: template: Serverless.gitlab-ci.yml functions:build: extends:

    .serverless:build:functions environment: production functions:deploy: extends: .serverless:deploy:functions environment: production ↑ Add this to your .gitlab-ci.yml to run CI pipeline workflow. https://gitlab.com/knative-examples/functions/blob/2741e54eb82f882179114590df72cb73074d1c48/.gitlab-ci.yml with Knative (thru gitlabktl) GitLab / OpenFaaS runtimes
  9. CI pipeline workflow of containers Include: template: Serverless.gitlab-ci.yml build: extends:

    .serverless:build:image deploy: extends: .serverless:deploy:image ↑ Add this to your .gitlab-ci.yml to run CI pipeline workflow. https://gitlab.com/knative-examples/knative-ruby-app/blob/ece26c9a98eb0c2cafb70c7904c9dbc35f0a0ded/.gitlab-ci.yml with Knative (thru gitlabktl) Your Dockerfile required
  10. Future of GitLab Serverless 1. Integrated with TriggerMesh’s Knative Lambda

    Runtime to emulate AWS Lambda on anywhere 2. Deeper Knative integration on GCP Anthos 3. (… Separate GitLab Serverless from GitLab itself)
  11. Crossplane integration • Cloud-agnostic ≒ everything prepared by us •

    → Use cloud (managed service) with (some) layer • Crossplane (by Upbound) ◦ Provides multiple-cloud seamless control plane. ◦ Enables independent CockroachDB, Elasticsearch, MongoDB datastore layers (DBaaS) available. https://crossplane.io/
  12. Progressive delivery • PD = Extension of continuous delivery (CD)

    with canaries ◦ by LaunchDarkly (2018) and Jenkins X (2019) • Automated with rolling back when key metrics are not fulfilled after deployed • An optional way to easily introduce CD
  13. Security - Application Security • Application security on serverless is

    as important as appsec on non-serverless ◦ 4 of top 6 attacks are web-application securities • Higher responsibility on serverless with shared responsibility model • For IAM/Networking, the same as cloud practice • Developer UX (DX): key to widely spread • → Unified & integrated platform is required
  14. User voice from around me • “Hard to see the

    logs!” ◦ https://gitlab.com/gitlab-org/security-products/tests/java-maven/-/jobs/402560464 • “Where is the security dashboard?”
  15. Uncovered area • Security check for IAM, bucket (S3, GCS)

    policies ◦ → Use Terraform, Ansible, CFn, or whatever you want • Capacity management ◦ → “memory”-based optimization required; no practice (to me) • Chaos engineering framework ◦ → Use general chaos engineering tools