Cloud-agnostic Serverless built with GitLab

2020-01-15 Tech Play Serverless #2
TECH PLAY SHIBUYA

Transcript

1. Cloud-agnostic Serverless built with GitLab @tnir - 2020-01-15 - Tech Play Serverless #2 @TECH PLAY SHIBUYA
2. Disclaimer • すべて個人の見解です。 • The views and opinions represented in this presentation are personal to the author of each respective talk and do not represent the views or opinions of any organization unless explicitly stated. All content provided on this presentation is for informational purposes only.
3. @tnir (Takuya Noguchi) • DevOps Tech Lead, Japan Digital Design • Core Team, GitLab • GitLab SuperStar 2018/2019, Superhero 2019 • Cloud Native Ambassador, CNCF • Co-organizer, Docker Tokyo • Founder & organizer, GitLab Tokyo • OSS contributors of multiple projects ◦ kubernetes/website, kubernetes-docs-ja, Django / django-ja, pandas, Mattermost, etc.
4. DB1 DB2 API Gateway DB3 Suppose that DB 2 is not either RDS/Aurora (MySQL, PostgreSQL etc.) or DynamoDB. (cf. RDS Proxy (re:Invent 2019)) Connection pooling problem in this architecture (if with Lambda)
5. Serverless
6. 「amazon lambdaの活用事例につい て...」(登壇依頼でのテーマ) • AWS Lambda since 2014 • 5+ years passed after re:Invent 2014 • → Cloud-agnostic (full) DevOps solution (my personal mission)
7. Serverless ≠ Lambda
8. Multi-cloud (cloud-agnostic) • Lambda looks nice！ • →Cloud vendor lock-in？ ◦ →e.g.) 2019/8 Service disruption at AWS Tokyo region ◦ Multiple regions, compliance with laws etc. ◦ Want to use Google Cloud Spanner (GCP) ◦ (Full-managed) Cloud Run GA’ed (Jul 2019): https://medium.com/google-cloud-jp/cloud-run-ga-fb31378cd0a1 • →cloud native ◦ →Kubernetes？ ▪ →Kubernetes specialist required, hard Ops • →Extract the common code as framework!!! 2019/8のAWS Tokyo障害: https://aws.amazon.com/jp/message/56489/
9. https://s.cncf.io
10. https://twitter.com/gitlab/status/1217133723818184704
11. GitLab Serverless
12. GitLab Serverless • Released in Jan 2019 ◦ https://www.publickey1.jp/blog/19/gitlab_serverlessgitlab_116knative.html • Status: Alpha (as Jan 2020) • Offerings: 1. Functions (TriggerMesh (Knative) / OpenFaaS) 2. Containers (TriggerMesh (Knative)) 3. AWS Lambda (with Serverless Framework) ← pre-defined runtime ← arbitrary runtime
13. CI pipeline workflow of Lambda image: node:latest stages: - deploy production: stage: deploy before_script: - npm config set prefix /usr/local - npm install -g serverless script: - serverless deploy --stage production --verbose environment: production ↑ Add this to your .gitlab-ci.yml to run CI pipeline workflow. https://gitlab.com/gitlab-org/serverless/examples/serverless-framework-js/blob/c4bab3616b0ccea96c88d8a28a1ca934ff55e0f8/.gitlab-ci.yml with Serverless Framework
14. CI pipeline workflow of Functions include: template: Serverless.gitlab-ci.yml functions:build: extends: .serverless:build:functions environment: production functions:deploy: extends: .serverless:deploy:functions environment: production ↑ Add this to your .gitlab-ci.yml to run CI pipeline workflow. https://gitlab.com/knative-examples/functions/blob/2741e54eb82f882179114590df72cb73074d1c48/.gitlab-ci.yml with Knative (thru gitlabktl) GitLab / OpenFaaS runtimes
15. CI pipeline workflow of containers Include: template: Serverless.gitlab-ci.yml build: extends: .serverless:build:image deploy: extends: .serverless:deploy:image ↑ Add this to your .gitlab-ci.yml to run CI pipeline workflow. https://gitlab.com/knative-examples/knative-ruby-app/blob/ece26c9a98eb0c2cafb70c7904c9dbc35f0a0ded/.gitlab-ci.yml with Knative (thru gitlabktl) Your Dockerfile required
16. Future of GitLab Serverless 1. Integrated with TriggerMesh’s Knative Lambda Runtime to emulate AWS Lambda on anywhere 2. Deeper Knative integration on GCP Anthos 3. (… Separate GitLab Serverless from GitLab itself)
17. Crossplane integration • Cloud-agnostic ≒ everything prepared by us • → Use cloud (managed service) with (some) layer • Crossplane (by Upbound) ◦ Provides multiple-cloud seamless control plane. ◦ Enables independent CockroachDB, Elasticsearch, MongoDB datastore layers (DBaaS) available. https://crossplane.io/
18. DevOps of “flexible” serverless with Crossplane
19. Progressive delivery • PD = Extension of continuous delivery (CD) with canaries ◦ by LaunchDarkly (2018) and Jenkins X (2019) • Automated with rolling back when key metrics are not fulfilled after deployed • An optional way to easily introduce CD
20. Observability
21. Monitoring at one place
22. Observability: with context with Jaeger
23. Security
24. https://blog.shiftleft.io/the-shiftleft-vision-68114e5f5efd
25. Security - Application Security • Application security on serverless is as important as appsec on non-serverless ◦ 4 of top 6 attacks are web-application securities • Higher responsibility on serverless with shared responsibility model • For IAM/Networking, the same as cloud practice • Developer UX (DX): key to widely spread • → Unified & integrated platform is required
26. User voice from around me • “Hard to see the logs!” ◦ https://gitlab.com/gitlab-org/security-products/tests/java-maven/-/jobs/402560464 • “Where is the security dashboard?”
27. Company-wide (app) security Will be released on Jan 22, 2020
28. GitLab DAST, same as legacy apps
29. Uncovered area • Security check for IAM, bucket (S3, GCS) policies ◦ → Use Terraform, Ansible, CFn, or whatever you want • Capacity management ◦ → “memory”-based optimization required; no practice (to me) • Chaos engineering framework ◦ → Use general chaos engineering tools
30. まとめ • クラウド非依存サーバレスへの挑戦 • DevOpsのチームへのアダプション〜ツールの改善の ループを回している（DevOpsツールの継続的改善） • サーバレスも考えることはたくさんなのに、実践からの 抽象化は難しい • 日本語圏でのGitLabの利用情報が少ないので試して 共有していただけると嬉しい
31. @tnir Takuya Noguchi Tw: @tn961ir • https://github.com/tnir • https://tnir.gitlab.io/
32. 懇親会の後半にGitLabグッズをプレゼント(5名)