Slide 1

Slide 1 text

CI/CD, Kubernetes, and Databases: Better Together Niraj Tolia Tom Manville @nirajtolia @tdmanv

Slide 2

Slide 2 text

about us page 02 Niraj Tolia Co-founder & CEO @ Kasten Previously at EMC, Maginatics, HP, CMU Tom Manville Founding Engineer @ Kasten Previously at Dropbox, Maginatics, U. Mich.

Slide 3

Slide 3 text

our goal: move fast and test with real data

Slide 4

Slide 4 text

what we will not cover in this talk page 04 Kubernetes Ready for Production Stateful Apps Presented at SNIA’s 2018 Storage Developer Conference Implementing a Data Protection Strategy KubeCon Seattle, Wednesday, December 12, 2:35pm

Slide 5

Slide 5 text

current state of databases in a cloud-native world page 05

Slide 6

Slide 6 text

cloud-native and databases why is there so much fear and risk? page 06 Still see database groups isolated from both dev and infra ops groups. Not part of app dev. DBAs and Ops Not built into CI/CD pipelines. Test datasets have manual imports and get stale quickly. Automation Gap Databases are isolated from the application, might have manual changes applied, treated as pets. Snowflakes

Slide 7

Slide 7 text

What should the future look like?

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

increasing agility with databases in a cloud-native environment page 09 Source Control Include all schema changes, upgrades changes, tools, etc. in the application repository Kubernetes to tie it all together! CI/CD Pipeline Automate testing all database changes and modifications Database Infrastructure Deliver database infrastructure and configuration as code

Slide 10

Slide 10 text

how kubernetes makes a difference page 010 Enforces Good DevOps Hygiene Immutability, config as code, automation makes repeatable and reliable testing easy Efficient, High Resource Utilization Declarative systems approach supports reliable use of multiple testing environments to test at scale Universal Control Plane Use the same management plane as you use for all other components of your application

Slide 11

Slide 11 text

ci/cd advantages for databases page 011 Catch issues early • Unit tests for coverage • Integration and staging environments for behavioral • Faster change iteration with automated testing • High velocity prod DB deployments Engineering agility • Enforces the the app and DB are always in sync • Higher-confidence releases Automated testing

Slide 12

Slide 12 text

But, it’s a database! So, what about the data?

Slide 13

Slide 13 text

Need to safely test with production data (but not in production!)

Slide 14

Slide 14 text

data based testing number of integration challenges page 014 Storage Integration Might need to integrate with volume- level storage APIs for efficiency. Database Integration For consistent data capture including w/ eventually consistent data stores Application Integration Polyglot persistence in micro-service based applications needs app-level coordination. So does data masking to protect sensitive data.

Slide 15

Slide 15 text

Supporting Data Mobility

Slide 16

Slide 16 text

page 016 kanister: A Kubernetes-native framework for application-level data management • Supports complex data management workflows • Easy to integrate against your CI/CD pipeline • Actions invoked via Custom Resources (CRs) • Easy to extend via simple “recipes” or Blueprints

Slide 17

Slide 17 text

kanister: the highlights page 017 Control Plane Integration • Ties K8s and DB control planes • Library support for complex workflows (e.g., scale up/down) • Filters • Masking • Incremental Capture Database Manipulation • File/Block integration via native API and CSI v1.0 • S3 API support for object stores Data Capture/Export Visit for more information

Slide 18

Slide 18 text

kanister workflow page 018 Blueprint (Custom K8s Resource) Stateful Application 1. ActionSet Creation 2. Blueprint Discovery 3. Action Execution KubeExec / KubeTask 4. Status Update Kanister Controller ActionSet (Custom K8s Resource)

Slide 19

Slide 19 text

kanister actionset (abridged) page 019 apiVersion: kind: ActionSet spec: actions: - name: backup blueprint: postgresql object: kind: StatefulSet name: postgresql-cluster namespace: default configMaps: ...

Slide 20

Slide 20 text

kanister blueprint (abridged) page 020 apiVersion: kind: Blueprint actions: backup: type: StatefulSet phases: - func: KubeExec args: - '{{ .StatefulSet.Namespace }}' - '{{ index .StatefulSet.Pods 0 }}' - postgresql-tools-sidecar - bash - -c - wal-e ... - func: ... restore: ...

Slide 21

Slide 21 text


Slide 22

Slide 22 text

demo: pipeline setup page 022 Application Code Config Definition Database Schema Source Control Integration Pipeline Deployment Pipeline Production Cluster Data Mobility

Slide 23

Slide 23 text

integration demo: data flow setup page 023 App Pod DB Pod Namespace: demo Production Kubernetes Cluster NS: kio DB Pod Integration Kubernetes Cluster NS: kio Object Storage Firewall App + Data Snapshot App Pod Namespace: test DB ⓵ App Export ⓶ App Import ⓸ Data Population ⓷ Test Invocation K10: Policy and Orchestration (e.g., Periodic Import or Export) + Kanister: Data Manipulation and Mobility

Slide 24

Slide 24 text

end-to-end demo

Slide 25

Slide 25 text

advanced topics (hopefully) coming soon to a conf. near you page 025 CD w/ schema changes Deploying schema changes (and rollbacks) can be a lot more involved. Backup/recovery is a critical part of this. Managed Services Apart from cost, these slides apply to managed services too but do track emerging best practices Masking and Sampling Kanister has support for injecting your own code to mask sensitive data or only extract a a subset Dataset Promotion There are situations where you might want to promote data from dev → staging → prod

Slide 26

Slide 26 text

kubernetes, ci/cd, and databases wrapping up page 26 01Automate your DB Pipeline Deploy database updates and changes with increased confidence 04Make DB Engineering Agile Integrate database teams into your DevOps and Agile journey. Break apart the silos! 02Leverage Kubernetes Deliver greater agility to your dev teams by allowing easy and reliable testing 03Use Real Data Test on production data to reduce code quality risk when running against synthetic or stale data Build & Standardize your DB Pipeline on Kubernetes!

Slide 27

Slide 27 text

page 027 Image is the cover art from Better Together, a Jack Johnson song Questions? You can also find us at: Booth S/E15 @kastenhq @nirajtolia @tdmanv