Postgres on Kubernetes - Dos and Don'ts

Slide 1

Slide 1 text

https://www.marketingdonut.co.uk/pr-and-promotion/exhibitions/dos-and-don-ts-when-exhibiting

Slide 2

Slide 2 text

PostgreSQL Kubernetes ❤

Slide 3

Slide 3 text

Chris Engelbert Devrel @ simplyblock Previous fun companies: - Ubisoft / Blue Byte - Hazelcast - Instana - clevabit - Timescale Interests: - Developer Relations - Anything Performance Engineering - Backend Technologies - Fairy Tales (AMD, Intel, Nvidia) @noctarius2k @noctarius2k@mastodon.online @noctarius.com

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Question 01

Slide 7

Slide 7 text

Question 01 Why you shouldn't run a database in Kubernetes?

Slide 8

Slide 8 text

Why not to run a database in Kubernetes?

Slide 9

Slide 9 text

Why not to run a database in Kubernetes?

Slide 10

Slide 10 text

Why not to run a database in Kubernetes? K8s is not designed with Databases in mind!

Slide 11

Slide 11 text

Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s!

Slide 12

Slide 12 text

Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s! Persistent Data will kill you! Too slow!

Slide 13

Slide 13 text

Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s! Persistent Data will kill you! Too slow! Nobody understands Kubernetes!

Slide 14

Slide 14 text

Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s! Persistent Data will kill you! Too slow! Nobody understands Kubernetes! What’s the bene fi t; databases don’t need autoscaling!

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Why not to run a database in Kubernetes? BURN IN HELL!

Slide 19

Slide 19 text

The Happy Place

Slide 20

Slide 20 text

Where are my gamers at? So we need to cheat!?

Slide 21

Slide 21 text

The Happy Place

Slide 22

Slide 22 text

Why?

Slide 23

Slide 23 text

No Cloud-Vendor Lock-In Why?

Slide 24

Slide 24 text

No Cloud-Vendor Lock-In Faster Time To Market Why?

Slide 25

Slide 25 text

No Cloud-Vendor Lock-In Faster Time To Market Decreasing cost Why?

Slide 26

Slide 26 text

No Cloud-Vendor Lock-In Faster Time To Market Decreasing cost Automation Why?

Slide 27

Slide 27 text

No Cloud-Vendor Lock-In Faster Time To Market Decreasing cost Automation Uni fi ed deployment architecture Why?

Slide 28

Slide 28 text

No Cloud-Vendor Lock-In Faster Time To Market Decreasing cost Automation Uni fi ed deployment architecture Need read-only replicas Why?

Slide 29

Slide 29 text

Let’s get something out of the way fi rst!

Slide 30

Slide 30 text

Call the Police!

Slide 31

Slide 31 text

Enable TLS Call the Police!

Slide 32

Slide 32 text

Enable TLS Use Kubernetes Secrets Call the Police!

Slide 33

Slide 33 text

Enable TLS Use Kubernetes Secrets Use Cert-Manager Call the Police!

Slide 34

Slide 34 text

Enable TLS Use Kubernetes Secrets Use Cert-Manager Encrypt Data-At-Rest Call the Police!

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Enable TLS Use Kubernetes Secrets Use Cert-Manager Encrypt Data-At-Rest Call the Police!

Slide 37

Slide 37 text

Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/

Slide 38

Slide 38 text

You want Continuous Backup and PITR Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/

Slide 39

Slide 39 text

You want Continuous Backup and PITR Roll your own pg_basebackup or pg_dump (don’t!) Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/

Slide 40

Slide 40 text

You want Continuous Backup and PITR Roll your own pg_basebackup or pg_dump (don’t!) Use tools like pgbackrest, barman, PGHoard, … Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/

Slide 41

Slide 41 text

You want Continuous Backup and PITR Roll your own pg_basebackup or pg_dump (don’t!) Use tools like pgbackrest, barman, PGHoard, … Upload backups to S3? Cost! Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/

Slide 42

Slide 42 text

Slide 43

Slide 43 text

PostgreSQL Con fi guration

Slide 44

Slide 44 text

PostgreSQL Con fi guration The PostgreSQL Con fi guration isn’t too much in fl uenced

Slide 45

Slide 45 text

shared_bu ff ers (maintenance_)work_mem e ff ective_cache_size PostgreSQL Con fi guration The PostgreSQL Con fi guration isn’t too much in fl uenced

Slide 46

Slide 46 text

shared_bu ff ers (maintenance_)work_mem e ff ective_cache_size PostgreSQL Con fi guration The PostgreSQL Con fi guration isn’t too much in fl uenced Use Huge Pages!

Slide 47

Slide 47 text

PostgreSQL Con fi guration https://www.youtube.com/watch?v=S0LEDGbAnn8 https://www.crunchydata.com/blog/optimize-postgresql-server-performance https://www.percona.com/blog/using-huge-pages-with-postgresql-running-inside-kubernetes/

Slide 48

Slide 48 text

Do you need more? Extensions!

Slide 49

Slide 49 text

Do you need PG Extensions? Do you need more? Extensions!

Slide 50

Slide 50 text

Do you need PG Extensions? Do you need more? Extensions! Is the extension part of the container image?

Slide 51

Slide 51 text

Do you need PG Extensions? Do you need more? Extensions! Is the extension part of the container image? If not, you need to build your own layer…

Slide 52

Slide 52 text

Do you need PG Extensions? Do you need more? Extensions! Is the extension part of the container image? If not, you need to build your own layer… or use some magic (more on this later).

Slide 53

Slide 53 text

Keep an Eye on PG and Kubernetes Versions Versions and Updates

Slide 54

Slide 54 text

So What is important or di ff erent?

Slide 55

Slide 55 text

Storage

Slide 56

Slide 56 text

Storage

Slide 57

Slide 57 text

Use Persistent Volumes Storage

Slide 58

Slide 58 text

Use Persistent Volumes Storage (local volumes are a bad idea)

Slide 59

Slide 59 text

Use Persistent Volumes Storage Should be dynamically provisioned (local volumes are a bad idea)

Slide 60

Slide 60 text

Use Persistent Volumes Storage Should be dynamically provisioned CSI provider enables encryption at rest (local volumes are a bad idea)

Slide 61

Slide 61 text

Use Persistent Volumes Storage Should be dynamically provisioned CSI provider enables encryption at rest High IOPS (SSD or NVMe) (local volumes are a bad idea)

Slide 62

Slide 62 text

Use Persistent Volumes Storage Should be dynamically provisioned CSI provider enables encryption at rest High IOPS (SSD or NVMe) Low Latency (local volumes are a bad idea)

Slide 63

Slide 63 text

Use Persistent Volumes Storage Should be dynamically provisioned CSI provider enables encryption at rest High IOPS (SSD or NVMe) Low Latency Database performance is as fast as your storage (local volumes are a bad idea)

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Storage www.storageclass.info/csidrivers

Slide 66

Slide 66 text

Requests, Limits, and Quotas Capacity Limits Requests Used

Slide 67

Slide 67 text

Requests, Limits, and Quotas Capacity Limits Requests Used Use Resource Requests, Limits, Quotas

Slide 68

Slide 68 text

Requests, Limits, and Quotas CPU and memory requests need to be accurate  to prevent contention and ensure predictable performance Capacity Limits Requests Used Use Resource Requests, Limits, Quotas

Slide 69

Slide 69 text

Requests, Limits, and Quotas CPU and memory requests need to be accurate  to prevent contention and ensure predictable performance Capacity Limits Requests Used https://codimite.ai/blog/kubernetes-resources-and-scaling-a-beginners-guide/ Use Resource Requests, Limits, Quotas

Slide 70

Slide 70 text

Make it big! Enable Huge Pages!

Slide 71

Slide 71 text

Make it big! Enable Huge Pages! In your OS and the Resource Descriptor.

Slide 72

Slide 72 text

Make it big! Enable Huge Pages! In your OS and the Resource Descriptor. https://www.percona.com/blog/using-huge-pages-with-postgresql-running-inside-kubernetes/

Slide 73

Slide 73 text

Resiliency and Overhead

Slide 74

Slide 74 text

Resiliency and Overhead High Availability

Slide 75

Slide 75 text

Patroni, repmgr, pg_auto_failover, … Resiliency and Overhead High Availability

Slide 76

Slide 76 text

Patroni, repmgr, pg_auto_failover, … Resiliency and Overhead High Availability https://medium.com/@kristi.anderson/whats-the-best-postgresql-high-availability-framework...

Slide 77

Slide 77 text

Resiliency and Overhead

Slide 78

Slide 78 text

Resiliency and Overhead Connection Pooling

Slide 79

Slide 79 text

Never use PostgreSQL without Connection Pooling! Resiliency and Overhead Connection Pooling

Slide 80

Slide 80 text

Never use PostgreSQL without Connection Pooling! Optimizes Overhead and Resource Utilization Resiliency and Overhead Connection Pooling

Slide 81

Slide 81 text

Never use PostgreSQL without Connection Pooling! Optimizes Overhead and Resource Utilization Handles failovers, central switching of Primary Resiliency and Overhead Connection Pooling

Slide 82

Slide 82 text

Slide 83

Slide 83 text

Never use PostgreSQL without Connection Pooling! Optimizes Overhead and Resource Utilization Handles failovers, central switching of Primary Enables easy use of Read-Replicas Resiliency and Overhead Connection Pooling PgBouncer, PgPool-II, pgagroal, PgCat, Odyssey, …

Slide 84

Slide 84 text

Slide 85

Slide 85 text

Where’s my Replicant?

Slide 86

Slide 86 text

Where’s my Replicant? Use available Kubernetes features

Slide 87

Slide 87 text

Where’s my Replicant? Use available Kubernetes features StatefulSet

Slide 88

Slide 88 text

Where’s my Replicant? Use available Kubernetes features StatefulSet

Slide 89

Slide 89 text

Networking and Access Control https://timeclock365.com/tc22-door-access-controller/

Slide 90

Slide 90 text

Use Network Policies Networking and Access Control https://timeclock365.com/tc22-door-access-controller/

Slide 91

Slide 91 text

Use Network Policies Enable TLS (you remember?!) Networking and Access Control https://timeclock365.com/tc22-door-access-controller/

Slide 92

Slide 92 text

Use Network Policies Enable TLS (you remember?!) Setup Security Policies Networking and Access Control https://timeclock365.com/tc22-door-access-controller/

Slide 93

Slide 93 text

Use Network Policies Enable TLS (you remember?!) Setup Security Policies Con fi gure RBAC (Role-Based Access Control) Networking and Access Control https://timeclock365.com/tc22-door-access-controller/

Slide 94

Slide 94 text

Use Network Policies Enable TLS (you remember?!) Setup Security Policies Con fi gure RBAC (Role-Based Access Control) Networking and Access Control Think about a policy manager such as OPA or kyverno https://timeclock365.com/tc22-door-access-controller/

Slide 95

Slide 95 text

Observability and Alerting

Slide 96

Slide 96 text

Observability and Alerting Like anything cloud, make sure you have monitoring (meaning observability) and alerting!

Slide 97

Slide 97 text

Prometheus Exporter, Log Collector, Aggregation, Analysis, Traceability, … Observability and Alerting Like anything cloud, make sure you have monitoring (meaning observability) and alerting!

Slide 98

Slide 98 text

Prometheus Exporter, Log Collector, Aggregation, Analysis, Traceability, … Observability and Alerting Like anything cloud, make sure you have monitoring (meaning observability) and alerting! Datadog, Instana, DynaTrace, Grafana, …

Slide 99

Slide 99 text

Operator

Slide 100

Slide 100 text

Operator Use a Postgres Kubernetes Operator

Slide 101

Slide 101 text

Operator Use a Postgres Kubernetes Operator Handles or con fi gures many of the typical tasks (HA, backup, …)

Slide 102

Slide 102 text

Operator Use a Postgres Kubernetes Operator Handles or con fi gures many of the typical tasks (HA, backup, …) Brings cloud-nativeness to PG

Slide 103

Slide 103 text

Operator Use a Postgres Kubernetes Operator Handles or con fi gures many of the typical tasks (HA, backup, …) Brings cloud-nativeness to PG Integrates PG into k8s

Slide 104

Slide 104 text

Operator If not, use Helm Charts Use a Postgres Kubernetes Operator Handles or con fi gures many of the typical tasks (HA, backup, …) Brings cloud-nativeness to PG Integrates PG into k8s

Slide 105

Slide 105 text

Operator CloudNativePG Crunchy Postgres for Kubernetes OnGres StackGres KubeDB Zalando Postgres Operator Supported versions 12, 13, 14, 15, 16 11, 12, 13, 14, 15, 16 12, 13, 14, 15, 16 9.6, 10, 11, 12, 13, 14 11, 12, 13, 14, 15, 16 Postgres Clusters ✔ ✔ ✔ ✔ ✔ Streaming replication ✔ ✔ ✔ ✔ ✔ Supports Extensions ✔ ✔ ✔ ✔ ✔

Slide 106

Slide 106 text

Operator CloudNativePG Crunchy Postgres for Kubernetes OnGres StackGres KubeDB Zalando Postgres Operator Hot Standby ✔ ✔ ✔ ✔ ✔ Warm Standby ✔ ✔ ✔ ✔ ✔ Automatic Failover ✔ ✔ ✔ ✔ ✔ Continuous Archiving ✔ ✔ ✔ ✔ ✔ Restore from  WAL archive ✔ ✔ ✔ ✔ ✔ Supports PITR ✔ ✔ ✔ ✔ ✔ Manual backups ✔ ✔ ✔ ✔ ✔ Scheduled backups ✔ ✔ ✔ ✔ ✔

Slide 107

Slide 107 text

Operator CloudNativePG Crunchy Postgres for Kubernetes OnGres StackGres KubeDB Zalando Postgres Operator Backups via Kubernetes ✔ ✘ ✔ ✔ ✘ Custom resources ✔ ✔ ✔ ✔ ✔ Uses default PG images ✘ ✔ ✔ ✘ ✘ CLI access ✔ ✔ ✔ ✔ ✘ WebUI ✘ ✘ ✔ ✔ ✘ Tolerations ✔ ✔ ✔ ✔ ✔ Node af fi nity ✔ ✔ ✔ ✔ ✔

Slide 108

Slide 108 text

Operator https://www.simplyblock.io/post/choosing-a-postgres-kubernetes-operator https://operatorhub.io/?keyword=postgres

Slide 109

Slide 109 text

Pinning and Tainting

Slide 110

Slide 110 text

Always use speci fi c, dedicated machines for your database. Pinning and Tainting

Slide 111

Slide 111 text

Always use speci fi c, dedicated machines for your database. Pinning and Tainting (except you’re running super small databases)

Slide 112

Slide 112 text

Always use speci fi c, dedicated machines for your database. Pin your database containers to those hosts. Pinning and Tainting (except you’re running super small databases)

Slide 113

Slide 113 text

Always use speci fi c, dedicated machines for your database. Pin your database containers to those hosts. Taint the hosts to prevent anything else from running on it. Pinning and Tainting (except you’re running super small databases)

Slide 114

Slide 114 text

Slide 115

Slide 115 text

Trust me, I’m Kelsey! https://x.com/kelseyhightower/status/1624081136073994240

Slide 116

Slide 116 text

Trust me, I’m Kelsey! https://x.com/kelseyhightower/status/1624081136073994240

Slide 117

Slide 117 text

More Resources Data on Kubernetes Community: https://dok.community Data on Kubernetes Whitepaper

Slide 118

Slide 118 text

Thank you very much! Questions? @noctarius2k @noctarius2k@mastodon.online @noctarius.com