Postgres on Kubernetes - Dos and Don'ts

Running databases in containers has been the biggest anti-pattern of the last decade. The world, however, moves on and stateful container workloads become more common, and so do databases in Kubernetes. People love the additional convenience when it comes to deployment, scalability, and operation.

With PostgreSQL on its way to become the world’s most beloved database, there certainly are quite some things to keep in mind when running it on k8s. Let us evaluate the important Dos and especially the Don’ts.

Presentation by Chris Engelbert from simplyblock (https://www.simplyblock.io).

Transcript

1. https://www.marketingdonut.co.uk/pr-and-promotion/exhibitions/dos-and-don-ts-when-exhibiting
2. PostgreSQL Kubernetes ❤
3. Chris Engelbert Devrel @ simplyblock Previous fun companies: - Ubisoft / Blue Byte - Hazelcast - Instana - clevabit - Timescale Interests: - Developer Relations - Anything Performance Engineering - Backend Technologies - Fairy Tales (AMD, Intel, Nvidia) @noctarius2k @noctarius2k@mastodon.online @noctarius.com
6. Question 01
7. Question 01 Why you shouldn't run a database in Kubernetes?
8. Why not to run a database in Kubernetes?
9. Why not to run a database in Kubernetes?
10. Why not to run a database in Kubernetes? K8s is not designed with Databases in mind!
11. Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s!
12. Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s! Persistent Data will kill you! Too slow!
13. Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s! Persistent Data will kill you! Too slow! Nobody understands Kubernetes!
14. Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s! Persistent Data will kill you! Too slow! Nobody understands Kubernetes! What’s the bene fi t; databases don’t need autoscaling!
15. Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s! Persistent Data will kill you! Too slow! Nobody understands Kubernetes! What’s the bene fi t; databases don’t need autoscaling! Databases and applications should be separated!
16. Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s! Persistent Data will kill you! Too slow! Nobody understands Kubernetes! What’s the bene fi t; databases don’t need autoscaling! Databases and applications should be separated! Not another layer of indirection / abstraction!
17. Why not to run a database in Kubernetes? K8s is not designed with Databases in mind! Never run Stateful Workloads in k8s! Persistent Data will kill you! Too slow! Nobody understands Kubernetes! What’s the bene fi t; databases don’t need autoscaling! Databases and applications should be separated! Not another layer of indirection / abstraction!
18. Why not to run a database in Kubernetes? BURN IN HELL!
19. The Happy Place
20. Where are my gamers at? So we need to cheat!?
21. The Happy Place
22. Why?
23. No Cloud-Vendor Lock-In Why?
24. No Cloud-Vendor Lock-In Faster Time To Market Why?
25. No Cloud-Vendor Lock-In Faster Time To Market Decreasing cost Why?
26. No Cloud-Vendor Lock-In Faster Time To Market Decreasing cost Automation Why?
27. No Cloud-Vendor Lock-In Faster Time To Market Decreasing cost Automation Uni fi ed deployment architecture Why?
28. No Cloud-Vendor Lock-In Faster Time To Market Decreasing cost Automation Uni fi ed deployment architecture Need read-only replicas Why?
29. Let’s get something out of the way fi rst!
30. Call the Police!
31. Enable TLS Call the Police!
32. Enable TLS Use Kubernetes Secrets Call the Police!
33. Enable TLS Use Kubernetes Secrets Use Cert-Manager Call the Police!
34. Enable TLS Use Kubernetes Secrets Use Cert-Manager Encrypt Data-At-Rest Call the Police!
36. Enable TLS Use Kubernetes Secrets Use Cert-Manager Encrypt Data-At-Rest Call the Police!
37. Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/
38. You want Continuous Backup and PITR Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/
39. You want Continuous Backup and PITR Roll your own pg_basebackup or pg_dump (don’t!) Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/
40. You want Continuous Backup and PITR Roll your own pg_basebackup or pg_dump (don’t!) Use tools like pgbackrest, barman, PGHoard, … Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/
41. You want Continuous Backup and PITR Roll your own pg_basebackup or pg_dump (don’t!) Use tools like pgbackrest, barman, PGHoard, … Upload backups to S3? Cost! Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/
42. You want Continuous Backup and PITR Roll your own pg_basebackup or pg_dump (don’t!) Use tools like pgbackrest, barman, PGHoard, … Upload backups to S3? Cost! Backup and Recovery https://www.ovhcloud.com/de/bare-metal/backup-storage/ 😅 Test Your Backups 😅
43. PostgreSQL Con fi guration
44. PostgreSQL Con fi guration The PostgreSQL Con fi guration isn’t too much in fl uenced
45. shared_bu ff ers (maintenance_)work_mem e ff ective_cache_size PostgreSQL Con fi guration The PostgreSQL Con fi guration isn’t too much in fl uenced
46. shared_bu ff ers (maintenance_)work_mem e ff ective_cache_size PostgreSQL Con fi guration The PostgreSQL Con fi guration isn’t too much in fl uenced Use Huge Pages!
47. PostgreSQL Con fi guration https://www.youtube.com/watch?v=S0LEDGbAnn8 https://www.crunchydata.com/blog/optimize-postgresql-server-performance https://www.percona.com/blog/using-huge-pages-with-postgresql-running-inside-kubernetes/
48. Do you need more? Extensions!
49. Do you need PG Extensions? Do you need more? Extensions!
50. Do you need PG Extensions? Do you need more? Extensions! Is the extension part of the container image?
51. Do you need PG Extensions? Do you need more? Extensions! Is the extension part of the container image? If not, you need to build your own layer…
52. Do you need PG Extensions? Do you need more? Extensions! Is the extension part of the container image? If not, you need to build your own layer… or use some magic (more on this later).
53. Keep an Eye on PG and Kubernetes Versions Versions and Updates
54. So What is important or di ff erent?
55. Storage
56. Storage
57. Use Persistent Volumes Storage
58. Use Persistent Volumes Storage (local volumes are a bad idea)
59. Use Persistent Volumes Storage Should be dynamically provisioned (local volumes are a bad idea)
60. Use Persistent Volumes Storage Should be dynamically provisioned CSI provider enables encryption at rest (local volumes are a bad idea)
61. Use Persistent Volumes Storage Should be dynamically provisioned CSI provider enables encryption at rest High IOPS (SSD or NVMe) (local volumes are a bad idea)
62. Use Persistent Volumes Storage Should be dynamically provisioned CSI provider enables encryption at rest High IOPS (SSD or NVMe) Low Latency (local volumes are a bad idea)
63. Use Persistent Volumes Storage Should be dynamically provisioned CSI provider enables encryption at rest High IOPS (SSD or NVMe) Low Latency Database performance is as fast as your storage (local volumes are a bad idea)
64. Use Persistent Volumes Storage Should be dynamically provisioned CSI provider enables encryption at rest High IOPS (SSD or NVMe) Low Latency Database performance is as fast as your storage (local volumes are a bad idea) I’d recommend a disaggregated storage!
65. Storage www.storageclass.info/csidrivers
66. Requests, Limits, and Quotas Capacity Limits Requests Used
67. Requests, Limits, and Quotas Capacity Limits Requests Used Use Resource Requests, Limits, Quotas
68. Requests, Limits, and Quotas CPU and memory requests need to be accurate
 to prevent contention and ensure predictable performance Capacity Limits Requests Used Use Resource Requests, Limits, Quotas
69. Requests, Limits, and Quotas CPU and memory requests need to be accurate
 to prevent contention and ensure predictable performance Capacity Limits Requests Used https://codimite.ai/blog/kubernetes-resources-and-scaling-a-beginners-guide/ Use Resource Requests, Limits, Quotas
70. Make it big! Enable Huge Pages!
71. Make it big! Enable Huge Pages! In your OS and the Resource Descriptor.
72. Make it big! Enable Huge Pages! In your OS and the Resource Descriptor. https://www.percona.com/blog/using-huge-pages-with-postgresql-running-inside-kubernetes/
73. Resiliency and Overhead
74. Resiliency and Overhead High Availability
75. Patroni, repmgr, pg_auto_failover, … Resiliency and Overhead High Availability
76. Patroni, repmgr, pg_auto_failover, … Resiliency and Overhead High Availability https://medium.com/@kristi.anderson/whats-the-best-postgresql-high-availability-framework...
77. Resiliency and Overhead
78. Resiliency and Overhead Connection Pooling
79. Never use PostgreSQL without Connection Pooling! Resiliency and Overhead Connection Pooling
80. Never use PostgreSQL without Connection Pooling! Optimizes Overhead and Resource Utilization Resiliency and Overhead Connection Pooling
81. Never use PostgreSQL without Connection Pooling! Optimizes Overhead and Resource Utilization Handles failovers, central switching of Primary Resiliency and Overhead Connection Pooling
82. Never use PostgreSQL without Connection Pooling! Optimizes Overhead and Resource Utilization Handles failovers, central switching of Primary Enables easy use of Read-Replicas Resiliency and Overhead Connection Pooling
83. Never use PostgreSQL without Connection Pooling! Optimizes Overhead and Resource Utilization Handles failovers, central switching of Primary Enables easy use of Read-Replicas Resiliency and Overhead Connection Pooling PgBouncer, PgPool-II, pgagroal, PgCat, Odyssey, …
84. Never use PostgreSQL without Connection Pooling! Optimizes Overhead and Resource Utilization Handles failovers, central switching of Primary Enables easy use of Read-Replicas Resiliency and Overhead Connection Pooling PgBouncer, PgPool-II, pgagroal, PgCat, Odyssey, … https://tembo.io/blog/postgres-connection-poolers
85. Where’s my Replicant?
86. Where’s my Replicant? Use available Kubernetes features
87. Where’s my Replicant? Use available Kubernetes features StatefulSet
88. Where’s my Replicant? Use available Kubernetes features StatefulSet
89. Networking and Access Control https://timeclock365.com/tc22-door-access-controller/
90. Use Network Policies Networking and Access Control https://timeclock365.com/tc22-door-access-controller/
91. Use Network Policies Enable TLS (you remember?!) Networking and Access Control https://timeclock365.com/tc22-door-access-controller/
92. Use Network Policies Enable TLS (you remember?!) Setup Security Policies Networking and Access Control https://timeclock365.com/tc22-door-access-controller/
93. Use Network Policies Enable TLS (you remember?!) Setup Security Policies Con fi gure RBAC (Role-Based Access Control) Networking and Access Control https://timeclock365.com/tc22-door-access-controller/
94. Use Network Policies Enable TLS (you remember?!) Setup Security Policies Con fi gure RBAC (Role-Based Access Control) Networking and Access Control Think about a policy manager such as OPA or kyverno https://timeclock365.com/tc22-door-access-controller/
95. Observability and Alerting
96. Observability and Alerting Like anything cloud, make sure you have monitoring (meaning observability) and alerting!
97. Prometheus Exporter, Log Collector, Aggregation, Analysis, Traceability, … Observability and Alerting Like anything cloud, make sure you have monitoring (meaning observability) and alerting!
98. Prometheus Exporter, Log Collector, Aggregation, Analysis, Traceability, … Observability and Alerting Like anything cloud, make sure you have monitoring (meaning observability) and alerting! Datadog, Instana, DynaTrace, Grafana, …
99. Operator
100. Operator Use a Postgres Kubernetes Operator
101. Operator Use a Postgres Kubernetes Operator Handles or con fi gures many of the typical tasks (HA, backup, …)
102. Operator Use a Postgres Kubernetes Operator Handles or con fi gures many of the typical tasks (HA, backup, …) Brings cloud-nativeness to PG
103. Operator Use a Postgres Kubernetes Operator Handles or con fi gures many of the typical tasks (HA, backup, …) Brings cloud-nativeness to PG Integrates PG into k8s
104. Operator If not, use Helm Charts Use a Postgres Kubernetes Operator Handles or con fi gures many of the typical tasks (HA, backup, …) Brings cloud-nativeness to PG Integrates PG into k8s
105. Operator CloudNativePG Crunchy Postgres for Kubernetes OnGres StackGres KubeDB Zalando Postgres Operator Supported versions 12, 13, 14, 15, 16 11, 12, 13, 14, 15, 16 12, 13, 14, 15, 16 9.6, 10, 11, 12, 13, 14 11, 12, 13, 14, 15, 16 Postgres Clusters ✔ ✔ ✔ ✔ ✔ Streaming replication ✔ ✔ ✔ ✔ ✔ Supports Extensions ✔ ✔ ✔ ✔ ✔
106. Operator CloudNativePG Crunchy Postgres for Kubernetes OnGres StackGres KubeDB Zalando Postgres Operator Hot Standby ✔ ✔ ✔ ✔ ✔ Warm Standby ✔ ✔ ✔ ✔ ✔ Automatic Failover ✔ ✔ ✔ ✔ ✔ Continuous Archiving ✔ ✔ ✔ ✔ ✔ Restore from
 WAL archive ✔ ✔ ✔ ✔ ✔ Supports PITR ✔ ✔ ✔ ✔ ✔ Manual backups ✔ ✔ ✔ ✔ ✔ Scheduled backups ✔ ✔ ✔ ✔ ✔
107. Operator CloudNativePG Crunchy Postgres for Kubernetes OnGres StackGres KubeDB Zalando Postgres Operator Backups via Kubernetes ✔ ✘ ✔ ✔ ✘ Custom resources ✔ ✔ ✔ ✔ ✔ Uses default PG images ✘ ✔ ✔ ✘ ✘ CLI access ✔ ✔ ✔ ✔ ✘ WebUI ✘ ✘ ✔ ✔ ✘ Tolerations ✔ ✔ ✔ ✔ ✔ Node af fi nity ✔ ✔ ✔ ✔ ✔
108. Operator https://www.simplyblock.io/post/choosing-a-postgres-kubernetes-operator https://operatorhub.io/?keyword=postgres
109. Pinning and Tainting
110. Always use speci fi c, dedicated machines for your database. Pinning and Tainting
111. Always use speci fi c, dedicated machines for your database. Pinning and Tainting (except you’re running super small databases)
112. Always use speci fi c, dedicated machines for your database. Pin your database containers to those hosts. Pinning and Tainting (except you’re running super small databases)
113. Always use speci fi c, dedicated machines for your database. Pin your database containers to those hosts. Taint the hosts to prevent anything else from running on it. Pinning and Tainting (except you’re running super small databases)
114. Always use speci fi c, dedicated machines for your database. Pin your database containers to those hosts. Taint the hosts to prevent anything else from running on it. Pinning and Tainting (except you’re running super small databases) (except the minimum necessary Kubernetes services, like KubeProxy)
115. Trust me, I’m Kelsey! https://x.com/kelseyhightower/status/1624081136073994240
116. Trust me, I’m Kelsey! https://x.com/kelseyhightower/status/1624081136073994240
117. More Resources Data on Kubernetes Community: https://dok.community Data on Kubernetes Whitepaper
118. Thank you very much! Questions? @noctarius2k @noctarius2k@mastodon.online @noctarius.com