Slide 1

Slide 1 text

1

Slide 2

Slide 2 text

Agenda for this Citus & Patroni talk @ Citus Con: An Event for Postgres 2023 ▪ What (business)problems do we solve? ▪ High Availability with Patroni ▪ Horizontal scalability with Citus ▪ PostgreSQL + Citus + Patroni for the win ▪ Live demo! ▪ Conclusion 1 2 3 4 5 6

Slide 3

Slide 3 text

As an application developer I want... • A database 3

Slide 4

Slide 4 text

As an application developer I want... • A database • A highly available database 4

Slide 5

Slide 5 text

As an application developer I want... • A database • A highly available database • A highly available database that scales 5

Slide 6

Slide 6 text

As an application developer I want... • A database • A highly available database • A highly available database that scales • A highly available database that scales horizontally! 6

Slide 7

Slide 7 text

As an application developer I want... • A database • A highly available database • A highly available database that scales • A highly available database that scales horizontally! • Highly available database that scales horizontally for free! 7

Slide 8

Slide 8 text

PostgreSQL: The World's Most Advanced Open Source Relational Database 8

Slide 9

Slide 9 text

High availability with Patroni 9 Primary Standby Etcd cluster

Slide 10

Slide 10 text

Vertical Scalability 10 Primary Standby Etcd cluster

Slide 11

Slide 11 text

Horizontal Scalability (sharding) 11 Primary Standby Primary Standby Primary Standby

Slide 12

Slide 12 text

Application Level Sharding 12 Shard 1 Shard 2 Shard 3 Shard 4 Application SELECT * FROM u WHERE id=4 SELECT * FROM u WHERE id=5 SELECT * FROM u WHERE id=6 SELECT * FROM u WHERE id=7 ID%4 == 0 ID%4 == 1 ID%4 == 2 ID%4 == 3

Slide 13

Slide 13 text

Disadvantages of Application Level Sharding • Must be implemented in the application • Separate connection (pool) for every shard • Must be introduced from day 1 • Depending on implementation adding new shards is either impossible or hard • If new shards are added, they are heavily disbalanced • Rebalancing (without downtime) is a challenge • Schema changes must be deployed on all shards 13

Slide 14

Slide 14 text

Sharding with Citus 14 Worker 1 Worker 2 Worker 3 Worker 4 Application Coordinator

Slide 15

Slide 15 text

Sharding with Citus: Pros & Cons Pros • Transparent sharding • A single connection (pool) • Query from any node! • Schema changes via coordinator • Start with small and scale-out • Rebalancing without downtime! 15 Cons • Increased query latency

Slide 16

Slide 16 text

Citus HA with Patroni 3.0 16 primary standby Citus worker 1 primary standby Citus coordinator primary standby Citus worker 2

Slide 17

Slide 17 text

Enabling Citus support in Patroni 3.0 scope: my-citus-cluster # must be the same on all nodes citus: group: X # 0 for coordinator and 1, 2, 3, etc for workers database: citus # must be the same on all nodes 17

Slide 18

Slide 18 text

Adding new workers to the Citus cluster 1. Adjust patroni.yaml, citus.group parameter 2. Start Patroni on the new node 3. SELECT * FROM citus_rebalance_start(); -- rebalance shards 4. SELECT * FROM citus_rebalance_status(); -- check rebalance progress 18

Slide 19

Slide 19 text

DEMO 19

Slide 20

Slide 20 text

20 Conclusion

Slide 21

Slide 21 text

Conclusion • Enterprise-grade sharding with Citus • Enterprise-grade HA with Patroni • Everything is open-source! • Start with small and scale • Vertically • Horizontally 21

Slide 22

Slide 22 text

Thank you! • Citus open source repo on GitHub aka.ms/citus • Patroni repo on GitHub github.com/zalando/patroni • Alexander on Twitter @cyberdemn