PGCon 2013 Keynote - Postgres at Disqus

Slide 1

Slide 1 text

Mike Clarke @mikeclarke Postgres at Disqus

Slide 2

Slide 2 text

what is disqus

Slide 3

Slide 3 text

what is disqus

Slide 4

Slide 4 text

the present

Slide 5

Slide 5 text

where we use Postgres ๏ All comment data ๏ Comments ๏ Threads ๏ Forums ๏ Imports ๏ All user data ๏ Users ๏ Proﬁles

Slide 6

Slide 6 text

where we DON’T we use it ๏ Stats and counters (Cassandra) ๏ Eventual consistency ๏ Availability ๏ Queue (RabbitMQ) ๏ Compatibility with existing libraries ๏ Availability

Slide 7

Slide 7 text

our main cluster ๏ 3+ TB of data ๏ Hardware ๏ 6 node cluster (1 master, 1 failover, 4 read-only slaves) ๏ 384G RAM ๏ 16xRAID10 800G SSDs ๏ Previously 32xRAID10 15K RPM drives ๏ Dual 2.9GHz OctoCore CPUs

Slide 8

Slide 8 text

our workload ๏ 50,000 transactions / second ๏ Even more queries / second ๏ millions of new posts + threads daily

Slide 9

Slide 9 text

the power of SSDs (before)

Slide 10

Slide 10 text

the power of SSDs (before)

Slide 11

Slide 11 text

the power of SSDs (after)

Slide 12

Slide 12 text

the power of SSDs (after)

Slide 13

Slide 13 text

pain points

Slide 14

Slide 14 text

logical replication ๏ Love + Hate for Slony ๏ Killer feature: ﬂexible replication sets ๏ Still running a patched 2.0.3 release ๏ Hate the lock-in & using 3rd party tool ๏ Tremendous risk associated with upgrade ๏ Unbelievably excited for BDR

Slide 15

Slide 15 text

connecting to postgres ๏ Lots of possibilities ๏ Connect directly to Postgres ๏ local pgBouncer - Postgres ๏ pgBouncer - pgBouncer - Postgres ๏ pgBouncer - HAProxy - Postgres ๏ HAProxy - pgBouncer - Postgres ๏ Zen of Python ๏ “There should be one-- and preferably only one --obvious way to do it.”

Slide 16

Slide 16 text

the future

Slide 17

Slide 17 text

a parallel to nginx ๏ right tool for the job... ๏ ...isn’t always obvious ๏ nginx can be used for: ๏ streaming HTTP connections ๏ resizing images on the ﬂy ๏ caching & transforming HTTP responses ๏ and more!

Slide 18

Slide 18 text

extensions

Slide 19

Slide 19 text

building your app on your database vs. building your app inside your database

Slide 20

Slide 20 text

real extensions ๏ already a bunch of awesome extensions today ๏ PostGIS (huge di erentiator for PG) ๏ pg_repack ๏ pl/v8 ๏ hstore ๏ ... and others, but we can do better

Slide 21

Slide 21 text

“magical pony” extensions ๏ PL/Proxy-style query routing ๏ move partitioning logic into postgres ๏ query progress indicator ๏ helps set biz analyst expectations ๏ pg_autoscale ๏ tools a la Heroku for everyone ๏ INDEXED BY support in SQL

Slide 22

Slide 22 text

tl;dr 1. SSDs are fantastic 2. connections & replication 3. extensions are a honking great idea -- let’s do more!

Slide 23

Slide 23 text

thanks! @mikeclarke