PGCon 2013 Keynote - Postgres at Disqus

PGCon 2013 Keynote - Postgres at Disqus

Overview of how Disqus uses Postgres at scale, some of the pain points we've encountered, and where we see the future of Postgres going in our use case.

E4a87fe047af5a706b99de6d3d81939a?s=128

Mike Clarke

May 23, 2013
Tweet

Transcript

  1. 5.

    where we use Postgres ๏ All comment data ๏ Comments

    ๏ Threads ๏ Forums ๏ Imports ๏ All user data ๏ Users ๏ Profiles
  2. 6.

    where we DON’T we use it ๏ Stats and counters

    (Cassandra) ๏ Eventual consistency ๏ Availability ๏ Queue (RabbitMQ) ๏ Compatibility with existing libraries ๏ Availability
  3. 7.

    our main cluster ๏ 3+ TB of data ๏ Hardware

    ๏ 6 node cluster (1 master, 1 failover, 4 read-only slaves) ๏ 384G RAM ๏ 16xRAID10 800G SSDs ๏ Previously 32xRAID10 15K RPM drives ๏ Dual 2.9GHz OctoCore CPUs
  4. 8.

    our workload ๏ 50,000 transactions / second ๏ Even more

    queries / second ๏ millions of new posts + threads daily
  5. 14.

    logical replication ๏ Love + Hate for Slony ๏ Killer

    feature: flexible replication sets ๏ Still running a patched 2.0.3 release ๏ Hate the lock-in & using 3rd party tool ๏ Tremendous risk associated with upgrade ๏ Unbelievably excited for BDR
  6. 15.

    connecting to postgres ๏ Lots of possibilities ๏ Connect directly

    to Postgres ๏ local pgBouncer - Postgres ๏ pgBouncer - pgBouncer - Postgres ๏ pgBouncer - HAProxy - Postgres ๏ HAProxy - pgBouncer - Postgres ๏ Zen of Python ๏ “There should be one-- and preferably only one --obvious way to do it.”
  7. 17.

    a parallel to nginx ๏ right tool for the job...

    ๏ ...isn’t always obvious ๏ nginx can be used for: ๏ streaming HTTP connections ๏ resizing images on the fly ๏ caching & transforming HTTP responses ๏ and more!
  8. 20.

    real extensions ๏ already a bunch of awesome extensions today

    ๏ PostGIS (huge di erentiator for PG) ๏ pg_repack ๏ pl/v8 ๏ hstore ๏ ... and others, but we can do better
  9. 21.

    “magical pony” extensions ๏ PL/Proxy-style query routing ๏ move partitioning

    logic into postgres ๏ query progress indicator ๏ helps set biz analyst expectations ๏ pg_autoscale ๏ tools a la Heroku for everyone ๏ INDEXED BY support in SQL
  10. 22.

    tl;dr 1. SSDs are fantastic 2. connections & replication 3.

    extensions are a honking great idea -- let’s do more!