Slide 1

Slide 1 text

Biggish Data with Rails and Postgres Ben Curtis / [email protected] / @stympy

Slide 2

Slide 2 text

Yeah, I’m cool like that

Slide 3

Slide 3 text

Setting the Stage So you want to store some data…

Slide 4

Slide 4 text

Use a Real Computer Big disks — Lots of RAM

Slide 5

Slide 5 text

Increase Read-Ahead blockdev --setra 2048 /dev/sda

Slide 6

Slide 6 text

Use a modern filesystem 2ndquadrant.com/media/pdfs/talks/RightingWrites.pdf

Slide 7

Slide 7 text

Tell PG about all that RAM github.com/gregs1104/pgtune

Slide 8

Slide 8 text

Vacuum Regularly devcenter.heroku.com/articles/heroku-postgres-database-tuning

Slide 9

Slide 9 text

Don’t dump it, ship it github.com/wal-e/wal-e

Slide 10

Slide 10 text

Some Common Problems PG, Y U SLOW?

Slide 11

Slide 11 text

$ rails console > Error.count()

Slide 12

Slide 12 text

db=# EXPLAIN (format yaml) SELECT count(*) FROM errors; QUERY PLAN -------------------------------------- - Plan: + Node Type: "Aggregate" + Total Cost: 49971.71 + Plan Rows: 1 + Plans: + - Node Type: "Seq Scan" + Relation Name: “errors" + Total Cost: 48172.96 + Plan Rows: 719496 +

Slide 13

Slide 13 text

$ rails console > user.errors.page(100)

Slide 14

Slide 14 text

db=# EXPLAIN ANALYZE SELECT id FROM errors OFFSET 500000 LIMIT 100; ! QUERY PLAN -------------------------------------- - Plan: + Node Type: "Limit" + Actual Rows: 100 + Plans: + - Node Type: "Seq Scan" + Relation Name: "errors" + Actual Rows: 500100 +

Slide 15

Slide 15 text

db=# EXPLAIN ANALYZE SELECT msg FROM errors WHERE id >= 500000 AND id < 500100; ! QUERY PLAN -------------------------------------------------------- - Plan: + Node Type: "Bitmap Heap Scan" + Relation Name: "errors" + Actual Rows: 100 + Plans: + - Node Type: "Bitmap Index Scan" + Index Name: "errors_pkey" + Total Cost: 5.42 + Actual Rows: 100 + Index Cond: "((id >= 500000) AND (id <= 500100))"

Slide 16

Slide 16 text

Long-running Migrations The only constant in life is change

Slide 17

Slide 17 text

class AddIndexToAsksActive < ActiveRecord::Migration disable_ddl_transaction! ! def change add_index :asks, :active, algorithm: :concurrently end end *Rails >= 4

Slide 18

Slide 18 text

Deadlock? Wha?

Slide 19

Slide 19 text

Too Many Locks wiki.postgresql.org/wiki/Lock_Monitoring

Slide 20

Slide 20 text

Intensive DB Queries www.postgresql.org/docs/current/static/warm-standby.html

Slide 21

Slide 21 text

Too Many DB Connections www.craigkerstiens.com/2014/05/22/on-connection-pooling

Slide 22

Slide 22 text

$ rails console > Error.where(“created_at < ‘1/1/2012’”). delete_all

Slide 23

Slide 23 text

$ rails console > Error.where(project_id: 5).delete_all

Slide 24

Slide 24 text

Partitioning for deletion & archival github.com/keithf4/pg_partman

Slide 25

Slide 25 text

Distributed & Unique IDs www.wekeroad.com/2014/05/29/a-better-id-generator- for-postgresql

Slide 26

Slide 26 text

Don’t fear the elephant

Slide 27

Slide 27 text

Thanks! Ben Curtis / bencurtis.com / @stympy speakerdeck.com/stympy