Beyond validates_presence_of: ensuring eventual consistency in distributed systems

Slide 1

Slide 1 text

Beyond :validates_presence_of

Slide 2

Slide 2 text

Hello! I am amy @cdwort

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

What I hope you’ll learn Causes Why would I let my data go wrong? Prevention How can I prevent data issues before I persist my data? Detection If things do go wrong, how can I make sure I know?

Slide 6

Slide 6 text

1. Causes How your data goes wrong

Slide 7

Slide 7 text

“ And why would you expect your data to be correct?

Slide 8

Slide 8 text

Tools for data correctness ▪ Database constraints and indexes ▪ ORM code

Slide 9

Slide 9 text

Database constraints and indexes

Slide 10

Slide 10 text

ALTER TABLE product ADD billing_record_id, int NOT NULL;

Slide 11

Slide 11 text

ORM code

Slide 12

Slide 12 text

class Product :validates_presence_of :billing_record end

Slide 13

Slide 13 text

class Product validates: :billing_record, presence: true end

Slide 14

Slide 14 text

JOBS!

Slide 15

Slide 15 text

class Product # :validates_presence_of :billing_record end def create @product = Product.new(params[:product]) Job::BillingCreator.enqueue(@product.id) respond_with @product.to_json end

Slide 16

Slide 16 text

SERVICES!

Slide 17

Slide 17 text

class Job::BillingCreator def perform(id) product = Product.find(id) data = { product: product.uuid, user: product.user.uuid } BillingServiceClient.new.post(data) end end

Slide 18

Slide 18 text

TCP!

Slide 19

Slide 19 text

Are you distinguishing your network failures? ▪ Cannot connect. ▪ Service partially completes work; never responds. ▪ Service completes work; network cuts out at response. ▪ Service completes work; sees client side timeout and rolls back.

Slide 20

Slide 20 text

“ What if my data cannot always be correct?

Slide 21

Slide 21 text

2. Prevention How to keep your data sane

Slide 22

Slide 22 text

Retry ▪ Idempotency ▪ Locks ▪ Creates OR Deletes ▪ Exponential Backoff & Circuit breakers

Slide 23

Slide 23 text

Roll Back ▪ Great if only your code has seen the action. ▪ What about external systems?

Slide 24

Slide 24 text

Roll Forward ▪ Job enqueued? ▪ Service called?

Slide 25

Slide 25 text

“ Okay, cool, what does this look like?

Slide 26

Slide 26 text

Transactions ▪ What strategy? ▪ Consider: Add your job queue to your database

Slide 27

Slide 27 text

Add Timestamps ▪ One timestamp per critical service call. ▪ Set this column in the same transaction as the service call.

Slide 28

Slide 28 text

Code Organization ▪ Write your failure code in the same place as your success code.

Slide 29

Slide 29 text

EmployeeCreator API Job

Slide 30

Slide 30 text

EmployeeCreator API Job Herokai Creator Engineer Creator

Slide 31

Slide 31 text

EmployeeCreator API Job Herokai Creator Engineer Creator

Slide 32

Slide 32 text

3. Detection Knowing when things have gone wrong

Slide 33

Slide 33 text

SQL with Timestamps

Slide 34

Slide 34 text

ALTER TABLE product ADD billing_record_id, int NOT NULL;

Slide 35

Slide 35 text

-- Things we have sold that have been cancelled that we are still billing for. SELECT * FROM billing_records, products WHERE billing_records.deleted_at IS NULL AND products.deleted_at IS NOT NULL AND billing_records.products_id = products.id AND products.deleted_at < NOW() - interval '15 minutes'|)

Slide 36

Slide 36 text

SQL with Timestamps ▪ SQL ▪ Automatic runs ▪ Drag-and-drop folders ▪ Alerting by default ▪ Documented remediation plans

Slide 37

Slide 37 text

Challenges ▪ Non-SQL stores

Slide 38

Slide 38 text

API API API API

Slide 39

Slide 39 text

API API API API

Slide 40

Slide 40 text

API API API API ETL

Slide 41

Slide 41 text

API API API API Auditing Code

Slide 42

Slide 42 text

Challenges ▪ Non-SQL stores ▫ Pull, transform & cache ▫ Write code, not SQL ▪ No timestamps ▫ Analysis events table ▫ SQL to determine coalescing

Slide 43

Slide 43 text

Event Stream

Slide 44

Slide 44 text

Buying a Heroku Add-on Someone wants to buy some Redis! Are they authenticated? Is that product available?

Slide 45

Slide 45 text

... Redis cluster provisioned Billing started User Response generated

Slide 46

Slide 46 text

Event Stream benefits ▪ Single format ▪ Black box testing

Slide 47

Slide 47 text

Event Stream challenges ▪ What if you emit the wrong events? ▪ What if you continue emitting events without doing the work? ▪ What if the stream consumer code is wrong?

Slide 48

Slide 48 text

HIGH RISK

Slide 49

Slide 49 text

What I hope you’ll learn Causes Why would I let my data go wrong? Prevention How can I prevent data issues before I persist my data? Detection If things do go wrong, how can I make sure I know?

Slide 50

Slide 50 text

Thanks!! amy @cdwort

Slide 51

Slide 51 text

Questions?

Slide 52

Slide 52 text

1.00 - 2.00 pm Heroku booth

Slide 53

Slide 53 text

Thanks!! amy @cdwort