Slide 1

Slide 1 text

Distributed ID Generation @nathankleyn

Slide 2

Slide 2 text

or, how to make your ID generation less like this:

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

and more like this:

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Requirements

Slide 7

Slide 7 text

3 main requirements

Slide 8

Slide 8 text

1 an ID for every event

Slide 9

Slide 9 text

2 IDs must be unique

Slide 10

Slide 10 text

3 it must scale well

Slide 11

Slide 11 text

Theory

Slide 12

Slide 12 text

Time

Slide 13

Slide 13 text

Time is hard. Really hard. (Diamond. Hard. See what I did there?)

Slide 14

Slide 14 text

Time oracles could solve this problem.

Slide 15

Slide 15 text

(No, not that oracle)

Slide 16

Slide 16 text

Time oracles could solve this problem.

Slide 17

Slide 17 text

NTP tries to solve this problem for the “rest of us”.

Slide 18

Slide 18 text

(No, not that NTP?)

Slide 19

Slide 19 text

NTP tries to solve this problem for the “rest of us”.

Slide 20

Slide 20 text

However, expect ±10ms. (at least)

Slide 21

Slide 21 text

That’s ±10ms per machine.

Slide 22

Slide 22 text

Hey Bob, what’s the time? It’s 1970! Back in my day... Fucking Bob always thinks it’s 1970. It’s clearly 1980. Look at my hair! No, Karen. It’s 1990. I’m not wearing this antiquated Disney shirt for kicks.

Slide 23

Slide 23 text

Time can move backwards or forwards.

Slide 24

Slide 24 text

1s ≠ 1000ms

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

Representing Numbers In Binary

Slide 27

Slide 27 text

Java has many numeric data types.

Slide 28

Slide 28 text

However it has no unsigned variants.* This is no longer strictly true in Java 8, however it’s a trick: they’re still signed types, but now there’s a bunch of functions to do unsigned operations on them (eg. unsignedDivide).

Slide 29

Slide 29 text

So in Java the MSB is for the sign: 1000000000000 1 when negative and 0 when positive. So the above is a short and is -4096.

Slide 30

Slide 30 text

Some languages make it hard to use the signed bit.

Slide 31

Slide 31 text

A Java long is 64-bits. So we have 63 usable bits.

Slide 32

Slide 32 text

The Epoch

Slide 33

Slide 33 text

An epoch is a marker of time relative to true time.

Slide 34

Slide 34 text

Unix time is an epoch measured relative to 00:00:00.000 1/1/1970.

Slide 35

Slide 35 text

We can define our own epoch.

Slide 36

Slide 36 text

So in 1 year it will be 31,536,000. Unix time will be ~1,456,348,092,000.

Slide 37

Slide 37 text

Why is this useful? Because it allows us to compress the storage of time.

Slide 38

Slide 38 text

Redis

Slide 39

Slide 39 text

Redis is awesome, fast and stable.

Slide 40

Slide 40 text

It supports scripting via Lua.

Slide 41

Slide 41 text

We can create a Lua script to make an ID inside Redis.

Slide 42

Slide 42 text

Redis is not distributed.* Redis clustering will arrive in v3.

Slide 43

Slide 43 text

We can round-robin between a bunch of Redis servers to achieve distribution.

Slide 44

Slide 44 text

k-sorting

Slide 45

Slide 45 text

k-sorting = “roughly sorting”

Slide 46

Slide 46 text

IDs should provide only k- sorting guarantees.

Slide 47

Slide 47 text

How It’s Done

Slide 48

Slide 48 text

Format

Slide 49

Slide 49 text

Our 64-bit IDs look like this:

Slide 50

Slide 50 text

ABBBBBBBBBBBBBBBB BBBBBBBBBBBBBBBBB BBBBBBBBCCCCCCCCC CDDDDDDDDDDDD

Slide 51

Slide 51 text

ABBBBBBBBBBBBBBBBBBBBBBBBB BBBBBBBBBBBBBBBBCCCCCCCCCC DDDDDDDDDDDD A is the reserved signed bit of a Java long (1 bit).

Slide 52

Slide 52 text

ABBBBBBBBBBBBBBBBBBBBBBBBB BBBBBBBBBBBBBBBBCCCCCCCCCC DDDDDDDDDDDD B is the timestamp in milliseconds since custom epoch bits (41 bits).

Slide 53

Slide 53 text

ABBBBBBBBBBBBBBBBBBBBBBBBB BBBBBBBBBBBBBBBBCCCCCCCCCC DDDDDDDDDDDD C is the logical shard ID (10 bits).

Slide 54

Slide 54 text

ABBBBBBBBBBBBBBBBBBBBBBBBB BBBBBBBBBBBBBBBBCCCCCCCCCC DDDDDDDDDDDD D is the sequence (12 bits).

Slide 55

Slide 55 text

The Timestamp

Slide 56

Slide 56 text

Represented in 41 bits using a custom epoch.

Slide 57

Slide 57 text

This allows ~69 years of continuous ID generation.

Slide 58

Slide 58 text

Note this is the first part of the ID, so it has the most bearing on sorting.

Slide 59

Slide 59 text

Sorting IDs sorts by time strictly, remainder of ID roughly (ie. k-sorted).

Slide 60

Slide 60 text

Logical Shard ID

Slide 61

Slide 61 text

We want to be able to have many Redis servers.

Slide 62

Slide 62 text

We allow 10 bits for this ID, so we can have up to 1024 ID generation machines.

Slide 63

Slide 63 text

We give a fixed ID to each Redis server and it stamps its IDs with this ever after.

Slide 64

Slide 64 text

The Sequence

Slide 65

Slide 65 text

What happens if you ask the same Redis server to generate multiple IDs in a millisecond?

Slide 66

Slide 66 text

The sequence ensures IDs are never duplicated when this happens.

Slide 67

Slide 67 text

We rotate a 12-bit number.

Slide 68

Slide 68 text

We roll back to 0 when it reaches 4905.

Slide 69

Slide 69 text

That means a maximum of 4096 IDs per node per millisecond.

Slide 70

Slide 70 text

If the sequence rolls over twice in the same millisecond, we block until the time changes.

Slide 71

Slide 71 text

Distributing The Load

Slide 72

Slide 72 text

Simple round-robin between the Redis servers.

Slide 73

Slide 73 text

Retry 5 times before failing.

Slide 74

Slide 74 text

Fin Questions?