the potential
dangers
of causal
consistency
and an explicit
solution
Peter Bailis, Alan Fekete, Ali Ghodsi,
Joseph M. Hellerstein, Ion Stoica
SOCC 2012
Slide 2
Slide 2 text
A Story
Slide 3
Slide 3 text
A Story
Slide 4
Slide 4 text
“Sally’s in
a coma!”
A Story
Slide 5
Slide 5 text
“Sally’s in
a coma!”
“Sally’s in
a coma!”
A Story
“Sally’s
okay!”
Slide 6
Slide 6 text
“Sally’s in
a coma!”
“Sally’s in
a coma!”
A Story
“Sally’s
okay!”
Slide 7
Slide 7 text
“Sally’s in
a coma!”
“Sally’s in
a coma!”
A Story
“Great
news!”
“Sally’s
okay!”
Slide 8
Slide 8 text
“Sally’s in
a coma!”
“Sally’s in
a coma!”
A Story
“Great
news!”
“Sally’s
okay!”
Slide 9
Slide 9 text
“Sally’s in
a coma!”
“Sally’s in
a coma!”
“Great
news!”
“Sally’s
okay!”
The Wrong Story
Slide 10
Slide 10 text
“Sally’s in
a coma!”
“Great
news!”
The Wrong Story
Slide 11
Slide 11 text
“Sally’s in
a coma!”
“Great
news!”
The Wrong Story
Slide 12
Slide 12 text
“Sally’s in
a coma!”
“Great
news!”
The Wrong Story
Slide 13
Slide 13 text
“Sally’s in
a coma!”
“Great
news!”
The Wrong Story
???
Slide 14
Slide 14 text
The Right Order
“Sally’s in
a coma!”
“Great
news!”
“Sally’s
okay!”
Slide 15
Slide 15 text
The Right Order
“Sally’s in
a coma!”
“Great
news!”
“Sally’s
okay!”
Slide 16
Slide 16 text
The Right Order
“Sally’s in
a coma!”
“Great
news!”
“Sally’s
okay!”
Slide 17
Slide 17 text
SOME DEPENDENCIES
METADATA AND MODERN SERVICES
(NOT) SCALING UP AND OUT
BETTER LIVING VIA APP CONTEXT
Slide 18
Slide 18 text
SOME DEPENDENCIES
METADATA AND MODERN SERVICES
(NOT) SCALING UP AND OUT
BETTER LIVING VIA APP CONTEXT
Slide 19
Slide 19 text
SOME DEPENDENCIES
METADATA AND MODERN SERVICES
(NOT) SCALING UP AND OUT
BETTER LIVING VIA APP CONTEXT
Slide 20
Slide 20 text
SOME DEPENDENCIES
METADATA AND MODERN SERVICES
(NOT) SCALING UP AND OUT
BETTER LIVING VIA APP CONTEXT
Slide 21
Slide 21 text
SOME DEPENDENCIES
METADATA AND MODERN SERVICES
(NOT) SCALING UP AND OUT
BETTER LIVING VIA APP CONTEXT
Slide 22
Slide 22 text
SOME DEPENDENCIES
METADATA AND MODERN SERVICES
(NOT) SCALING UP AND OUT
BETTER LIVING VIA APP CONTEXT
Slide 23
Slide 23 text
No content
Slide 24
Slide 24 text
1978
“Time, Clocks, and the Ordering of Events
in a Distributed System” by Leslie Lamport
Slide 25
Slide 25 text
1978
“Time, Clocks, and the Ordering of Events
in a Distributed System” by Leslie Lamport
1994
“Causal Memory” by Ahamad et al.
Slide 26
Slide 26 text
1978
“Time, Clocks, and the Ordering of Events
in a Distributed System” by Leslie Lamport
1994
“Causal Memory” by Ahamad et al.
2011-12
Slide 27
Slide 27 text
2011-12
Slide 28
Slide 28 text
2011-12
Slide 29
Slide 29 text
2011-12
IEEE CAP
There are
many hard,
trade-offs,
Slide 30
Slide 30 text
2011-12
IEEE CAP SOSP: COPS
There are
many hard,
trade-offs,
but
causal
consistency
can work
well
Slide 31
Slide 31 text
2011-12
IEEE CAP SOSP: COPS Texas: CAC
There are
many hard,
trade-offs,
but
causal
consistency
can work
well
and it’s
the best
you can
do*!
Slide 32
Slide 32 text
No content
Slide 33
Slide 33 text
“Sally’s
okay!”
Slide 34
Slide 34 text
“Sally’s
okay!”
Slide 35
Slide 35 text
“Sally’s
okay!”
Slide 36
Slide 36 text
“Sally’s
okay!”
Slide 37
Slide 37 text
“Sally’s
okay!”
Slide 38
Slide 38 text
“Sally’s
okay!”
Slide 39
Slide 39 text
“Great
news!”
“Sally’s
okay!”
Slide 40
Slide 40 text
“Great
news!”
“Sally’s
okay!”
Slide 41
Slide 41 text
“Great
news!”
“Sally’s
okay!”
Slide 42
Slide 42 text
“Great
news!”
“Sally’s
okay!”
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
No content
Slide 45
Slide 45 text
No content
Slide 46
Slide 46 text
Deliver writes
in the “right”
order
Slide 47
Slide 47 text
Deliver writes
in the “right”
order
“Great
news!”
“Sally’s
okay!”
“Sally’s
in a coma!”
Slide 48
Slide 48 text
Causal Consistency
reads obey partial order
called “happens-before”
Slide 49
Slide 49 text
No content
Slide 50
Slide 50 text
No content
Slide 51
Slide 51 text
No content
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
No content
Slide 55
Slide 55 text
No content
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
Write value
Slide 59
Slide 59 text
Write value
Happens-before
metadata
Slide 60
Slide 60 text
Write value
Happens-before
metadata
Slide 61
Slide 61 text
No content
Slide 62
Slide 62 text
No content
Slide 63
Slide 63 text
No content
Slide 64
Slide 64 text
Have we seen all
dependencies?
Slide 65
Slide 65 text
Have we seen all
dependencies?
Slide 66
Slide 66 text
Have we seen all
dependencies?
Y
Slide 67
Slide 67 text
Have we seen all
dependencies?
Slide 68
Slide 68 text
Have we seen all
dependencies?
Y
Slide 69
Slide 69 text
Have we seen all
dependencies?
Yes: apply write
Slide 70
Slide 70 text
Have we seen all
dependencies?
Yes: apply write
Slide 71
Slide 71 text
Have we seen all
dependencies?
No: wait
Yes: apply write
Slide 72
Slide 72 text
Sounds great, right?
Slide 73
Slide 73 text
SOME DEPENDENCIES
METADATA AND MODERN SERVICES
(NOT) SCALING UP AND OUT
BETTER LIVING VIA APP CONTEXT
Slide 74
Slide 74 text
Causal Consistency
reads obey partial order
called “happens-before”
Slide 75
Slide 75 text
Causal Consistency
traditional happens-before:
potential causality
reads obey partial order
called “happens-before”
Slide 76
Slide 76 text
If you wish to make an
apple pie from scratch,
you must first invent
the universe.
-Carl Sagan
Slide 77
Slide 77 text
No content
Slide 78
Slide 78 text
“Great
news!”
Slide 79
Slide 79 text
“Great
news!”
“Sally’s
okay!”
Slide 80
Slide 80 text
“Great
news!”
“Sally’s
okay!”
“Great
picture!”
“Rad
party!”
“Lol!”
“Can’t wait
for SOCC!”
“Want to go
skiing?”
Slide 81
Slide 81 text
“Great
news!”
“Sally’s
okay!”
“Great
picture!”
“Rad
party!”
“Lol!”
“Can’t wait
for SOCC!”
“Want to go
skiing?”
“I hope my
paper gets in.”
“You
who?”
“I love
Tahoe!”
“Coming tonight?
Slide 82
Slide 82 text
“Great
news!”
“Sally’s
okay!”
“Great
picture!”
“Rad
party!”
“Lol!”
“Can’t wait
for SOCC!”
“Want to go
skiing?”
“Snow
rocks!”
“I hear the
PC is great!”
“You.”
“Hello,
“I hope my
paper gets in.”
“You
who?”
“I love
Tahoe!”
“Coming tonight?
Slide 83
Slide 83 text
“Great
news!”
“Sally’s
okay!”
“Great
picture!”
“Rad
party!”
“Lol!”
“Can’t wait
for SOCC!”
“Want to go
skiing?”
“Snow
rocks!”
“I hear the
PC is great!”
“You.”
“Hello,
“Who’s
there?”
“Are you
submitting?”
“Have you
met Larry?”
“Great
food here”
“I hope my
paper gets in.”
“You
who?”
“I love
Tahoe!”
“Coming tonight?
Slide 84
Slide 84 text
“Great
news!”
“Sally’s
okay!”
“Great
picture!”
“Rad
party!”
“Lol!”
“Can’t wait
for SOCC!”
“Want to go
skiing?”
“Snow
rocks!”
“I hear the
PC is great!”
“You.”
“Hello,
“Who’s
there?”
“Are you
submitting?”
“Have you
met Larry?”
“Great
food here”
“Knock, knock.”
“I hope my
paper gets in.”
“You
who?”
“I love
Tahoe!”
“Coming tonight?
Slide 85
Slide 85 text
“Great
news!”
“Sally’s
okay!”
“Great
picture!”
“Rad
party!”
“Lol!”
“Can’t wait
for SOCC!”
“Want to go
skiing?”
“Snow
rocks!”
“I hear the
PC is great!”
“You.”
“Hello,
“Who’s
there?”
“Are you
submitting?”
“Have you
met Larry?”
“Great
food here”
“Knock, knock.”
“I hope my
paper gets in.”
“You
who?”
“I love
Tahoe!”
“Coming tonight?
Slide 86
Slide 86 text
Twitter.com
20 Tweets
at login
20 more
auto-scroll
600+ per min
Slide 87
Slide 87 text
Twitter.com
quickly
approaches
asymptotic limit
Slide 88
Slide 88 text
r
e
c
a
p
Slide 89
Slide 89 text
potential danger
Potential causality graphs are huge,
limiting local apply rates and throughput
r
e
c
a
p
Slide 90
Slide 90 text
SOME DEPENDENCIES
METADATA AND MODERN SERVICES
(NOT) SCALING UP AND OUT
BETTER LIVING VIA APP CONTEXT
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
Unstable queue!
Sustained throughput
limited to slowest DC
Slide 99
Slide 99 text
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
Unstable queue!
Sustained throughput
limited to slowest DC
Slide 100
Slide 100 text
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
Sustained throughput
limited to slowest DC
Slide 101
Slide 101 text
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
Adding DCs doesn’t help
slowest site
Sustained throughput
limited to slowest DC
Slide 102
Slide 102 text
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
During partitions/failures,
sustainable throughput is
zero writes/s
zero
Sustained throughput
limited to slowest DC
Slide 103
Slide 103 text
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
During partitions/failures,
sustainable throughput is
zero writes/s
zero
Slide 104
Slide 104 text
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
During partitions/failures,
sustainable throughput is
zero writes/s
zero
Slide 105
Slide 105 text
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
During partitions/failures,
sustainable throughput is
zero writes/s
zero
Metadata garbage collection
stalls
stalls
Slide 106
Slide 106 text
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
During partitions/failures,
sustainable throughput is
zero writes/s
zero
Metadata garbage collection
stalls
stalls
Slide 107
Slide 107 text
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
DANGER!
During partitions/failures,
sustainable throughput is
zero writes/s
zero
Metadata garbage collection
stalls
stalls
Slide 108
Slide 108 text
r
e
c
a
p
Slide 109
Slide 109 text
r
e
c
a
p
potential danger
Write throughput limited to slowest DC
Violation 㱺 arbitrarily high visibility latency
Adding DCs does not increase throughput
Slide 110
Slide 110 text
Potential dangers spell trouble for causality
bad news
Slide 111
Slide 111 text
Potential dangers spell trouble for causality
to it?
Are live without we
destined
bad news
Slide 112
Slide 112 text
Potential dangers spell trouble for causality
to it?
Are live without
we destined
bad news
Slide 113
Slide 113 text
SOME DEPENDENCIES
METADATA AND MODERN SERVICES
(NOT) SCALING UP AND OUT
BETTER LIVING VIA APP CONTEXT
Slide 114
Slide 114 text
Observation
Why track dependencies that don’t matter?
Slide 115
Slide 115 text
Observation
Why track dependencies that don’t matter?
Slide 116
Slide 116 text
Observation
Why track dependencies that don’t matter?
Need to know application semantics
Slide 117
Slide 117 text
Explicit Causality
app-defined “happens-before”
transitivity enforced
subset of potential causality
Slide 118
Slide 118 text
Explicit Causality
app-defined “happens-before”
transitivity enforced
subset of potential causality
not a new idea
(e.g., Cheriton and Skeen SOSP 1993,
Ladin et al. PODC 1990)
but...
Slide 119
Slide 119 text
Explicit Matters
Slide 120
Slide 120 text
Explicit Matters
Twitter
28% of Tweets in conversations
69% of convos are depth two
average depth is 10.7
[Ye and Wu SocInfo 2010, Ritter et al. HLT 2010]
Slide 121
Slide 121 text
Explicit Matters
Twitter
28% of Tweets in conversations
69% of convos are depth two
average depth is 10.7
[Ye and Wu SocInfo 2010, Ritter et al. HLT 2010]
reply-to degree and depth are limited
Slide 122
Slide 122 text
Explicit Matters
Twitter
28% of Tweets in conversations
69% of convos are depth two
average depth is 10.7
[Ye and Wu SocInfo 2010, Ritter et al. HLT 2010]
reply-to degree and depth are limited
109 smaller graph for a year of Tweets
Slide 123
Slide 123 text
Explicit API
put(key, value)
Slide 124
Slide 124 text
put_after(key, value, deps)
Explicit API
Slide 125
Slide 125 text
put_after(key, value, deps)
Explicit API
Slide 126
Slide 126 text
put_after(key, value, deps)
Explicit API
(possibly empty) set of
references to other writes
Slide 127
Slide 127 text
put_after(key, value, deps)
Explicit API
track what matters
(possibly empty) set of
references to other writes
Slide 128
Slide 128 text
put_after(key, value, deps)
Explicit API
track what matters
frequently in data model already
(possibly empty) set of
references to other writes
Slide 129
Slide 129 text
put_after(key, value, deps)
Explicit API
track what matters
frequently in data model already
can simulate fencing
(possibly empty) set of
references to other writes
Slide 130
Slide 130 text
put_after(key, value, deps)
Explicit API
track what matters
frequently in data model already
can simulate fencing
(possibly empty) set of
references to other writes
won’t track non-explicit references
Slide 131
Slide 131 text
NATURAL EXTENSION
NOT ROCKET SCIENCE
Slide 132
Slide 132 text
NATURAL EXTENSION
NOT ROCKET SCIENCE
http://twitter.com/rbranson/status/256795094387142657
Slide 133
Slide 133 text
No content
Slide 134
Slide 134 text
potential
causality
Slide 135
Slide 135 text
No content
Slide 136
Slide 136 text
A WORLD OF
PHYSICAL
OPERATIONS
MESSAGES
REGISTERS
READS
WRITES
MODEST
SCALE
Slide 137
Slide 137 text
No content
Slide 138
Slide 138 text
No content
Slide 139
Slide 139 text
A WORLD OF
HUGE SCALE
MASSIVE
INTERACTIVITY
PLANET-WIDE
NETWORKS
REAL-WORLD
INTERFACES
UBICOMP
BIG DATA
potential
dangers
huge causality graphs
explicit
causality
semantic context to the rescue
consider modern apps
helps with #1, indirectly with #2
throughput scalability limited