Slide 1
Slide 1 text
Realtime Database for
high traffic production application
@sota1235
2018/9/1 GDG DevFest Tokyo 2018
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
For what?
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
Mercari channel
• Live e-commerce
• Working on Mercari
app
• You can buy/sell
anything through
streaming
https://www.mercari.com/jp/mercari-channel/
Slide 7
Slide 7 text
• Messages
• Likes
• Notifications
• Item list
Slide 8
Slide 8 text
Today
• Why Realtime Database?
• How to build app
• How to run app on production
Slide 9
Slide 9 text
Note
• Today, Iʼll share some techniques
• I believe itʼll help you to use Realtime
Database
Slide 10
Slide 10 text
Why Realtime Database?
Slide 11
Slide 11 text
Tight schedule Easy to use
Sync data
in real time
What we needed
Slide 12
Slide 12 text
Points
• Fully managed storage
• Powerful SDK
• Sync data in real time
Slide 13
Slide 13 text
Fully managed
• 1 JSON data in 1 database
• Multiple databases per 1 Firebase Project
• Make project and ship it
Slide 14
Slide 14 text
Powerful SDK
https://firebase.google.com/docs/guides/
Slide 15
Slide 15 text
// Set the configuration for your app
const config = {
apiKey: "apiKey",
authDomain: "projectId.firebaseapp.com",
databaseURL: "https://databaseName.firebaseio.com",
storageBucket: "bucket.appspot.com"
};
firebase.initializeApp(config);
// Get a reference to the database service
const database = firebase.database();
Initializing
Slide 16
Slide 16 text
const starCountRef =
firebase.database().ref('posts/starCount');
starCountRef.on('value', (snapshot) => {
console.log(snapshot.val());
});
Slide 17
Slide 17 text
Sync data in real time
Subscribe
Publish
Slide 18
Slide 18 text
How to build app
Slide 19
Slide 19 text
Architecture design
Schema design
Slide 20
Slide 20 text
Architecture design
Architecture design
Schema design
Slide 21
Slide 21 text
Read
Write
・Using SDK to read data
・Techniques not to read all data
・Updating data with REST API
・Managing frequency of query
Slide 22
Slide 22 text
Read data
• Getting data through SDK
• Thatʼs it
• We need only 1 hack
Slide 23
Slide 23 text
$PNNFOU
$PNNFOU
$PNNFOU
Time
Slide 24
Slide 24 text
$PNNFOU
$PNNFOU
$PNNFOU
Time
Start seeing
streaming
Slide 25
Slide 25 text
$PNNFOU
$PNNFOU
$PNNFOU
Time
Start seeing
streaming
SDK got this
even though
it’s unnecessary ☹
Slide 26
Slide 26 text
Little hack
• SDK gets one data when connecting
• But sometimes itʼs unnecessary
• So filtering data by timestamp
Slide 27
Slide 27 text
Write data
• Updating data through REST API
• Managing frequency of query
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
REST API
• Two ways to write data
• REST API or WebSocket(w/SDK)
• In our case, REST API matched
Slide 31
Slide 31 text
Authentication
• Database token
• Deprecated
• Google OAuth2
• Recommended
Slide 32
Slide 32 text
Cache auth token
• Cache auth token on cache store
• In case of sending request to REST API, it
will work
• OAuth2 authentication will be overhead
Slide 33
Slide 33 text
keep-alive with chocon
• We use chocon for using keep-alive
• PHP process canʼt use it
• Realtime Database instances are in US
• 1 request means RTT to the Pacific Ocean
• https://github.com/kazeburo/chocon
Slide 34
Slide 34 text
Keep alive with chocon
+ ,
keep-alive
Slide 35
Slide 35 text
Frequency managing
• Not sending all data from user
Slide 36
Slide 36 text
Frequency managing
• We donʼt need to send these kinds of data
every time
• Likes, Audience count, Notifications
• So we manage frequency of sending such
data
Slide 37
Slide 37 text
Mercari API
Like
count=1
count=1
Slide 38
Slide 38 text
Mercari API
Like
count=1
count=1
Like
Like
Like
count=2
count=3
count=4
Slide 39
Slide 39 text
Mercari API
Like
count=1
count=1
Like
Like
Like
count=2
count=3
count=4
Slide 40
Slide 40 text
Mercari API
Like
count=1
count=1
Like
Like
Like
count=2
count=3
count=4
Check frequency before
sending data
N query/sec
Slide 41
Slide 41 text
Mercari API
Like
count=1
count=1
Like
Like
Like
count=2
count=3
count=4 count=4
N sec
Slide 42
Slide 42 text
Motivation
• We donʼt need to update UI 1000 times
• The most important point is experience
• Actually, spec is limited for 1 instance
• 1000 update per sec
Slide 43
Slide 43 text
Architecture design
Schema design
Schema design
Slide 44
Slide 44 text
Realtime Database is
schema-less
Then, how to design?
Slide 45
Slide 45 text
Rule
• Settings for 1 database
• Written in JSON
• Validation
• Permission/Authorization
Slide 46
Slide 46 text
{
"rules": {
".write": false,
"lives": {
"$live_id": {
"messages": {
"$message_id": {
".validate": "newData.hasChildren(['user', 'text'])"
}
},
"notifications": {
"buy_item": {
".validate": "newData.hasChildren(['text'])"
}
}
}
}
}
}
Sample rule
Slide 47
Slide 47 text
Our strategy
• Only writing data from our server
• Controlling read permission by using flag
data on database
Slide 48
Slide 48 text
Only writing from server
• Pros
• Not need to be careful about permission
for client
• Cons
• Server needs to handle high traffic
Slide 49
Slide 49 text
{
"rules": {
".write": false,
"lives": {
"$live_id": {
"messages": {
"$message_id": {
".validate": "newData.hasChildren(['user', 'text'])"
}
},
"notifications": {
"buy_item": {
".validate": "newData.hasChildren(['text'])"
}
}
}
}
}
}
Read only Rule
Slide 50
Slide 50 text
{
"rules": {
".write": false,
"lives": {
"$live_id": {
"messages": {
"$message_id": {
".validate": "newData.hasChildren(['user', 'text'])"
}
},
"notifications": {
"buy_item": {
".validate": "newData.hasChildren(['text'])"
}
}
}
}
}
}
Read only Rule
Slide 51
Slide 51 text
{
"rules": {
".write": false,
"lives": {
"$live_id": {
"messages": {
"$message_id": {
".validate": "newData.hasChildren(['user', 'text'])"
}
},
"notifications": {
"buy_item": {
".validate": "newData.hasChildren(['text'])"
}
}
}
}
}
}
Read only Rule
Tips:
Admin user can ignore
all rules
Slide 52
Slide 52 text
Controlling permission
dynamically
• Rule can refer data on schema
• It means we can manage permission by
data
Slide 53
Slide 53 text
{
"rules": {
".write": false,
"lives": {
"$live_id": {
"messages": {
"$message_id": {
".validate": "newData.hasChildren(['user', 'image', 'text'])"
}
},
"notifications": {
"buy_item": {
".validate": "newData.hasChildren(['text'])"
}
}
}
},
"alive_lives": {
".write": false,
".read": false
}
}
}
Slide 54
Slide 54 text
{
"rules": {
".write": false,
"lives": {
"$live_id": {
"messages": {
"$message_id": {
".validate": "newData.hasChildren(['user', 'image', 'text'])"
}
},
"notifications": {
"buy_item": {
".validate": "newData.hasChildren(['text'])"
}
}
}
},
"alive_lives": {
".write": false,
".read": false
}
}
}
Data for client
Data for managing permission
Slide 55
Slide 55 text
{
"rules": {
".write": false,
"lives": {
"$live_id": {
".read": "root.child('alive_lives').child($live_id).val() === true",
"messages": {
"$message_id": {
".validate": "newData.hasChildren(['user', 'image', 'text'])"
}
},
"notifications": {
"buy_item": {
".validate": "newData.hasChildren(['text'])"
}
}
}
},
"alive_lives": {
".write": false,
".read": false
}
}
}
Slide 56
Slide 56 text
{
"rules": {
".write": false,
"lives": {
"$live_id": {
".read": "root.child('alive_lives').child($live_id).val() === true",
"messages": {
"$message_id": {
".validate": "newData.hasChildren(['user', 'image', 'text'])"
}
},
"notifications": {
"buy_item": {
".validate": "newData.hasChildren(['text'])"
}
}
}
},
"alive_lives": {
".write": false,
".read": false
}
}
}
Slide 57
Slide 57 text
"root.child('alive_lives').child($live_id).val() === true”,
Refer data on database
Slide 58
Slide 58 text
"root.child('alive_lives').child($live_id).val() === true”,
Permission condition
Slide 59
Slide 59 text
{
"lives": {
"1": { ... },
"2": { ... }
},
"alive_lives": {
"1": true,
"2": false
}
}
alive_lives[ʻ1ʼ] is true
So it can be read
Slide 60
Slide 60 text
{
"lives": {
"1": { ... },
"2": { ... }
},
"alive_lives": {
"1": true,
"2": false
}
}
alive_lives[ʻ1ʼ] is false
So it canʼt be read
Slide 61
Slide 61 text
Easier to control permission
• Updating rule every time will be overhead
• So adding data to manage permission is
good technique
Slide 62
Slide 62 text
How to run on production
Slide 63
Slide 63 text
Scalability Monitoring Troubleshooting
What we needed
Slide 64
Slide 64 text
Scalability
Slide 65
Slide 65 text
Scalability
• Mercari is high traffic service
• Mercari channel also should think about
for high traffic
Slide 66
Slide 66 text
KPI of Mercari channel
• Over 1500 streamers/day
• Over 30K viewers/day
• 5-6M yen sales per 1 stream
※not strict numbers
Slide 67
Slide 67 text
Can Realtime Database
be scale automatically?
Slide 68
Slide 68 text
No, we need to scale by
ourselves
Slide 69
Slide 69 text
Again, limitation of database
• There is limitation for each database
• 1000 query/sec
• 1M connection
Slide 70
Slide 70 text
Vertical sharding
Slide 71
Slide 71 text
subscribe
Some data
Slide 72
Slide 72 text
subscribe
Some data
Slide 73
Slide 73 text
For scalability
• We use over 15+ database instances in
production
• Client switch DB by data from Mercari API
• For now, Cloud Firestore can be good
choice for this problem
Slide 74
Slide 74 text
Monitoring
Slide 75
Slide 75 text
Monitoring
• In house monitoring tool
• Stackdriver
Slide 76
Slide 76 text
In house
• Application logging on Kibana
• Queue status on Mackarel
Slide 77
Slide 77 text
Kibana
Slide 78
Slide 78 text
Mackerel
Slide 79
Slide 79 text
Stackdriver
• Monitoring tool provided by Google
• Some metrics of Realtime Database
• Aggregating all instances data to 1
dashboard
Slide 80
Slide 80 text
Metrics
• Connection count
• Database load
• Network usage
• etc…
https://cloud.google.com/monitoring/api/metrics_gcp#gcp-firebasedatabase
Slide 81
Slide 81 text
For our app
• Connection count
• Database load
• REST API hit
Slide 82
Slide 82 text
Console
Slide 83
Slide 83 text
Troubleshooting
Slide 84
Slide 84 text
Only one truth
Slide 85
Slide 85 text
All services can be in
trouble
Slide 86
Slide 86 text
What we need
• Expect Cloud services as failing
sometimes
• Think about what we can for that
Slide 87
Slide 87 text
Notice
Operation
・Notice to some problem
・Investigation
・Fix bug
・Make service maintenance
Problem
・Some incident of Cloud service
・System alert
Slide 88
Slide 88 text
Notice
Operation
・Notice to some problem
・Investigation
・Fix bug
・Make service maintenance
Problem
・Some incident of Cloud service
・System alert
Slide 89
Slide 89 text
Problem from official
• There is official status dashboard
• RSS/JSON feed supported
Slide 90
Slide 90 text
Problem from system
• Customized Stackdriver
• Mackerel
Slide 91
Slide 91 text
Notice
Operation
・Notice to some problem
・Investigation
・Fix bug
・Make service maintenance
Problem
・Some incident of Cloud service
・System alert
Slide 92
Slide 92 text
Alert channel
• All information will be passed to Slack
channel
• Channel includes all team members
Slide 93
Slide 93 text
Notification from system
• Stackdriver
• Mackerel
Slide 94
Slide 94 text
Notification from official feed
Slide 95
Slide 95 text
When recovered
Slide 96
Slide 96 text
Small technique
• Official alert including both incident and
recovery informations
• There is a staff who can't read English
• So we customized feed bot
Slide 97
Slide 97 text
Customized message
4XJUDIFNPKJ
PS
Slide 98
Slide 98 text
Customized message
POMZNFOUJPOGPS
JODJEFOU
Slide 99
Slide 99 text
Customized message
"VUPUSBOTMBUJPO
Slide 100
Slide 100 text
Notice
Operation
・Notice to some problem
・Investigation
・Fix bug
・Make service maintenance
Problem
・Some incident of Cloud service
・System alert
Slide 101
Slide 101 text
Maintenance mode
• System flag to manage maintenance
mode
• Team member can edit
• If the effect of incident is too large for
customers, we'll edit it
Slide 102
Slide 102 text
Appendix
Slide 103
Slide 103 text
Cloud Firestore vs
Realtime Database
• You SHOULD check this article
• https://firebase.googleblog.com/2017/10/
cloud-firestore-for-rtdb-developers.html
Slide 104
Slide 104 text
For more detail
https://speakerdeck.com/sota1235/realtime-messaging-with-firebase-number-phpcon2017
Slide 105
Slide 105 text
Thank you