Realtime Database for high traffic production application

046baac588d91fd78a85b189847a151d?s=47 Sota Sugiura
September 01, 2018

Realtime Database for high traffic production application

2018/9/1
GDG DevFest Tokyo 2018

046baac588d91fd78a85b189847a151d?s=128

Sota Sugiura

September 01, 2018
Tweet

Transcript

  1. 2.
  2. 3.
  3. 5.
  4. 6.

    Mercari channel • Live e-commerce • Working on Mercari app

    • You can buy/sell anything through streaming https://www.mercari.com/jp/mercari-channel/
  5. 8.
  6. 9.

    Note • Today, Iʼll share some techniques • I believe

    itʼll help you to use Realtime Database
  7. 13.

    Fully managed • 1 JSON data in 1 database •

    Multiple databases per 1 Firebase Project • Make project and ship it
  8. 15.

    // Set the configuration for your app const config =

    { apiKey: "apiKey", authDomain: "projectId.firebaseapp.com", databaseURL: "https://databaseName.firebaseio.com", storageBucket: "bucket.appspot.com" }; firebase.initializeApp(config); // Get a reference to the database service const database = firebase.database(); Initializing
  9. 21.

    Read Write ・Using SDK to read data ・Techniques not to

    read all data ・Updating data with REST API ・Managing frequency of query
  10. 26.

    Little hack • SDK gets one data when connecting •

    But sometimes itʼs unnecessary • So filtering data by timestamp
  11. 28.
  12. 29.
  13. 30.

    REST API • Two ways to write data • REST

    API or WebSocket(w/SDK) • In our case, REST API matched
  14. 32.

    Cache auth token • Cache auth token on cache store

    • In case of sending request to REST API, it will work • OAuth2 authentication will be overhead
  15. 33.

    keep-alive with chocon • We use chocon for using keep-alive

    • PHP process canʼt use it • Realtime Database instances are in US • 1 request means RTT to the Pacific Ocean • https://github.com/kazeburo/chocon
  16. 36.

    Frequency managing • We donʼt need to send these kinds

    of data every time • Likes, Audience count, Notifications • So we manage frequency of sending such data
  17. 40.

    Mercari API Like count=1 count=1 Like Like Like count=2 count=3

    count=4 Check frequency before sending data N query/sec
  18. 42.

    Motivation • We donʼt need to update UI 1000 times

    • The most important point is experience • Actually, spec is limited for 1 instance • 1000 update per sec
  19. 45.

    Rule • Settings for 1 database • Written in JSON

    • Validation • Permission/Authorization
  20. 46.

    { "rules": { ".write": false, "lives": { "$live_id": { "messages":

    { "$message_id": { ".validate": "newData.hasChildren(['user', 'text'])" } }, "notifications": { "buy_item": { ".validate": "newData.hasChildren(['text'])" } } } } } } Sample rule
  21. 47.

    Our strategy • Only writing data from our server •

    Controlling read permission by using flag data on database
  22. 48.

    Only writing from server • Pros • Not need to

    be careful about permission for client • Cons • Server needs to handle high traffic
  23. 49.

    { "rules": { ".write": false, "lives": { "$live_id": { "messages":

    { "$message_id": { ".validate": "newData.hasChildren(['user', 'text'])" } }, "notifications": { "buy_item": { ".validate": "newData.hasChildren(['text'])" } } } } } } Read only Rule
  24. 50.

    { "rules": { ".write": false, "lives": { "$live_id": { "messages":

    { "$message_id": { ".validate": "newData.hasChildren(['user', 'text'])" } }, "notifications": { "buy_item": { ".validate": "newData.hasChildren(['text'])" } } } } } } Read only Rule
  25. 51.

    { "rules": { ".write": false, "lives": { "$live_id": { "messages":

    { "$message_id": { ".validate": "newData.hasChildren(['user', 'text'])" } }, "notifications": { "buy_item": { ".validate": "newData.hasChildren(['text'])" } } } } } } Read only Rule Tips: Admin user can ignore all rules
  26. 52.

    Controlling permission dynamically • Rule can refer data on schema

    • It means we can manage permission by data
  27. 53.

    { "rules": { ".write": false, "lives": { "$live_id": { "messages":

    { "$message_id": { ".validate": "newData.hasChildren(['user', 'image', 'text'])" } }, "notifications": { "buy_item": { ".validate": "newData.hasChildren(['text'])" } } } }, "alive_lives": { ".write": false, ".read": false } } }
  28. 54.

    { "rules": { ".write": false, "lives": { "$live_id": { "messages":

    { "$message_id": { ".validate": "newData.hasChildren(['user', 'image', 'text'])" } }, "notifications": { "buy_item": { ".validate": "newData.hasChildren(['text'])" } } } }, "alive_lives": { ".write": false, ".read": false } } } Data for client Data for managing permission
  29. 55.

    { "rules": { ".write": false, "lives": { "$live_id": { ".read":

    "root.child('alive_lives').child($live_id).val() === true", "messages": { "$message_id": { ".validate": "newData.hasChildren(['user', 'image', 'text'])" } }, "notifications": { "buy_item": { ".validate": "newData.hasChildren(['text'])" } } } }, "alive_lives": { ".write": false, ".read": false } } }
  30. 56.

    { "rules": { ".write": false, "lives": { "$live_id": { ".read":

    "root.child('alive_lives').child($live_id).val() === true", "messages": { "$message_id": { ".validate": "newData.hasChildren(['user', 'image', 'text'])" } }, "notifications": { "buy_item": { ".validate": "newData.hasChildren(['text'])" } } } }, "alive_lives": { ".write": false, ".read": false } } }
  31. 59.

    { "lives": { "1": { ... }, "2": { ...

    } }, "alive_lives": { "1": true, "2": false } } alive_lives[ʻ1ʼ] is true So it can be read
  32. 60.

    { "lives": { "1": { ... }, "2": { ...

    } }, "alive_lives": { "1": true, "2": false } } alive_lives[ʻ1ʼ] is false So it canʼt be read
  33. 61.

    Easier to control permission • Updating rule every time will

    be overhead • So adding data to manage permission is good technique
  34. 65.
  35. 66.

    KPI of Mercari channel • Over 1500 streamers/day • Over

    30K viewers/day • 5-6M yen sales per 1 stream ※not strict numbers
  36. 69.

    Again, limitation of database • There is limitation for each

    database • 1000 query/sec • 1M connection
  37. 73.

    For scalability • We use over 15+ database instances in

    production • Client switch DB by data from Mercari API • For now, Cloud Firestore can be good choice for this problem
  38. 77.
  39. 78.
  40. 79.

    Stackdriver • Monitoring tool provided by Google • Some metrics

    of Realtime Database • Aggregating all instances data to 1 dashboard
  41. 80.

    Metrics • Connection count • Database load • Network usage

    • etc… https://cloud.google.com/monitoring/api/metrics_gcp#gcp-firebasedatabase
  42. 82.
  43. 86.
  44. 87.

    Notice Operation ・Notice to some problem ・Investigation ・Fix bug ・Make

    service maintenance Problem ・Some incident of Cloud service ・System alert
  45. 88.

    Notice Operation ・Notice to some problem ・Investigation ・Fix bug ・Make

    service maintenance Problem ・Some incident of Cloud service ・System alert
  46. 91.

    Notice Operation ・Notice to some problem ・Investigation ・Fix bug ・Make

    service maintenance Problem ・Some incident of Cloud service ・System alert
  47. 92.

    Alert channel • All information will be passed to Slack

    channel • Channel includes all team members
  48. 96.

    Small technique • Official alert including both incident and recovery

    informations • There is a staff who can't read English • So we customized feed bot
  49. 100.

    Notice Operation ・Notice to some problem ・Investigation ・Fix bug ・Make

    service maintenance Problem ・Some incident of Cloud service ・System alert
  50. 101.

    Maintenance mode • System flag to manage maintenance mode •

    Team member can edit • If the effect of incident is too large for customers, we'll edit it
  51. 102.
  52. 103.

    Cloud Firestore vs Realtime Database • You SHOULD check this

    article • https://firebase.googleblog.com/2017/10/ cloud-firestore-for-rtdb-developers.html
  53. 105.