30,000,000 Requests in a Hour in the Cloud

6c18ceafef161be26ae441469b29c475?s=47 Terrence Ryan
September 03, 2016
220

30,000,000 Requests in a Hour in the Cloud

6c18ceafef161be26ae441469b29c475?s=128

Terrence Ryan

September 03, 2016
Tweet

Transcript

  1. ‹#› @tpryan Terrence Ryan Developer Advocate 30,000,000 Requests An Hour

    In The Cloud
  2. ‹#› @tpryan Who are you?

  3. ‹#› @tpryan 01 Introduction What was I trying to do

  4. None
  5. ‹#› @tpryan Demo: App Engine

  6. ‹#› @tpryan App Engine • Writes to Firebase • Writes

    to Cloud Storage • Run the Visualizer
  7. ‹#› @tpryan Compute Engine • Load Generators • Apache Bench

    • Remote Command Execution
  8. ‹#› @tpryan Firebase • Realtime Display • Tracking all load

  9. ‹#› @tpryan Cloud Storage • Proof of activity • Object

    notification
  10. ‹#› @tpryan App Engine • Process Notifications from Cloud Storage

  11. ‹#› @tpryan Planned / //Write to Firebase //Write to Cloud

    Storage //Receive realtime
 //updates /visualizer /storage //Write to Firebase //Display updates //Store file //Send Notification http://load-demo.appspot.com
  12. ‹#› @tpryan Actual / //Write to Firebase //Write to Cloud

    Storage //Receive realtime
 //updates /visualizer /storage //Write to Firebase //Display updates //Store file //Send Notification http://load-demo.appspot.com
  13. @briandorsey 30,000,000 requests in one hour

  14. @briandorsey What do you call a site that gets 30,000,000

    hits an hour?
  15. @briandorsey

  16. @briandorsey I should try and do that and mean it

  17. ‹#› @tpryan 02 Ramping up What is Google Cloud Platform

  18. @briandorsey 30,000,000 qph /60/60 8,333 qps

  19. @briandorsey Ramp the demo up to 8,333 qps

  20. ‹#› @tpryan Demo: App Engine

  21. ‹#› @tpryan URLFetch • Network abstraction layer for http •

    Most service calls
  22. ‹#› @tpryan Source Code $instance = $_SERVER['INSTANCE_ID']; $urlToPatch = $fbBaseURL

    . "appengine/" . $instance . ".json"; $urlToPost = $fbBaseURL . "appengine/" . $instance . “/sessions.json"; $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->updated = time(); $instance_json = json_encode($instance_data); $ch = curl_init(); $output = patch($ch, $urlToPatch, $instance_json); $output = post($ch, $urlToPost, $instance_json); $name = json_decode($output)->name; $filename = $gcsBaseURL . $instance . "/file" . $name . ".txt"; file_put_contents($filename, $name); $instance = $_SERVER['INSTANCE_ID']; $urlToPatch = $fbBaseURL . "appengine/" . $instance . ".json"; $urlToPost = $fbBaseURL . "appengine/" . $instance . “/sessions.json"; $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->updated = time(); $instance_json = json_encode($instance_data); $ch = curl_init(); $output = patch($ch, $urlToPatch, $instance_json); $output = post($ch, $urlToPost, $instance_json); $name = json_decode($output)->name; $filename = $gcsBaseURL . $instance . "/file" . $name . ".txt"; file_put_contents($filename, $name);
  23. ‹#› @tpryan Source Code function patch($ch, $url, $json){ curl_setopt($ch, CURLOPT_URL,

    $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PATCH"); curl_setopt($ch, CURLOPT_POSTFIELDS, $json); return curl_exec($ch); } function post($ch, $url, $json){ curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST"); curl_setopt($ch, CURLOPT_POSTFIELDS, $json); return curl_exec($ch); }
  24. ‹#› @tpryan

  25. ‹#› @tpryan

  26. ‹#› @tpryan

  27. @briandorsey 660,000 qpm /60 11,000 qps /3 3,666 qps

  28. ‹#› @tpryan 03 Fix 1 Sockets

  29. ‹#› @tpryan

  30. ‹#› @tpryan

  31. ‹#› @tpryan

  32. ‹#› @tpryan Demo: App Engine

  33. ‹#› @tpryan Reasons for failure • Sockets are lighter •

    But implementation is not configurable • So write your own socket client?
  34. ‹#› @tpryan 04 Fix 2 Memcache

  35. ‹#› @tpryan Assumptions • Visualization needs to be realtime •

    But does it?
  36. ‹#› @tpryan Memcache on App Engine

  37. ‹#› @tpryan Pseudo Code // store instances request count //

    Get list of keys // Display request counts Bad Idea
  38. ‹#› @tpryan

  39. ‹#› @tpryan Source Code $memcache = new Memcached; $instance =

    $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $result = $memcache->increment($instance); if ($result == 1 ){ $result = $memcache->append("instances", "|" . $instance); if ($memcache->getResultCode() == 14){ $result = $memcache->set("instances", $instance); } } $filename = $gcsBaseURL . $instance . "/file" . $request . ".txt"; file_put_contents($filename, $filename);
  40. ‹#› @tpryan Close but no cigar • Losing instances •

    Analysis revealed: • All requests were firing • All requests were registering • Cause: Memcache strings were getting too long
  41. ‹#› @tpryan Source Code $memcache = new Memcached; $instance =

    $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $memcache->increment("total"); $result = $memcache->increment($instance); if ($result == 1 ){ $result = $memcache->append("instances", "|" . $instance); if ($memcache->getResultCode() == 14){ $result = $memcache->set("instances", $instance); } } $filename = $gcsBaseURL . $instance . "/file" . $request . ".txt"; file_put_contents($filename, $filename); $memcache = new Memcached; $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $memcache->increment("total"); $result = $memcache->increment($instance); if ($result == 1 ){ $result = $memcache->append("instances", "|" . $instance); if ($memcache->getResultCode() == 14){ $result = $memcache->set("instances", $instance); } } $filename = $gcsBaseURL . $instance . "/file" . $request . ".txt"; file_put_contents($filename, $filename);
  42. ‹#› @tpryan Source Code function whichInstanceList($instance){ $last = substr($instance, -1);

    return "instances-" . $last; }
  43. ‹#› @tpryan Demo: App Engine

  44. ‹#› @tpryan 05 Conclusion What did I learn

  45. @briandorsey Wait, you’re giving up realtime without a fight?

  46. ‹#› @tpryan 05 Fix 3 Brining back the realtime

  47. ‹#› @tpryan Current / //Write to Memcache //Write to Cloud

    Storage //Receive realtime
 //updates /visualizer http://load-demo.appspot.com //Display Updates //Store request details
  48. ‹#› @tpryan Current / //Write to Memcache //Write to Cloud

    Storage //Receive realtime
 //updates /visualizer http://load-demo.appspot.com //Display Updates //Store request details //Read memcache //Write to Firebase /realtime.php
  49. ‹#› @tpryan Pseudo Code // Get list of instances //

    Get list of number of requests for each instance // Send data for each request to firebase Bad Idea
  50. ‹#› @tpryan How Firebase works • appengine • instance1 •

    request1 • request2 • request3 • request4 • request5 • request6 • instance2
  51. ‹#› @tpryan How Firebase works { "appengine": { "instance1": {

    "request1": "", "request2": "", "request3": "", "request4": "", "request5": "", "request6": "" }, "instance2": {} } }
  52. ‹#› @tpryan How Firebase works { "appengine": { "instance1": {

    "request1": "", "request2": "", "request3": "", "request4": "", "request5": "", "request6": "" }, "instance2": {} } } /appengine/instance1/request2.json GET POST PATCH
  53. ‹#› @tpryan How Firebase works { "appengine": { "instance1": {

    "request1": "", "request2": "", "request3": "", "request4": "", "request5": "", "request6": "" }, "instance2": {} } } /appengine/instance1.json GET POST PATCH
  54. ‹#› @tpryan How Firebase works { "appengine": { "instance1": {

    "request1": "", "request2": "", "request3": "", "request4": "", "request5": "", "request6": "" }, "instance2": {} } } /appengine.json GET POST PATCH
  55. ‹#› @tpryan

  56. ‹#› @tpryan Batch Memcache calls getMulti()

  57. ‹#› @tpryan Let’s send too much stuff • appengine •

    00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request1 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request2 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request3 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request4 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request5 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request6 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • 00c61b117cf44784cd0c142018b2ada185ce6d3472a963342388ef123a30
  58. ‹#› @tpryan Let’s send too much stuff • appengine •

    00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • 6 • 00c61b117cf44784cd0c142018b2ada185ce6d3472a963342388ef123a30 • 1
  59. ‹#› @tpryan

  60. ‹#› @tpryan

  61. ‹#› @tpryan

  62. ‹#› @tpryan

  63. ‹#› @tpryan

  64. ‹#› @tpryan Source Code $task1 = new PushTask('/realtime.php'); $task1->add("realtimesynch");

  65. ‹#› @tpryan Demo: App Engine

  66. ‹#› @tpryan 06 Conclusion What did I learn

  67. @briandorsey Safety Quotas are not just something to be gotten

    around
  68. @briandorsey Bundle up data calls when you can

  69. @briandorsey Be prepared to make compromises

  70. @briandorsey Time for a language change?

  71. ‹#› @tpryan Thank You terrenceryan.com @tpryan This preso: http://bit.ly/tpryan-30m