Upgrade to Pro — share decks privately, control downloads, hide ads and more …

30,000,000 Requests in a Hour in the Cloud

Terrence Ryan
September 03, 2016
250

30,000,000 Requests in a Hour in the Cloud

Terrence Ryan

September 03, 2016
Tweet

Transcript

  1. ‹#›
    @tpryan
    Terrence Ryan
    Developer Advocate
    30,000,000 Requests An Hour
    In The Cloud

    View Slide

  2. ‹#›
    @tpryan
    Who are you?

    View Slide

  3. ‹#›
    @tpryan
    01 Introduction
    What was I trying to do

    View Slide

  4. View Slide

  5. ‹#›
    @tpryan
    Demo: App Engine

    View Slide

  6. ‹#›
    @tpryan
    App Engine
    • Writes to Firebase
    • Writes to Cloud Storage
    • Run the Visualizer

    View Slide

  7. ‹#›
    @tpryan
    Compute Engine
    • Load Generators
    • Apache Bench
    • Remote Command Execution

    View Slide

  8. ‹#›
    @tpryan
    Firebase
    • Realtime Display
    • Tracking all load

    View Slide

  9. ‹#›
    @tpryan
    Cloud Storage
    • Proof of activity
    • Object notification

    View Slide

  10. ‹#›
    @tpryan
    App Engine
    • Process Notifications from
    Cloud Storage

    View Slide

  11. ‹#›
    @tpryan
    Planned
    /
    //Write to Firebase
    //Write to Cloud Storage
    //Receive realtime

    //updates
    /visualizer /storage
    //Write to Firebase
    //Display updates
    //Store file
    //Send Notification
    http://load-demo.appspot.com

    View Slide

  12. ‹#›
    @tpryan
    Actual
    /
    //Write to Firebase
    //Write to Cloud Storage
    //Receive realtime

    //updates
    /visualizer /storage
    //Write to Firebase
    //Display updates
    //Store file
    //Send Notification
    http://load-demo.appspot.com

    View Slide

  13. @briandorsey
    30,000,000 requests
    in one hour

    View Slide

  14. @briandorsey
    What do you call a site that
    gets 30,000,000 hits an hour?

    View Slide

  15. @briandorsey

    View Slide

  16. @briandorsey
    I should try and do
    that and mean it

    View Slide

  17. ‹#›
    @tpryan
    02 Ramping up
    What is Google Cloud Platform

    View Slide

  18. @briandorsey
    30,000,000 qph
    /60/60
    8,333 qps

    View Slide

  19. @briandorsey
    Ramp the demo up to
    8,333 qps

    View Slide

  20. ‹#›
    @tpryan
    Demo: App Engine

    View Slide

  21. ‹#›
    @tpryan
    URLFetch
    • Network abstraction layer for http
    • Most service calls

    View Slide

  22. ‹#›
    @tpryan
    Source Code
    $instance = $_SERVER['INSTANCE_ID'];
    $urlToPatch = $fbBaseURL . "appengine/" . $instance . ".json";
    $urlToPost = $fbBaseURL . "appengine/" . $instance . “/sessions.json";
    $instance_data = new stdClass();
    $instance_data->instance = $instance;
    $instance_data->updated = time();
    $instance_json = json_encode($instance_data);
    $ch = curl_init();
    $output = patch($ch, $urlToPatch, $instance_json);
    $output = post($ch, $urlToPost, $instance_json);
    $name = json_decode($output)->name;
    $filename = $gcsBaseURL . $instance . "/file" . $name . ".txt";
    file_put_contents($filename, $name);
    $instance = $_SERVER['INSTANCE_ID'];
    $urlToPatch = $fbBaseURL . "appengine/" . $instance . ".json";
    $urlToPost = $fbBaseURL . "appengine/" . $instance . “/sessions.json";
    $instance_data = new stdClass();
    $instance_data->instance = $instance;
    $instance_data->updated = time();
    $instance_json = json_encode($instance_data);
    $ch = curl_init();
    $output = patch($ch, $urlToPatch, $instance_json);
    $output = post($ch, $urlToPost, $instance_json);
    $name = json_decode($output)->name;
    $filename = $gcsBaseURL . $instance . "/file" . $name . ".txt";
    file_put_contents($filename, $name);

    View Slide

  23. ‹#›
    @tpryan
    Source Code
    function patch($ch, $url, $json){
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PATCH");
    curl_setopt($ch, CURLOPT_POSTFIELDS, $json);
    return curl_exec($ch);
    }
    function post($ch, $url, $json){
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
    curl_setopt($ch, CURLOPT_POSTFIELDS, $json);
    return curl_exec($ch);
    }

    View Slide

  24. ‹#›
    @tpryan

    View Slide

  25. ‹#›
    @tpryan

    View Slide

  26. ‹#›
    @tpryan

    View Slide

  27. @briandorsey
    660,000 qpm
    /60
    11,000 qps
    /3
    3,666 qps

    View Slide

  28. ‹#›
    @tpryan
    03 Fix 1
    Sockets

    View Slide

  29. ‹#›
    @tpryan

    View Slide

  30. ‹#›
    @tpryan

    View Slide

  31. ‹#›
    @tpryan

    View Slide

  32. ‹#›
    @tpryan
    Demo: App Engine

    View Slide

  33. ‹#›
    @tpryan
    Reasons for failure
    • Sockets are lighter
    • But implementation is not configurable
    • So write your own socket client?

    View Slide

  34. ‹#›
    @tpryan
    04 Fix 2
    Memcache

    View Slide

  35. ‹#›
    @tpryan
    Assumptions
    • Visualization needs to be realtime
    • But does it?

    View Slide

  36. ‹#›
    @tpryan
    Memcache on App Engine

    View Slide

  37. ‹#›
    @tpryan
    Pseudo Code
    // store instances request count
    // Get list of keys
    // Display request counts
    Bad Idea

    View Slide

  38. ‹#›
    @tpryan

    View Slide

  39. ‹#›
    @tpryan
    Source Code
    $memcache = new Memcached;
    $instance = $_SERVER['INSTANCE_ID'];
    $request = $_SERVER['REQUEST_LOG_ID'];
    $result = $memcache->increment($instance);
    if ($result == 1 ){
    $result = $memcache->append("instances", "|" . $instance);
    if ($memcache->getResultCode() == 14){
    $result = $memcache->set("instances", $instance);
    }
    }
    $filename = $gcsBaseURL . $instance . "/file" . $request . ".txt";
    file_put_contents($filename, $filename);

    View Slide

  40. ‹#›
    @tpryan
    Close but no cigar
    • Losing instances
    • Analysis revealed:
    • All requests were firing
    • All requests were registering
    • Cause: Memcache strings were getting too long

    View Slide

  41. ‹#›
    @tpryan
    Source Code
    $memcache = new Memcached;
    $instance = $_SERVER['INSTANCE_ID'];
    $request = $_SERVER['REQUEST_LOG_ID'];
    $memcache->increment("total");
    $result = $memcache->increment($instance);
    if ($result == 1 ){
    $result = $memcache->append("instances", "|" . $instance);
    if ($memcache->getResultCode() == 14){
    $result = $memcache->set("instances", $instance);
    }
    }
    $filename = $gcsBaseURL . $instance . "/file" . $request . ".txt";
    file_put_contents($filename, $filename);
    $memcache = new Memcached;
    $instance = $_SERVER['INSTANCE_ID'];
    $request = $_SERVER['REQUEST_LOG_ID'];
    $memcache->increment("total");
    $result = $memcache->increment($instance);
    if ($result == 1 ){
    $result = $memcache->append("instances", "|" . $instance);
    if ($memcache->getResultCode() == 14){
    $result = $memcache->set("instances", $instance);
    }
    }
    $filename = $gcsBaseURL . $instance . "/file" . $request . ".txt";
    file_put_contents($filename, $filename);

    View Slide

  42. ‹#›
    @tpryan
    Source Code
    function whichInstanceList($instance){
    $last = substr($instance, -1);
    return "instances-" . $last;
    }

    View Slide

  43. ‹#›
    @tpryan
    Demo: App Engine

    View Slide

  44. ‹#›
    @tpryan
    05 Conclusion
    What did I learn

    View Slide

  45. @briandorsey
    Wait, you’re giving up
    realtime without
    a fight?

    View Slide

  46. ‹#›
    @tpryan
    05 Fix 3
    Brining back the realtime

    View Slide

  47. ‹#›
    @tpryan
    Current
    /
    //Write to Memcache
    //Write to Cloud Storage
    //Receive realtime

    //updates
    /visualizer
    http://load-demo.appspot.com
    //Display Updates
    //Store request details

    View Slide

  48. ‹#›
    @tpryan
    Current
    /
    //Write to Memcache
    //Write to Cloud Storage
    //Receive realtime

    //updates
    /visualizer
    http://load-demo.appspot.com
    //Display Updates
    //Store request details
    //Read memcache
    //Write to Firebase
    /realtime.php

    View Slide

  49. ‹#›
    @tpryan
    Pseudo Code
    // Get list of instances
    // Get list of number of requests for each instance
    // Send data for each request to firebase
    Bad Idea

    View Slide

  50. ‹#›
    @tpryan
    How Firebase works
    • appengine
    • instance1
    • request1
    • request2
    • request3
    • request4
    • request5
    • request6
    • instance2

    View Slide

  51. ‹#›
    @tpryan
    How Firebase works
    {
    "appengine": {
    "instance1": {
    "request1": "",
    "request2": "",
    "request3": "",
    "request4": "",
    "request5": "",
    "request6": ""
    },
    "instance2": {}
    }
    }

    View Slide

  52. ‹#›
    @tpryan
    How Firebase works
    {
    "appengine": {
    "instance1": {
    "request1": "",
    "request2": "",
    "request3": "",
    "request4": "",
    "request5": "",
    "request6": ""
    },
    "instance2": {}
    }
    }
    /appengine/instance1/request2.json
    GET
    POST
    PATCH

    View Slide

  53. ‹#›
    @tpryan
    How Firebase works
    {
    "appengine": {
    "instance1": {
    "request1": "",
    "request2": "",
    "request3": "",
    "request4": "",
    "request5": "",
    "request6": ""
    },
    "instance2": {}
    }
    }
    /appengine/instance1.json
    GET
    POST
    PATCH

    View Slide

  54. ‹#›
    @tpryan
    How Firebase works
    {
    "appengine": {
    "instance1": {
    "request1": "",
    "request2": "",
    "request3": "",
    "request4": "",
    "request5": "",
    "request6": ""
    },
    "instance2": {}
    }
    }
    /appengine.json
    GET
    POST
    PATCH

    View Slide

  55. ‹#›
    @tpryan

    View Slide

  56. ‹#›
    @tpryan
    Batch Memcache calls
    getMulti()

    View Slide

  57. ‹#›
    @tpryan
    Let’s send too much stuff
    • appengine
    • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0
    • request1
    • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0
    • request2
    • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0
    • request3
    • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0
    • request4
    • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0
    • request5
    • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0
    • request6
    • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0
    • 00c61b117cf44784cd0c142018b2ada185ce6d3472a963342388ef123a30

    View Slide

  58. ‹#›
    @tpryan
    Let’s send too much stuff
    • appengine
    • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0
    • 6
    • 00c61b117cf44784cd0c142018b2ada185ce6d3472a963342388ef123a30
    • 1

    View Slide

  59. ‹#›
    @tpryan

    View Slide

  60. ‹#›
    @tpryan

    View Slide

  61. ‹#›
    @tpryan

    View Slide

  62. ‹#›
    @tpryan

    View Slide

  63. ‹#›
    @tpryan

    View Slide

  64. ‹#›
    @tpryan
    Source Code
    $task1 = new PushTask('/realtime.php');
    $task1->add("realtimesynch");

    View Slide

  65. ‹#›
    @tpryan
    Demo: App Engine

    View Slide

  66. ‹#›
    @tpryan
    06 Conclusion
    What did I learn

    View Slide

  67. @briandorsey
    Safety Quotas are not just
    something to be gotten around

    View Slide

  68. @briandorsey
    Bundle up data calls
    when you can

    View Slide

  69. @briandorsey
    Be prepared to make
    compromises

    View Slide

  70. @briandorsey
    Time for a language change?

    View Slide

  71. ‹#›
    @tpryan
    Thank You
    terrenceryan.com
    @tpryan
    This preso: http://bit.ly/tpryan-30m

    View Slide