30,000,000 Requests in a Hour in the Cloud

Transcript

1. ‹#› @tpryan Terrence Ryan Developer Advocate 30,000,000 Requests An Hour In The Cloud
2. ‹#› @tpryan Who are you?
3. ‹#› @tpryan 01 Introduction What was I trying to do
5. ‹#› @tpryan Demo: App Engine
6. ‹#› @tpryan App Engine • Writes to Firebase • Writes to Cloud Storage • Run the Visualizer
7. ‹#› @tpryan Compute Engine • Load Generators • Apache Bench • Remote Command Execution
8. ‹#› @tpryan Firebase • Realtime Display • Tracking all load
9. ‹#› @tpryan Cloud Storage • Proof of activity • Object notification
10. ‹#› @tpryan App Engine • Process Notifications from Cloud Storage
11. ‹#› @tpryan Planned / //Write to Firebase //Write to Cloud Storage //Receive realtime
 //updates /visualizer /storage //Write to Firebase //Display updates //Store file //Send Notification http://load-demo.appspot.com
12. ‹#› @tpryan Actual / //Write to Firebase //Write to Cloud Storage //Receive realtime
 //updates /visualizer /storage //Write to Firebase //Display updates //Store file //Send Notification http://load-demo.appspot.com
13. @briandorsey 30,000,000 requests in one hour
14. @briandorsey What do you call a site that gets 30,000,000 hits an hour?
15. @briandorsey
16. @briandorsey I should try and do that and mean it
17. ‹#› @tpryan 02 Ramping up What is Google Cloud Platform
18. @briandorsey 30,000,000 qph /60/60 8,333 qps
19. @briandorsey Ramp the demo up to 8,333 qps
20. ‹#› @tpryan Demo: App Engine
21. ‹#› @tpryan URLFetch • Network abstraction layer for http • Most service calls
22. ‹#› @tpryan Source Code $instance = $_SERVER['INSTANCE_ID']; $urlToPatch = $fbBaseURL . "appengine/" . $instance . ".json"; $urlToPost = $fbBaseURL . "appengine/" . $instance . “/sessions.json"; $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->updated = time(); $instance_json = json_encode($instance_data); $ch = curl_init(); $output = patch($ch, $urlToPatch, $instance_json); $output = post($ch, $urlToPost, $instance_json); $name = json_decode($output)->name; $filename = $gcsBaseURL . $instance . "/file" . $name . ".txt"; file_put_contents($filename, $name); $instance = $_SERVER['INSTANCE_ID']; $urlToPatch = $fbBaseURL . "appengine/" . $instance . ".json"; $urlToPost = $fbBaseURL . "appengine/" . $instance . “/sessions.json"; $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->updated = time(); $instance_json = json_encode($instance_data); $ch = curl_init(); $output = patch($ch, $urlToPatch, $instance_json); $output = post($ch, $urlToPost, $instance_json); $name = json_decode($output)->name; $filename = $gcsBaseURL . $instance . "/file" . $name . ".txt"; file_put_contents($filename, $name);
23. ‹#› @tpryan Source Code function patch($ch, $url, $json){ curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PATCH"); curl_setopt($ch, CURLOPT_POSTFIELDS, $json); return curl_exec($ch); } function post($ch, $url, $json){ curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST"); curl_setopt($ch, CURLOPT_POSTFIELDS, $json); return curl_exec($ch); }
24. ‹#› @tpryan
25. ‹#› @tpryan
26. ‹#› @tpryan
27. @briandorsey 660,000 qpm /60 11,000 qps /3 3,666 qps
28. ‹#› @tpryan 03 Fix 1 Sockets
29. ‹#› @tpryan
30. ‹#› @tpryan
31. ‹#› @tpryan
32. ‹#› @tpryan Demo: App Engine
33. ‹#› @tpryan Reasons for failure • Sockets are lighter • But implementation is not configurable • So write your own socket client?
34. ‹#› @tpryan 04 Fix 2 Memcache
35. ‹#› @tpryan Assumptions • Visualization needs to be realtime • But does it?
36. ‹#› @tpryan Memcache on App Engine
37. ‹#› @tpryan Pseudo Code // store instances request count // Get list of keys // Display request counts Bad Idea
38. ‹#› @tpryan
39. ‹#› @tpryan Source Code $memcache = new Memcached; $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $result = $memcache->increment($instance); if ($result == 1 ){ $result = $memcache->append("instances", "|" . $instance); if ($memcache->getResultCode() == 14){ $result = $memcache->set("instances", $instance); } } $filename = $gcsBaseURL . $instance . "/file" . $request . ".txt"; file_put_contents($filename, $filename);
40. ‹#› @tpryan Close but no cigar • Losing instances • Analysis revealed: • All requests were firing • All requests were registering • Cause: Memcache strings were getting too long
41. ‹#› @tpryan Source Code $memcache = new Memcached; $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $memcache->increment("total"); $result = $memcache->increment($instance); if ($result == 1 ){ $result = $memcache->append("instances", "|" . $instance); if ($memcache->getResultCode() == 14){ $result = $memcache->set("instances", $instance); } } $filename = $gcsBaseURL . $instance . "/file" . $request . ".txt"; file_put_contents($filename, $filename); $memcache = new Memcached; $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $memcache->increment("total"); $result = $memcache->increment($instance); if ($result == 1 ){ $result = $memcache->append("instances", "|" . $instance); if ($memcache->getResultCode() == 14){ $result = $memcache->set("instances", $instance); } } $filename = $gcsBaseURL . $instance . "/file" . $request . ".txt"; file_put_contents($filename, $filename);
42. ‹#› @tpryan Source Code function whichInstanceList($instance){ $last = substr($instance, -1); return "instances-" . $last; }
43. ‹#› @tpryan Demo: App Engine
44. ‹#› @tpryan 05 Conclusion What did I learn
45. @briandorsey Wait, you’re giving up realtime without a fight?
46. ‹#› @tpryan 05 Fix 3 Brining back the realtime
47. ‹#› @tpryan Current / //Write to Memcache //Write to Cloud Storage //Receive realtime
 //updates /visualizer http://load-demo.appspot.com //Display Updates //Store request details
48. ‹#› @tpryan Current / //Write to Memcache //Write to Cloud Storage //Receive realtime
 //updates /visualizer http://load-demo.appspot.com //Display Updates //Store request details //Read memcache //Write to Firebase /realtime.php
49. ‹#› @tpryan Pseudo Code // Get list of instances // Get list of number of requests for each instance // Send data for each request to firebase Bad Idea
50. ‹#› @tpryan How Firebase works • appengine • instance1 • request1 • request2 • request3 • request4 • request5 • request6 • instance2
51. ‹#› @tpryan How Firebase works { "appengine": { "instance1": { "request1": "", "request2": "", "request3": "", "request4": "", "request5": "", "request6": "" }, "instance2": {} } }
52. ‹#› @tpryan How Firebase works { "appengine": { "instance1": { "request1": "", "request2": "", "request3": "", "request4": "", "request5": "", "request6": "" }, "instance2": {} } } /appengine/instance1/request2.json GET POST PATCH
53. ‹#› @tpryan How Firebase works { "appengine": { "instance1": { "request1": "", "request2": "", "request3": "", "request4": "", "request5": "", "request6": "" }, "instance2": {} } } /appengine/instance1.json GET POST PATCH
54. ‹#› @tpryan How Firebase works { "appengine": { "instance1": { "request1": "", "request2": "", "request3": "", "request4": "", "request5": "", "request6": "" }, "instance2": {} } } /appengine.json GET POST PATCH
55. ‹#› @tpryan
56. ‹#› @tpryan Batch Memcache calls getMulti()
57. ‹#› @tpryan Let’s send too much stuff • appengine • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request1 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request2 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request3 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request4 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request5 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • request6 • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • 00c61b117cf44784cd0c142018b2ada185ce6d3472a963342388ef123a30
58. ‹#› @tpryan Let’s send too much stuff • appengine • 00c61b117c1d99526c06775beef85526a00dbdd295ffef80dbe702a836b0 • 6 • 00c61b117cf44784cd0c142018b2ada185ce6d3472a963342388ef123a30 • 1
59. ‹#› @tpryan
60. ‹#› @tpryan
61. ‹#› @tpryan
62. ‹#› @tpryan
63. ‹#› @tpryan
64. ‹#› @tpryan Source Code $task1 = new PushTask('/realtime.php'); $task1->add("realtimesynch");
65. ‹#› @tpryan Demo: App Engine
66. ‹#› @tpryan 06 Conclusion What did I learn
67. @briandorsey Safety Quotas are not just something to be gotten around
68. @briandorsey Bundle up data calls when you can
69. @briandorsey Be prepared to make compromises
70. @briandorsey Time for a language change?
71. ‹#› @tpryan Thank You terrenceryan.com @tpryan This preso: http://bit.ly/tpryan-30m