Guzzle Promiseを使った
非同期処理によるAPIコールの高速化

Guzzle Promiseを使った
非同期処理によるAPIコールの高速化

JavaScript で非同期処理を実現する Promise という機構はご存知でしょうか?

今回は「Promise の考え方を PHP で実装した Guzzle Promise」を使って、大量の API コールを高速化したときの経験についてお話してみたいと思います。

Talked:
- http://phpcon.fukuoka.jp/

Sample codes:
- https://github.com/suzuki/guzzle-promise-sample

8e528456ff66ec543952daa815353a01?s=128

Norio Suzuki

May 21, 2016
Tweet

Transcript

  1. Guzzle PromiseΛ࢖ͬͨ
 ඇಉظॲཧʹΑΔAPIίʔϧͷߴ଎Խ PHPΧϯϑΝϨϯε෱Ԭ2016 2016-05-21-Sat @suzuki

  2. About me

  3. My favorite account is "suzuki" GitHub: @suzuki Twitter: @suzuki

  4. PHP Conference Kansai 2015 https://speakerdeck.com/suzuki/automatically-run-the-javascript-test-in-multiple-browsers

  5. https://twitter.com/hasegawayosuke/status/704143316611235840

  6. None
  7. None
  8. Story

  9. Background • ͦΖͦΖೖࣾ1ϲ݄ʹͳΔࠒͷ࿩ • ؆୯ͳमਖ਼Ҋ݅͸͍͔ͭ͘ऴΘΒͤͨ • ৽͍͠Ҋ݅ʹΞαΠϯ͞Εͨ

  10. Requirement • 1࣌ؒʹ1ճͷόονॲཧͰ࣮ߦ͢Δ • 1ճͷॲཧʹ͖ͭ࠷େ100ສ݅ͷσʔλΛ
 APIͰૹ৴͢Δ • APIίʔϧ1ճͰɺ1݅ͷσʔλૹ৴͕Մೳ → 1࣌ؒʹ࠷େ100ສ݅ͷAPIίʔϧ͕ඞཁ

  11. Think of the system • ࣮૷ݴޠ • PHPͰ͸ແཧʁ • Go΍Scala౳͕ద੾ͩͱͯ͠΋࣮૷ܦݧ͸ͳ͍

    • ίʔϧઌͷAPI • ॳΊͯར༻͢ΔAPI • ΞαΠϯ࣌఺Ͱ͸ಈ࡞͕೺ѲͰ͖͍ͯͳ͍ → ॳΊͯͷݴޠͰॳΊͯͷAPIίʔϧ͢Δඞཁ͕ʂʁ
  12. Policy to implement • ׳Ε͍ͯΔݴޠͰ·ͣ͸ಈ͔͢ • εϐʔυ͕଍Γͳ͚Ε͹ଞͷݴޠͰ࣮૷͢Δ → PHP +

    Guzzle ͰͷϦϑΝϨϯε࣮૷Λ͢Δ
  13. Guzzle

  14. Topic: What is guzzle ? http://eow.alc.co.jp/search?q=Guzzle

  15. http://docs.guzzlephp.org/en/latest/

  16. Usage use GuzzleHttp\Client; $client = new Client(); $response = $client->request(

    'GET', 'http://localhost:8008/' );
  17. GET, Magic method $response = $client->request( 'GET', 'http://localhost:8008/' ); $response

    = $client->get( 'http://localhost:8008/' );
  18. Other magic methods $response = $client->get('http://localhost/'); $response = $client->delete('http://localhost/'); $response

    = $client->head('http://localhost/'); $response = $client->options('http://localhost/'); $response = $client->patch('http://localhost/'); $response = $client->post('http://localhost/'); $response = $client->put('http://localhost/');
  19. Using Request object use GuzzleHttp\Client; use GuzzleHttp\Psr7\Request; $client = new

    Client(); $request = new Request( 'GET', 'http://localhost:8008' ); $response = $client->send($request);
  20. Response object $response = $client->request( 'GET', 'http://localhost:8008/' ); echo get_class($response);

    => GuzzleHttp\Psr7\Response
  21. None
  22. Test Server

  23. server.js var http = require('http'); var sleep = require('sleep-async')(); var

    server = http.createServer(); var sleepMiliSecond = 1000; server.on('request', function(req, res) { console.log('Received.'); sleep.sleep(sleepMiliSecond, function() { res.writeHead(200, {'Content-Type': 'text/json'}); res.write('{"status": "ok"}'); res.end(); }); }); server.listen(8008, 'localhost');
  24. server.js • ҙਤతʹ1ඵͷsleepΛೖΕ͍ͯΔ • গ͠஗ΊͷAPIΛγϛϡϨʔτ

  25. Run server $ npm install $ node server.js

  26. Request using Curl

  27. sample01.php use GuzzleHttp\Client; $client = new Client(); $response = $client->request(

    'GET', 'http://localhost:8008/' ); echo $response->getBody() . PHP_EOL;
  28. Request using Guzzle

  29. Summary of sample01.php • 1ճͷϦΫΤετͳͷͰɺॲཧͷ஗͞ʢ1ඵͷ sleepʣ͸͋·ΓؾʹͳΒͳ͍ • Ͱ͸ճ਺Λଟͯ͘͠ΈΔͱͲ͏ͳΔͷ͔ʁ

  30. sample02.php for ($i = 0; $i < 10; ++$i) {

    $response = $client->request( 'GET', 'http://localhost:8008' ); echo $response->getBody() . PHP_EOL; }
  31. Run sample02.php

  32. Summary of sample02.php • ෳ਺ճͷAPIίʔϧ΋໰୊ͳ͘Ͱ͖ͨ • ͨͩ͠εϐʔυ͸஗͍ • 1ඵͷsleep͕10ճߦΘΕΔ

  33. ࣮Ҋ݅Ͱͷεϐʔυײ • ͜ͷํ๏ʹ͍ۙܗͰͷϦϑΝϨϯε࣮૷Λͨ͠ • 10,000ϦΫΤετʹ10਺෼͔͔ͬͨ • 50,000ϦΫΤετ͕1࣌ؒܦաͯ͠΋ऴΘΒͣ • ఘΊͯڧ੍ऴྃͨ͠ →

    ͜ͷ··Ͱ͸ཁ݅Λຬͨͤͳ͍
  34. ղܾํ๏͸ʁ • σʔλΛ෼ׂ͢Δʁ • ෼ׂ਺ͷ਺͚ͩεΫϦϓτΛىಈʁ • ෳ਺ϓϩηεͰ࣮ߦ͢Δʁ • forkͯ͠ؤுΔʁ •

    ඇಉظϦΫΤετ͕͋ͬͨɺ͜ΕͰͳΜͱ͔ͳΔʂʁ → Guzzle ͷ Async ϦΫΤετΛ࢖ͬͯΈΔ͜ͱʹ
  35. Async Requests

  36. http://docs.guzzlephp.org/en/latest/quickstart.html#async-requests

  37. Sync and Async methods $client->get() $client->getAsync() $client->request() $client->requestAsync()

  38. sample03.php for ($i = 0; $i < 10; ++$i) {

    $response = $client->requestAsync( 'GET', 'http://localhost:8008/' ); echo $response->getBody() . PHP_EOL; }
  39. Run sample03.php

  40. Summary of sample03.php PHP Fatal error: Call to undefined method

    GuzzleHttp\Promise\Promise::getBody()
  41. Not $response

  42. Promise object $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); echo get_class($promise);

    => GuzzleHttp\Promise\Promise
  43. sample04.php for ($i = 0; $i < 10; ++$i) {

    $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); // echo $response->getBody() . PHP_EOL; }
  44. Run sample04.php

  45. Summary of sample04.php • PHP Fatal error͸ग़ͳ͘ͳͬͨ • ࣮ߦʹ͸੒ޭ͍ͯ͠Δʁ •

    αʔόଆʹʮReceived.ʯͷද͕ࣔग़͍ͯͳ͍ • ϦΫΤετ͕ಧ͍͍ͯͳ͍ʁʁ • Promise ͬͯͳʹʁ ͋ͷ Promise ͷ͜ͱʁʁ
  46. Promise

  47. Famous in JavaScript http://azu.github.io/promises-book/

  48. • ඇಉظॲཧΛந৅Խͨ͠ΦϒδΣΫτ 1SPNJTF What is Promise ? ඇಉظॲཧͷ࣮ߦ ੒ޭͨ࣌͠ SFTPMWF

    ࣦഊͨ࣌͠ SFKFDU
  49. Method Chains 1SPNJTF then then 1SPNJTF 1SPNJTF 1SPNJTF catch

  50. Promise overview var promise = new Promise(function(resolve, reject) { //

    Async routines if (isSuccess) { resolve(value); } else { reject(error); } });
  51. Method Chains sample var promise = new Promise(...); promise .then(function(value)

    { if (isSuccess) { resolve(value); } else { reject(error); }; }) .then(function(value) { if (isSuccess) { resolve(value); } else { reject(error); }; }) .catch(function(error) { // handle error }) ; 1SPNJTF 1SPNJTF 1SPNJTF 1SPNJTF then then catch
  52. Status of Promise ඇಉظॲཧͷ࣮ߦ ॲཧͷ੒൱ TUBUVT ࣮ߦલ  1FOEJOH ࣮ߦޙ

    ੒ޭ 'VMpMMFE ࣮ߦޙ ࣦഊ 3FKFDUFE
  53. Spec: Promises/A+ https://promisesaplus.com/

  54. Spec: ECMAScript 2015 (ES6) http://www.ecma-international.org/ecma-262/6.0/

  55. Guzzle Promise

  56. Guzzle Promises https://github.com/guzzle/promises#guzzle-promises

  57. Promise is required by Guzzle https://github.com/guzzle/guzzle/blob/master/composer.json#L18

  58. Guzzle Promise overview use GuzzleHttp\Promise\Promise; $promise = new Promise(function() use

    (&$promise) { // Async routine if ($isSuccess) { $promise->resolve($value); } else { $promise->reject($error); } });
  59. Async Request using Promise use GuzzleHttp\Client; use Psr\Http\Message\ResponseInterface; use GuzzleHttp\Exception\RequestException;

    $client = new Client(); $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); $promise ->then(function(ResponseInterface $response) { echo $response->getBody() . PHP_EOL; }) ->otherwise(function(RequestException $e) { echo $e->getMessage() . PHP_EOL; }) ; $promise->wait();
  60. sample05.php for ($i = 0; $i < 10; ++$i) {

    $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); $promise ->then(function(ResponseInterface $response) { echo $response->getBody() . PHP_EOL; }) ; $promise->wait(); }
  61. Run sample05.php

  62. Tooooo slow …

  63. sample05.php for ($i = 0; $i < 10; ++$i) {

    $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); $promise ->then(function(ResponseInterface $response) { echo $response->getBody() . PHP_EOL; }) ; $promise->wait(); } Not Async
  64. sample06.php use GuzzleHttp\Promise; $promises = []; for ($i = 0;

    $i < 10; ++$i) { $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); $promise ->then(function(ResponseInterface $response) { echo $response->getBody() . PHP_EOL; }); $promises[] = $promise; } Promise\all($promises)->wait();
  65. Run sample06.php

  66. Fast !!

  67. sample07.php $promises = []; for ($i = 0; $i <

    10000; ++$i) { $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); $promise ->then(function(ResponseInterface $response) { echo $response->getBody() . PHP_EOL; }) ; $promises[] = $promise; } Promise\all($promises)->wait();
  68. Run sample07.php

  69. Summary of sample07.php PHP Fatal error: Allowed memory size of

    134217728 bytes exhausted (tried to allocate 72 bytes)
  70. Why exhausted memory limit ? • $promises ഑ྻʹ10000ݸͷ Promise Φϒ

    δΣΫτ͕ೖͬͨ··ղ์͞Ε͍ͯͳ͍ • ͦΕͧΕͷ Promise ΦϒδΣΫτ͸ঢ়ଶΛอ ͍࣋ͯ͠ΔͷͰͦΕͳΓʹॏ͍σʔλྔ → Կ͔ղܾํ๏͸ʁ
  71. Generator

  72. http://php.net/manual/en/language.generators.overview.php

  73. Generator $requests = function() { for ($i = 0; $i

    < 10; ++$i) { yield new Request( 'GET', 'http://localhost:8008/' ); } };
  74. How to use Generator sample08.php use GuzzleHttp\Client; use GuzzleHttp\Psr7\Request; $client

    = new Client(); $requests = function() { for ($i = 0; $i < 10; ++$i) { yield new Request('GET', 'http://localhost:8008/'); } }; foreach ($requests() as $request) { $response = $client->send($request); echo $response->getBody() . PHP_EOL; }
  75. Topic: Generator can use in ES6 https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Generator

  76. Pool

  77. Pool overview use GuzzleHttp\Pool; $pool = new Pool($client, $requests(), [

    'concurrency' => 5, 'fulfilled' => function ($response, $index) { // when success }, 'rejected' => function ($reason, $index) { // when failed }, ]); Generator
  78. Pool config LFZ UZQF QVSQPTF DPODVSSFODZ JOUFHFS NBYDPODVSSFODZQBSBNFUFS GVMpMMFE DBMMBCMFGVODUJPO

    $BMMXIFO1SPNJTFJTSFTPMWFE SFKFDUFE DBMMBCMFGVODUJPO $BMMXIFO1SPNJTFJTSFKFDUFE
  79. sample09.php $client = new Client(); $requests = function() { for

    ($i = 0; $i < 10; ++$i) { yield new Request('GET', 'http://localhost:8008/'); } }; $pool = new Pool($client, $requests(), [ 'concurrency' => 5, 'fulfilled' => function(ResponseInterface $response, $index) { echo $response->getBody() . PHP_EOL; }, ]); $promise = $pool->promise(); $promise->wait();
  80. Run sample09.php

  81. Summary of sample09.php • Ұؾʹ5ͭͷϦΫΤετ͕౤͛ΒΕ͍ͯΔ • concurrency = 5 Ͱ΋͍ͩͿ଎͍

    • Ͱ͸ concurrency Λ૿΍ͤ͹΋ͬͱ଎͍ʁ
  82. Concurrency = 10 sample10.php $pool = new Pool($client, $requests(), [

    'concurrency' => 10, 'fulfilled' => function(ResponseInterface $response, $index) { echo $response->getBody() . PHP_EOL; }, ]);
  83. Run sample11.php

  84. Summary of sample10.php • concurrency = 5 ΑΓ΋͞Βʹ଎͍ • Ͱ͸ɺ΋ͬͱ΋ͬͱ૿΍ͤ͹͍͍ͷͰ͸ʁ

  85. sample11.php $client = new Client(); $requests = function() { for

    ($i = 0; $i < 10000; ++$i) { yield new Request('GET', 'http://localhost:8008/'); } }; $pool = new Pool($client, $requests(), [ 'concurrency' => 1000, 'fulfilled' => function(ResponseInterface $response, $index) { echo sprintf("%5d: %s\n", $index, $response->getBody()); }, ]); $promise = $pool->promise(); $promise->wait();
  86. Run sample11.php

  87. Summary of sample11.php • 10000ճͷϦΫΤετʹ੒ޭ • ϝϞϦෆ଍ʹͳΒͳ͔ͬͨ • concurrency =

    1000 Ͱ΋༨༟ʁ • Ͳ͜·Ͱ૿΍ͤΔ΋ͷͳͷ͔ʁ
  88. None
  89. None
  90. None
  91. ࣮Ҋ݅Ͱͷ Pool ར༻ • concurrency = 100 Λ࠾༻ • APIαʔό΁ͷಉ࣌ΞΫηε਺͸ڐ༰ൣғ

    • ϝϞϦͷ࢖༻ྔ΋ڐ༰ൣғ • ໿50ສϦΫΤετΛ30෼͔͔Βͣʹ׬ྃՄೳ → 1࣌ؒ100ສϦΫΤετΛΫϦΞ
  92. Conclusion

  93. Promise • JavaScriptͰ࢖ΘΕ͍ͯΔPromise͕PHPʹ΋ • JavaScriptͷ஌͕ࣝPHPͰ׆͔ͤΔ • PHPͰ΋࢖ΘΕ͍ͯΔGenerator͕JSʹ΋ • PHPͷ஌͕ࣝJavaScriptͰ׆͔ͤΔ →

    ೔ࠒ͔Βෳ਺ͷݴޠΛ࢖͓ͬͯ͘ͱศར
  94. Guzzle • ෳ਺ͷϦΫΤετ͕ඞཁͳΒAsyncΛར༻ • Promiseͷ֓೦Ͱܧଓ͢Δॲཧ͕΍Γ΍͍͢ • PoolΛ࢖͑͹઀ଓ਺੍ޚ΋؆୯ʹ • concurrency͸APIଆͱϝϞϦʹ༏͍͠਺஋Ͱ →

    ਺ेສ݅Λૹ৴͢Δ৔߹Ͱ΋҆ఆՔಇ
  95. Company • ೖࣾ1ϲ݄Ͱʢ৽ਓʹ͸ʣ೉қ౓ߴΊͷҊ͕݅ ߱ͬͯ͘Δ͜ͱ΋͋Δ • ݫ͕͋͠͞Δձࣾ • ৘ใڞ༗͢Δͱ஥͔ؒΒΞυόΠε͕߱ͬͯ͘ Δ͜ͱ΋͋Δ •

    ༏͠͞΋͋Δձࣾ
  96. We are hiring ! https://www.mercari.com/jp/jobs/

  97. Thank you

  98. Appendix

  99. Sample codes • https://github.com/suzuki/guzzle-promise-sample

  100. PHP side links • Guzzle Documentation • http://docs.guzzlephp.org/en/latest/ • Guzzle

    Promises • https://github.com/guzzle/promises • PHP Documentation: Generator • http://php.net/manual/en/language.generators.overview.php • AWS SDK for PHP • https://docs.aws.amazon.com/aws-sdk-php/v3/guide/
  101. JavaScript side links • JavaScript Promiseͷຊ • http://azu.github.io/promises-book/ •Promise/A+ •

    https://promisesaplus.com/ •ECMAScript 2015 (ES6) • http://www.ecma-international.org/ecma-262/6.0/ •MDN Generator • https://developer.mozilla.org/en-US/docs/Web/JavaScript/ Reference/Global_Objects/Generator
  102. References • Asynchronous API Interaction with Guzzle • https://speakerdeck.com/jeremeamia/asynchronous-api- interaction-with-guzzle

    • JavaScript Promises - Thinking Sync in an Async World • https://speakerdeck.com/kerrick/javascript-promises- thinking-sync-in-an-async-world • guzzleͰඇಉظϦΫΤετΛฒྻॲཧͰ͍ͬͺ͍౤͛Δ • http://hacknote.jp/archives/16095/
  103. Topic: Use case of Pool https://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/commands.html#commandpool

  104. Topic: Use case of Promise https://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/promises.html