$30 off During Our Annual Pro Sale. View Details »

Guzzle Promiseを使った
非同期処理によるAPIコールの高速化

Guzzle Promiseを使った
非同期処理によるAPIコールの高速化

JavaScript で非同期処理を実現する Promise という機構はご存知でしょうか?

今回は「Promise の考え方を PHP で実装した Guzzle Promise」を使って、大量の API コールを高速化したときの経験についてお話してみたいと思います。

Talked:
- http://phpcon.fukuoka.jp/

Sample codes:
- https://github.com/suzuki/guzzle-promise-sample

Norio Suzuki

May 21, 2016
Tweet

More Decks by Norio Suzuki

Other Decks in Technology

Transcript

  1. Guzzle PromiseΛ࢖ͬͨ

    ඇಉظॲཧʹΑΔAPIίʔϧͷߴ଎Խ
    PHPΧϯϑΝϨϯε෱Ԭ2016
    2016-05-21-Sat
    @suzuki

    View Slide

  2. About me

    View Slide

  3. My favorite account is "suzuki"
    GitHub: @suzuki Twitter: @suzuki

    View Slide

  4. PHP Conference Kansai 2015
    https://speakerdeck.com/suzuki/automatically-run-the-javascript-test-in-multiple-browsers

    View Slide

  5. https://twitter.com/hasegawayosuke/status/704143316611235840

    View Slide

  6. View Slide

  7. View Slide

  8. Story

    View Slide

  9. Background
    • ͦΖͦΖೖࣾ1ϲ݄ʹͳΔࠒͷ࿩
    • ؆୯ͳमਖ਼Ҋ݅͸͍͔ͭ͘ऴΘΒͤͨ
    • ৽͍͠Ҋ݅ʹΞαΠϯ͞Εͨ

    View Slide

  10. Requirement
    • 1࣌ؒʹ1ճͷόονॲཧͰ࣮ߦ͢Δ
    • 1ճͷॲཧʹ͖ͭ࠷େ100ສ݅ͷσʔλΛ

    APIͰૹ৴͢Δ
    • APIίʔϧ1ճͰɺ1݅ͷσʔλૹ৴͕Մೳ
    → 1࣌ؒʹ࠷େ100ສ݅ͷAPIίʔϧ͕ඞཁ

    View Slide

  11. Think of the system
    • ࣮૷ݴޠ
    • PHPͰ͸ແཧʁ
    • Go΍Scala౳͕ద੾ͩͱͯ͠΋࣮૷ܦݧ͸ͳ͍
    • ίʔϧઌͷAPI
    • ॳΊͯར༻͢ΔAPI
    • ΞαΠϯ࣌఺Ͱ͸ಈ࡞͕೺ѲͰ͖͍ͯͳ͍
    → ॳΊͯͷݴޠͰॳΊͯͷAPIίʔϧ͢Δඞཁ͕ʂʁ

    View Slide

  12. Policy to implement
    • ׳Ε͍ͯΔݴޠͰ·ͣ͸ಈ͔͢
    • εϐʔυ͕଍Γͳ͚Ε͹ଞͷݴޠͰ࣮૷͢Δ
    → PHP + Guzzle ͰͷϦϑΝϨϯε࣮૷Λ͢Δ

    View Slide

  13. Guzzle

    View Slide

  14. Topic: What is guzzle ?
    http://eow.alc.co.jp/search?q=Guzzle

    View Slide

  15. http://docs.guzzlephp.org/en/latest/

    View Slide

  16. Usage
    use GuzzleHttp\Client;
    $client = new Client();
    $response = $client->request(
    'GET',
    'http://localhost:8008/'
    );

    View Slide

  17. GET, Magic method
    $response = $client->request(
    'GET',
    'http://localhost:8008/'
    );
    $response = $client->get(
    'http://localhost:8008/'
    );

    View Slide

  18. Other magic methods
    $response = $client->get('http://localhost/');
    $response = $client->delete('http://localhost/');
    $response = $client->head('http://localhost/');
    $response = $client->options('http://localhost/');
    $response = $client->patch('http://localhost/');
    $response = $client->post('http://localhost/');
    $response = $client->put('http://localhost/');

    View Slide

  19. Using Request object
    use GuzzleHttp\Client;
    use GuzzleHttp\Psr7\Request;
    $client = new Client();
    $request = new Request(
    'GET',
    'http://localhost:8008'
    );
    $response = $client->send($request);

    View Slide

  20. Response object
    $response = $client->request(
    'GET',
    'http://localhost:8008/'
    );
    echo get_class($response);
    => GuzzleHttp\Psr7\Response

    View Slide

  21. View Slide

  22. Test Server

    View Slide

  23. server.js
    var http = require('http');
    var sleep = require('sleep-async')();
    var server = http.createServer();
    var sleepMiliSecond = 1000;
    server.on('request', function(req, res) {
    console.log('Received.');
    sleep.sleep(sleepMiliSecond, function() {
    res.writeHead(200, {'Content-Type': 'text/json'});
    res.write('{"status": "ok"}');
    res.end();
    });
    });
    server.listen(8008, 'localhost');

    View Slide

  24. server.js
    • ҙਤతʹ1ඵͷsleepΛೖΕ͍ͯΔ
    • গ͠஗ΊͷAPIΛγϛϡϨʔτ

    View Slide

  25. Run server
    $ npm install
    $ node server.js

    View Slide

  26. Request using Curl

    View Slide

  27. sample01.php
    use GuzzleHttp\Client;
    $client = new Client();
    $response = $client->request(
    'GET',
    'http://localhost:8008/'
    );
    echo $response->getBody() . PHP_EOL;

    View Slide

  28. Request using Guzzle

    View Slide

  29. Summary of sample01.php
    • 1ճͷϦΫΤετͳͷͰɺॲཧͷ஗͞ʢ1ඵͷ
    sleepʣ͸͋·ΓؾʹͳΒͳ͍
    • Ͱ͸ճ਺Λଟͯ͘͠ΈΔͱͲ͏ͳΔͷ͔ʁ

    View Slide

  30. sample02.php
    for ($i = 0; $i < 10; ++$i) {
    $response = $client->request(
    'GET',
    'http://localhost:8008'
    );
    echo $response->getBody() . PHP_EOL;
    }

    View Slide

  31. Run sample02.php

    View Slide

  32. Summary of sample02.php
    • ෳ਺ճͷAPIίʔϧ΋໰୊ͳ͘Ͱ͖ͨ
    • ͨͩ͠εϐʔυ͸஗͍
    • 1ඵͷsleep͕10ճߦΘΕΔ

    View Slide

  33. ࣮Ҋ݅Ͱͷεϐʔυײ
    • ͜ͷํ๏ʹ͍ۙܗͰͷϦϑΝϨϯε࣮૷Λͨ͠
    • 10,000ϦΫΤετʹ10਺෼͔͔ͬͨ
    • 50,000ϦΫΤετ͕1࣌ؒܦաͯ͠΋ऴΘΒͣ
    • ఘΊͯڧ੍ऴྃͨ͠
    → ͜ͷ··Ͱ͸ཁ݅Λຬͨͤͳ͍

    View Slide

  34. ղܾํ๏͸ʁ
    • σʔλΛ෼ׂ͢Δʁ
    • ෼ׂ਺ͷ਺͚ͩεΫϦϓτΛىಈʁ
    • ෳ਺ϓϩηεͰ࣮ߦ͢Δʁ
    • forkͯ͠ؤுΔʁ
    • ඇಉظϦΫΤετ͕͋ͬͨɺ͜ΕͰͳΜͱ͔ͳΔʂʁ
    → Guzzle ͷ Async ϦΫΤετΛ࢖ͬͯΈΔ͜ͱʹ

    View Slide

  35. Async Requests

    View Slide

  36. http://docs.guzzlephp.org/en/latest/quickstart.html#async-requests

    View Slide

  37. Sync and Async methods
    $client->get()
    $client->getAsync()
    $client->request()
    $client->requestAsync()

    View Slide

  38. sample03.php
    for ($i = 0; $i < 10; ++$i) {
    $response = $client->requestAsync(
    'GET',
    'http://localhost:8008/'
    );
    echo $response->getBody() . PHP_EOL;
    }

    View Slide

  39. Run sample03.php

    View Slide

  40. Summary of sample03.php
    PHP Fatal error:
    Call to undefined method
    GuzzleHttp\Promise\Promise::getBody()

    View Slide

  41. Not $response

    View Slide

  42. Promise object
    $promise = $client->requestAsync(
    'GET',
    'http://localhost:8008/'
    );
    echo get_class($promise);
    => GuzzleHttp\Promise\Promise

    View Slide

  43. sample04.php
    for ($i = 0; $i < 10; ++$i) {
    $promise = $client->requestAsync(
    'GET',
    'http://localhost:8008/'
    );
    // echo $response->getBody() . PHP_EOL;
    }

    View Slide

  44. Run sample04.php

    View Slide

  45. Summary of sample04.php
    • PHP Fatal error͸ग़ͳ͘ͳͬͨ
    • ࣮ߦʹ͸੒ޭ͍ͯ͠Δʁ
    • αʔόଆʹʮReceived.ʯͷද͕ࣔग़͍ͯͳ͍
    • ϦΫΤετ͕ಧ͍͍ͯͳ͍ʁʁ
    • Promise ͬͯͳʹʁ ͋ͷ Promise ͷ͜ͱʁʁ

    View Slide

  46. Promise

    View Slide

  47. Famous in JavaScript
    http://azu.github.io/promises-book/

    View Slide

  48. • ඇಉظॲཧΛந৅Խͨ͠ΦϒδΣΫτ
    1SPNJTF
    What is Promise ?
    ඇಉظॲཧͷ࣮ߦ
    ੒ޭͨ࣌͠
    SFTPMWF
    ࣦഊͨ࣌͠
    SFKFDU

    View Slide

  49. Method Chains
    1SPNJTF
    then then
    1SPNJTF 1SPNJTF 1SPNJTF
    catch

    View Slide

  50. Promise overview
    var promise = new Promise(function(resolve, reject) {
    // Async routines
    if (isSuccess) {
    resolve(value);
    } else {
    reject(error);
    }
    });

    View Slide

  51. Method Chains sample
    var promise = new Promise(...);
    promise
    .then(function(value) {
    if (isSuccess) {
    resolve(value);
    } else {
    reject(error);
    };
    })
    .then(function(value) {
    if (isSuccess) {
    resolve(value);
    } else {
    reject(error);
    };
    })
    .catch(function(error) {
    // handle error
    })
    ;
    1SPNJTF
    1SPNJTF
    1SPNJTF
    1SPNJTF
    then
    then
    catch

    View Slide

  52. Status of Promise
    ඇಉظॲཧͷ࣮ߦ ॲཧͷ੒൱ TUBUVT
    ࣮ߦલ 1FOEJOH
    ࣮ߦޙ ੒ޭ 'VMpMMFE
    ࣮ߦޙ ࣦഊ 3FKFDUFE

    View Slide

  53. Spec: Promises/A+
    https://promisesaplus.com/

    View Slide

  54. Spec: ECMAScript 2015 (ES6)
    http://www.ecma-international.org/ecma-262/6.0/

    View Slide

  55. Guzzle Promise

    View Slide

  56. Guzzle Promises
    https://github.com/guzzle/promises#guzzle-promises

    View Slide

  57. Promise is required by Guzzle
    https://github.com/guzzle/guzzle/blob/master/composer.json#L18

    View Slide

  58. Guzzle Promise overview
    use GuzzleHttp\Promise\Promise;
    $promise = new Promise(function() use (&$promise) {
    // Async routine
    if ($isSuccess) {
    $promise->resolve($value);
    } else {
    $promise->reject($error);
    }
    });

    View Slide

  59. Async Request using Promise
    use GuzzleHttp\Client;
    use Psr\Http\Message\ResponseInterface;
    use GuzzleHttp\Exception\RequestException;
    $client = new Client();
    $promise = $client->requestAsync(
    'GET',
    'http://localhost:8008/'
    );
    $promise
    ->then(function(ResponseInterface $response) {
    echo $response->getBody() . PHP_EOL;
    })
    ->otherwise(function(RequestException $e) {
    echo $e->getMessage() . PHP_EOL;
    })
    ;
    $promise->wait();

    View Slide

  60. sample05.php
    for ($i = 0; $i < 10; ++$i) {
    $promise = $client->requestAsync(
    'GET',
    'http://localhost:8008/'
    );
    $promise
    ->then(function(ResponseInterface $response) {
    echo $response->getBody() . PHP_EOL;
    })
    ;
    $promise->wait();
    }

    View Slide

  61. Run sample05.php

    View Slide

  62. Tooooo slow …

    View Slide

  63. sample05.php
    for ($i = 0; $i < 10; ++$i) {
    $promise = $client->requestAsync(
    'GET',
    'http://localhost:8008/'
    );
    $promise
    ->then(function(ResponseInterface $response) {
    echo $response->getBody() . PHP_EOL;
    })
    ;
    $promise->wait();
    }
    Not Async

    View Slide

  64. sample06.php
    use GuzzleHttp\Promise;
    $promises = [];
    for ($i = 0; $i < 10; ++$i) {
    $promise = $client->requestAsync(
    'GET',
    'http://localhost:8008/'
    );
    $promise
    ->then(function(ResponseInterface $response) {
    echo $response->getBody() . PHP_EOL;
    });
    $promises[] = $promise;
    }
    Promise\all($promises)->wait();

    View Slide

  65. Run sample06.php

    View Slide

  66. Fast !!

    View Slide

  67. sample07.php
    $promises = [];
    for ($i = 0; $i < 10000; ++$i) {
    $promise = $client->requestAsync(
    'GET',
    'http://localhost:8008/'
    );
    $promise
    ->then(function(ResponseInterface $response) {
    echo $response->getBody() . PHP_EOL;
    })
    ;
    $promises[] = $promise;
    }
    Promise\all($promises)->wait();

    View Slide

  68. Run sample07.php

    View Slide

  69. Summary of sample07.php
    PHP Fatal error:
    Allowed memory size of 134217728 bytes
    exhausted (tried to allocate 72 bytes)

    View Slide

  70. Why exhausted memory limit ?
    • $promises ഑ྻʹ10000ݸͷ Promise Φϒ
    δΣΫτ͕ೖͬͨ··ղ์͞Ε͍ͯͳ͍
    • ͦΕͧΕͷ Promise ΦϒδΣΫτ͸ঢ়ଶΛอ
    ͍࣋ͯ͠ΔͷͰͦΕͳΓʹॏ͍σʔλྔ
    → Կ͔ղܾํ๏͸ʁ

    View Slide

  71. Generator

    View Slide

  72. http://php.net/manual/en/language.generators.overview.php

    View Slide

  73. Generator
    $requests = function() {
    for ($i = 0; $i < 10; ++$i) {
    yield new Request(
    'GET',
    'http://localhost:8008/'
    );
    }
    };

    View Slide

  74. How to use Generator
    sample08.php
    use GuzzleHttp\Client;
    use GuzzleHttp\Psr7\Request;
    $client = new Client();
    $requests = function() {
    for ($i = 0; $i < 10; ++$i) {
    yield new Request('GET', 'http://localhost:8008/');
    }
    };
    foreach ($requests() as $request) {
    $response = $client->send($request);
    echo $response->getBody() . PHP_EOL;
    }

    View Slide

  75. Topic: Generator can use in ES6
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Generator

    View Slide

  76. Pool

    View Slide

  77. Pool overview
    use GuzzleHttp\Pool;
    $pool = new Pool($client, $requests(), [
    'concurrency' => 5,
    'fulfilled' => function ($response, $index) {
    // when success
    },
    'rejected' => function ($reason, $index) {
    // when failed
    },
    ]);
    Generator

    View Slide

  78. Pool config
    LFZ UZQF QVSQPTF
    DPODVSSFODZ JOUFHFS NBYDPODVSSFODZQBSBNFUFS
    GVMpMMFE DBMMBCMFGVODUJPO $BMMXIFO1SPNJTFJTSFTPMWFE
    SFKFDUFE DBMMBCMFGVODUJPO $BMMXIFO1SPNJTFJTSFKFDUFE

    View Slide

  79. sample09.php
    $client = new Client();
    $requests = function() {
    for ($i = 0; $i < 10; ++$i) {
    yield new Request('GET', 'http://localhost:8008/');
    }
    };
    $pool = new Pool($client, $requests(), [
    'concurrency' => 5,
    'fulfilled' => function(ResponseInterface $response, $index) {
    echo $response->getBody() . PHP_EOL;
    },
    ]);
    $promise = $pool->promise();
    $promise->wait();

    View Slide

  80. Run sample09.php

    View Slide

  81. Summary of sample09.php
    • Ұؾʹ5ͭͷϦΫΤετ͕౤͛ΒΕ͍ͯΔ
    • concurrency = 5 Ͱ΋͍ͩͿ଎͍
    • Ͱ͸ concurrency Λ૿΍ͤ͹΋ͬͱ଎͍ʁ

    View Slide

  82. Concurrency = 10
    sample10.php
    $pool = new Pool($client, $requests(), [
    'concurrency' => 10,
    'fulfilled' => function(ResponseInterface $response, $index) {
    echo $response->getBody() . PHP_EOL;
    },
    ]);

    View Slide

  83. Run sample11.php

    View Slide

  84. Summary of sample10.php
    • concurrency = 5 ΑΓ΋͞Βʹ଎͍
    • Ͱ͸ɺ΋ͬͱ΋ͬͱ૿΍ͤ͹͍͍ͷͰ͸ʁ

    View Slide

  85. sample11.php
    $client = new Client();
    $requests = function() {
    for ($i = 0; $i < 10000; ++$i) {
    yield new Request('GET', 'http://localhost:8008/');
    }
    };
    $pool = new Pool($client, $requests(), [
    'concurrency' => 1000,
    'fulfilled' => function(ResponseInterface $response, $index)
    {
    echo sprintf("%5d: %s\n", $index, $response->getBody());
    },
    ]);
    $promise = $pool->promise();
    $promise->wait();

    View Slide

  86. Run sample11.php

    View Slide

  87. Summary of sample11.php
    • 10000ճͷϦΫΤετʹ੒ޭ
    • ϝϞϦෆ଍ʹͳΒͳ͔ͬͨ
    • concurrency = 1000 Ͱ΋༨༟ʁ
    • Ͳ͜·Ͱ૿΍ͤΔ΋ͷͳͷ͔ʁ

    View Slide

  88. View Slide

  89. View Slide

  90. View Slide

  91. ࣮Ҋ݅Ͱͷ Pool ར༻
    • concurrency = 100 Λ࠾༻
    • APIαʔό΁ͷಉ࣌ΞΫηε਺͸ڐ༰ൣғ
    • ϝϞϦͷ࢖༻ྔ΋ڐ༰ൣғ
    • ໿50ສϦΫΤετΛ30෼͔͔Βͣʹ׬ྃՄೳ
    → 1࣌ؒ100ສϦΫΤετΛΫϦΞ

    View Slide

  92. Conclusion

    View Slide

  93. Promise
    • JavaScriptͰ࢖ΘΕ͍ͯΔPromise͕PHPʹ΋
    • JavaScriptͷ஌͕ࣝPHPͰ׆͔ͤΔ
    • PHPͰ΋࢖ΘΕ͍ͯΔGenerator͕JSʹ΋
    • PHPͷ஌͕ࣝJavaScriptͰ׆͔ͤΔ
    → ೔ࠒ͔Βෳ਺ͷݴޠΛ࢖͓ͬͯ͘ͱศར

    View Slide

  94. Guzzle
    • ෳ਺ͷϦΫΤετ͕ඞཁͳΒAsyncΛར༻
    • Promiseͷ֓೦Ͱܧଓ͢Δॲཧ͕΍Γ΍͍͢
    • PoolΛ࢖͑͹઀ଓ਺੍ޚ΋؆୯ʹ
    • concurrency͸APIଆͱϝϞϦʹ༏͍͠਺஋Ͱ
    → ਺ेສ݅Λૹ৴͢Δ৔߹Ͱ΋҆ఆՔಇ

    View Slide

  95. Company
    • ೖࣾ1ϲ݄Ͱʢ৽ਓʹ͸ʣ೉қ౓ߴΊͷҊ͕݅
    ߱ͬͯ͘Δ͜ͱ΋͋Δ
    • ݫ͕͋͠͞Δձࣾ
    • ৘ใڞ༗͢Δͱ஥͔ؒΒΞυόΠε͕߱ͬͯ͘
    Δ͜ͱ΋͋Δ
    • ༏͠͞΋͋Δձࣾ

    View Slide

  96. We are hiring !
    https://www.mercari.com/jp/jobs/

    View Slide

  97. Thank you

    View Slide

  98. Appendix

    View Slide

  99. Sample codes
    • https://github.com/suzuki/guzzle-promise-sample

    View Slide

  100. PHP side links
    • Guzzle Documentation
    • http://docs.guzzlephp.org/en/latest/
    • Guzzle Promises
    • https://github.com/guzzle/promises
    • PHP Documentation: Generator
    • http://php.net/manual/en/language.generators.overview.php
    • AWS SDK for PHP
    • https://docs.aws.amazon.com/aws-sdk-php/v3/guide/

    View Slide

  101. JavaScript side links
    • JavaScript Promiseͷຊ
    • http://azu.github.io/promises-book/
    •Promise/A+
    • https://promisesaplus.com/
    •ECMAScript 2015 (ES6)
    • http://www.ecma-international.org/ecma-262/6.0/
    •MDN Generator
    • https://developer.mozilla.org/en-US/docs/Web/JavaScript/
    Reference/Global_Objects/Generator

    View Slide

  102. References
    • Asynchronous API Interaction with Guzzle
    • https://speakerdeck.com/jeremeamia/asynchronous-api-
    interaction-with-guzzle
    • JavaScript Promises - Thinking Sync in an Async World
    • https://speakerdeck.com/kerrick/javascript-promises-
    thinking-sync-in-an-async-world
    • guzzleͰඇಉظϦΫΤετΛฒྻॲཧͰ͍ͬͺ͍౤͛Δ
    • http://hacknote.jp/archives/16095/

    View Slide

  103. Topic: Use case of Pool
    https://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/commands.html#commandpool

    View Slide

  104. Topic: Use case of Promise
    https://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/promises.html

    View Slide