Upgrade to Pro — share decks privately, control downloads, hide ads and more …

10倍速いNode.js並列プログラミング

 10倍速いNode.js並列プログラミング

Node.jsのプログラミングを並列化して10倍速く動かした話です。
Node学園 31時限目 のLT
https://nodejs.connpass.com/event/90936/

shigeru. nakajima

June 29, 2018
Tweet

More Decks by shigeru. nakajima

Other Decks in Technology

Transcript

  1. 10ഒ଎͍Node.jsฒྻϓϩάϥϛϯά
    Shigeru Nakajima a.k.a. @ledsun
    Luxiar co., ltd
    NodeֶԂ LT 2018/06/29

    View full-size slide

  2. Node εΫϨΠϐϯά͓͡͞Μ
    ʮnode εΫϨΠϐϯάʯͰάάͬͯ

    View full-size slide

  3. PANQ
    panq http://www.panq.jp/ ͱ͍͏αʔϏεΛ࡞ͬ
    ͍ͯΔ
    QiitaͷهࣄΛࢀর਺ͰධՁ
    Qiita APIͰ͸ࢀর਺͸औΕͳ͍
    εΫϨΠϐϯάͰऔಘ
    8000هࣄΛूΊͨ

    View full-size slide

  4. NodeͱεΫϨΠϐϯά
    εΫϨΠϐϯά
    = HTMLμ΢ϯϩʔυɹ + ɹHTMLύʔε
    = Network IOɹɹɹɹɹ+ɹ CPU work

    View full-size slide

  5. εΫϨΠϐϯά͸ඇಉظϓϩ
    άϥϛϯάϞσϧͱΑ͘߹͏
    request('http://example.com/', (e, res, body) => {
    const doc = libxmljs.parseHtmlString(body)
    const data = doc.get('(//dl[@class="newsList"])[1]/dt[1]/text()')
    })
    ;ͭ͏ʹॻ͘ͱޮ཰తʹಈ͘

    View full-size slide

  6. ࠶ύʔε
    • ධՁ߲໨Λมߋ͍ͨ͠
    • 8000htmlϑΝΠϧΛ࠶ύʔε

    View full-size slide

  7. ΋ͬͱ଎͍ͨ͘͠

    View full-size slide

  8. C++ Addon: libxmljs
    • ύʔεॲཧΛ଎͘͢Δ
    • parse5Λlibxmljsʹม͑Δ
    • 1200s(20min) => 160s
    • 7.5ഒ଎͘ͳͬͨ
    • parse5: pure js
    • libxmljs: libxml2 (c) ͷϥούʔ

    View full-size slide

  9. ΋ͬͱ଎͍ͨ͘͠

    View full-size slide

  10. ॲཧ಺༰ͷ෼ੳ
    • SSD؀ڥͰFileಡΈࠐΈ͸շ଎
    • IO଴ͪ͸ຆͲແ͍
    • CPUॲཧ͕େ൒Λ઎ΊΔ
    • ୯ମͷCPUॲཧΛ଎͘͢Δͷ͸೉͍͠

    View full-size slide

  11. ฒྻϓϩάϥϛϯά
    ݱ୅ͷύιίϯ͸ෳ਺ίΞ͕ࡌ͍ͬͯΔ
    ෳ਺ͷίΞͰCPUॲཧΛฒྻʹ૸ΒͤΔ
    => ฒྻϓϩάϥϛϯά

    View full-size slide

  12. ຊ೔ͷ͓୊
    Node.jsϓϩάϥϜΛฒྻԽͯ͠ੑೳΛ্͛Δ

    View full-size slide

  13. ฒྻԽ
    • 1୆ͷϚγϯͷෳ਺ͷίΞΛ࢖ͬͯɺಉ࣌ʹෳ
    ਺ͷܭࢉΛ࣮ߦ
    • ෳ਺Ϛγϯͷ࿩͸ѻΘͳ͍

    View full-size slide

  14. ϓϩηε
    • ฒྻϓϩάϥϛϯάͱ͍͑͹εϨου
    • Node.jsʹ͸εϨου͸ͳ͍
    • ϓϩηεΛ࢖͏

    View full-size slide

  15. child_process.fork()
    ϫʔΧʔϓϩηεΛཱͯΔ
    const processes = []
    for (var i = 0; i < number; i++) {
    processes.push(fork(program, [], {
    stdio: ['ignore', 'ignore', process.stderr, 'ipc']
    }))
    }

    View full-size slide

  16. ֤ϓϩηεʹৼΓ෼͚Δ
    // ϑΥϧμ಺ͷϑΝΠϧҰཡΛऔಘ
    const dir = `${process.cwd()}/data/public/cache`
    const stream = readdirp({
    root: dir,
    fileFilter: '*.html'
    })
    // ࢠϓϩηεͰύʔε
    let count = 0
    stream.on('data', (data) => {
    // ϥ΢ϯυϩϏϯͰϑΝΠϧΛ഑෍
    count++;
    processes[count % processes.length].send(data)
    })

    View full-size slide

  17. ݁Ռɿ80sʂ͖Ε͍ʹ2ഒ଎

    View full-size slide

  18. ΋ͬͱ଎͍ͨ͘͠

    View full-size slide

  19. Node.jsʹεϨου͸ͳ͍ͱ
    ݴͬͨͳɺ͋Ε͸ӕͩ
    https://github.com/xk/node-threads-a-gogo
    • C++ AddonͰωΠςΟϒεϨουΛىಈ

    View full-size slide

  20. ໰୊ɿNode 6.x Ͱಈ͘
    • Node 6.xͰ͔͠ಈ͔ͳ͍
    • Node 6.x ͩͱͦΕ͚ͩͰ30%͙Β͍஗͍
    • node-threads-a-gogo ͷ Node 10ରԠ
    • 4࣌ؒؤுͬͯఘΊͨ

    View full-size slide

  21. Node.jsʹεϨου͸ͳ͍ͱ
    ݴͬͨͳɺ͋Ε͸ӕͩʢ̎ʣ
    https://nodejs.org/api/worker_threads.html
    • v10.5.0@6/20 Ҏ߱
    • ΤΫεϖϦϝϯλϧ
    • --experimental-worker
    • ϑϥάΛ͚ͭΔͱಡΈࠐ·ΕΔϞδϡʔϧ

    View full-size slide

  22. worker_threads.Worker
    child_process.fork()ͱେମҰॹ
    const workers = []
    for (var i = 0; i < number; i++) {
    workers.push(new Worker(program))
    }

    View full-size slide

  23. threadʹৼΓ෼͚
    // ϑΥϧμ಺ͷϑΝΠϧҰཡΛऔಘ
    const dir = `${process.cwd()}/data/public/cache`
    const stream = readdirp({
    root: dir,
    fileFilter: '*.html'
    })
    // ϫʔΧʔͰύʔε
    let count = 0
    stream.on('data', (data) => {
    // ϥ΢ϯυϩϏϯͰϑΝΠϧΛ഑෍
    count++;
    workers[count % workers.length].postMessage(data)
    })

    View full-size slide

  24. ໰୊ɿC++ Addon͸ಈ͔ͳ
    ͍
    • libxmljs͕࢖͑ͳ͍
    • 7.5ഒ͕ɺ஗͘ͳΔ
    • https://github.com/nodejs/node/issues/
    21481

    View full-size slide

  25. ΋ͬͱ଎͍ͨ͘͠

    View full-size slide

  26. ίΞΛ૿΍ͤ͹ฒྻ਺΋
    • 2ίΞͰ2ഒ଎͕ݶքͳΒɺͨ͘͞ΜίΞͷ͋
    ΔϚγϯΛ࢖͑͹͍͍
    • AWS EC2 c4.8xlarge
    • 36ίΞ => 80s͕2.2sʹ!?
    • 2.016USD/࣌ؒ

    View full-size slide

  27. c4.8xlarge
    • ͦ΋ͦ΋CPU͕͍͢͝
    • 1ฒྻͰ90sʢx1.8ʣ
    • 12ฒྻͰ8s
    • ݶքʢ36ίΞΛ࢖͍੾Εͳ͍ʣ

    View full-size slide

  28. ϘτϧωοΫෆ໌
    • ಡΈࠐΈIO͕ϘτϧωοΫʁ
    • SSDϚγϯʢi3.4xlargeʣΛ࢖ͬͯ΋܏޲͕
    มΘΒͳ͍
    • ϓϩηεͷݶքʁ
    • εϨουΛࢼ͔ͨͬͨ͠

    View full-size slide

  29. ຊ೔ͷ·ͱΊ
    • libxmljsΛ࢖ͬͯ 7.5ഒ
    • 2$/hͷϚγϯͰ 1.8ഒ
    • ฒྻϓϩάϥϛϯάͰ 11.3 ഒ
    • worker threads΁ͷҠߦ͕؆୯
    ฒྻϓϩάϥϛϯά͸͓͍͍͠

    View full-size slide

  30. ฒྻϓϩάϥϛϯά͠Α͏
    worker threadsΛָ͠Έʹ͠·͠ΐʔ

    View full-size slide