Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
10倍速いNode.js並列プログラミング
Search
shigeru. nakajima
June 29, 2018
Technology
12
4.1k
10倍速いNode.js並列プログラミング
Node.jsのプログラミングを並列化して10倍速く動かした話です。
Node学園 31時限目 のLT
https://nodejs.connpass.com/event/90936/
shigeru. nakajima
June 29, 2018
Tweet
Share
More Decks by shigeru. nakajima
See All by shigeru. nakajima
Rubyで書いたテトリスをブラウザで動かしてみた
ledsun
0
1.4k
ruby.wasm に関する進捗報告
ledsun
0
860
Hacking Guide of the ruby.wasm
ledsun
0
1.2k
私の作ったruby.wasm アプリケーション
ledsun
0
530
Load gem from browser
ledsun
2
1.4k
パラメタライズドテスト
ledsun
0
460
ゆっくり動くと速く動ける / If you move slowly, you can move more fast
ledsun
0
900
WebSockets and Ruby : avoiding the pitfalls of multithreading
ledsun
0
900
Test asynchronous functions with RSpec
ledsun
0
2.5k
Other Decks in Technology
See All in Technology
開発生産性大幅アップ!Postman VS Code拡張機能
nagix
2
390
ChatworkのSRE部って実は 半分くらいPlatform Engineering部かもしれない
saramune
0
160
require(ESM)とECMAScript仕様
uhyo
3
780
20分で完全に理解するGrafanaダッシュボード
hamadakoji
3
690
Delivering Millions of Messages within seconds @ Duolingo
pelelgrino
0
350
Tellus の衛星データを見てみよう #mf_fukuoka
kongmingstrap
0
220
IaCジェネレーターとBedrockで詳細設計書を生成してみた
tsukasa_ishimaru
2
280
Reducing Cross-Zone Egress at Spotify with Custom gRPC Load Balancing Recap
koh_naga
0
210
AWSに詳しくない人でも始められるコスト最適化ガイド
yuhta28
1
250
自己改善からチームを動かす! 「セルフエンジニアリングマネージャー」のすゝめ
shoota
6
800
複雑な構成要素を持つUIとの向き合い方 〜新・支出グラフでの実例〜 / B43 TECH TALK
nakamuuu
0
140
LayerXにおけるLLMプロダクト開発の今までとこれから
layerx
PRO
1
400
Featured
See All Featured
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
21
1.6k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
125
32k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
659
120k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
25
2.3k
Become a Pro
speakerdeck
PRO
11
4.5k
The Brand Is Dead. Long Live the Brand.
mthomps
49
29k
For a Future-Friendly Web
brad_frost
172
9k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
30
6k
Fireside Chat
paigeccino
21
2.6k
The Cost Of JavaScript in 2023
addyosmani
16
3.9k
Rails Girls Zürich Keynote
gr2m
91
13k
Clear Off the Table
cherdarchuk
84
310k
Transcript
10ഒ͍Node.jsฒྻϓϩάϥϛϯά Shigeru Nakajima a.k.a. @ledsun Luxiar co., ltd NodeֶԂ LT
2018/06/29
Node εΫϨΠϐϯά͓͡͞Μ ʮnode εΫϨΠϐϯάʯͰάάͬͯ
PANQ panq http://www.panq.jp/ ͱ͍͏αʔϏεΛ࡞ͬ ͍ͯΔ QiitaͷهࣄΛࢀরͰධՁ Qiita APIͰࢀরऔΕͳ͍ εΫϨΠϐϯάͰऔಘ 8000هࣄΛूΊͨ
NodeͱεΫϨΠϐϯά εΫϨΠϐϯά = HTMLμϯϩʔυɹ + ɹHTMLύʔε = Network IOɹɹɹɹɹ+ɹ CPU
work
εΫϨΠϐϯάඇಉظϓϩ άϥϛϯάϞσϧͱΑ͘߹͏ request('http://example.com/', (e, res, body) => { const doc
= libxmljs.parseHtmlString(body) const data = doc.get('(//dl[@class="newsList"])[1]/dt[1]/text()') }) ;ͭ͏ʹॻ͘ͱޮతʹಈ͘
࠶ύʔε • ධՁ߲Λมߋ͍ͨ͠ • 8000htmlϑΝΠϧΛ࠶ύʔε
20min
ͬͱ͍ͨ͘͠
C++ Addon: libxmljs • ύʔεॲཧΛ͘͢Δ • parse5Λlibxmljsʹม͑Δ • 1200s(20min) =>
160s • 7.5ഒ͘ͳͬͨ • parse5: pure js • libxmljs: libxml2 (c) ͷϥούʔ
ͬͱ͍ͨ͘͠
ॲཧ༰ͷੳ • SSDڥͰFileಡΈࠐΈշ • IOͪຆͲແ͍ • CPUॲཧ͕େΛΊΔ • ୯ମͷCPUॲཧΛ͘͢Δͷ͍͠
ฒྻϓϩάϥϛϯά ݱͷύιίϯෳίΞ͕ࡌ͍ͬͯΔ ෳͷίΞͰCPUॲཧΛฒྻʹΒͤΔ => ฒྻϓϩάϥϛϯά
ຊͷ͓ Node.jsϓϩάϥϜΛฒྻԽͯ͠ੑೳΛ্͛Δ
ฒྻԽ • 1ͷϚγϯͷෳͷίΞΛͬͯɺಉ࣌ʹෳ ͷܭࢉΛ࣮ߦ • ෳϚγϯͷѻΘͳ͍
ϓϩηε • ฒྻϓϩάϥϛϯάͱ͍͑εϨου • Node.jsʹεϨουͳ͍ • ϓϩηεΛ͏
child_process.fork() ϫʔΧʔϓϩηεΛཱͯΔ const processes = [] for (var i =
0; i < number; i++) { processes.push(fork(program, [], { stdio: ['ignore', 'ignore', process.stderr, 'ipc'] })) }
֤ϓϩηεʹৼΓ͚Δ // ϑΥϧμͷϑΝΠϧҰཡΛऔಘ const dir = `${process.cwd()}/data/public/cache` const stream =
readdirp({ root: dir, fileFilter: '*.html' }) // ࢠϓϩηεͰύʔε let count = 0 stream.on('data', (data) => { // ϥϯυϩϏϯͰϑΝΠϧΛ count++; processes[count % processes.length].send(data) })
݁Ռɿ80sʂ͖Ε͍ʹ2ഒ
ͬͱ͍ͨ͘͠
Node.jsʹεϨουͳ͍ͱ ݴͬͨͳɺ͋Εӕͩ https://github.com/xk/node-threads-a-gogo • C++ AddonͰωΠςΟϒεϨουΛىಈ
ɿNode 6.x Ͱಈ͘ • Node 6.xͰ͔͠ಈ͔ͳ͍ • Node 6.x ͩͱͦΕ͚ͩͰ30%͙Β͍͍
• node-threads-a-gogo ͷ Node 10ରԠ • 4࣌ؒؤுͬͯఘΊͨ
Node.jsʹεϨουͳ͍ͱ ݴͬͨͳɺ͋Εӕͩʢ̎ʣ https://nodejs.org/api/worker_threads.html • v10.5.0@6/20 Ҏ߱ • ΤΫεϖϦϝϯλϧ • --experimental-worker
• ϑϥάΛ͚ͭΔͱಡΈࠐ·ΕΔϞδϡʔϧ
worker_threads.Worker child_process.fork()ͱେମҰॹ const workers = [] for (var i =
0; i < number; i++) { workers.push(new Worker(program)) }
threadʹৼΓ͚ // ϑΥϧμͷϑΝΠϧҰཡΛऔಘ const dir = `${process.cwd()}/data/public/cache` const stream =
readdirp({ root: dir, fileFilter: '*.html' }) // ϫʔΧʔͰύʔε let count = 0 stream.on('data', (data) => { // ϥϯυϩϏϯͰϑΝΠϧΛ count++; workers[count % workers.length].postMessage(data) })
ɿC++ Addonಈ͔ͳ ͍ • libxmljs͕͑ͳ͍ • 7.5ഒ͕ɺ͘ͳΔ • https://github.com/nodejs/node/issues/ 21481
ͬͱ͍ͨ͘͠
ίΞΛ૿ͤฒྻ • 2ίΞͰ2ഒ͕ݶքͳΒɺͨ͘͞ΜίΞͷ͋ ΔϚγϯΛ͍͍͑ • AWS EC2 c4.8xlarge • 36ίΞ
=> 80s͕2.2sʹ!? • 2.016USD/࣌ؒ
None
c4.8xlarge • ͦͦCPU͕͍͢͝ • 1ฒྻͰ90sʢx1.8ʣ • 12ฒྻͰ8s • ݶքʢ36ίΞΛ͍Εͳ͍ʣ
ϘτϧωοΫෆ໌ • ಡΈࠐΈIO͕ϘτϧωοΫʁ • SSDϚγϯʢi3.4xlargeʣΛ͕ͬͯ มΘΒͳ͍ • ϓϩηεͷݶքʁ • εϨουΛࢼ͔ͨͬͨ͠
ຊͷ·ͱΊ • libxmljsΛͬͯ 7.5ഒ • 2$/hͷϚγϯͰ 1.8ഒ • ฒྻϓϩάϥϛϯάͰ 11.3
ഒ • worker threadsͷҠߦ͕؆୯ ฒྻϓϩάϥϛϯά͓͍͍͠
ฒྻϓϩάϥϛϯά͠Α͏ worker threadsΛָ͠Έʹ͠·͠ΐʔ