10倍速いNode.js並列プログラミング

by shigeru. nakajima

Slide 1

Slide 1 text

10ഒ଎͍Node.jsฒྻϓϩάϥϛϯά Shigeru Nakajima a.k.a. @ledsun Luxiar co., ltd NodeֶԂ LT 2018/06/29

Slide 2

Slide 2 text

Node εΫϨΠϐϯά͓͡͞Μ ʮnode εΫϨΠϐϯάʯͰάάͬͯ

Slide 3

Slide 3 text

PANQ panq http://www.panq.jp/ ͱ͍͏αʔϏεΛ࡞ͬ ͍ͯΔ QiitaͷهࣄΛࢀর਺ͰධՁ Qiita APIͰ͸ࢀর਺͸औΕͳ͍ εΫϨΠϐϯάͰऔಘ 8000هࣄΛूΊͨ

Slide 4

Slide 4 text

NodeͱεΫϨΠϐϯά εΫϨΠϐϯά = HTMLμ΢ϯϩʔυɹ + ɹHTMLύʔε = Network IOɹɹɹɹɹ+ɹ CPU work

Slide 5

Slide 5 text

εΫϨΠϐϯά͸ඇಉظϓϩ άϥϛϯάϞσϧͱΑ͘߹͏ request('http://example.com/', (e, res, body) => { const doc = libxmljs.parseHtmlString(body) const data = doc.get('(//dl[@class="newsList"])[1]/dt[1]/text()') }) ;ͭ͏ʹॻ͘ͱޮ཰తʹಈ͘

Slide 6

Slide 6 text

࠶ύʔε • ධՁ߲໨Λมߋ͍ͨ͠ • 8000htmlϑΝΠϧΛ࠶ύʔε

Slide 7

Slide 7 text

20min

Slide 8

Slide 8 text

΋ͬͱ଎͍ͨ͘͠

Slide 9

Slide 9 text

C++ Addon: libxmljs • ύʔεॲཧΛ଎͘͢Δ • parse5Λlibxmljsʹม͑Δ • 1200s(20min) => 160s • 7.5ഒ଎͘ͳͬͨ • parse5: pure js • libxmljs: libxml2 (c) ͷϥούʔ

Slide 10

Slide 10 text

΋ͬͱ଎͍ͨ͘͠

Slide 11

Slide 11 text

ॲཧ಺༰ͷ෼ੳ • SSD؀ڥͰFileಡΈࠐΈ͸շ଎ • IO଴ͪ͸ຆͲແ͍ • CPUॲཧ͕େ൒Λ઎ΊΔ • ୯ମͷCPUॲཧΛ଎͘͢Δͷ͸೉͍͠

Slide 12

Slide 12 text

ฒྻϓϩάϥϛϯά ݱ୅ͷύιίϯ͸ෳ਺ίΞ͕ࡌ͍ͬͯΔ ෳ਺ͷίΞͰCPUॲཧΛฒྻʹ૸ΒͤΔ => ฒྻϓϩάϥϛϯά

Slide 13

Slide 13 text

ຊ೔ͷ͓୊ Node.jsϓϩάϥϜΛฒྻԽͯ͠ੑೳΛ্͛Δ

Slide 14

Slide 14 text

ฒྻԽ • 1୆ͷϚγϯͷෳ਺ͷίΞΛ࢖ͬͯɺಉ࣌ʹෳ ਺ͷܭࢉΛ࣮ߦ • ෳ਺Ϛγϯͷ࿩͸ѻΘͳ͍

Slide 15

Slide 15 text

ϓϩηε • ฒྻϓϩάϥϛϯάͱ͍͑͹εϨου • Node.jsʹ͸εϨου͸ͳ͍ • ϓϩηεΛ࢖͏

Slide 16

Slide 16 text

child_process.fork() ϫʔΧʔϓϩηεΛཱͯΔ const processes = [] for (var i = 0; i < number; i++) { processes.push(fork(program, [], { stdio: ['ignore', 'ignore', process.stderr, 'ipc'] })) }

Slide 17

Slide 17 text

֤ϓϩηεʹৼΓ෼͚Δ // ϑΥϧμ಺ͷϑΝΠϧҰཡΛऔಘ const dir = `${process.cwd()}/data/public/cache` const stream = readdirp({ root: dir, fileFilter: '*.html' }) // ࢠϓϩηεͰύʔε let count = 0 stream.on('data', (data) => { // ϥ΢ϯυϩϏϯͰϑΝΠϧΛ഑෍ count++; processes[count % processes.length].send(data) })

Slide 18

Slide 18 text

݁Ռɿ80sʂ͖Ε͍ʹ2ഒ଎

Slide 19

Slide 19 text

΋ͬͱ଎͍ͨ͘͠

Slide 20

Slide 20 text

Node.jsʹεϨου͸ͳ͍ͱ ݴͬͨͳɺ͋Ε͸ӕͩ https://github.com/xk/node-threads-a-gogo • C++ AddonͰωΠςΟϒεϨουΛىಈ

Slide 21

Slide 21 text

໰୊ɿNode 6.x Ͱಈ͘ • Node 6.xͰ͔͠ಈ͔ͳ͍ • Node 6.x ͩͱͦΕ͚ͩͰ30%͙Β͍஗͍ • node-threads-a-gogo ͷ Node 10ରԠ • 4࣌ؒؤுͬͯఘΊͨ

Slide 22

Slide 22 text

Node.jsʹεϨου͸ͳ͍ͱ ݴͬͨͳɺ͋Ε͸ӕͩʢ̎ʣ https://nodejs.org/api/worker_threads.html • v10.5.0@6/20 Ҏ߱ • ΤΫεϖϦϝϯλϧ • --experimental-worker • ϑϥάΛ͚ͭΔͱಡΈࠐ·ΕΔϞδϡʔϧ

Slide 23

Slide 23 text

worker_threads.Worker child_process.fork()ͱେମҰॹ const workers = [] for (var i = 0; i < number; i++) { workers.push(new Worker(program)) }

Slide 24

Slide 24 text

threadʹৼΓ෼͚ // ϑΥϧμ಺ͷϑΝΠϧҰཡΛऔಘ const dir = `${process.cwd()}/data/public/cache` const stream = readdirp({ root: dir, fileFilter: '*.html' }) // ϫʔΧʔͰύʔε let count = 0 stream.on('data', (data) => { // ϥ΢ϯυϩϏϯͰϑΝΠϧΛ഑෍ count++; workers[count % workers.length].postMessage(data) })

Slide 25

Slide 25 text

໰୊ɿC++ Addon͸ಈ͔ͳ ͍ • libxmljs͕࢖͑ͳ͍ • 7.5ഒ͕ɺ஗͘ͳΔ • https://github.com/nodejs/node/issues/ 21481

Slide 26

Slide 26 text

΋ͬͱ଎͍ͨ͘͠

Slide 27

Slide 27 text

ίΞΛ૿΍ͤ͹ฒྻ਺΋ • 2ίΞͰ2ഒ଎͕ݶքͳΒɺͨ͘͞ΜίΞͷ͋ ΔϚγϯΛ࢖͑͹͍͍ • AWS EC2 c4.8xlarge • 36ίΞ => 80s͕2.2sʹ!? • 2.016USD/࣌ؒ

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

c4.8xlarge • ͦ΋ͦ΋CPU͕͍͢͝ • 1ฒྻͰ90sʢx1.8ʣ • 12ฒྻͰ8s • ݶքʢ36ίΞΛ࢖͍੾Εͳ͍ʣ

Slide 30

Slide 30 text

ϘτϧωοΫෆ໌ • ಡΈࠐΈIO͕ϘτϧωοΫʁ • SSDϚγϯʢi3.4xlargeʣΛ࢖ͬͯ΋܏޲͕ มΘΒͳ͍ • ϓϩηεͷݶքʁ • εϨουΛࢼ͔ͨͬͨ͠

Slide 31

Slide 31 text

ຊ೔ͷ·ͱΊ • libxmljsΛ࢖ͬͯ 7.5ഒ • 2$/hͷϚγϯͰ 1.8ഒ • ฒྻϓϩάϥϛϯάͰ 11.3 ഒ • worker threads΁ͷҠߦ͕؆୯ ฒྻϓϩάϥϛϯά͸͓͍͍͠

Slide 32

Slide 32 text

ฒྻϓϩάϥϛϯά͠Α͏ worker threadsΛָ͠Έʹ͠·͠ΐʔ