Slide 1

Slide 1 text

Amazon KinesisͰ ετϦʔϜղੳ (ג)ϓϨΠυ @nashibao

Slide 2

Slide 2 text

0. Who are you?

Slide 3

Slide 3 text

@nashibaoͱ͍͍·͢ ࣲࢁ௚थ ৘೤త෼ࢄܕػցֶशతΤϯδχΞ(31) ! ~201208: େֶͱ͔Ͱ໱ʑͱ͢ΔɽMapReduceͱߦྻ෼ղ͕޷෺ɽ 201209: Node.jsʹ໨֮ΊΔ 201304: ݩָఱ(31)ͱىۀ σʔλղੳ&CRMαʔϏε 201309: ത࢜υϩοϓΞ΢τ

Slide 4

Slide 4 text

(ג)ϓϨΠυͱ͍͍·͢ http://plaid.co.jp EC޲͚ͷղੳ&CRMαʔϏεΛεςϧε։ൃதɽ ! ݱࡏ։ൃ͸ͨͬͨ2ਓ(t-nakamura)ͳͷͰɺ ԠืΛຊ౰ʹ͓଴͓ͪͯ͠Γ·͢ʂ ෲյ͢·Ͱ೑͕৯͑ΔձࣾʂͦΕ͕ϓϨΠυʂ ׻ܴձ͸γϡϥείͰ͢ʂ

Slide 5

Slide 5 text

Amazon KinesisΛετϦʔϜ ղੳʹ࢖͑Δ͔ݕূ͓ͨ͠࿩

Slide 6

Slide 6 text

1. Kinesisͱ͸ʁ 2. Node.js͔ΒKinesisΛ࢖͏ 3. ύϑΥʔϚϯεݕূ 4. ײ૝

Slide 7

Slide 7 text

1. Kinesisͱ͸ʁ

Slide 8

Slide 8 text

Amazon Kinesis

Slide 9

Slide 9 text

“a fully managed service for real-time processing of streaming data at massive scale.” ͲͰ͔͍σʔλ͔Β ϦΞϧλΠϜॲཧ͕ग़དྷΔ ϑϧϚωʔδυͳαʔϏε

Slide 10

Slide 10 text

5JCTF 5JCTF 5JCTF -# -# -# 2TQFWEGT -KPGUKU#RRNKECVKQPU 5VTGCO 2TQFWEGT 2TQFWEGT 2TQFWEGT 2TQFWEGT 哋┮؃䠉㏰׵ا揉ⓧ

Slide 11

Slide 11 text

5JCTF 5JCTF 5JCTF -# -# -# WEGT -KPGUKU#RRNKECVKQPU 5VTGCO WEGT WEGT WEGT EGT ϑϧϚωʔδυͳ෦෼ ! • 24࣌ؒอ࣋ • ࠶औಘՄʢex: Τϥʔ࣌ʣ • 1000tps/shard put • 5tps/shard get

Slide 12

Slide 12 text

5JCTF 5JCTF 5JCTF -# -# -# WEGT -KPGUKU#RRNKECVKQPU 5VTGCO WEGT WEGT WEGT EGT ϑϧϚωʔδυͳ෦෼ ! • 24࣌ؒอ࣋ • ࠶औಘՄʢex: Τϥʔ࣌ʣ • 1000tps/shard put • 5tps/shard get • hashKey(ࣗ༝)ͰৼΓ෼͚

Slide 13

Slide 13 text

5JCTF 5JCTF 5JCTF -# -# -# WEGT -KPGUKU#RRNKECVKQPU 5VTGCO WEGT WEGT WEGT EGT େྔʹॻ͖ࠐΜͰ ·ͱΊͯ औಘ͢ΔͨΊͷΩϡʔ? ॻ͖ࠐΈΛΩϟογϡ͢Δ ड͚ࡼͷΑ͏ͳΠϝʔδ

Slide 14

Slide 14 text

ෛՙूதͷड͚ࡼʹ Ͱ͖Δʁ

Slide 15

Slide 15 text

5KVG# 5KVG$ &$ &$ ! 5VTGCO #PCN[VKEU 5GOKDCVEJ/CR4GFWEG ! .QIIGT Ý ݱঢ় YGDUQEMGV

Slide 16

Slide 16 text

5KVG# 5KVG$ &$ &$ ! 5VTGCO #PCN[VKEU 5GOKDCVEJ/CR4GFWEG ! .QIIGT Ý ݱঢ় ෛՙ͕ूத͢Δͱ͞͹͘ͷେมʂ YGDUQEMGV

Slide 17

Slide 17 text

5KVG# 5KVG$ &$ &$ ! 5VTGCO #PCN[VKEU 5GOKDCVEJ/CR4GFWEG ! .QIIGT Ý Kinesis? YGDUQEMGV

Slide 18

Slide 18 text

5KVG# 5KVG$ &$ ! 5VTGCO #PCN[VKEU 5GOKDCVEJ/CR4GFWEG Kinesis? -KPGUKU ! Ý YGDUQEMGV

Slide 19

Slide 19 text

2.Node.js͔Βѻ͏

Slide 20

Slide 20 text

plaidev/kinesis-wrapper.js # install npm install kinesis-wrapper.js # stream ϕʔε mhart/kinesis ! # ΫϥγΧϧͳͷ͕ཉ͍͠ͷɽɽ͔ͩΒॻ͍ͨͷɽɽ plaidev/kinesis-wrapper.js

Slide 21

Slide 21 text

plaidev/kinesis-wrapper.js # install npm install kinesis-wrapper.js # stream ϕʔε mhart/kinesis ! # bacon.js ϕʔε andyhorng/kinesis ! # ΫϥγΧϧͷ͕ཉ͍͠ͷɽɽ͔ͩΒॻ͍ͨͷɽɽ plaidev/kinesis-wrapper.js

Slide 22

Slide 22 text

1. putRecordͰ౤͛ࠐΉ var kinesis = require('..'); ! # ετϦʔϜͱͭͳ͛Δ var stream = kinesis.stream('hoge'); ! # ‘key’͸MD5ͰϋογϡԽ͞ΕͯγϟʔυʹৼΓ෼͚ΒΕΔ stream.putRecord('key', {date: new Date()}, function(err, res){ console.log(err, res); });

Slide 23

Slide 23 text

2. getRecordsͰϙʔϦϯά var kinesis = require('..'); ! # ετϦʔϜͱͭͳ͛Δ var stream = kinesis.stream('hoge'); ! # ϙʔϦϯάͯ͠औಘɽγϟʔυ͝ͱ΋ग़དྷ·͢ stream.getRecords(function(err, records){ console.log(new Date(), records[0].val.date); });

Slide 24

Slide 24 text

͍ͭͰʹ ಉظతʹฒྻԽ͢Δ • async.jsʁ • ͜Ε͔Β͸yieldʂ • node-thunkify • nashibao/co-gate΋ѱ͘ͳ͍Αʂ

Slide 25

Slide 25 text

co-gateϞδϡʔϧ ैདྷͷίʔϧόοΫܗࣜͷؔ਺Λͦͷ··ฒ ྻԽɾಉظతʹॻ͚Δ ICVG ECNNDCEM ECNNDCEM ECNNDCEM ECNNDCEM KP QWV TGUWNVU

Slide 26

Slide 26 text

var co = require('co'); var Gate = require('co-gate'); ! co(function *(){ ! var gate = new Gate(); ! async_function(gate.in()); ! var results = yield gate.out(); ! })(); co-gateϞδϡʔϧ

Slide 27

Slide 27 text

3. େྔʹ౤͛ࠐΉʢ࠶ʣ co(function *(){ var gate = new Gate(); ! for (var i = 0; i < 100; i++){ stream.putRecord('key', {date: new Date()}, gate.in()); } ! var res = yield gate.out(); }

Slide 28

Slide 28 text

4. ϙʔϦϯάʢ࠶ʣ co(function *(){ var gate = new Gate(); ! stream.getRecords(gate.in()); ! var records = yield gate.out(); })();

Slide 29

Slide 29 text

3. KinesisͷύϑΥʔϚϯε

Slide 30

Slide 30 text

ݕূ؀ڥɾ৚݅ • EC2(c3-4xlarge)1୆͔Β౤͛ͯݕূɽ • ΫϥΠΞϯτଆͷλΠϜελϯϓͰtpsΛܭଌ • 1~3shard • ৄ͘͠͸ɺ΋͏গ͠ݕূඞཁ

Slide 31

Slide 31 text

putRecord / shard UQTͱ͔ೖͬͨΓ͢Δɽɽ

Slide 32

Slide 32 text

put & getRecords UQTఔ౓Ͱݻ·ͬͯऔಘͰ͖Δ

Slide 33

Slide 33 text

ࢥͬͨΑΓॻ͖ࠐΊΔɽͰ΋ ͪΐͬͱ஗͍? • max3000~ tps/shard put • ॻ͖ࠐΈ͗͢Δͱ݁ہऔΓग़͢·Ͱʹ͔͔࣌ؒΔɽ • ݁ہ400~700tpsఔ౓·Ͱ͍͚Δ͜ͱ͸֬ೝ • avg1sec get • ϙʔϦϯάࠐΈͰ͜ͷఔ౓ • 200msҎ಺ʹฦͬͯ͘ΔΘ͚͡Όͳ͍ • ॻ͖ࠐΜͰऔΓग़͢·Ͱʹͳ͔ͥϥά͕͋ͬͨΓͳ͔ͬͨΓ

Slide 34

Slide 34 text

ײ૝ • ϑϧϚωʔδυʁ • KA͸ࣗ෼Ͱ؅ཧɽγϟʔυʹϫʔΧʔ৐ͤΒΕͳ͍ͷ͔ɽɽ ! • ϦΞϧλΠϜʁ • ϙʔϦϯά • औΓग़͢ͷʹϥά͕͚ͬ͜͏͋Δ ! • ύϑΥʔϚϯε • write͸ߴ͍ɽγϟʔυ૿΍ͤ͹͔ͳΓ͍͚ͦ͏

Slide 35

Slide 35 text

͋Γ͕ͱ͏͍͟͝·ͨ͠ɽ

Slide 36

Slide 36 text

http://plaid.co.jp