Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Amazon Kinesisでストリーム解析 - Node学園12時限目

nashibao
April 24, 2014

Amazon Kinesisでストリーム解析 - Node学園12時限目

Node学園12時限目で話します.
Amazon Kinesisをストリーム解析に使えるか検証した時の話です.
http://plaid.co.jp

nashibao

April 24, 2014
Tweet

More Decks by nashibao

Other Decks in Technology

Transcript

  1. Amazon KinesisͰ
    ετϦʔϜղੳ
    (ג)ϓϨΠυ @nashibao

    View Slide

  2. 0. Who are you?

    View Slide

  3. @nashibaoͱ͍͍·͢
    ࣲࢁ௚थ
    ৘೤త෼ࢄܕػցֶशతΤϯδχΞ(31)
    !
    ~201208: େֶͱ͔Ͱ໱ʑͱ͢ΔɽMapReduceͱߦྻ෼ղ͕޷෺ɽ
    201209: Node.jsʹ໨֮ΊΔ
    201304: ݩָఱ(31)ͱىۀ σʔλղੳ&CRMαʔϏε
    201309: ത࢜υϩοϓΞ΢τ

    View Slide

  4. (ג)ϓϨΠυͱ͍͍·͢
    http://plaid.co.jp
    EC޲͚ͷղੳ&CRMαʔϏεΛεςϧε։ൃதɽ
    !
    ݱࡏ։ൃ͸ͨͬͨ2ਓ(t-nakamura)ͳͷͰɺ
    ԠืΛຊ౰ʹ͓଴͓ͪͯ͠Γ·͢ʂ
    ෲյ͢·Ͱ೑͕৯͑ΔձࣾʂͦΕ͕ϓϨΠυʂ
    ׻ܴձ͸γϡϥείͰ͢ʂ

    View Slide

  5. Amazon KinesisΛετϦʔϜ
    ղੳʹ࢖͑Δ͔ݕূ͓ͨ͠࿩

    View Slide

  6. 1. Kinesisͱ͸ʁ
    2. Node.js͔ΒKinesisΛ࢖͏
    3. ύϑΥʔϚϯεݕূ
    4. ײ૝

    View Slide

  7. 1. Kinesisͱ͸ʁ

    View Slide

  8. Amazon Kinesis

    View Slide

  9. “a fully managed service for real-time processing of
    streaming data at massive scale.”
    ͲͰ͔͍σʔλ͔Β
    ϦΞϧλΠϜॲཧ͕ग़དྷΔ
    ϑϧϚωʔδυͳαʔϏε

    View Slide

  10. 5JCTF
    5JCTF
    5JCTF
    -#
    -#
    -#
    2TQFWEGT
    -KPGUKU#RRNKECVKQPU
    5VTGCO
    2TQFWEGT
    2TQFWEGT
    2TQFWEGT
    2TQFWEGT
    哋┮؃䠉㏰׵ا揉ⓧ

    View Slide

  11. 5JCTF
    5JCTF
    5JCTF
    -#
    -#
    -#
    WEGT
    -KPGUKU#RRNKECVKQPU
    5VTGCO
    WEGT
    WEGT
    WEGT
    EGT ϑϧϚωʔδυͳ෦෼
    !
    • 24࣌ؒอ࣋
    • ࠶औಘՄʢex: Τϥʔ࣌ʣ
    • 1000tps/shard put
    • 5tps/shard get

    View Slide

  12. 5JCTF
    5JCTF
    5JCTF
    -#
    -#
    -#
    WEGT
    -KPGUKU#RRNKECVKQPU
    5VTGCO
    WEGT
    WEGT
    WEGT
    EGT ϑϧϚωʔδυͳ෦෼
    !
    • 24࣌ؒอ࣋
    • ࠶औಘՄʢex: Τϥʔ࣌ʣ
    • 1000tps/shard put
    • 5tps/shard get
    • hashKey(ࣗ༝)ͰৼΓ෼͚

    View Slide

  13. 5JCTF
    5JCTF
    5JCTF
    -#
    -#
    -#
    WEGT
    -KPGUKU#RRNKECVKQPU
    5VTGCO
    WEGT
    WEGT
    WEGT
    EGT
    େྔʹॻ͖ࠐΜͰ
    ·ͱΊͯ
    औಘ͢ΔͨΊͷΩϡʔ?
    ॻ͖ࠐΈΛΩϟογϡ͢Δ
    ड͚ࡼͷΑ͏ͳΠϝʔδ

    View Slide

  14. ෛՙूதͷड͚ࡼʹ
    Ͱ͖Δʁ

    View Slide



  15. 5KVG#
    5KVG$
    &$
    &$
    !
    5VTGCO
    #PCN[VKEU
    5GOKDCVEJ/CR4GFWEG
    !
    .QIIGT
    Ý
    ݱঢ়
    YGDUQEMGV

    View Slide



  16. 5KVG#
    5KVG$
    &$
    &$
    !
    5VTGCO
    #PCN[VKEU
    5GOKDCVEJ/CR4GFWEG
    !
    .QIIGT
    Ý
    ݱঢ়
    ෛՙ͕ूத͢Δͱ͞͹͘ͷେมʂ
    YGDUQEMGV

    View Slide



  17. 5KVG#
    5KVG$
    &$
    &$
    !
    5VTGCO
    #PCN[VKEU
    5GOKDCVEJ/CR4GFWEG
    !
    .QIIGT
    Ý
    Kinesis?
    YGDUQEMGV

    View Slide



  18. 5KVG#
    5KVG$
    &$
    !
    5VTGCO
    #PCN[VKEU
    5GOKDCVEJ/CR4GFWEG
    Kinesis?
    -KPGUKU !
    Ý
    YGDUQEMGV

    View Slide

  19. 2.Node.js͔Βѻ͏

    View Slide

  20. plaidev/kinesis-wrapper.js
    # install
    npm install kinesis-wrapper.js
    # stream ϕʔε
    mhart/kinesis
    !
    # ΫϥγΧϧͳͷ͕ཉ͍͠ͷɽɽ͔ͩΒॻ͍ͨͷɽɽ
    plaidev/kinesis-wrapper.js

    View Slide

  21. plaidev/kinesis-wrapper.js
    # install
    npm install kinesis-wrapper.js
    # stream ϕʔε
    mhart/kinesis
    !
    # bacon.js ϕʔε
    andyhorng/kinesis
    !
    # ΫϥγΧϧͷ͕ཉ͍͠ͷɽɽ͔ͩΒॻ͍ͨͷɽɽ
    plaidev/kinesis-wrapper.js

    View Slide

  22. 1. putRecordͰ౤͛ࠐΉ
    var kinesis = require('..');
    !
    # ετϦʔϜͱͭͳ͛Δ
    var stream = kinesis.stream('hoge');
    !
    # ‘key’͸MD5ͰϋογϡԽ͞ΕͯγϟʔυʹৼΓ෼͚ΒΕΔ
    stream.putRecord('key', {date: new Date()}, function(err, res){
    console.log(err, res);
    });

    View Slide

  23. 2. getRecordsͰϙʔϦϯά
    var kinesis = require('..');
    !
    # ετϦʔϜͱͭͳ͛Δ
    var stream = kinesis.stream('hoge');
    !
    # ϙʔϦϯάͯ͠औಘɽγϟʔυ͝ͱ΋ग़དྷ·͢
    stream.getRecords(function(err, records){
    console.log(new Date(), records[0].val.date);
    });

    View Slide

  24. ͍ͭͰʹ
    ಉظతʹฒྻԽ͢Δ
    • async.jsʁ
    • ͜Ε͔Β͸yieldʂ
    • node-thunkify
    • nashibao/co-gate΋ѱ͘ͳ͍Αʂ

    View Slide

  25. co-gateϞδϡʔϧ
    ैདྷͷίʔϧόοΫܗࣜͷؔ਺Λͦͷ··ฒ
    ྻԽɾಉظతʹॻ͚Δ
    ICVG
    ECNNDCEM
    ECNNDCEM
    ECNNDCEM
    ECNNDCEM KP QWV
    TGUWNVU

    View Slide

  26. var co = require('co');
    var Gate = require('co-gate');
    !
    co(function *(){
    !
    var gate = new Gate();
    !
    async_function(gate.in());
    !
    var results = yield gate.out();
    !
    })();
    co-gateϞδϡʔϧ

    View Slide

  27. 3. େྔʹ౤͛ࠐΉʢ࠶ʣ
    co(function *(){
    var gate = new Gate();
    !
    for (var i = 0; i < 100; i++){
    stream.putRecord('key', {date: new Date()}, gate.in());
    }
    !
    var res = yield gate.out();
    }

    View Slide

  28. 4. ϙʔϦϯάʢ࠶ʣ
    co(function *(){
    var gate = new Gate();
    !
    stream.getRecords(gate.in());
    !
    var records = yield gate.out();
    })();

    View Slide

  29. 3. KinesisͷύϑΥʔϚϯε

    View Slide

  30. ݕূ؀ڥɾ৚݅
    • EC2(c3-4xlarge)1୆͔Β౤͛ͯݕূɽ
    • ΫϥΠΞϯτଆͷλΠϜελϯϓͰtpsΛܭଌ
    • 1~3shard
    • ৄ͘͠͸ɺ΋͏গ͠ݕূඞཁ

    View Slide

  31. putRecord / shard
    UQTͱ͔ೖͬͨΓ͢Δɽɽ

    View Slide

  32. put & getRecords
    UQTఔ౓Ͱݻ·ͬͯऔಘͰ͖Δ

    View Slide

  33. ࢥͬͨΑΓॻ͖ࠐΊΔɽͰ΋
    ͪΐͬͱ஗͍?
    • max3000~ tps/shard put
    • ॻ͖ࠐΈ͗͢Δͱ݁ہऔΓग़͢·Ͱʹ͔͔࣌ؒΔɽ
    • ݁ہ400~700tpsఔ౓·Ͱ͍͚Δ͜ͱ͸֬ೝ
    • avg1sec get
    • ϙʔϦϯάࠐΈͰ͜ͷఔ౓
    • 200msҎ಺ʹฦͬͯ͘ΔΘ͚͡Όͳ͍
    • ॻ͖ࠐΜͰऔΓग़͢·Ͱʹͳ͔ͥϥά͕͋ͬͨΓͳ͔ͬͨΓ

    View Slide

  34. ײ૝
    • ϑϧϚωʔδυʁ
    • KA͸ࣗ෼Ͱ؅ཧɽγϟʔυʹϫʔΧʔ৐ͤΒΕͳ͍ͷ͔ɽɽ
    !
    • ϦΞϧλΠϜʁ
    • ϙʔϦϯά
    • औΓग़͢ͷʹϥά͕͚ͬ͜͏͋Δ
    !
    • ύϑΥʔϚϯε
    • write͸ߴ͍ɽγϟʔυ૿΍ͤ͹͔ͳΓ͍͚ͦ͏

    View Slide

  35. ͋Γ͕ͱ͏͍͟͝·ͨ͠ɽ

    View Slide

  36. http://plaid.co.jp

    View Slide