$30 off During Our Annual Pro Sale. View Details »

ITS を支える
情報集約基盤
アーキテクチャ / ITS Tech Study #02

ITS を支える
情報集約基盤
アーキテクチャ / ITS Tech Study #02

Hiroyoshi HOUCHI

June 08, 2018
Tweet

More Decks by Hiroyoshi HOUCHI

Other Decks in Technology

Transcript

  1. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Hiroyoshi HOUCHI Office

    of Technology Development System Management Unit
 DeNA Co., Ltd. *54Λࢧ͑Δ
 ৘ใू໿ج൫
 ΞʔΩςΫνϟ *545FDI4UVEZ"84*P5 1
  2. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ࣗݾ঺հ ์஍޺Ղ !IJYJ@IZJ

     w ೥݄dݱΦʔϓϯϓϥοτϑΥʔϜγεςϜࣄۀ෦  4PDJBM"1* ։ൃ   ࣾ಺޲͚.PCJMF#BB4 ։ൃϦʔμ   ڠۀҊ݅ ։ൃΠϯϑϥ   "OE"QQ Ϋϥ΢υΞʔΩςΫνϟઃܭ  w ೥݄dΦʔτϞʔςΟϒࣄۀຊ෦  Ϋϥ΢υγεςϜΞʔΩςΫτɾڞ௨ج൫ w ೥݄dγεςϜຊ෦  શࣾԣஅΫϥ΢υγεςϜΞʔΩςΫτ Λ΍Δͭ΋Γ 
 ΦʔτϞʔςΟϒ΋Ҿ͖ଓ͖ 2
  3. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ຊ೔ͷൃද಺༰ 3

  4. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. *54Λࢧ͑Δ৘ใू໿ج൫ΞʔΩςΫνϟ ⁃ ं྆ొ࿥ػೳ

    ⁃ ं྆ɾαʔόʹର͢ΔೝূೝՄ ⁃ ৘ใͷऩू ⁃ ं྆ͦͷ΋ͷͷ৘ใͷऩू ⁃ ΤοδίϯϐϡʔςΟϯάͷ෼ੳ৘ใͷऩू ⁃ ं྆৘ใͷఏڙ ⁃ ं྆৘ใͷϦΞϧλΠϜऔಘػೳͷఏڙ ⁃ δΦؔ਺Λ༻͍ͨं྆৘ใͷఏڙ ⁃ ෼ੳγεςϜ΁ͷ৘ใఏڙ 4
  5. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. *54Λࢧ͑Δ৘ใू໿ج൫ΞʔΩςΫνϟ ⁃ ं྆ొ࿥ػೳ

    ⁃ ं྆ɾαʔόʹର͢ΔೝূೝՄ ⁃ ৘ใͷऩू ⁃ ं྆ͦͷ΋ͷͷ৘ใͷऩू ⁃ ΤοδίϯϐϡʔςΟϯάͷ෼ੳ৘ใͷऩू ⁃ ं྆৘ใͷఏڙ ⁃ ं྆৘ใͷϦΞϧλΠϜऔಘػೳͷఏڙ ⁃ δΦؔ਺Λ༻͍ͨं྆৘ใͷఏڙ ⁃ ෼ੳγεςϜ΁ͷ৘ใఏڙ 5
  6. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ࢀߟࢿྉ λΫϕϧʹ͍ͭͯ ⁃

    AWS Summit 2018 Tokyo - DeNA ΦʔτϞʔςΟϒʹ͓͚Δ AWS ׆༻ࣄྫ 
 ٕज़෦෼ʹ͍ͭͯ ⁃ DeNA TechCon 2018 – AWS IoTΛ༻͍ͨDeNAΦʔτϞʔςΟϒΞʔΩςΫ νϟ ݕࡧΩʔϫʔυʮ AWS IoT DeNA ʯ(ൃද౰೔ Google ݕࡧ࣌఺) 6
  7. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ൃද෦෼ͷΞʔΩςΫνϟਤ 7 3.

    ෼ੳ৘ใऩू 2. ं྆৘ใऩू 1. ෼ੳγεςϜ౷߹
  8. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ෼ੳγεςϜ౷߹ 8

    8
  9. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ෼ੳγεςϜ౷߹ 9

    { "topic": "pub/sub topic", "data": { "hoge": "fuga" } } ※ format ʹ͍ͭͯ͸গ͠ lambda Ͱٵऩͯ͠Δ෦෼͋Γ
  10. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ෼ੳγεςϜ౷߹ 10

    • όοϑΝϦϯά͞Εͨ৘ใΛ Lambda Ͱऔಘ • σʔλϑϩʔ্ Kinesis Data Streams → Lambda
 ࣮ࡍ͸ Lambda → Kinesis Data Streams
  11. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ෼ੳγεςϜ౷߹ 11

    • topic Ͱάϧʔϐϯά • Cloud Pub/Sub ΁͸
 ·ͱΊͯˍฒྻ Publish [ { "topic": "pub/sub topic", "data": { "hoge": "fuga" } }, …… ]
  12. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ෼ੳγεςϜ౷߹෦ʹ͍ͭͯ •

    ޻෉఺ • جຊతʹ Network ͷ४උ + I/O ͕࣌ؒͱͯ΋͔͔Δ෦෼ • ୯ൃ࣮ߦͰ͸ͳͯ͘ɺόοϑΝϦϯά͢Δ͜ͱʹΑͬͯ Lambda ͷ࣮ߦճ਺ + ૯࣮ߦ࣌ؒΛେ෯ʹݮΒ͢͜ͱ(=ίετݮ)͕Ͱ͖Δ (※ ं྆୆਺ rps͙Β͍͘Δ ) • ΄΅΄΅ϦΞϧλΠϜʹ৘ใΛૹΔ͜ͱ͕Ͱ͖Δ • ߟྀ఺ • Kinesis Data Streams ͷγϟʔυ਺ • ผख๏ • ϦΞϧλΠϜੑΛ΋͏গ͠ଛͳ͍͍ͬͯͷͰ͋Ε͹ S3 Λ༻͍ͭͭɺ
 ఆظతʹ GCS ʹૹ৴͢Δ͜ͱ΋Մ • ͦ͏ͨ͠৔߹ɺ΋ͬͱίετΛݮΒ͢͜ͱ͕Ͱ͖Δ 12
  13. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 13

  14. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ͷલͷɺࣄલ৘ใ… • ं͔྆Βͷૹ৴ϑΥʔϚοτ͸

    bson (TechCon #33) • ࣄલʹߦ͏ं྆ొ࿥(TechCon #25)ʹΑΓҎԼͷΑ͏ʹͳΔ • ThingName ͸ূ໌ॻID • ThingShadow ʹ͸ҎԼͷ৘ใؚ͕·Ε͍ͯΔ (TechCon #29) 14 \ TUBUF\ SFQPSUFE\ EFWJDF\ !JE999999 ^ ^ ^ ^
  15. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 15

  16. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 16

    Topic p/@s.b 
 (providers/taxibell/@status.bson) Payload ҎԼͷΑ͏ͳ json ͷ bson දݱ {"@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": "1528437407"}
  17. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 17

    SQL SELECT aws_lambda(“${bson2json lambda arn}”, { “bson”: encode(*, “base64”)) as state.reported.status FROM p/@s.b Action ҎԼ΁ͷ Republish $$aws/things/${principal()}/shadow/update
  18. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 18

    ݩͷσʔλ { "state": { "reported": { "device": { "@id": "XXXXXX" } }, "status": {"hoge":"fuga" } } ೖྗσʔλ { "state": { "reported": { "status": { "@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": "1528437407" } } ݁Ռσʔλ { "state": { "reported": { "device": { "@id": "XXXXXX" }, "status": { "@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": “1528437407”, “hoge”: “fuga” } }
  19. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 19

    Topic $aws/things/+/shadow/update/documents Payload { "current": { "state": { "reported": { "device": { "@id": "XXXXXX" }, "status": { "@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": "1528437407", "hoge":"fuga" } } , "desired": {} } }, "previous": {ུ} }
  20. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 20

    SQL SELECT {'data': {'device': current.state.reported.device, 'reported': current.state.reported.status}, 'topic': ‘${topicname}'} FROM '$aws/things/+/shadow/update/documents' Action Kinesis
  21. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू෦ʹ͍ͭͯ •

    ޻෉఺ • ৘ใͷ౷߹ • ThingShadow ͷػೳΛ༗ޮ׆༻͢Δ͜ͱͰɺࠩ෼͚ͩͰ͸ͳ͘ɺͦͷ࣌ͷं྆ ͷ͢΂ͯͷঢ়ଶΛ౉͢͜ͱ͕Մೳ • ΫϥΠΞϯτ͸ࠩ෼͚ͩͳͷͰ௨৴ྔతʹ༏͍͠ • ෼ੳ؀ڥʹͯ৘ใͷิ׬͕ඞཁͳ͘ͳΔ • ඞཁͳ෦෼ͷΈͷ Lambda ʹ཈͑Δ • Lambda Ͱ ThingShadow ͷߋ৽·Ͱ͢Δ͜ͱ͸Մೳ • ҰํͰ ThingShadow ͷؾ෼Ͱ࣮ߦ͕࣌ؒ૿͑ͯ͠·͏ʹίετ͕૿Ճ͢ΔՄೳ ੑ͕͋Δ 21
  22. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ෼ੳ৘ใऩू 22

  23. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ෼ੳ৘ใऩू 23

    Topic p/@a.b 
 (providers/taxibell/@analytics.bson) Payload ҎԼͷΑ͏ͳ json ͷ bson දݱ { "@t": "${ࣝผࢠ}", "@d" { "accuracy": "100", "raw": { "hoge": "huga" }, "res": { "hoge": "huga" } } } ※ ͜͜Ͱͷ @t ͸ type ͷུ
  24. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ෼ੳ৘ใऩू 24

    SQL SELECT “${topicname} as topic, aws_lambda(“${bson2json lambda arn}”, { “bson”: encode(*, “base64”)) as data, get_thing_shadow(principal(), “${role arn}”) as thing FROM p/ @a.b Action Kinesis
  25. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ෼ੳ৘ใऩूʹ͍ͭͯ •

    ޻෉఺ • ৘ใͷิ׬ • ୯७ͳ෼ੳͷΈͰ͸ͳ͘ɺݱࡏͷं྆ͷ৘ใ΋ؚΊͯૹ৴ • ෼ੳ؀ڥʹͯ৘ใͷิ׬͕ඞཁͳ͘ͳΔ • ඞཁͳ෦෼ͷΈͷ Lambda • Thing Shadow ͷऔಘͰ͢Β SQL 25
  26. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ޻෉఺ • Ͱ͖ΔݶΓϚωʔδϝϯτػೳΛ࢖͏

    • Lambda ΋Ϛωʔδϝϯτ͕ͩɺ΋ͬͱ͍͍΋ͷ͸΋ͬͱ͋Δ • ཪଆͰಈ͍͍ͯΔ༷ʑͳγεςϜΛ༗ޮ׆༻͢Δ • ThingShadow ͕ߋ৽͞Εͨͱ͖ʹ͸৭ʑ Publish ͞ΕΔɻ • ࠓճ͸ update/documents ͷΈ. update/delta ΋৭ʑເ͕޿͕Δ • ৽͍͠ػೳ΋ϝϦοτΛߟ͑ͯ࢖͏ • SQL ͷؔ਺ͷҰͭͰ͋Δ aws_lambda ͸ൺֱత৽͍͠ػೳɻ • අ༻࡟ݮʹେ͍ʹ໾ཱͬͯ͘ΕͯΔ (͸͡Ί͸͍Βͳ͍ͳʔͬͯࢥͬͯͨ) 26
  27. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ຊεϥΠυ • ৘ใू໿ج൫γεςϜ಺ͷ


    ৘ใΛ෼ੳ͢Δ·Ͱͷ෦෼ͷൃදΛͤͯ͞௖͖·ͨ͠ • ͦͷଞ෦෼ʹڵຯ͋Γ·ͨ͠ΒҎԼΛࢀর͍ͩ͘͞ ⁃ AWS Summit 2018 Tokyo - DeNA ΦʔτϞʔςΟϒʹ͓͚Δ AWS ׆༻ࣄྫ ⁃ DeNA TechCon 2018 – AWS IoTΛ༻͍ͨDeNAΦʔτϞʔςΟϒΞʔΩςΫ νϟ ⁃ ݕࡧΩʔϫʔυʮAWS IoT DeNAʯ ※ ΞʔΩςΫνϟͷத਎͸೔ʑਐԽ͓ͯ͠Γ·͢ 27
  28. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ࣦഊஊ 28

  29. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ੲͷࣦഊ఺ (1)

    • Lambda ࢖͍·͘Γ • ੲ͸౷ҰԽΛؾʹͯ͠ɺશ෦ Lambda ͰॲཧΛͯͨ͠ • ं͔྆Βͷೖྗ͕ ਺ઍ rps. • தΛશ෦ lambda ʹ͢ΔͱඵؒສϨϕϧͷlambda͕࣮ߦ͞Ε͍ͯͨɻ • ߴ͍ɻ 29
  30. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ੲͷࣦഊ఺ (2)

    • ThingShadow ͷ index ࢖͍·͘Γ • ݕࡧͰ͖Δͷ͸ඇৗʹศརͩͬͨ (͜ͷ৚݅ͷं͚ͩநग़͍ͨ͠ͱ͔) • ਺ઍ rps ͷ index ͷߋ৽͕͔͔Δ • ߴ͍ɻ • ஗͘ͳΔʢ͸ͣʣ 30
  31. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ੲͷࣦഊ఺ (3)

    • Admin API (؅ཧܥͷAPI) ࢖͍·͘Γ • εϧʔϓοτΛٻΊͨ API ͡Όͳ͍ͷʹɺ਺ઍ rps Ͱόϯόϯୟ͍ͯ·ͨ͠ • έʔε͕࡞੒͞ΕΔͱ͍͏ܦݧΛ͠·ͨ͠ 31