Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
ITS を支える 情報集約基盤 アーキテクチャ / ITS Tech Study #02
Hiroyoshi HOUCHI
June 08, 2018
Technology
1
530
ITS を支える 情報集約基盤 アーキテクチャ / ITS Tech Study #02
Hiroyoshi HOUCHI
June 08, 2018
Tweet
Share
More Decks by Hiroyoshi HOUCHI
See All by Hiroyoshi HOUCHI
aws-dev-day-2020-f9-cdk-bestpractice
hixi
0
170
CDK 利用者から見る CDK
hixi
3
610
SIerIoT vol.7
hixi
1
490
AWS IoT を用いた DeNA オートモーティブアーキテクチャ / DeNA Techcon 2018 Houchi Hiroyoshi
hixi
1
33k
sake-game-gcp-5.pdf
hixi
0
62
appengine-ja-night-35.pdf
hixi
0
42
Other Decks in Technology
See All in Technology
toilを解消した話
asumaywy
0
220
大きくなるチームを支える技術 / Technology to support a growing SCX team
ku00
0
150
srenext2022-skaru
mixi_engineers
PRO
1
1.1k
Who owns the Service Level?
chaspy
5
1.5k
テスト自動化の成功を支えるチームと仕組み/TestAutomation
goyoki
7
1.5k
Learning from AWS Customer Security Incidents [2022]
ramimac
0
1.6k
ニフティでSRE推進活動を始めて取り組んできたこと
niftycorp
2
890
CTOのためのQAのつくりかた #scrumniigata / SigSQA How to create QA for CTOs and VPoEs
caori_t
0
360
[SRE NEXT 2022]KaaS桶狭間の戦い 〜Yahoo! JAPANのSLI/SLOを用いた統合監視〜
srenext
0
750
220524_開発PM勉強会vol.11_MNTSQ
kkawase
0
120
Data Warehouse or Data Lake, which one do I choose?
ahana
0
160
2022年最新版 GatsbyJS + TypeScript + microCMS でブログを作る。
hanetsuki
1
1.1k
Featured
See All Featured
Streamline your AJAX requests with AmplifyJS and jQuery
dougneiner
125
8.5k
Atom: Resistance is Futile
akmur
255
20k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1M
Rails Girls Zürich Keynote
gr2m
86
12k
The Language of Interfaces
destraynor
148
20k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
920
How STYLIGHT went responsive
nonsquared
85
3.9k
Making Projects Easy
brettharned
98
4.3k
In The Pink: A Labor of Love
frogandcode
130
21k
Stop Working from a Prison Cell
hatefulcrawdad
261
17k
Ruby is Unlike a Banana
tanoku
91
9.2k
How to Ace a Technical Interview
jacobian
265
21k
Transcript
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Hiroyoshi HOUCHI Office
of Technology Development System Management Unit DeNA Co., Ltd. *54Λࢧ͑Δ ใूج൫ ΞʔΩςΫνϟ *545FDI4UVEZ"84*P5 1
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ࣗݾհ ์Ղ !IJYJ@IZJ
w ݄dݱΦʔϓϯϓϥοτϑΥʔϜγεςϜࣄۀ෦ 4PDJBM"1* ։ൃ ͚ࣾ.PCJMF#BB4 ։ൃϦʔμ ڠۀҊ݅ ։ൃΠϯϑϥ "OE"QQ ΫϥυΞʔΩςΫνϟઃܭ w ݄dΦʔτϞʔςΟϒࣄۀຊ෦ ΫϥυγεςϜΞʔΩςΫτɾڞ௨ج൫ w ݄dγεςϜຊ෦ શࣾԣஅΫϥυγεςϜΞʔΩςΫτ ΛΔͭΓ ΦʔτϞʔςΟϒҾ͖ଓ͖ 2
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ຊͷൃද༰ 3
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. *54Λࢧ͑Δใूج൫ΞʔΩςΫνϟ ⁃ ं྆ొػೳ
⁃ ं྆ɾαʔόʹର͢ΔೝূೝՄ ⁃ ใͷऩू ⁃ ं྆ͦͷͷͷใͷऩू ⁃ ΤοδίϯϐϡʔςΟϯάͷੳใͷऩू ⁃ ं྆ใͷఏڙ ⁃ ं྆ใͷϦΞϧλΠϜऔಘػೳͷఏڙ ⁃ δΦؔΛ༻͍ͨं྆ใͷఏڙ ⁃ ੳγεςϜͷใఏڙ 4
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. *54Λࢧ͑Δใूج൫ΞʔΩςΫνϟ ⁃ ं྆ొػೳ
⁃ ं྆ɾαʔόʹର͢ΔೝূೝՄ ⁃ ใͷऩू ⁃ ं྆ͦͷͷͷใͷऩू ⁃ ΤοδίϯϐϡʔςΟϯάͷੳใͷऩू ⁃ ं྆ใͷఏڙ ⁃ ं྆ใͷϦΞϧλΠϜऔಘػೳͷఏڙ ⁃ δΦؔΛ༻͍ͨं྆ใͷఏڙ ⁃ ੳγεςϜͷใఏڙ 5
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ࢀߟࢿྉ λΫϕϧʹ͍ͭͯ ⁃
AWS Summit 2018 Tokyo - DeNA ΦʔτϞʔςΟϒʹ͓͚Δ AWS ׆༻ࣄྫ ٕज़෦ʹ͍ͭͯ ⁃ DeNA TechCon 2018 – AWS IoTΛ༻͍ͨDeNAΦʔτϞʔςΟϒΞʔΩςΫ νϟ ݕࡧΩʔϫʔυʮ AWS IoT DeNA ʯ(ൃද Google ݕࡧ࣌) 6
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ൃද෦ͷΞʔΩςΫνϟਤ 7 3.
ੳใऩू 2. ं྆ใऩू 1. ੳγεςϜ౷߹
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ੳγεςϜ౷߹ 8
8
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ੳγεςϜ౷߹ 9
{ "topic": "pub/sub topic", "data": { "hoge": "fuga" } } ※ format ʹ͍ͭͯগ͠ lambda Ͱٵऩͯ͠Δ෦͋Γ
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ੳγεςϜ౷߹ 10
• όοϑΝϦϯά͞ΕͨใΛ Lambda Ͱऔಘ • σʔλϑϩʔ্ Kinesis Data Streams → Lambda ࣮ࡍ Lambda → Kinesis Data Streams
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ੳγεςϜ౷߹ 11
• topic Ͱάϧʔϐϯά • Cloud Pub/Sub ·ͱΊͯˍฒྻ Publish [ { "topic": "pub/sub topic", "data": { "hoge": "fuga" } }, …… ]
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ੳγεςϜ౷߹෦ʹ͍ͭͯ •
• جຊతʹ Network ͷ४උ + I/O ͕࣌ؒͱ͔͔ͯΔ෦ • ୯ൃ࣮ߦͰͳͯ͘ɺόοϑΝϦϯά͢Δ͜ͱʹΑͬͯ Lambda ͷ࣮ߦճ + ૯࣮ߦ࣌ؒΛେ෯ʹݮΒ͢͜ͱ(=ίετݮ)͕Ͱ͖Δ (※ ं྆ rps͙Β͍͘Δ ) • ΄΅΄΅ϦΞϧλΠϜʹใΛૹΔ͜ͱ͕Ͱ͖Δ • ߟྀ • Kinesis Data Streams ͷγϟʔυ • ผख๏ • ϦΞϧλΠϜੑΛ͏গ͠ଛͳ͍͍ͬͯͷͰ͋Ε S3 Λ༻͍ͭͭɺ ఆظతʹ GCS ʹૹ৴͢Δ͜ͱՄ • ͦ͏ͨ͠߹ɺͬͱίετΛݮΒ͢͜ͱ͕Ͱ͖Δ 12
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆ใऩू 13
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ͷલͷɺࣄલใ… • ं͔྆Βͷૹ৴ϑΥʔϚοτ
bson (TechCon #33) • ࣄલʹߦ͏ं྆ొ(TechCon #25)ʹΑΓҎԼͷΑ͏ʹͳΔ • ThingName ূ໌ॻID • ThingShadow ʹҎԼͷใؚ͕·Ε͍ͯΔ (TechCon #29) 14 \ TUBUF\ SFQPSUFE\ EFWJDF\ !JE999999 ^ ^ ^ ^
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆ใऩू 15
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆ใऩू 16
Topic p/@s.b (providers/taxibell/@status.bson) Payload ҎԼͷΑ͏ͳ json ͷ bson දݱ {"@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": "1528437407"}
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆ใऩू 17
SQL SELECT aws_lambda(“${bson2json lambda arn}”, { “bson”: encode(*, “base64”)) as state.reported.status FROM p/@s.b Action ҎԼͷ Republish $$aws/things/${principal()}/shadow/update
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆ใऩू 18
ݩͷσʔλ { "state": { "reported": { "device": { "@id": "XXXXXX" } }, "status": {"hoge":"fuga" } } ೖྗσʔλ { "state": { "reported": { "status": { "@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": "1528437407" } } ݁Ռσʔλ { "state": { "reported": { "device": { "@id": "XXXXXX" }, "status": { "@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": “1528437407”, “hoge”: “fuga” } }
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆ใऩू 19
Topic $aws/things/+/shadow/update/documents Payload { "current": { "state": { "reported": { "device": { "@id": "XXXXXX" }, "status": { "@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": "1528437407", "hoge":"fuga" } } , "desired": {} } }, "previous": {ུ} }
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆ใऩू 20
SQL SELECT {'data': {'device': current.state.reported.device, 'reported': current.state.reported.status}, 'topic': ‘${topicname}'} FROM '$aws/things/+/shadow/update/documents' Action Kinesis
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆ใऩू෦ʹ͍ͭͯ •
• ใͷ౷߹ • ThingShadow ͷػೳΛ༗ޮ׆༻͢Δ͜ͱͰɺ͚ࠩͩͰͳ͘ɺͦͷ࣌ͷं྆ ͷͯ͢ͷঢ়ଶΛ͢͜ͱ͕Մೳ • ΫϥΠΞϯτ͚ࠩͩͳͷͰ௨৴ྔతʹ༏͍͠ • ੳڥʹͯใͷิ͕ඞཁͳ͘ͳΔ • ඞཁͳ෦ͷΈͷ Lambda ʹ͑Δ • Lambda Ͱ ThingShadow ͷߋ৽·Ͱ͢Δ͜ͱՄೳ • ҰํͰ ThingShadow ͷؾͰ࣮ߦ͕࣌ؒ૿͑ͯ͠·͏ʹίετ͕૿Ճ͢ΔՄೳ ੑ͕͋Δ 21
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ੳใऩू 22
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ੳใऩू 23
Topic p/@a.b (providers/taxibell/@analytics.bson) Payload ҎԼͷΑ͏ͳ json ͷ bson දݱ { "@t": "${ࣝผࢠ}", "@d" { "accuracy": "100", "raw": { "hoge": "huga" }, "res": { "hoge": "huga" } } } ※ ͜͜Ͱͷ @t type ͷུ
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ੳใऩू 24
SQL SELECT “${topicname} as topic, aws_lambda(“${bson2json lambda arn}”, { “bson”: encode(*, “base64”)) as data, get_thing_shadow(principal(), “${role arn}”) as thing FROM p/ @a.b Action Kinesis
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ੳใऩूʹ͍ͭͯ •
• ใͷิ • ୯७ͳੳͷΈͰͳ͘ɺݱࡏͷं྆ͷใؚΊͯૹ৴ • ੳڥʹͯใͷิ͕ඞཁͳ͘ͳΔ • ඞཁͳ෦ͷΈͷ Lambda • Thing Shadow ͷऔಘͰ͢Β SQL 25
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • Ͱ͖ΔݶΓϚωʔδϝϯτػೳΛ͏
• Lambda Ϛωʔδϝϯτ͕ͩɺͬͱ͍͍ͷͬͱ͋Δ • ཪଆͰಈ͍͍ͯΔ༷ʑͳγεςϜΛ༗ޮ׆༻͢Δ • ThingShadow ͕ߋ৽͞Εͨͱ͖ʹ৭ʑ Publish ͞ΕΔɻ • ࠓճ update/documents ͷΈ. update/delta ৭ʑເ͕͕Δ • ৽͍͠ػೳϝϦοτΛߟ͑ͯ͏ • SQL ͷؔͷҰͭͰ͋Δ aws_lambda ൺֱత৽͍͠ػೳɻ • අ༻ݮʹେ͍ʹཱͬͯ͘ΕͯΔ (͡Ί͍Βͳ͍ͳʔͬͯࢥͬͯͨ) 26
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ຊεϥΠυ • ใूج൫γεςϜͷ
ใΛੳ͢Δ·Ͱͷ෦ͷൃදΛ͖ͤͯ͞·ͨ͠ • ͦͷଞ෦ʹڵຯ͋Γ·ͨ͠ΒҎԼΛࢀর͍ͩ͘͞ ⁃ AWS Summit 2018 Tokyo - DeNA ΦʔτϞʔςΟϒʹ͓͚Δ AWS ׆༻ࣄྫ ⁃ DeNA TechCon 2018 – AWS IoTΛ༻͍ͨDeNAΦʔτϞʔςΟϒΞʔΩςΫ νϟ ⁃ ݕࡧΩʔϫʔυʮAWS IoT DeNAʯ ※ ΞʔΩςΫνϟͷதʑਐԽ͓ͯ͠Γ·͢ 27
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ࣦഊஊ 28
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ੲͷࣦഊ (1)
• Lambda ͍·͘Γ • ੲ౷ҰԽΛؾʹͯ͠ɺશ෦ Lambda ͰॲཧΛͯͨ͠ • ं͔྆Βͷೖྗ͕ ઍ rps. • தΛશ෦ lambda ʹ͢ΔͱඵؒສϨϕϧͷlambda͕࣮ߦ͞Ε͍ͯͨɻ • ߴ͍ɻ 29
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ੲͷࣦഊ (2)
• ThingShadow ͷ index ͍·͘Γ • ݕࡧͰ͖Δͷඇৗʹศརͩͬͨ (͜ͷ݅ͷं͚ͩநग़͍ͨ͠ͱ͔) • ઍ rps ͷ index ͷߋ৽͕͔͔Δ • ߴ͍ɻ • ͘ͳΔʢͣʣ 30
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ੲͷࣦഊ (3)
• Admin API (ཧܥͷAPI) ͍·͘Γ • εϧʔϓοτΛٻΊͨ API ͡Όͳ͍ͷʹɺઍ rps Ͱόϯόϯୟ͍ͯ·ͨ͠ • έʔε͕࡞͞ΕΔͱ͍͏ܦݧΛ͠·ͨ͠ 31