AWSが支えるEightリコメンデーションエンジンの裏側

"84͕ࢧ͑Δ &JHIUϦίϝϯσʔγϣϯΤϯδϯ ͷཪଆ 4BOTBOגࣜձࣾɹ&JHIUࣄۀ෦  ླ໦߁׮ "84%FW%BZ5PLZP %BZ

ླ໦߁׮ w 3VCZͰαʔϏεج൫΍"1*࡞ͬͨΓ w ࠷ۙ͸1ZUIPOͰ-BNCEB࣮૷ͱ͔ w ւ֎ཱྀߦ޷͖ ϦίϝϯσʔγϣϯνʔϜ νʔϜϦʔμ݉ςΫχΧϧɾΞʔΩςΫτ

ܭαʔϏεҎ্

αʔόʔϨε Ϗοάσʔλ º

ΞδΣϯμ &JHIUʹ͓͚ΔϦίϝϯσʔγϣϯ ϦίϝϯυΞʔΩςΫνϟ࡮৽ Ϧίϝϯυσʔλચ͍ସ͑

&JHIUʹ͓͚Δ Ϧίϝϯσʔγϣϯ

ݸ ਓ ޲ ͚ ໊ ࢗ Ξ ϓ Ϧ

ʮ ໊ ࢗ Λ Ϗ δ ω ε ͷ ͭ
ͳ ͕ Γ ʯ ʹ ม ͑Δ

໊ ࢗ ͷ σ ʔ λ Խ

ͭ ͳ ͕ ͬ ͨ ૬ ख ͱ ί ϛ
ϡ χέ ʔ γ ϣ ϯ

໊ࢗަ׵

- - - ೥໊ؒࢗަ׵"  ໿

- - - ໿ ԯຕ

Ϗοάσʔλ

໊ࢗ؅ཧ͔Β ϏδωεωοτϫʔΫ΁

ͭͳ͕Γ

w ໊ࢗ৘ใͷΞοϓσʔτ w ϑΟʔυ౤ߘӾཡ w ϝοηʔδૹड৴ ωοτϫʔΫ ׆ੑԽ ͭͳ͕Δ͜ͱͰ

ͭͳ͕ΓΛՃ଎ͤ͞Δٕज़

ར༻ऀҰਓͻͱΓͷᅂ޷ʹ߹Θͤ  ίϯςϯπΛਪન Ϧίϝϯσʔγϣϯ

ਓ෺Λਪન

໊ࢗަ׵ϦΫΤετ⬆ ͭͳ͕Γ⬆ ঝೝ⬆ ʮͭͳ͕ΓʯΛޮ཰తʹੜΈग़͢

Ϧίϝϯσʔγϣϯ Τϯδϯ

࣮͸ɺ  ೥΄Ͳલ͔Βಋೖ

چϦίϝϯσʔγϣϯΤϯδϯ ໊ࢗ௥Ճ ͭͳ͕ͬͨ ੜ ੒ ୭͕୭ʹԿΛ͔ͨ͠ ϢʔβʔΞΫςΟϏςΟ Ϧίϝϯυ  ܭࢉΤϯδϯ ຖ೔
ຖि Ұ౓ɺόονॲཧͰߋ৽

͔͠͠ɺ

3FETIJGU w ແବʹෳࡶͳΫΤϦ $MPVE4FBSDI w શจݕࡧͱͯ͠࢖ͬͯͳ͔ͬͨ@TDPSF w ߋ৽·ͰͷϨΠςϯγʔ

ͭ͗͸͗ ΞʔΩςΫνϟ

૿͑ͳ͍ ͭͳ͕Γ

w ύϑΥʔϚϯε⬇ w ΦϖϨʔγϣϯίετ⬆

࠷దͳ"84αʔϏε બఆͷॏཁੑ

Ϧίϝϯυ ΞʔΩςΫνϟ࡮৽

࡮৽ͷνϟϯε౸དྷ ࡢ೥݄

Ϧίϝϯυʹ࠷దͳ ΞʔΩςΫνϟઃܭͱ͸

Ϧίϝϯσʔγϣϯͷཁ σʔλ෼ੳ ΞϧΰϦζϜ ϦΞϧλΠϜϑΟʔυόοΫ γχΞσʔλαΠΤϯςΟετᐌ͘

௚ۙͷग़ձ͍Λେࣄʹ σʔλͷ઱౓͕ྑ͍΄Ͳɺ Ϣʔβʔ͸ϦΫΤετঝೝΛ͠΍͘͢ͳΔ ͭͳ͕Δ

lϦΞϧλΠϜz   ϦίϝϯσʔγϣϯΤϯδϯ

։ൃظؒ  ϲ݄

৚݅ • 2ϲ݄Ͱ͖Δ͜ͱʹϑΥʔΧε ɾεέʔϥϏϦςΟ ˠϦΞϧλΠϜੑΛ୲อ͠ɺ๲େͳσʔλΛॲཧͰ͖Δ͜ͱ

ϨʔςΟϯάσʔλ ୭͕୭ʹԿΛ͔ͨ͠ ϢʔβʔΞΫςΟϏςΟ ϦΞϧλΠϜੑΛ୲อ͢ΔͨΊʹ தؒσʔλΛ࣋ͭ ੜϩά Ϩίʔυʹ৘ใΛ·ͱΊΔ ୭͕ "͞Μ
9͞Μ ୭ʹ #͞Μ :͞Μ ͭͳ͕Γঢ়ଶ ͭͳ͕͍ͬͯΔ ʜ ͍͍Ͷճ਺ ճ ʜ ϝοηʔδճ਺ ճ ʜ ౤ߘγΣΞճ਺ ճ ʜ '#ͭͳ͕Γ ͳ͠ ʜ

ߏ૝ ɾϩά෦෼͸طଘͷ3FETIJGUͷσʔλΛ׆༻ wதؒσʔλߋ৽ʹ,JOFTJT -BNCEB͕࢖͑ͦ͏ w஌ݟ͕͋Δ%ZOBNP%#ΛετϨʔδʹ wγϟʔυ਺ΩϟύγςΟΛ্͛Δ͜ͱͰεέʔϧ

͜Ε᷿ͬͯͰ࿩୊ͷ

αʔόʔϨε ΞʔΩςΫνϟ

ܹಈͷϲ݄Λܦͯ ɻɻɻ ৭ʑলུ

৽ Ϧίϝϯσʔγϣϯ Τϯδϯ ஀ੜ

ར༻"84αʔϏε ετϨʔδ ίϯγϡʔϚ -BNCEB %ZOBNP%# &MBTUJ$BDIF 3FETIJGU ετϦʔϜ %ZOBNP%#
4USFBNT ,JOFTJT4USFBNT

logging Streaming Triggered Write Send Message Polled Get Write Triggered
Set Ϣʔβʔ ΞΫςΟϏςΟ Get Set 8FCαʔόʔ #BUDIαʔόʔ Ϧίϝϯυ ݁Ռ Triggered ϩάऩू ϨʔςΟϯάσʔλ  தؒσʔλ ੜ੒ ϨʔςΟϯάσʔλ  Ωϟογϡੜ੒ ਪનσʔλܭࢉɾ ੜ੒ Triggered

΄΅ αʔόʔϨε ΞʔΩςΫνϟ

Ϧίϝϯυ݁Ռ͕ ϦΞϧλΠϜͰੜ੒͞ΕΔ

΄Ͳɺ ؁͘͸ͳ͔ͬͨʂʂ

αʔόʔϨε͸  ເͷπʔϧͰ͸ͳ͔ͬͨ

αʔόϨεʹର͢Δޡղ w ύϑΥʔϚϯε⬆ w ΦϖϨʔγϣϯίετ⬇

αʔόϨεʹର͢Δޡղ w ύϑΥʔϚϯε⬆ w ΦϖϨʔγϣϯίετ⬇ ❌ ❌

ύϑΥʔϚϯε

ύϑΥʔϚϯε໰୊ ϢʔβΞΫςΟϏςΟ ϨʔςΟϯάσʔλ ෼ؒͷΞΫςΟϏςΟ͕ ࣌ؒͨͬͯ΋ॲཧͰ͖ͳ͍ ໿݅NJO

Ͳ͏͔ͨ͠ʁ

ղܾํ๏ ,JOFTJT 4USFBNT 1VU3FDPSETͰϦΫΤετΛ·ͱΊΔ -BNCEB ϝϞϦઃఆͰϦιʔεΛ૿΍͢ %ZOBNP%# #BUDIXSJUFͰ·ͱΊͯॻ͖ࠐΈ

1VUSFDPSETͰ·ͱΊͯϦΫΤετ Put record Put records #FGPSF "GUFS

ϝϞϦઃఆ͸ίϯϐϡʔςΟϯάϦιʔεͷઃఆ .# .#

#BUDIXSJUFͰҰׅอଘ Put item Batch write item #FGPSF "GUFS

ύϑΥʔϚϯε վળ✨

ΦϖϨʔγϣϯ ίετ

'VODUJPO਺ͷ૿Ճ໰୊ ॲཧͷ୯७Խ͍ͨ͠ 'VODUJPO਺ͷ૿Ճ ؅ཧ͖͠Εͳ͍ ιʔείʔυ؅ཧͱ'VODUJPO਺ͷ τϨʔυΦϑΛղܾ͢Δ

ετϦʔϜίϯγϡʔϚ໰୊ ετϦʔϜͱίϯγϡʔϚͷόϥϯεΛͱΔ 3FBE1SPWJTJPOFE 5ISPVHIQVU&YDFFEFE͕ ൃੜ͠΍͘͢ͳΔ ετϦʔϜ਺ͷ૿Ճ ؅ཧ͖͠Εͳ͍ ίετ૿

Ͳ͏͔ͨ͠ʁ

ղܾํ๏ -BNCEB ετϦʔϜͷ൑ผʹΑΓɺ -BNCEBΛ࡟ݮ͢Δ ,JOFTJT 4USFBNT ֨ೲ͢ΔύϥϝʔλΛ޻෉ͯ͠ɺ  4USFBNͱ-BNCEBΛ࡟ݮ͢Δ

ετϦʔϜͷ ൑ผ✨

ҟͳΔετϦʔϜ͔Β ಉ͡Α͏ͳॲཧΛ͍ͤͨ͞έʔε ͲͷετϦʔϜ͔Λ൑ผ͠ɺॲཧΛ෼ذ -BNCEBΛ·ͱΊɺετϦʔϜΛ൑ผ { ‘eventSource’: ‘aws:kinesis’, ‘eventSourceARN’: ‘arn:aws:kinesis:Stream1’, }
{ ‘eventSource’: ‘aws:dynamodb’, ‘eventSourceARN’: ‘arn:aws:dynamodb:xxxx’, } { ‘eventSource’: ‘aws:kinesis’, ‘eventSourceARN’: ‘arn:aws:kinesis:Stream2’, }

ύϥϝʔλΛ ޻෉✨

ಉ͡ύϥϝʔλͰ ҟͳΔॲཧΛ͍ͤͨ͞έʔε {‘data’: 1} {‘data’: 1} {‘data’: 1} {‘key’: ‘func1’,
’data’: 1} {‘key’: ‘func2’, ’data’: 1} {‘key’: ‘func3’, ’data’: 1} ύϥϝʔλʹΑΓॲཧΛ෼ذ {’data’: 1} 4USFBNΛॻ͖໭͢ ෼਎ͷज़

࠷খݶͰ ؅ཧ͠΍͘͢✨

w ύϑΥʔϚϯε⬆ w ΦϖϨʔγϣϯίετ⬇

αʔόʔϨεԽ ϦΞϧλΠϜԽ

lϦΞϧλΠϜz   ϦίϝϯσʔγϣϯΤϯδϯ

Ϧίϝϯυσʔλ ચ͍ସ͑

ϨʔςΟϯάσʔλ ͷ௠෗Խ ࢒Δ՝୊

&JHIUຊαʔϏεଆͷσʔλߏ଄ͷมߋ σʔλෆ੔߹ͷൃੜ ϦίϝϯυΞϧΰϦζϜͷมߋ ݪҼ

࣮ࡍʹൃੜ

୭͕ "͞Μ 9͞Μ ୭ʹ #͞Μ :͞Μ ͭͳ͕Γঢ়ଶ ͭͳ͕͍ͬͯΔ ʜ ͍͍Ͷճ਺
ճ ʜ ϝοηʔδճ਺ ճ ʜ ౤ߘγΣΞճ਺ ճ ʜ '#ͭͳ͕Γ ͳ͠ ʜ ࠓ·Ͱؤுͬͯ஝ੵͨ͠΋ͷ͸ແʹؐΔ ❌ ALTER TABLE ͱ͔ͳ͍ͷͰ

Ͳ͏͔ͨ͠ʁ

શաڈϩάΛूܭͯ͠ ϨʔςΟϯάσʔλΛ ࡞Γ௚͢✨

%BUB1JQFMJOFΛར༻ چςʔϒϧ  ෆཁʹͳͬͨΒ࡟আ ৽ςʔϒϧ 3FETIJGU ❌ Redshift Copy Activity Hive
Activity ૯*UFN਺ສ௒ 8SJUF$BQBDJUZ ʹઃఆͯ͠ ਺͔͔࣌ؒΔ 

γεςϜશମͰߟ͑Δͱ

Set Ϣʔβʔ ΞΫςΟϏςΟ Get Set 8FCαʔόʔ #BUDIαʔόʔ Ϧίϝϯυ ݁Ռ Triggered Triggered

Set Ϣʔβʔ ΞΫςΟϏςΟ Get Set 8FCαʔόʔ #BUDIαʔόʔ Ϧίϝϯυ ݁Ռ Triggered Triggered -BNCEBఀࢭ

Set Ϣʔβʔ ΞΫςΟϏςΟ Get Set 8FCαʔόʔ #BUDIαʔόʔ Ϧίϝϯυ ݁Ռ Triggered Triggered -BNCEBఀࢭ ৽σʔλੜ੒

Set Ϣʔβʔ ΞΫςΟϏςΟ Get Set 8FCαʔόʔ #BUDIαʔόʔ Ϧίϝϯυ ݁Ռ Triggered Triggered -BNCEBఀࢭ Replace ςʔϒϧೖΕସ͑ ৽σʔλੜ੒

Set Ϣʔβʔ ΞΫςΟϏςΟ Get Set 8FCαʔόʔ #BUDIαʔόʔ Ϧίϝϯυ ݁Ռ Triggered Triggered -BNCEBఀࢭ Replace ςʔϒϧೖΕସ͑ ৽σʔλੜ੒ Ωϟογϡ ɹ΢ΥʔϜΞοϓ

Set Ϣʔβʔ ΞΫςΟϏςΟ Get Set 8FCαʔόʔ #BUDIαʔόʔ Ϧίϝϯυ ݁Ռ Triggered Triggered -BNCEBఀࢭ Replace ςʔϒϧೖΕସ͑ -BNCEB࠶։ ৽σʔλੜ੒ Ωϟογϡ ɹ΢ΥʔϜΞοϓ

৺ଁόΠύεखज़ͷΑ͏

͔͠΋

ର৅͸ςʔϒϧଘࡏ ࢦఆͨ͠΋ͷ͚ͩચ͍ସ͍͑ͨ ϨʔςΟϯάσʔλ  *UFN਺໿ສ ϨʔςΟϯάσʔλ *UFN਺໿ສ ϨʔςΟϯάσʔλ  *UFN਺໿ສ

͜Μͳͷ खͰ ΍ͬͯΒΕͳ͍

μ΢ϯλΠϜແ͠Ͱߦ͏ʹ͸ ϫʔΫϑϩʔ؅ཧ͕ඞਢ

αʔόʔϨεͰઈର׬͍݁ͨ͠

ࡢ೥ ݄

ح੻తͳλΠϛϯάͰ ਆΞοϓσʔτʂʂ

4UFQ'VODUJPOT✨

ෳ਺ͷ-BNCEBॲཧΛ·ͱΊͯ ϫʔΫϑϩʔ؅ཧͰ͖ΔΑ͏ʹ͢ΔαʔϏε

%BUB1JQFMJOF͸ ׆͔͍ͨ͠

0O %BUB1JQFMJOF 4UFQ'VODUJPOT αʔόʔϨεͰσʔλચ͍ସ͑

՝୊ w 4UFQ'VODUJPOT͔Β%BUB1JQFMJOFͷঢ়ଶ؅ཧ w -BNCEBͷ5JNFPVU໰୊ w ચ͍ସ͑ର৅ςʔϒϧͷࢦఆํ๏ w Ωϟογϡͷ΢ΥʔϜΞοϓ໰୊ w
μ΢ϯλΠϜແ͠Ͱચ͍ସ͑

Ͳ͏͔ͨ͠ʁ

ղܾํ๏ %BUB1JQFMJOFͷঢ়ଶ؅ཧ %BUB1JQFMJOFͷঢ়ଶΛอ࣋ -BNCEBͷ5JNFPVU໰୊ 3FUSZΛ׆༻ ચ͍ସ͑ର৅ςʔϒϧͷ ࢦఆํ๏ 1BSBMMFM$IPJDFΛ׆༻ Ωϟογϡͷ
΢ΥʔϜΞοϓ໰୊ ɹɹ Ͱߴ଎ʹ΢ΥʔϜΞοϓ μ΢ϯλΠϜແ͠Ͱ ચ͍ସ͑ "U5JNFTUBNQࢦఆͰτϦΨʔઃఆ

%BUB1JQFMJOFͷ ঢ়ଶ؅ཧ

4UFQ'VODUJPOT ಺෦ͷঢ়ଶ͔͠஌Βͳ͍ Step1 Step2 Step3 ❓

%ZOBNP%#ʹ %BUB1JQFMJOFͷ׬ྃঢ়ଶΛ࣋ͨͤΔ Activate Alarm Trigger Put record Polling Step1
Step2 Step3 %BUB1JQFMJOFঢ়ଶอ࣋ςʔϒϧ

-BNCEBͷ λΠϜΞ΢τ໰୊

Step1 Step2 Step3 σʔλͷΠϯϙʔτʹ ਺͔͔࣌ؒΔ λΠϜΞ΢τൃੜ

׬͍ྃͯ͠ͳ͍৔߹͸ Τϥʔѻ͍ͱ͠ɺఆִؒͰϦτϥΠ Polling "Retry" : [{ "ErrorEquals": [ "States.ALL" ],
"IntervalSeconds": 300, "MaxAttempts": 100, "BackoffRate": 1.0 }], If: error Step1 Step2 Step3 else: success %BUB1JQFMJOF ঢ়ଶอ࣋ςʔϒϧ Retry

ચ͍ସ͑ର৅ͷ ࢦఆํ๏

Choice Choice Choice Parallel 1BSBMMFMΛ࢖༻͠ฒྻʹ $IPJDFͰॲཧΛ෼ذ ϨʔςΟϯά ςʔϒϧ ϨʔςΟϯά
ςʔϒϧ ϨʔςΟϯά ςʔϒϧ

ೖྗ͢Δ+40/Λ޻෉͠ɺ ࢦఆςʔϒϧͷΈચ͍ସ͑ Choice Choice Choice Parallel { "RatingTable1": false,
"RatingTable2": true, "RatingTable3": false }

Ωϟογϡͷ ΢ΥʔϜΞοϓ໰୊

Ϣʔβʔ ΞΫςΟϏςΟ 8FCαʔόʔ Ϧίϝϯυ ݁Ռ Get %ZOBNP%#ʹ ϑΣʔϧΦʔόʔ ❌
4MPXEPXO⬇ Get Item ΢ΥʔϜΞοϓ ͍ͯ͠ͳ͍ͱ

%BUB1JQFMJOF %ZOBNP%#4USFBN -BNCEB✨

ΩʔσʔλΛ ҰׅΠϯϙʔτ͠ɺ ετϦʔϜʹҰؾʹྲྀ͢ %ZOBNP%#4USFBNܦ༝Ͱ ΩϟογϡΛͻͨ͢Βߴ଎ੜ੒ Triggered Caching [ :key1,
:key2, :key3, :key4, ], Rating data table )BTILFZ DPVOU TDPSF LFZ LFZ LFZ LFZ Get by :key1 {hash_key: :key1, count:1, score:1} -BNCEB͕ෳ਺ىಈ ΢ΥʔϜΞοϓ༻  ςϯϙϥϦςʔϒϧ

μ΢ϯλΠϜແ͠ Ͱચ͍ସ͑

-BNCEBఀࢭલ -BNCEB࠶։ޙ શաڈϩά͔Β%BUB1JQFMJOFͰ ϨʔςΟϯάσʔλΛҰׅੜ੒ ετϦʔϜ͔Β-BNCEBͰ ϨʔςΟϯάσʔλΛ౎౓ੜ੒

"U5JNFTUBNQͰτϦΨʔઃఆ Latest Oldest refreshed_at Starting Position: At Timestamp =
refreshed_at ࢦఆ࣌ࠁ SFGSFTIFE@BU ΑΓ΋ ৽͍͠΋ͷΛॲཧ

σʔλҰׅੜ੒ͱ-BNCEBॲཧ ͷ଍ฒΈΛἧ͑Δ SFGSFTIFE@BU SFGSFTIFE@BU Starting Position:   At Timestamp import
all put each

͍ͭʹ શࣗಈԽ

ࣄલ४උ  ɾ-BNCEBఀࢭ ϨʔςΟϯάσʔλચ͍ସ͑ ɾ৽σʔλҰׅੜ੒  ɾ%ZOBNP%#ςʔϒϧ࡞੒  ɾσʔλΠϯϙʔτ ɾςʔϒϧ੾Γସ͑ Ωϟογϡ΢ΥʔϜΞοϓ ɾϋογϡΩʔςʔϒϧ࡞੒
ɾϋογϡΩʔΠϯϙʔτ ɾ΢ΥʔϜΞοϓ࣮ߦ ࣄޙॲཧ  ɾ-BNCEB࠶։ ɾෆཁϦιʔε࡟আ

w ܭສ݅ w ໿࣌ؒ

ΞʔΩςΫνϟͷ શ๴͕

Set Ϣʔβʔ ΞΫςΟϏςΟ Get Set 8FCαʔόʔ #BUDIαʔόʔ Ϧίϝϯυ ݁Ռ Triggered Triggered Replace σʔλચ͍ସ͑ػߏ 4UFQ'VODUJPOT

·ͱΊ

αʔόʔϨεͷՄೳੑ

αʔόʔϨεͷར఺ ؾܰ͞ εέʔϥϏϦςΟ ॊೈੑ

ਖ਼͘͠࢖͏͜ͱͷՁ஋

AWSが支えるEightリコメンデーションエンジンの裏側

AWSが支えるEightリコメンデーションエンジンの裏側

More Decks by Yasunori Suzuki

Other Decks in Technology

Featured

Transcript