solving of multi-armed bandit problem in advertisement recommendation

Slide 1

Slide 1 text

޿ࠂϨίϝϯυʹ͓͚Δ ଟ࿹όϯσΟοτ໰୊ͷద༻ͱղ๏ )JSPLB[V:PTIJEB!ZPQQJCMPH 4QFFF *OD ΦϨγΧφΠτ

Slide 2

Slide 2 text

import “github.com/yoppi” w !ZPQQJCMPH4QFFF *OD w 6;06UFDIMFBEFOHJOFFS w ϨίϝϯυΤϯδϯ࡞ͬͨΓ w Ξυαʔό࡞ͬͨΓ w ޿ࠂۀք͸೥͘Β͍ʢಈը޿ࠂɺωΠςΟϒΞυʣ w (Pɺ+BWB4DSJQUɺ3VCZ

Slide 3

Slide 3 text

ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ w ଟ࿹όϯσΟοτ໰୊ w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏ w ద༻݁Ռ w ·ͱΊͱࠓޙͷ՝୊

Slide 4

Slide 4 text

ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ w ଟ࿹όϯσΟοτ໰୊ w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏ w ద༻݁Ռ w ·ͱΊͱࠓޙͷ՝୊

Slide 5

Slide 5 text

6;06ͷ޿ࠂϨίϝϯυ ޿ࠂೖߘ ޿ࠂ഑৴ ΞυωοτϫʔΫ ʜ ʜ ϝσΟΞ ޿ࠂ୅ཧళ

Slide 6

Slide 6 text

6;06ͷ޿ࠂϨίϝϯυ هࣄɾίϯςϯπ ʮ͋ͳͨʹ͓͢͢Ίͷهࣄʯ ϝσΟΞͷهࣄԼ෦ʹ  ΢ΟδΣοτͱ͍͏ܗͰ  ޿ࠂΛ഑৴͢Δ

Slide 7

Slide 7 text

6;06ͷ޿ࠂϨίϝϯυ

Slide 8

Slide 8 text

6;06ͷ޿ࠂϨίϝϯυ ޿ࠂ ޿ࠂ ޿ࠂ/ ਺͋Δ޿ࠂͰԿΛ഑৴͢Ε͹ΫϦοΫͯ͘͠ΕΔͷ͔ʁ ʜ

Slide 9

Slide 9 text

޿ࠂϨίϝϯυͷछྨ w ίϯςϯπϕʔε഑৴ ‣ ͦͷϝσΟΞͷهࣄͱ޿ࠂͷྨࣅ౓͔ΒϨίϝϯυ͢Δ w ϦλʔήςΟϯά഑৴ ‣ ޿ࠂओͷ঎඼Λݟͨ͋ͱʹϢʔβʹରͯͦ͠ͷ঎඼ΛϨί ϝϯυ͢Δ w λʔήςΟϯά഑৴ ‣ ϢʔβͷଐੑΛ%.1౳͔Βਪଌ͠ྨࣅ͢Δଐੑͷ޿ࠂΛϨ ίϝϯυ͢Δ w աڈͷ੒Ռʹجͮ͘഑৴ ‣ ͦͷ࿮ʹաڈʹ഑৴͞Εͨ৘ใΛ΋ͱʹ੒Ռ͕Α͘ͳΔΑ ͏ʹϨίϝϯυ͢Δ

Slide 10

Slide 10 text

ࠓճͷϨίϝϯυΤϯδϯͷ໨త w 6;06ͷച্ʢF$1.ʣΛ্͍͛ͨ w ܭࢉྔΛݮΒ͠ޮ཰Α͘ϨίϝϯυγεςϜΛӡ༻ ͍ͨ͠

Slide 11

Slide 11 text

޿ࠂϨίϝϯυͷछྨ w ίϯςϯπϕʔε഑৴ w ͦͷϝσΟΞͷهࣄͱ޿ࠂͷྨࣅ౓͔ΒϨίϝϯυ͢Δ w ϦλʔήςΟϯά഑৴ w ޿ࠂओͷ঎඼Λݟͨ͋ͱʹϢʔβʹରͯͦ͠ͷ঎඼ΛϨί ϝϯυ͢Δ w λʔήςΟϯά഑৴ w ϢʔβͷଐੑΛ%.1౳͔Βਪଌ͠ྨࣅ͢Δଐੑͷ޿ࠂΛϨ ίϝϯυ͢Δ w աڈͷ੒Ռʹجͮ͘഑৴ w ͦͷ࿮ʹաڈʹ഑৴͞Εͨ৘ใΛ΋ͱʹ੒Ռ͕Α͘ͳΔΑ ͏ʹϨίϝϯυ͢Δ

Slide 12

Slide 12 text

ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ w ଟ࿹όϯσΟοτ໰୊ w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏ w ద༻݁Ռ w ·ͱΊͱࠓޙͷ՝୊

Slide 13

Slide 13 text

ଟ࿹όϯσΟοτ໰୊ ࿹͕Kݸ ࣌ؒtʹ͓͍ͯ࿹i(t)Λબ୒͠ ใुXi(t)ΛಘΔ ͦΕΛ܁Γฦͯ͠ใुΛ࠷େ Խ͍ͤͨ͞ i(t) Xi(t)

Slide 14

Slide 14 text

ଟ࿹όϯσΟοτ໰୊ w બ୒ࢶͷू߹͔ΒҰͭΛબ୒ͨ͠ͱ͖ʹɺͦͷબ୒ࢶʹ ର͢ΔใुΛಘΔ͕ଞͷબ୒ࢶͷใु৘ใ͸ಘΒΕͳ͍ w ͦͷঢ়گԼͰ܁Γฦ͠બ୒ͨ͠ͱ͖ʹใुͷ࿨Λ࠷େԽ ͢Δ͜ͱΛ໨ࢦ͢ ‣ ࿹iͷਅͷظ଴஋µi Λ஌͍ͬͯΔͳΒظ଴஋͕࠷େʹͳ Δ࿹ΛҾ͖ଓ͚Ε͹͍͍͕ͦΕ͸લఏ৚݅ͱͯ͠Ͱ͖ ͳ͍ w ͦΕ·ͰʹಘͨใुΛ΋ͱʹબ୒ࢶΛܾఆ͢ΔઓུΛํ ࡦʢQPMJDZʣͱݺͿ

Slide 15

Slide 15 text

ଟ࿹όϯσΟοτ໰୊ ʲྫ୊ʳ εϩοτϚγϯΛճ(T= 100) Ҿ͚Δͱ͖ʹ ใुΛ࠷େԽͤ͞Δʹ͸ʁ ࿹K = 4

Slide 16

Slide 16 text

ଟ࿹όϯσΟοτ໰୊ ࿹K = 4 ʲࡶͳํࡦʳ nճશ෦ͷ࿹ΛҾ͍ͯ੒Ռͷྑ͔ͬͨ࿹Λɺ ࢒Γͷճ਺(100-4n) Ҿ͖ଓ͚Δ

Slide 17

Slide 17 text

ଟ࿹όϯσΟοτ໰୊ ࿹K = 4 n = 3 ͜ͷͱ͖ճ੒Ռ͕Α͔ͬͨ࿹ΛҾ͖ଓ͚Δ͕ɺ ͔͔ͨͩճͷ୳ࡧͰ͸ਅͷ੒ՌΛݕূ͢Δ͜ͱ͸ ೉ͦ͠͏ i* = argmaxi∈{1, 2, 3, 4}µi

Slide 18

Slide 18 text

ଟ࿹όϯσΟοτ໰୊ ࿹K = 4 n = 20 ͜ͷͱ͖ճ੒Ռ͕Α͔ͬͨ࿹ΛҾ͖ଓ͚Δ͕ɺ ใुͷྑ͍࿹Λར༻͢Δճ਺͕গͳ͘ɺ ใुΛ࠷େԽͤ͞Δ͜ͱ͸೉͍͔͠΋ i* = argmaxi∈{1, 2, 3, 4}µi

Slide 19

Slide 19 text

୳ࡧͱ஌ࣝར༻ͷτϨʔυΦϑ w ୳ࡧʢFYQMPSBUJPOʣ w ଞͷ࿹ͷ΄͏͕੒Ռ͕ྑ͍ͷͰ͸ͱ༧ଌͯ͠Ҿ͍ͯΈΔ w ͨͩɺʮ΋ͬͱଞʹྑ͍࿹͕͋ΔͷͰ͸ʁʯΛ܁Γฦ͢ͱɺ੒Ռ ͷ͍͍࿹ΛҾ͘ճ਺͕গͳ͘ͳΓύϑΥʔϚϯεΛͩ͢͜ͱ͕Ͱ͖ ͳ͍ w ஌ࣝར༻ʢFYQMPJUBUJPOʣ w ͋Δظؒͷ͍͔ͭ͘ͷ࿹ͷ੒Ռ͕࠷΋ྑ͍΋ͷΛ͞Βʹબ୒͠ଓ͚ Δ w ͨͩɺʮ͜ͷ࿹࠷ߴʂʯͱͳͬͯͦΕ͹͔Γબ୒͍ͯ͠Δͱɺଞʹ ΋ͬͱύϑΥʔϚϯε͕ྑ͍࿹Λݟ͚ͭΒΕͳ͍ w ୳ࡧͱ஌ࣝར༻ΛͦΕͧΕద੾ʹ࣮ߦ͢ΔํࡦΛݕ౼͢Δ͜ͱ͕ॏཁ

Slide 20

Slide 20 text

ଟ࿹όϯσΟοτ໰୊ͷओཁͳํࡦ w ЏHSFFEZ w ୯७ͳ૯౰Γ݁Ռ͔Βྑ͔ͬͨ΋ͷΛҾ͖ଓ͚Δ w 6$#ʢ6QQFSDPOpEFODFCPVOEʣ w ඪຊฏۉʹิਖ਼߲ΛՃ͑ͨ΋ͷΛείΞͱͯ͠࿹ΛબͿ w 54ʢ5IPNQTPO4BNQMJOHʣ w աڈʹ؍ଌͰ͖ͨ݁Ռ͔ΒϕΠζ౷ܭͰະདྷͷظ଴஋Λ༧ ଌ͢Δ

Slide 21

Slide 21 text

ଟ࿹όϯσΟοτ໰୊ͷओཁͳํࡦ w ЏHSFFEZ w 6$# w 5IPNQTPO4BNQMJOH

Slide 22

Slide 22 text

5IPNQTPO4BNQMJOH w ʮ֤࿹͕ظ଴஋࠷େͱͳΔ֬཰ʯΛ΋ͱʹϥϯμϜͰ࿹Λબ୒͢Δํ ࡦʢ֬཰Ұக๏ʣ w ͜ͷ֬཰ͷ࿮૊ΈΛϕΠζ౷ܭͰఆࣜԽͨ͠΋ͷ͕5IPNQTPO 4BNQMJOH w ظ଴஋µi ͕ԿΒ͔ͷࣄલ෼෍͔Β੒Γཱ͍ͬͯΔͱ͠ɺਅͷظ଴஋µi ͷࣄޙ෼෍ΛٻΊ࿹Λબ୒͢Δ w ྫ͑͹޿ࠂΛྫʹڍ͛Δͱɺ޿ࠂ͕ΫϦοΫ͞ΕͨɺΫϦοΫ͞ Εͳ͔ͬͨͱ͍͏஋ʹ෼ྨͰ͖ΔͷͰࣄલ෼෍͸ϕϧψʔΠ෼෍ ʢBer(µ): µ∈[0, 1]ʣͱͳΔ w ܭࢉΛ؆ૉʹ͢ΔͨΊɺϕϧψʔΠ෼෍ͷڞ໾ࣄલ෼෍Ͱ͋Δ#FUB ෼෍Λ༻͍Δ

Slide 23

Slide 23 text

5IPNQTPO4BNQMJOHͷٙࣅίʔυ ύϥϝʔλ: α>0, β>0 1: ֤iʹ͍ͭͯni ←0, mi ←0 2: for i = 1, 2,...,T do 3: ^μi = Beta(mi+α, (ni-mi)+β) 4: i ← argmaxi∈{1,2,...,K}^μi 5: ࿹iΛͻ͍ͯใुXi(t)∈{0,1}Λ؍ଌ͢Δ 6: ni ← ni + 1, mi ← mi + Xi(t) 7: end for

Slide 24

Slide 24 text

ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ w ଟ࿹όϯσΟοτ໰୊ w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏ w ద༻݁Ռ w ·ͱΊͱࠓޙͷ՝୊

Slide 25

Slide 25 text

6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ w ଟ࿹όϯσΟοτ໰୊Λ޿ࠂϨίϝϯυΞϧΰϦζϜͱͯ͠ 6;06ద༻ w ࿹͕޿ࠂʹରԠ͢Δ w ํࡦͱͯ͠5IPNQTPO4BNQMJOHΛ࠾༻ w ࣄલ෼෍ͱࣄޙ෼෍͸ͦͷ޿ࠂͷ$53ͷ෼෍ w ࿮ຖͷ഑৴݁Ռ $53 Λࣄલ෼෍ͱͯ͠ࣄޙ෼෍Λܭࢉ͠ɺ഑ ৴͢Δ޿ࠂΛબ୒͢Δ w #FUB෼෍ͷܭࢉ࣌ʹ͓͚ΔЋЌ͸ώϡʔϦεςΟοΫʹܾΊ ଧͪ

Slide 26

Slide 26 text

5IPNQTPO4BNQMJOHΛબΜͩ༁ w όονߋ৽ʹؤ݈Ͱ͋Δ w ຊ൪ӡ༻͢Δࡍʹ͸σʔλαΠζ͔Β͘Δܭࢉྔͷ੍໿ʹΑΓ ใु݁ՌΛϦΞϧλΠϜͰ͸؍ଌ͢Δ͜ͱ͕೉͘͠஗Ԇͯ͠ߋ৽ ͢Δ͜ͱ͕ଟ͍ w ͕ɺ࿹ͷબ୒࣌ʹʮظ଴஋࠷େͱͳΔ֬཰ʹ͕ͨͬͯ͠ϥϯμ ϜͰબ୒͢ΔʯͨΊɺ੒Ռͷѱ͍࿹ʹภΔ͜ͱ͕গͳ͍ w ܭࢉྔ͕গͳ͍ w جຊతͳϩδοΫ͸ࣄલ෼෍͔Βࣄޙ෼෍Λ#FUB෼෍Λܭࢉ͢ Δ͚ͩͰΑ͍ͷͰܭࢉྔΛ཈͑ΒΕΔ w ଟ͘ͷϞσϧͰཧ࿦ݶքΛୡ੒Ͱ͖Δ͜ͱ͕஌ΒΕ͍ͯΔ

Slide 27

Slide 27 text

54Λ޿ࠂʹద༻ͨ͠৔߹ͷٙࣅίʔυ ύϥϝʔλ: α>0, β>0 1: ֤޿ࠂiʹ͍ͭͯ impressioni ←0, clicki ←0 2: for i = 1, 2,...,T do 3: ^μi = Beta(clicki+α, (impressioni-clicki)+β) 4: i←argmaxi∈{1,2,…,K}^μi*CPCi*1000 5: ޿ࠂάϧʔϓiΛͻ͍ͯใुXi(t)∈{0, 1} Λ؍ଌ͢Δ 6: impressioni ←impressioni+1, clicki ←clicki+Xi(t) 7: end for

Slide 28

Slide 28 text

ЋЌͬͯͲ͏ܾΊΕ͹͍͍ͷʁ w ࣄલ෼෍ʢ৽نʹೖߘ͞Εͨ޿ࠂʣ͕ͳ͍޿ࠂʹ͓͍ ͯҙຯΛͳ͢஋ w ؾΛ͚ͭΔ΂͖ͱ͜Ζ͸ฏۉͱ෼ࢄͰΦʔμʔ͕มΘ Δ

Slide 29

Slide 29 text

ϨίϝϯυγεςϜߏ੒ ޿ࠂϚελ ഑৴ϩά ഑৴αʔόɾ࣮ੵूܭαʔό ഑৴σʔλ 0O.FN $BDIF ഑৴σʔλ࡞੒όον 0O.FN $BDIF .FEJB "EWFSUJTFS

Slide 30

Slide 30 text

(PWT1ZUIPO w ݁ہͷͱ͜Ζ5IPNQTPO4BNQMJOH͸ࣄલ෼෍͔Βࣄ ޙ෼෍ΛٻΊΔͱ͖ʹ#FUB෼෍Λܭࢉ͢Δ͚ͩͰΑ͍ w ߴ౓ͳ਺ֶతͳܭࢉ͸ٻΊΒΕͳ͍ w ѹ౗తʹ(PͰܭࢉ͢Δ΄͏͕ճͷόονʹΑΔ࣮ߦ ࣌ؒΛ୹͘Ͱ͖Δ w όϯσΟοτ໰୊Λղ͘ͱ͖ʹ͸࿹ͷߋ৽ִؒΛͳ Δ΂͘খ͘͞ͳΔ΄͏͕ྑ͍ w ͦͷͨΊ(PΛ࠾༻࣮ͯ͠૷

Slide 31

Slide 31 text

(PWT1ZUIPO ࣮ߦ࣌ؒ T ࿹਺ / (P 1ZUIPO

Slide 32

Slide 32 text

ద༻݁Ռ w 6;06Λಋೖ͍ͯ͠ΔϝσΟΞʹద༻ w ഑৴͍ͯ͠Δɺ͔ͭɺJNQSFTTJPO͕ଟ͍ϝσΟΞ w 6;06શମͷF$1.Λ΄Ͳ޲্Ͱ͖ͨ

Slide 33

Slide 33 text

·ͱΊͱࠓޙͷ՝୊ w ໨తͱ͍ͯͨ͠ɺܭࢉྔͷ࡟ݮٴͼF$1.ͷ޲্Λୡ੒Ͱ͖ͨ w ϋΠύʔύϥϝʔλͷ࠷దԽ w ЋЌͷ஋Λࠓ͸ώϡʔϦεςΟοΫʹఆ͍ٛͯ͠Δঢ়ଶͳͷͰ ࿮͝ͱͷ࠷దͳ஋ΛٻΊ͍ͨ w બ୒͢Δ࿹ͷߋ৽ස౓ͷ࠷దԽ w શͯͷ࿮ʹ͓͍ͯ࣌ؒݻఆͰ࿹Λߋ৽͍ͯ͠ΔͷͰ࿮͝ͱͷ࠷ దͳߋ৽ස౓ΛٻΊ͍ͨ w $73ΛՃຯͨ͠࿹ͷ༧ଌ w $53ͷΈ༧ଌͨ͠΋ͷͰ࿹ΛબΜͰ͍ΔͷͰ$73༧ଌ΋౿·͑ ࿹Λબ୒͍ͨ͠

Slide 34

Slide 34 text

ࢀߟจݙ w ੢ྛ޹ ΞυωοτϫʔΫʹ͓͚Δ޿ࠂ഑৴ܭըͷ ࠷దԽ ਓ޻஌ೳֶձࢽ7PM/P w ଟ࿹όϯσΟοτ໰୊ͷཧ࿦ͱΞϧΰϦζϜ IUUQJCJTNMPSHBSDIJWFJCJT JCJT@CBOEJUQEGT w ຊଟ३໵ʰόϯσΟοτ໰୊ͷཧ࿦ͱΞϧΰϦζ Ϝʱߨஊࣾ