solving of multi-armed bandit problem in advertisement recommendation

Transcript

1. ޿ࠂϨίϝϯυʹ͓͚Δ
   ଟ࿹όϯσΟοτ໰୊ͷద༻ͱղ๏ )JSPLB[V
   :PTIJEB
   !ZPQQJCMPH
   4QFFF
   *OD
   ΦϨγΧφΠτ
2. import “github.com/yoppi” w !ZPQQJCMPH
   4QFFF
   *OD
   w 6;06
   UFDI
   MFBE
   FOHJOFFS
   w ϨίϝϯυΤϯδϯ࡞ͬͨΓ
   w Ξυαʔό࡞ͬͨΓ
   w ޿ࠂۀք͸೥͘Β͍ʢಈը޿ࠂɺωΠςΟϒΞυʣ
   w (Pɺ+BWB4DSJQUɺ3VCZ
3. ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ
   w ଟ࿹όϯσΟοτ໰୊
   w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏
   w ద༻݁Ռ
   w ·ͱΊͱࠓޙͷ՝୊
4. ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ
   w ଟ࿹όϯσΟοτ໰୊
   w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏
   w ద༻݁Ռ
   w ·ͱΊͱࠓޙͷ՝୊
5. 6;06ͷ޿ࠂϨίϝϯυ ޿ࠂೖߘ ޿ࠂ഑৴ ΞυωοτϫʔΫ ʜ ʜ ϝσΟΞ ޿ࠂ୅ཧళ
6. 6;06ͷ޿ࠂϨίϝϯυ هࣄɾίϯςϯπ ʮ͋ͳͨʹ͓͢͢Ίͷهࣄʯ ϝσΟΞͷهࣄԼ෦ʹ
 ΢ΟδΣοτͱ͍͏ܗͰ
 ޿ࠂΛ഑৴͢Δ
7. 6;06ͷ޿ࠂϨίϝϯυ
8. 6;06ͷ޿ࠂϨίϝϯυ ޿ࠂ ޿ࠂ ޿ࠂ/ ਺͋Δ޿ࠂͰԿΛ഑৴͢Ε͹ΫϦοΫͯ͘͠ΕΔͷ͔ʁ ʜ
9. ޿ࠂϨίϝϯυͷछྨ w ίϯςϯπϕʔε഑৴
   ‣ ͦͷϝσΟΞͷهࣄͱ޿ࠂͷྨࣅ౓͔ΒϨίϝϯυ͢Δ
   w ϦλʔήςΟϯά഑৴
   ‣ ޿ࠂओͷ঎඼Λݟͨ͋ͱʹϢʔβʹରͯͦ͠ͷ঎඼ΛϨί ϝϯυ͢Δ
   w λʔήςΟϯά഑৴
   ‣ ϢʔβͷଐੑΛ%.1౳͔Βਪଌ͠ྨࣅ͢Δଐੑͷ޿ࠂΛϨ ίϝϯυ͢Δ
   w աڈͷ੒Ռʹجͮ͘഑৴
   ‣ ͦͷ࿮ʹաڈʹ഑৴͞Εͨ৘ใΛ΋ͱʹ੒Ռ͕Α͘ͳΔΑ ͏ʹϨίϝϯυ͢Δ
10. ࠓճͷϨίϝϯυΤϯδϯͷ໨త w 6;06ͷച্ʢF$1.ʣΛ্͍͛ͨ
    w ܭࢉྔΛݮΒ͠ޮ཰Α͘ϨίϝϯυγεςϜΛӡ༻ ͍ͨ͠
11. ޿ࠂϨίϝϯυͷछྨ w ίϯςϯπϕʔε഑৴
    w ͦͷϝσΟΞͷهࣄͱ޿ࠂͷྨࣅ౓͔ΒϨίϝϯυ͢Δ
    w ϦλʔήςΟϯά഑৴
    w ޿ࠂओͷ঎඼Λݟͨ͋ͱʹϢʔβʹରͯͦ͠ͷ঎඼ΛϨί ϝϯυ͢Δ
    w λʔήςΟϯά഑৴
    w ϢʔβͷଐੑΛ%.1౳͔Βਪଌ͠ྨࣅ͢Δଐੑͷ޿ࠂΛϨ ίϝϯυ͢Δ
    w աڈͷ੒Ռʹجͮ͘഑৴
    w ͦͷ࿮ʹաڈʹ഑৴͞Εͨ৘ใΛ΋ͱʹ੒Ռ͕Α͘ͳΔΑ ͏ʹϨίϝϯυ͢Δ
12. ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ
    w ଟ࿹όϯσΟοτ໰୊
    w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏
    w ద༻݁Ռ
    w ·ͱΊͱࠓޙͷ՝୊
13. ଟ࿹όϯσΟοτ໰୊ ࿹͕Kݸ ࣌ؒtʹ͓͍ͯ࿹i(t)Λબ୒͠ ใुXi(t)ΛಘΔ
    ͦΕΛ܁Γฦͯ͠ใुΛ࠷େ Խ͍ͤͨ͞ i(t) Xi(t)
14. ଟ࿹όϯσΟοτ໰୊ w બ୒ࢶͷू߹͔ΒҰͭΛબ୒ͨ͠ͱ͖ʹɺͦͷબ୒ࢶʹ ର͢ΔใुΛಘΔ͕ଞͷબ୒ࢶͷใु৘ใ͸ಘΒΕͳ͍
    w ͦͷঢ়گԼͰ܁Γฦ͠બ୒ͨ͠ͱ͖ʹใुͷ࿨Λ࠷େԽ ͢Δ͜ͱΛ໨ࢦ͢
    ‣ ࿹iͷਅͷظ଴஋µi Λ஌͍ͬͯΔͳΒظ଴஋͕࠷େʹͳ Δ࿹ΛҾ͖ଓ͚Ε͹͍͍͕ͦΕ͸લఏ৚݅ͱͯ͠Ͱ͖ ͳ͍
    w ͦΕ·ͰʹಘͨใुΛ΋ͱʹબ୒ࢶΛܾఆ͢ΔઓུΛํ ࡦʢQPMJDZʣͱݺͿ
15. ଟ࿹όϯσΟοτ໰୊ ʲྫ୊ʳ
    εϩοτϚγϯΛճ
    (T= 100) Ҿ͚Δͱ͖ʹ ใुΛ࠷େԽͤ͞Δʹ͸ʁ ࿹
    K = 4
16. ଟ࿹όϯσΟοτ໰୊ ࿹
    K = 4 ʲࡶͳํࡦʳ
    nճશ෦ͷ࿹ΛҾ͍ͯ੒Ռͷྑ͔ͬͨ࿹Λɺ
    ࢒Γͷճ਺
    (100-4n) Ҿ͖ଓ͚Δ
17. ଟ࿹όϯσΟοτ໰୊ ࿹
    K = 4 n = 3 ͜ͷͱ͖ճ੒Ռ͕Α͔ͬͨ࿹ΛҾ͖ଓ͚Δ͕ɺ
    ͔͔ͨͩճͷ୳ࡧͰ͸ਅͷ੒ՌΛݕূ͢Δ͜ͱ͸
    ೉ͦ͠͏ i* = argmaxi∈{1, 2, 3, 4}µi
18. ଟ࿹όϯσΟοτ໰୊ ࿹
    K = 4 n = 20 ͜ͷͱ͖ճ੒Ռ͕Α͔ͬͨ࿹ΛҾ͖ଓ͚Δ͕ɺ
    ใुͷྑ͍࿹Λར༻͢Δճ਺͕গͳ͘ɺ
    ใुΛ࠷େԽͤ͞Δ͜ͱ͸೉͍͔͠΋ i* = argmaxi∈{1, 2, 3, 4}µi
19. ୳ࡧͱ஌ࣝར༻ͷτϨʔυΦϑ w ୳ࡧʢFYQMPSBUJPOʣ
    w ଞͷ࿹ͷ΄͏͕੒Ռ͕ྑ͍ͷͰ͸ͱ༧ଌͯ͠Ҿ͍ͯΈΔ
    w ͨͩɺʮ΋ͬͱଞʹྑ͍࿹͕͋ΔͷͰ͸ʁʯΛ܁Γฦ͢ͱɺ੒Ռ ͷ͍͍࿹ΛҾ͘ճ਺͕গͳ͘ͳΓύϑΥʔϚϯεΛͩ͢͜ͱ͕Ͱ͖ ͳ͍
    w ஌ࣝར༻ʢFYQMPJUBUJPOʣ
    w ͋Δظؒͷ͍͔ͭ͘ͷ࿹ͷ੒Ռ͕࠷΋ྑ͍΋ͷΛ͞Βʹબ୒͠ଓ͚ Δ
    w ͨͩɺʮ͜ͷ࿹࠷ߴʂʯͱͳͬͯͦΕ͹͔Γબ୒͍ͯ͠Δͱɺଞʹ ΋ͬͱύϑΥʔϚϯε͕ྑ͍࿹Λݟ͚ͭΒΕͳ͍
    w ୳ࡧͱ஌ࣝར༻ΛͦΕͧΕద੾ʹ࣮ߦ͢ΔํࡦΛݕ౼͢Δ͜ͱ͕ॏཁ
20. ଟ࿹όϯσΟοτ໰୊ͷओཁͳํࡦ w ЏHSFFEZ
    w ୯७ͳ૯౰Γ݁Ռ͔Βྑ͔ͬͨ΋ͷΛҾ͖ଓ͚Δ
    w 6$#ʢ6QQFSDPOpEFODF
    CPVOEʣ
    w ඪຊฏۉʹิਖ਼߲ΛՃ͑ͨ΋ͷΛείΞͱͯ͠࿹ΛબͿ
    w 54ʢ5IPNQTPO
    4BNQMJOHʣ
    w աڈʹ؍ଌͰ͖ͨ݁Ռ͔ΒϕΠζ౷ܭͰະདྷͷظ଴஋Λ༧ ଌ͢Δ
21. ଟ࿹όϯσΟοτ໰୊ͷओཁͳํࡦ w ЏHSFFEZ
    w 6$#
    
    w 5IPNQTPO
    4BNQMJOH
22. 5IPNQTPO
    4BNQMJOH w ʮ֤࿹͕ظ଴஋࠷େͱͳΔ֬཰ʯΛ΋ͱʹϥϯμϜͰ࿹Λબ୒͢Δํ ࡦʢ֬཰Ұக๏ʣ
    w ͜ͷ֬཰ͷ࿮૊ΈΛϕΠζ౷ܭͰఆࣜԽͨ͠΋ͷ͕5IPNQTPO
    4BNQMJOH
    w ظ଴஋µi ͕ԿΒ͔ͷࣄલ෼෍͔Β੒Γཱ͍ͬͯΔͱ͠ɺਅͷظ଴஋µi ͷࣄޙ෼෍ΛٻΊ࿹Λબ୒͢Δ
    w ྫ͑͹޿ࠂΛྫʹڍ͛Δͱɺ޿ࠂ͕ΫϦοΫ͞ΕͨɺΫϦοΫ͞ Εͳ͔ͬͨͱ͍͏஋ʹ෼ྨͰ͖ΔͷͰࣄલ෼෍͸ϕϧψʔΠ෼෍ ʢBer(µ): µ∈[0, 1]ʣͱͳΔ
    w ܭࢉΛ؆ૉʹ͢ΔͨΊɺϕϧψʔΠ෼෍ͷڞ໾ࣄલ෼෍Ͱ͋Δ#FUB ෼෍Λ༻͍Δ
23. 5IPNQTPO
    4BNQMJOHͷٙࣅίʔυ ύϥϝʔλ: α>0, β>0 1: ֤iʹ͍ͭͯni ←0, mi ←0 2: for i = 1, 2,...,T do 3: ^μi = Beta(mi+α, (ni-mi)+β) 4: i ← argmaxi∈{1,2,...,K}^μi 5: ࿹iΛͻ͍ͯใुXi(t)∈{0,1}Λ؍ଌ͢Δ 6: ni ← ni + 1, mi ← mi + Xi(t) 7: end for
24. ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ
    w ଟ࿹όϯσΟοτ໰୊
    w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏
    w ద༻݁Ռ
    w ·ͱΊͱࠓޙͷ՝୊
25. 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ w ଟ࿹όϯσΟοτ໰୊Λ޿ࠂϨίϝϯυΞϧΰϦζϜͱͯ͠ 6;06ద༻
    w ࿹͕޿ࠂʹରԠ͢Δ
    w ํࡦͱͯ͠5IPNQTPO
    4BNQMJOHΛ࠾༻
    w ࣄલ෼෍ͱࣄޙ෼෍͸ͦͷ޿ࠂͷ$53ͷ෼෍
    w ࿮ຖͷ഑৴݁Ռ $53 Λࣄલ෼෍ͱͯ͠ࣄޙ෼෍Λܭࢉ͠ɺ഑ ৴͢Δ޿ࠂΛબ୒͢Δ
    w #FUB෼෍ͷܭࢉ࣌ʹ͓͚ΔЋ Ќ͸ώϡʔϦεςΟοΫʹܾΊ ଧͪ
26. 5IPNQTPO
    4BNQMJOHΛબΜͩ༁ w όονߋ৽ʹؤ݈Ͱ͋Δ
    w ຊ൪ӡ༻͢Δࡍʹ͸σʔλαΠζ͔Β͘Δܭࢉྔͷ੍໿ʹΑΓ ใु݁ՌΛϦΞϧλΠϜͰ͸؍ଌ͢Δ͜ͱ͕೉͘͠஗Ԇͯ͠ߋ৽ ͢Δ͜ͱ͕ଟ͍
    w ͕ɺ࿹ͷબ୒࣌ʹʮظ଴஋࠷େͱͳΔ֬཰ʹ͕ͨͬͯ͠ϥϯμ ϜͰબ୒͢ΔʯͨΊɺ੒Ռͷѱ͍࿹ʹภΔ͜ͱ͕গͳ͍
    w ܭࢉྔ͕গͳ͍
    w جຊతͳϩδοΫ͸ࣄલ෼෍͔Βࣄޙ෼෍Λ#FUB෼෍Λܭࢉ͢ Δ͚ͩͰΑ͍ͷͰܭࢉྔΛ཈͑ΒΕΔ
    w ଟ͘ͷϞσϧͰཧ࿦ݶքΛୡ੒Ͱ͖Δ͜ͱ͕஌ΒΕ͍ͯΔ
27. 54Λ޿ࠂʹద༻ͨ͠৔߹ͷٙࣅίʔυ ύϥϝʔλ: α>0, β>0 1: ֤޿ࠂiʹ͍ͭͯ impressioni ←0, clicki ←0 2: for i = 1, 2,...,T do 3: ^μi = Beta(clicki+α, (impressioni-clicki)+β) 4: i←argmaxi∈{1,2,…,K}^μi*CPCi*1000 5: ޿ࠂάϧʔϓiΛͻ͍ͯใुXi(t)∈{0, 1} Λ؍ଌ͢Δ 6: impressioni ←impressioni+1, clicki ←clicki+Xi(t) 7: end for
28. Ћ ЌͬͯͲ͏ܾΊΕ͹͍͍ͷʁ w ࣄલ෼෍ʢ৽نʹೖߘ͞Εͨ޿ࠂʣ͕ͳ͍޿ࠂʹ͓͍ ͯҙຯΛͳ͢஋
    w ؾΛ͚ͭΔ΂͖ͱ͜Ζ͸ฏۉͱ෼ࢄͰΦʔμʔ͕มΘ Δ
29. ϨίϝϯυγεςϜߏ੒ ޿ࠂϚελ ഑৴ϩά ഑৴αʔόɾ࣮ੵूܭαʔό ഑৴σʔλ 0O.FN $BDIF ഑৴σʔλ࡞੒όον 0O.FN $BDIF .FEJB "EWFSUJTFS
30. (P
    WT
    1ZUIPO w ݁ہͷͱ͜Ζ5IPNQTPO
    4BNQMJOH͸ࣄલ෼෍͔Βࣄ ޙ෼෍ΛٻΊΔͱ͖ʹ#FUB෼෍Λܭࢉ͢Δ͚ͩͰΑ͍
    w ߴ౓ͳ਺ֶతͳܭࢉ͸ٻΊΒΕͳ͍
    w ѹ౗తʹ(PͰܭࢉ͢Δ΄͏͕ճͷόονʹΑΔ࣮ߦ ࣌ؒΛ୹͘Ͱ͖Δ
    w όϯσΟοτ໰୊Λղ͘ͱ͖ʹ͸࿹ͷߋ৽ִؒΛͳ Δ΂͘খ͘͞ͳΔ΄͏͕ྑ͍
    w ͦͷͨΊ(PΛ࠾༻࣮ͯ͠૷
31. (P
    WT
    1ZUIPO ࣮ߦ࣌ؒ T      ࿹਺ /   (P 1ZUIPO
32. ద༻݁Ռ w 6;06Λಋೖ͍ͯ͠ΔϝσΟΞʹద༻
    w ഑৴͍ͯ͠Δɺ͔ͭɺJNQSFTTJPO͕ଟ͍ϝσΟΞ
    w 6;06શମͷF$1.Λ΄Ͳ޲্Ͱ͖ͨ
33. ·ͱΊͱࠓޙͷ՝୊ w ໨తͱ͍ͯͨ͠ɺܭࢉྔͷ࡟ݮٴͼF$1.ͷ޲্Λୡ੒Ͱ͖ͨ
    w ϋΠύʔύϥϝʔλͷ࠷దԽ
    w Ћ Ќͷ஋Λࠓ͸ώϡʔϦεςΟοΫʹఆ͍ٛͯ͠Δঢ়ଶͳͷͰ ࿮͝ͱͷ࠷దͳ஋ΛٻΊ͍ͨ
    w બ୒͢Δ࿹ͷߋ৽ස౓ͷ࠷దԽ
    w શͯͷ࿮ʹ͓͍ͯ࣌ؒݻఆͰ࿹Λߋ৽͍ͯ͠ΔͷͰ࿮͝ͱͷ࠷ దͳߋ৽ස౓ΛٻΊ͍ͨ
    w $73ΛՃຯͨ͠࿹ͷ༧ଌ
    w $53ͷΈ༧ଌͨ͠΋ͷͰ࿹ΛબΜͰ͍ΔͷͰ$73༧ଌ΋౿·͑ ࿹Λબ୒͍ͨ͠
34. ࢀߟจݙ w ੢ྛ
    ޹
    ΞυωοτϫʔΫʹ͓͚Δ޿ࠂ഑৴ܭըͷ ࠷దԽ
    ਓ޻஌ೳֶձࢽ
    7PM
    /P
    w ଟ࿹όϯσΟοτ໰୊ͷཧ࿦ͱΞϧΰϦζϜ
    IUUQJCJTNMPSHBSDIJWFJCJT JCJT@CBOEJUQEGT
    w ຊଟ
    ३໵ʰόϯσΟοτ໰୊ͷཧ࿦ͱΞϧΰϦζ Ϝʱߨஊࣾ