Save 37% off PRO during our Black Friday Sale! »

solving of multi-armed bandit problem in advertisement recommendation

Ad6c3628aba9c357f42e412468e39732?s=47 yoppi
April 20, 2018
4.7k

solving of multi-armed bandit problem in advertisement recommendation

Ad6c3628aba9c357f42e412468e39732?s=128

yoppi

April 20, 2018
Tweet

Transcript

  1. ޿ࠂϨίϝϯυʹ͓͚Δ ଟ࿹όϯσΟοτ໰୊ͷద༻ͱղ๏ )JSPLB[V:PTIJEB!ZPQQJCMPH 4QFFF *OD ΦϨγΧφΠτ

  2. import “github.com/yoppi” w !ZPQQJCMPH4QFFF *OD w 6;06UFDIMFBEFOHJOFFS w ϨίϝϯυΤϯδϯ࡞ͬͨΓ w

    Ξυαʔό࡞ͬͨΓ w ޿ࠂۀք͸೥͘Β͍ʢಈը޿ࠂɺωΠςΟϒΞυʣ w (Pɺ+BWB4DSJQUɺ3VCZ
  3. ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ w ଟ࿹όϯσΟοτ໰୊ w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏ w ద༻݁Ռ w

    ·ͱΊͱࠓޙͷ՝୊
  4. ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ w ଟ࿹όϯσΟοτ໰୊ w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏ w ద༻݁Ռ w

    ·ͱΊͱࠓޙͷ՝୊
  5. 6;06ͷ޿ࠂϨίϝϯυ ޿ࠂೖߘ ޿ࠂ഑৴ ΞυωοτϫʔΫ ʜ ʜ ϝσΟΞ ޿ࠂ୅ཧళ

  6. 6;06ͷ޿ࠂϨίϝϯυ هࣄɾίϯςϯπ ʮ͋ͳͨʹ͓͢͢Ίͷهࣄʯ ϝσΟΞͷهࣄԼ෦ʹ
 ΢ΟδΣοτͱ͍͏ܗͰ
 ޿ࠂΛ഑৴͢Δ

  7. 6;06ͷ޿ࠂϨίϝϯυ

  8. 6;06ͷ޿ࠂϨίϝϯυ ޿ࠂ ޿ࠂ ޿ࠂ/ ਺͋Δ޿ࠂͰԿΛ഑৴͢Ε͹ΫϦοΫͯ͘͠ΕΔͷ͔ʁ ʜ

  9. ޿ࠂϨίϝϯυͷछྨ w ίϯςϯπϕʔε഑৴ ‣ ͦͷϝσΟΞͷهࣄͱ޿ࠂͷྨࣅ౓͔ΒϨίϝϯυ͢Δ w ϦλʔήςΟϯά഑৴ ‣ ޿ࠂओͷ঎඼Λݟͨ͋ͱʹϢʔβʹରͯͦ͠ͷ঎඼ΛϨί ϝϯυ͢Δ

    w λʔήςΟϯά഑৴ ‣ ϢʔβͷଐੑΛ%.1౳͔Βਪଌ͠ྨࣅ͢Δଐੑͷ޿ࠂΛϨ ίϝϯυ͢Δ w աڈͷ੒Ռʹجͮ͘഑৴ ‣ ͦͷ࿮ʹաڈʹ഑৴͞Εͨ৘ใΛ΋ͱʹ੒Ռ͕Α͘ͳΔΑ ͏ʹϨίϝϯυ͢Δ
  10. ࠓճͷϨίϝϯυΤϯδϯͷ໨త w 6;06ͷച্ʢF$1.ʣΛ্͍͛ͨ w ܭࢉྔΛݮΒ͠ޮ཰Α͘ϨίϝϯυγεςϜΛӡ༻ ͍ͨ͠

  11. ޿ࠂϨίϝϯυͷछྨ w ίϯςϯπϕʔε഑৴ w ͦͷϝσΟΞͷهࣄͱ޿ࠂͷྨࣅ౓͔ΒϨίϝϯυ͢Δ w ϦλʔήςΟϯά഑৴ w ޿ࠂओͷ঎඼Λݟͨ͋ͱʹϢʔβʹରͯͦ͠ͷ঎඼ΛϨί ϝϯυ͢Δ

    w λʔήςΟϯά഑৴ w ϢʔβͷଐੑΛ%.1౳͔Βਪଌ͠ྨࣅ͢Δଐੑͷ޿ࠂΛϨ ίϝϯυ͢Δ w աڈͷ੒Ռʹجͮ͘഑৴ w ͦͷ࿮ʹաڈʹ഑৴͞Εͨ৘ใΛ΋ͱʹ੒Ռ͕Α͘ͳΔΑ ͏ʹϨίϝϯυ͢Δ
  12. ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ w ଟ࿹όϯσΟοτ໰୊ w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏ w ద༻݁Ռ w

    ·ͱΊͱࠓޙͷ՝୊
  13. ଟ࿹όϯσΟοτ໰୊ ࿹͕Kݸ ࣌ؒtʹ͓͍ͯ࿹i(t)Λબ୒͠ ใुXi(t)ΛಘΔ ͦΕΛ܁Γฦͯ͠ใुΛ࠷େ Խ͍ͤͨ͞ i(t) Xi(t)

  14. ଟ࿹όϯσΟοτ໰୊ w બ୒ࢶͷू߹͔ΒҰͭΛબ୒ͨ͠ͱ͖ʹɺͦͷબ୒ࢶʹ ର͢ΔใुΛಘΔ͕ଞͷબ୒ࢶͷใु৘ใ͸ಘΒΕͳ͍ w ͦͷঢ়گԼͰ܁Γฦ͠બ୒ͨ͠ͱ͖ʹใुͷ࿨Λ࠷େԽ ͢Δ͜ͱΛ໨ࢦ͢ ‣ ࿹iͷਅͷظ଴஋µi Λ஌͍ͬͯΔͳΒظ଴஋͕࠷େʹͳ

    Δ࿹ΛҾ͖ଓ͚Ε͹͍͍͕ͦΕ͸લఏ৚݅ͱͯ͠Ͱ͖ ͳ͍ w ͦΕ·ͰʹಘͨใुΛ΋ͱʹબ୒ࢶΛܾఆ͢ΔઓུΛํ ࡦʢQPMJDZʣͱݺͿ
  15. ଟ࿹όϯσΟοτ໰୊ ʲྫ୊ʳ εϩοτϚγϯΛճ(T= 100) Ҿ͚Δͱ͖ʹ ใुΛ࠷େԽͤ͞Δʹ͸ʁ ࿹K = 4

  16. ଟ࿹όϯσΟοτ໰୊ ࿹K = 4 ʲࡶͳํࡦʳ nճશ෦ͷ࿹ΛҾ͍ͯ੒Ռͷྑ͔ͬͨ࿹Λɺ ࢒Γͷճ਺(100-4n) Ҿ͖ଓ͚Δ

  17. ଟ࿹όϯσΟοτ໰୊ ࿹K = 4 n = 3 ͜ͷͱ͖ճ੒Ռ͕Α͔ͬͨ࿹ΛҾ͖ଓ͚Δ͕ɺ ͔͔ͨͩճͷ୳ࡧͰ͸ਅͷ੒ՌΛݕূ͢Δ͜ͱ͸ ೉ͦ͠͏

    i* = argmaxi∈{1, 2, 3, 4}µi
  18. ଟ࿹όϯσΟοτ໰୊ ࿹K = 4 n = 20 ͜ͷͱ͖ճ੒Ռ͕Α͔ͬͨ࿹ΛҾ͖ଓ͚Δ͕ɺ ใुͷྑ͍࿹Λར༻͢Δճ਺͕গͳ͘ɺ ใुΛ࠷େԽͤ͞Δ͜ͱ͸೉͍͔͠΋

    i* = argmaxi∈{1, 2, 3, 4}µi
  19. ୳ࡧͱ஌ࣝར༻ͷτϨʔυΦϑ w ୳ࡧʢFYQMPSBUJPOʣ w ଞͷ࿹ͷ΄͏͕੒Ռ͕ྑ͍ͷͰ͸ͱ༧ଌͯ͠Ҿ͍ͯΈΔ w ͨͩɺʮ΋ͬͱଞʹྑ͍࿹͕͋ΔͷͰ͸ʁʯΛ܁Γฦ͢ͱɺ੒Ռ ͷ͍͍࿹ΛҾ͘ճ਺͕গͳ͘ͳΓύϑΥʔϚϯεΛͩ͢͜ͱ͕Ͱ͖ ͳ͍ w

    ஌ࣝར༻ʢFYQMPJUBUJPOʣ w ͋Δظؒͷ͍͔ͭ͘ͷ࿹ͷ੒Ռ͕࠷΋ྑ͍΋ͷΛ͞Βʹબ୒͠ଓ͚ Δ w ͨͩɺʮ͜ͷ࿹࠷ߴʂʯͱͳͬͯͦΕ͹͔Γબ୒͍ͯ͠Δͱɺଞʹ ΋ͬͱύϑΥʔϚϯε͕ྑ͍࿹Λݟ͚ͭΒΕͳ͍ w ୳ࡧͱ஌ࣝར༻ΛͦΕͧΕద੾ʹ࣮ߦ͢ΔํࡦΛݕ౼͢Δ͜ͱ͕ॏཁ
  20. ଟ࿹όϯσΟοτ໰୊ͷओཁͳํࡦ w ЏHSFFEZ w ୯७ͳ૯౰Γ݁Ռ͔Βྑ͔ͬͨ΋ͷΛҾ͖ଓ͚Δ w 6$#ʢ6QQFSDPOpEFODFCPVOEʣ w ඪຊฏۉʹิਖ਼߲ΛՃ͑ͨ΋ͷΛείΞͱͯ͠࿹ΛબͿ w

    54ʢ5IPNQTPO4BNQMJOHʣ w աڈʹ؍ଌͰ͖ͨ݁Ռ͔ΒϕΠζ౷ܭͰະདྷͷظ଴஋Λ༧ ଌ͢Δ
  21. ଟ࿹όϯσΟοτ໰୊ͷओཁͳํࡦ w ЏHSFFEZ w 6$# w 5IPNQTPO4BNQMJOH

  22. 5IPNQTPO4BNQMJOH w ʮ֤࿹͕ظ଴஋࠷େͱͳΔ֬཰ʯΛ΋ͱʹϥϯμϜͰ࿹Λબ୒͢Δํ ࡦʢ֬཰Ұக๏ʣ w ͜ͷ֬཰ͷ࿮૊ΈΛϕΠζ౷ܭͰఆࣜԽͨ͠΋ͷ͕5IPNQTPO 4BNQMJOH w ظ଴஋µi ͕ԿΒ͔ͷࣄલ෼෍͔Β੒Γཱ͍ͬͯΔͱ͠ɺਅͷظ଴஋µi

    ͷࣄޙ෼෍ΛٻΊ࿹Λબ୒͢Δ w ྫ͑͹޿ࠂΛྫʹڍ͛Δͱɺ޿ࠂ͕ΫϦοΫ͞ΕͨɺΫϦοΫ͞ Εͳ͔ͬͨͱ͍͏஋ʹ෼ྨͰ͖ΔͷͰࣄલ෼෍͸ϕϧψʔΠ෼෍ ʢBer(µ): µ∈[0, 1]ʣͱͳΔ w ܭࢉΛ؆ૉʹ͢ΔͨΊɺϕϧψʔΠ෼෍ͷڞ໾ࣄલ෼෍Ͱ͋Δ#FUB ෼෍Λ༻͍Δ
  23. 5IPNQTPO4BNQMJOHͷٙࣅίʔυ ύϥϝʔλ: α>0, β>0 1: ֤iʹ͍ͭͯni ←0, mi ←0 2:

    for i = 1, 2,...,T do 3: ^μi = Beta(mi+α, (ni-mi)+β) 4: i ← argmaxi∈{1,2,...,K}^μi 5: ࿹iΛͻ͍ͯใुXi(t)∈{0,1}Λ؍ଌ͢Δ 6: ni ← ni + 1, mi ← mi + Xi(t) 7: end for
  24. ࠓ೔࿩͢͜ͱ w 6;06ͷ޿ࠂϨίϝϯυ w ଟ࿹όϯσΟοτ໰୊ w 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ͱղ๏ w ద༻݁Ռ w

    ·ͱΊͱࠓޙͷ՝୊
  25. 6;06ͷ޿ࠂϨίϝϯυ΁ͷద༻ w ଟ࿹όϯσΟοτ໰୊Λ޿ࠂϨίϝϯυΞϧΰϦζϜͱͯ͠ 6;06ద༻ w ࿹͕޿ࠂʹରԠ͢Δ w ํࡦͱͯ͠5IPNQTPO4BNQMJOHΛ࠾༻ w ࣄલ෼෍ͱࣄޙ෼෍͸ͦͷ޿ࠂͷ$53ͷ෼෍

    w ࿮ຖͷ഑৴݁Ռ $53 Λࣄલ෼෍ͱͯ͠ࣄޙ෼෍Λܭࢉ͠ɺ഑ ৴͢Δ޿ࠂΛબ୒͢Δ w #FUB෼෍ͷܭࢉ࣌ʹ͓͚ΔЋ Ќ͸ώϡʔϦεςΟοΫʹܾΊ ଧͪ
  26. 5IPNQTPO4BNQMJOHΛબΜͩ༁ w όονߋ৽ʹؤ݈Ͱ͋Δ w ຊ൪ӡ༻͢Δࡍʹ͸σʔλαΠζ͔Β͘Δܭࢉྔͷ੍໿ʹΑΓ ใु݁ՌΛϦΞϧλΠϜͰ͸؍ଌ͢Δ͜ͱ͕೉͘͠஗Ԇͯ͠ߋ৽ ͢Δ͜ͱ͕ଟ͍ w ͕ɺ࿹ͷબ୒࣌ʹʮظ଴஋࠷େͱͳΔ֬཰ʹ͕ͨͬͯ͠ϥϯμ ϜͰબ୒͢ΔʯͨΊɺ੒Ռͷѱ͍࿹ʹภΔ͜ͱ͕গͳ͍

    w ܭࢉྔ͕গͳ͍ w جຊతͳϩδοΫ͸ࣄલ෼෍͔Βࣄޙ෼෍Λ#FUB෼෍Λܭࢉ͢ Δ͚ͩͰΑ͍ͷͰܭࢉྔΛ཈͑ΒΕΔ w ଟ͘ͷϞσϧͰཧ࿦ݶքΛୡ੒Ͱ͖Δ͜ͱ͕஌ΒΕ͍ͯΔ
  27. 54Λ޿ࠂʹద༻ͨ͠৔߹ͷٙࣅίʔυ ύϥϝʔλ: α>0, β>0 1: ֤޿ࠂiʹ͍ͭͯ impressioni ←0, clicki ←0

    2: for i = 1, 2,...,T do 3: ^μi = Beta(clicki+α, (impressioni-clicki)+β) 4: i←argmaxi∈{1,2,…,K}^μi*CPCi*1000 5: ޿ࠂάϧʔϓiΛͻ͍ͯใुXi(t)∈{0, 1} Λ؍ଌ͢Δ 6: impressioni ←impressioni+1, clicki ←clicki+Xi(t) 7: end for
  28. Ћ ЌͬͯͲ͏ܾΊΕ͹͍͍ͷʁ w ࣄલ෼෍ʢ৽نʹೖߘ͞Εͨ޿ࠂʣ͕ͳ͍޿ࠂʹ͓͍ ͯҙຯΛͳ͢஋ w ؾΛ͚ͭΔ΂͖ͱ͜Ζ͸ฏۉͱ෼ࢄͰΦʔμʔ͕มΘ Δ

  29. ϨίϝϯυγεςϜߏ੒ ޿ࠂϚελ ഑৴ϩά ഑৴αʔόɾ࣮ੵूܭαʔό ഑৴σʔλ 0O.FN $BDIF ഑৴σʔλ࡞੒όον 0O.FN $BDIF

    .FEJB "EWFSUJTFS
  30. (PWT1ZUIPO w ݁ہͷͱ͜Ζ5IPNQTPO4BNQMJOH͸ࣄલ෼෍͔Βࣄ ޙ෼෍ΛٻΊΔͱ͖ʹ#FUB෼෍Λܭࢉ͢Δ͚ͩͰΑ͍ w ߴ౓ͳ਺ֶతͳܭࢉ͸ٻΊΒΕͳ͍ w ѹ౗తʹ(PͰܭࢉ͢Δ΄͏͕ճͷόονʹΑΔ࣮ߦ ࣌ؒΛ୹͘Ͱ͖Δ w

    όϯσΟοτ໰୊Λղ͘ͱ͖ʹ͸࿹ͷߋ৽ִؒΛͳ Δ΂͘খ͘͞ͳΔ΄͏͕ྑ͍ w ͦͷͨΊ(PΛ࠾༻࣮ͯ͠૷
  31. (PWT1ZUIPO ࣮ߦ࣌ؒ T      ࿹਺ /

      (P 1ZUIPO
  32. ద༻݁Ռ w 6;06Λಋೖ͍ͯ͠ΔϝσΟΞʹద༻ w ഑৴͍ͯ͠Δɺ͔ͭɺJNQSFTTJPO͕ଟ͍ϝσΟΞ w 6;06શମͷF$1.Λ΄Ͳ޲্Ͱ͖ͨ

  33. ·ͱΊͱࠓޙͷ՝୊ w ໨తͱ͍ͯͨ͠ɺܭࢉྔͷ࡟ݮٴͼF$1.ͷ޲্Λୡ੒Ͱ͖ͨ w ϋΠύʔύϥϝʔλͷ࠷దԽ w Ћ Ќͷ஋Λࠓ͸ώϡʔϦεςΟοΫʹఆ͍ٛͯ͠Δঢ়ଶͳͷͰ ࿮͝ͱͷ࠷దͳ஋ΛٻΊ͍ͨ w

    બ୒͢Δ࿹ͷߋ৽ස౓ͷ࠷దԽ w શͯͷ࿮ʹ͓͍ͯ࣌ؒݻఆͰ࿹Λߋ৽͍ͯ͠ΔͷͰ࿮͝ͱͷ࠷ దͳߋ৽ස౓ΛٻΊ͍ͨ w $73ΛՃຯͨ͠࿹ͷ༧ଌ w $53ͷΈ༧ଌͨ͠΋ͷͰ࿹ΛબΜͰ͍ΔͷͰ$73༧ଌ΋౿·͑ ࿹Λબ୒͍ͨ͠
  34. ࢀߟจݙ w ੢ྛ޹ ΞυωοτϫʔΫʹ͓͚Δ޿ࠂ഑৴ܭըͷ ࠷దԽ ਓ޻஌ೳֶձࢽ7PM/P w ଟ࿹όϯσΟοτ໰୊ͷཧ࿦ͱΞϧΰϦζϜ IUUQJCJTNMPSHBSDIJWFJCJT JCJT@CBOEJUQEGT

    w ຊଟ३໵ʰόϯσΟοτ໰୊ͷཧ࿦ͱΞϧΰϦζ Ϝʱߨஊࣾ