非定常な多腕バンディット問題における変化検出アプローチの線形モデルへの拡張/Extension of Change Detection Method for Non-Stationary Linear Multi Armed Bandit

2020.07.10 第50回 情報処理学会 インターネットと運用技術研究会
https://www.iot.ipsj.or.jp/meeting/50-program/

Transcript

1. ࡾ୐ ༔հ, ܀ྛ ݈ଠ࿠ / Pepabo R&D Institute, GMO Pepabo, Inc.
   2020.07.10 ୈ50ճ ৘ใॲཧֶձ Πϯλʔωοτͱӡ༻ٕज़ݚڀձ
   ඇఆৗͳଟ࿹όϯσΟοτ໰୊ʹ͓͚Δ
   มԽݕग़ΞϓϩʔνͷઢܗϞσϧ΁ͷ֦ு
   

   View Slide

2. 1. ͸͡Ίʹ
   2. ؔ࿈ݚڀ
   3. ఏҊख๏
   4. ධՁ
   5. ·ͱΊ
   2
   ໨࣍
   

   View Slide

3. 1.
   ͸͡Ίʹ
   

   View Slide

4. • WebαʔϏεʹ͓͚Δ৘ใաଟ໰୊Λղܾ͢ΔɺਪનγεςϜͷಋೖ
   • → ͳΜΒ͔ͷํࡦʹج͖ͮଟ਺ͷબ୒ࢶ͔Βར༻ऀ͕ڵຯΛ࣋ͭ΋ͷΛఏ
   Ҋ͢ΔγεςϜ
   4
   എܠ
   • ঎඼
   • ޿ࠂ
   • ਪનΞϧΰϦζϜ
   

   View Slide

5. • WebαʔϏεʹ͓͚Δ৘ใաଟ໰୊Λղܾ͢ΔɺਪનγεςϜͷಋೖ
   • → ͳΜΒ͔ͷํࡦʹج͖ͮଟ਺ͷબ୒ࢶ͔Βར༻ऀ͕ڵຯΛ࣋ͭ΋ͷΛఏ
   Ҋ͢ΔγεςϜ
   5
   എܠ
   • γεςϜʹͱͬͯར༻ऀͷᅂ޷͸ະ஌ʢਪଌͷͨΊͷ৘ใ஝ੵ͕ඞཁʣ
   • ͋Δ࣌఺Ͱ༗༻ͱߟ͑ΒΕΔ৘ใΛʮ׆༻ʯͨ͠ఏҊΛߦ͍ͭͭɺΑΓ༗༻ͳ
   ఏҊʹͭͳ͕Δ৘ใͷʮ୳ࡧʯ͕ٻΊΒΕΔʢػձଛࣦͷճආʣ
   ӡ༻্ͷ՝୊
   

   View Slide

6. • ͋Δ࣌఺ʹ͓͚Δ࠷ળͳબ୒ͷͨΊɺ׆༻ͱ୳ࡧͷτϨʔυΦϑΛղܾ͍ͨ͠
   • → ଟ࿹όϯσΟοτ໰୊ͱΈͳͯ͠ɺಉ໰୊ͷղ๏Λద༻[1]
   6
   ݚڀͷ໨త
   >ࡾ୐༔հ ็߃ݑ 4ZOBQTFจ຺ʹԠͯ͡ܧଓతʹਪનख๏ͷબ୒Λ࠷దԽ͢ΔਪનγεςϜ ిࢠ৘ใ௨৴ֶձ࿦จࢽ%
   7PM+% /P QQ /PW UPBQQFBS
   
   

   View Slide

7. • ͋Δ࣌఺ʹ͓͚Δ࠷ળͳબ୒ͷͨΊɺ׆༻ͱ୳ࡧͷτϨʔυΦϑΛղܾ͍ͨ͠
   • → ଟ࿹όϯσΟοτ໰୊ͱΈͳͯ͠ɺಉ໰୊ͷղ๏Λద༻
   7
   ݚڀͷ໨త
   • ҰํͰɺैདྷͷಉ໰୊ͷղ๏Ͱ͸ɺར༻ऀͷᅂ޷͕ʮଟ༷͔ͭܧଓతʹมԽʯ
   ͢ΔΑ͏ͳ؀ڥʹॆ෼ʹରԠɾ௥ै͢Δ͜ͱ͕Ͱ͖ͳ͍ʢؔ࿈ݚڀͰઆ໌ʣ
   • ͜ͷ؀ڥʹରԠՄೳͳղ๏ΛఏҊ͢Δ͜ͱͰɺར༻ऀͷᅂ޷ͷଟ༷ੑ΍มԽʹ
   ௥ै͢ΔఏҊΛՄೳʹ͍ͨ͠
   

   View Slide

8. 2.
   ؔ࿈ݚڀ
   

   View Slide

9. • ʮ࿹ʯͱݺ͹ΕΔෳ਺ͷީิ͔ΒಘΒΕΔใुΛ࠷େԽ͢Δ໰୊
   • ϓϨΠϠʔ͸Ұ౓ͷࢼߦͰ1ͭͷ࿹Λબ୒͠ɺใुΛಘΔ
   • ͦΕͧΕͷ࿹͸͋Δใु෼෍ʹै͍ใुΛੜ੒
   • ͨͩ͠ɺϓϨΠϠʔ͸͜ͷใु෼෍Λࢼߦͷ݁Ռ͔Βਪଌ͢Δඞཁ͕͋Δ
   9
   ଟ࿹όϯσΟοτ໰୊
   • ϓϨΠϠʔ͸͋Δ࣌఺ͷ࿹ͷධՁʹج͖ͮʮ׆༻ʯͱʮ୳ࡧʯΛฒߦͯ͠ߦ͏
   • ͜ͷτϨʔυΦϑΛղফ͢ΔͨΊʹ༷ʑͳղ๏͕ఏҊ͞Ε͍ͯΔ
   

   View Slide

10. ଟ࿹όϯσΟοτ໰୊ͷ૝ఆ͢Δ࿹ͷ֬཰෼෍ͷछྨ
    10
    Arm0
    Arm1
    Arm2
    User(s) System
    ਪఆͨ֬͠཰෼෍
    ਅͷ֬཰෼෍
    Recommend
    Click
    

    View Slide

11. • ࿹͝ͱͷใु෼෍͸ৗʹಉ͡Ͱ͋Δͱ͍͏Ծఆ
    • → ঢ়گ΍ଐੑ͝ͱʹใु෼෍͕ҟͳΔͷͰ͸ͳ͍͔ʁ
    • ྫʣ೥୅͝ͱʹਓؾͷ঎඼͕ҧ͏ɺ࠷ۙɺಉ͡ΧςΰϦͷ঎඼Λങͬͨ
    • → ࣌ؒͷܦաʹΑͬͯ΋ใु෼෍͕มԽ͢ΔͷͰ͸ͳ͍͔ʁ
    • ྫʣ৽͍͠χϡʔεهࣄɺҰ࣌తͳྲྀߦ
    11
    ଟ࿹όϯσΟοτ໰୊ͷ֦ு
    • ʮઢܗʯ·ͨ͸ʮඇఆৗʯͳଟ࿹όϯσΟοτ໰୊ͱ֦ͯ͠ு͕ߦΘΕ͍ͯΔ
    

    View Slide

12. • ࿹͝ͱʹෳ਺ͷจ຺͕͋Γɺจ຺ʹԠͯ͡ใु෼෍͕ܾ·Δଟ࿹όϯσΟοτ
    ໰୊ͷઃఆ
    • ຊݚڀใࠂͰ͸ɺจ຺͸ɺෳ਺ͷཁҼͷύϥϝʔλͷ૊Έ߹ΘͤͰදݱ͞
    Εͨঢ়ଶͷ͜ͱΛࢦ͢
    • → ཁҼύϥϝʔλͷ஋͕{0,1}ͷ৔߹ɺจ຺͸ཁҼ਺ ʹରͯ͠ ύλʔϯ
    d 2d
    12
    ઢܗͳଟ࿹όϯσΟοτ໰୊
    • ઢܗͳଟ࿹όϯσΟοτ໰୊ͷղ๏Ͱ͸ɺจ຺ͷ֬཰෼෍Ͱ͸ͳ͘ɺཁҼ͝ͱ
    ͷ܎਺ʢઢܗύϥϝʔλʣΛਪఆ͢Δ͜ͱͰ֤จ຺ʹ͓͚ΔใुΛ༧૝͢Δ
    

    View Slide

13. ઢܗͳଟ࿹όϯσΟοτ໰୊
    13
    Arm0
    Arm1
    Arm2
    User(s) System
    ਪఆͨ֬͠཰෼෍
    ਅͷ֬཰෼෍
    Recommend
    Click
    Context = 0 Context = 0
    

    View Slide

14. ઢܗͳଟ࿹όϯσΟοτ໰୊
    14
    Arm0
    Arm1
    Arm2
    User(s) System
    ਪఆͨ֬͠཰෼෍
    ਅͷ֬཰෼෍
    Recommend
    Click
    Context = 1 Context = 1
    

    View Slide

15. • ಉ͡จ຺ʹ͓͍ͯ΋࣌ؒܦաʹΑͬͯใु෼෍͕มԽ͢Δଟ࿹όϯσΟοτ໰
    ୊ͷ໰୊ઃఆ
    • पظతͳมԽͰ͋Ε͹ཁҼͷύϥϝʔλʹؚΊΔ͜ͱͰରԠͰ͖Δ͕ෆن
    ଇͳ৔߹ʢඇఆৗͳมԽͷ৔߹ʣ͸͜ͷݶΓͰ͸ͳ͍
    15
    ඇఆৗͳଟ࿹όϯσΟοτ໰୊
    • ඇఆৗͳଟ࿹όϯσΟοτ໰୊ͷղ๏Ͱ͸ɺաڈʹ؍ଌͨ͠ใुʹଊΘΕͣ࿹
    ͷධՁΛਝ଎ʹߋ৽͢Δ͜ͱͰ֤จ຺ʹ͓͚ΔใुΛ༧૝͢Δ
    ׂҾܕΞϓϩʔν มԽݕग़ܕΞϓϩʔν
    

    View Slide

16. ඇఆৗͳଟ࿹όϯσΟοτ໰୊
    16
    Arm0
    Arm1
    Arm2
    User(s) System
    ਪఆͨ֬͠཰෼෍
    ਅͷ֬཰෼෍
    Recommend
    Click
    t = 0~ t = 100~ t = 0~ t = 100~
    t = 99 t = 99
    

    View Slide

17. ඇఆৗͳଟ࿹όϯσΟοτ໰୊
    17
    Arm0
    Arm1
    Arm2
    User(s) System
    ਪఆͨ֬͠཰෼෍
    ਅͷ֬཰෼෍
    Recommend
    Click
    t = 0~ t = 100~ t = 0~ t = 100~
    t = 199 t = 199
    

    View Slide

18. • ࿹ͷใु෼෍ͷมԽΛલఏʹɺܧଓతʹ࿹ͷධՁΛߋ৽͢ΔΞϓϩʔν
    • աڈͷࢼߦճ਺ɺใुʹରׂͯ͠Ҿ཰Λಋೖ
    • → ৽͍͠؍ଌ݁ՌΛॏཁࢹ
    • UCB1΍Thomson SamplingΛ֦ுͨ͠ղ๏͕ఏҊ͞Ε͍ͯΔ
    • Discounted UCB, Sliding Window UCB
    • Dynamic Thompson Sampling, Discounted TS
    18
    ඇఆৗͳଟ࿹όϯσΟοτ໰୊ͷղ๏ - ׂҾܕ
    ✓
    Trialt Trialt 1
    · · · Trialt n
    Rewardt Rewardt 1
    · · · Rewardt n
    ◆
    0
    B
    B
    B
    @
    ↵1
    ↵2
    .
    .
    .
    ↵n
    1
    C
    C
    C
    A
    AAADbHichVHNbtNAEB7H0BYDbfg5IFVIUaNG5dBoHSoVcarEhQtSmzZtUV2i9WaTrLpZW+tNUDB+AV6AAyeQOCAegAfgwgtw6BMghMSllXrpgbFjCDRQxrL3m2/2m/3W44dSRIaQQ6tgX7g4NT1zybl85ersXPHa9e0o6GvGGyyQgd71acSlULxhhJF8N9Sc9nzJd/yDB2l9Z8B1JAK1ZYYh3+/RjhJtwahBqln84kneNkuO5/OOUDHVmg6TWGIkzpYWVDZjk5QqpZ942U0zj7UCE/1Oq8TznDp/SnUrV/xKzkjGvEocj6tWfqjjadHpmjvOXx0xrFMZdukTFw/KYS2Fg7TvmFMpnOzaLJZJlWRRmgRuDsqQx3pQ/AAetCAABn3oAQcFBrEEChE+e+ACgRC5fYiR04hEVueQgIPaPu7iuIMie4DfDmZ7OaswT3tGmZrhKRJfjcoSLJLP5B05Ip/Ie/KVnP6zV5z1SL0McfVHWh42517c2jz5r6qHq4HuWHWuZwNtuJd5Feg9zJj0FmykHzx7ebR5v74YV8gb8g39vyaH5CPeQA2O2dsNXn91jh8fveAfwwG5Z8cxCbZrVfdutbaxUl57nI9qBuZhAZZwHquwBg9hHRrArEdWZD23ksJ3+6Y9b98ebS1YueYG/BF25QcNWuOW
    

    View Slide

19. • ࿹ͷใु෼෍ͷมԽΛܖػʹɺ࿹Λ࠶ධՁ͢ΔΞϓϩʔν
    • ᶃ ؍ଌͨ͠ใुΛ࣌ܥྻσʔλͱΈͳ͠ɺ౎౓ɺมԽݕग़ΛࢼΈΔ
    • ᶄ มԽΛݕग़ͨ͠৔߹ɺมԽޙͷࢼߦճ਺ɺใुͰ࿹ͷධՁΛߋ৽
    • มԽݕग़͸طଘͷख๏Λར༻ՄೳʢPage-Hinkley test, ADWINʣ
    • UCB1΍Thomson SamplingͱมԽݕग़Λ૊Έ߹Θͤͨղ๏͕ఏҊ͞Ε͍ͯΔ
    • Adapt-EvE, S-TS-ADWIN
    19
    ඇఆৗͳଟ࿹όϯσΟοτ໰୊ͷղ๏ - มԽݕग़ܕ
    ✓
    Trialt Trialt 1
    · · · Trialt n
    Rewardt Rewardt 1
    · · · Rewardt n
    ◆
    AAADF3ichVE9b9RAEB2br8QEckCDFCk6cUo4ipzWASmIKhJNyuSSy4fi6LT2zV1W2Vtb672LDst/gJIGoVRBooj4AZQpkFBaCor8BEQZJCgoGPsMgUSEsex983be7FuPH0kRG8aOLfvS5StXr42MOtfHbtwcL926vRqHPR1gIwhlqNd9HqMUChtGGInrkUbe9SWu+TtPs/21PupYhGrFDCLc6vKOEm0RcENUs/TCk9g2VcfzsSNUwrXmgzSRFKmzogWXzcSk5enyLzzjZpkXtEIT/0mr1POcOu5y3SoUv5MzklNepY6HqlUc6nhadLbNA6dZqrAay6N8HrgFqEARi2HpHXjQghAC6EEXEBQYwhI4xPRsggsMIuK2ICFOExL5PkIKDml7VIVUwYndoW+Hss2CVZRnPeNcHdApkl5NyjJMsU/sgJ2wI/aWfWY//tkryXtkXga0+kMtRs3x53eXv/1X1aXVwPap6kLPBtrwOPcqyHuUM9ktgqG+/+zlyfKT+lQyzV6zL+R/nx2z93QD1f8avFnC+t4FfnzyQn+MBuSeHcd5sDpbcx/WZpceVeY3ilGNwATcgyrNYw7mYQEWoUH9v1uT1n2rar+yD+0P9tGw1LYKzR34K+yPPwFXgsYt
    มԽલ
    มԽޙ
    

    View Slide

20. 3.
    ఏҊख๏
    

    View Slide

21. • ଟ༷͔ͭܧଓతʹมԽ͢Δ؀ڥΛલఏͱ͍ͯ͠ͳ͍
    21
    ैདྷख๏ͷ՝୊
    ઢܗʢෳ਺ͷจ຺ʣ ඇఆৗ
    

    View Slide

22. • ଟ༷͔ͭܧଓతʹมԽ͢Δ؀ڥΛલఏͱ͍ͯ͠ͳ͍
    • → ఏҊख๏Ͱ͸ઢܗͳख๏ͱඇఆৗͳख๏Λ૊Έ߹ΘͤΔ͜ͱͰରԠ
    22
    ैདྷख๏ͷ՝୊
    

    View Slide

23. • ඇఆৗͳଟ࿹όϯσΟοτ໰୊ͷղ๏Λઢܗʹ֦ு
    • → ैདྷख๏Λ൚༻తʹར༻Ͱ͖ɺׂҾܕΞϓϩʔνͱൺֱͯ͠΋ྑ͍ධՁ
    ͕ಘΒΕ͍ͯΔɺมԽݕग़ܕΞϓϩʔνͰ͋ΔS-TS-ADWIN[2]ͷ֦ு
    23
    ఏҊख๏
    >&EPVBSE'PVDI
    F +VOQFJ,PNJZBNB BOE,MFNFOT#P
    „IN4DBMJOHNVMUJBSNFECBOEJUBMHPSJUINT*O1SPDFFEJOHTPGUIFUI"$.
    4*(,%%*OUFSOBUJPOBM$POGFSFODFPO,OPXMFEHF%JTDPWFSZ%BUB.JOJOH QQr
    

    View Slide

24. • ᶃ ؍ଌ͢΂͖࣌ܥྻͷࢦ਺తͳ૿Ճ
    • ैདྷख๏Ͱ͸୯Ұͷจ຺Ͱൃੜ͢ΔใुͷมԽΛܭଌ
    • ཁҼͷύϥϝʔλ਺ʹରͯ͠ࢦ਺తʹ૿Ճ͢Δจ຺͝ͱʹ࣌ܥྻΛܭଌ͢Δ
    ͜ͱʹΑΔਪનγεςϜͷϝϞϦ࢖༻ྔͷ૿Ճ
    • ᶄ ௥ैੑʹର͢Δվળͷ༨஍
    • ׆༻ͱ୳ࡧͷτϨʔυΦϑͷղফͷ࿮૊Έ͕ैདྷख๏ͷ··
    • → ࢼߦճ਺ʹରͯ͠Ұఆ
    24
    ैདྷख๏Λઢܗ͔ͭඇఆৗ΁֦ு͢Δࡍͷ՝୊
    

    View Slide

25. • ඇఆৗͳଟ࿹όϯσΟοτ໰୊ͷղ๏Λઢܗʹ֦ு
    • ैདྷख๏Λ൚༻తʹར༻Ͱ͖ɺׂҾܕΞϓϩʔνͱൺֱͯ͠΋ྑ͍ධՁ͕ಘ
    ΒΕ͍ͯΔɺมԽݕग़ܕΞϓϩʔνͰ͋ΔS-TS-ADWINͷ֦ு
    • ᶃͷରࡦ: จ຺ͷύλʔϯ͝ͱͰ͸ͳ͘ɺਪఆͨ͠ઢܗύϥϝʔλͷ࿨͔Β
    มԽݕग़ΛࢼΈΔ
    • ᶄͷରࡦ: ୳ࡧϋΠύʔύϥϝʔλͷಈతͳௐ੔Λಋೖ͠ɺੵۃతͳ׆༻ͱ
    ୳ࡧͷ੾Γସ͑ΛࢼΈΔ
    25
    ఏҊख๏
    

    View Slide

26. ઢܗύϥϝʔλͷਪఆͱ
    มԽݕग़
    

    View Slide

27. • ࢼߦ࣌ͷίϯςΩετ৘ใ ͱใु ͔Β࿹͝ͱͷઢܗύϥϝʔλΛਪఆ
    • ίϯςΩετ৘ใͷཁҼ਺ʢ=ઢܗύϥϝʔλͷཁҼ਺ʣ͸ ࣍ݩͱ͢Δ
    b r
    d
    27
    ઢܗύϥϝʔλͷਪఆ
    Bi = Bi + bbT
    AAACpnichVFNLwNBGH66vqq+iovEBQ2RSJopEiKRCBcn0VIlSu2uKZPuV3anTdj0LPEHHJxIHERc+QEu/oCDnyCOJC4O3m43EQTvZmaeeeZ93n1mXs0xhCcZe4woDY1NzS3R1lhbe0dnV7y7Z82zy67Os7pt2O66pnrcEBbPSiENvu64XDU1g+e00kLtPFfhridsa1UeOHzLVPcsURS6KokqxAfnC2KWxlheM32tWp+3/bypyn2v6K9Wq4V4giVZEAM/QSoECYSxbMdvkccubOgowwSHBUnYgAqPvk2kwOAQtwWfOJeQCM45qoiRtkxZnDJUYks079FuM2Qt2tdqeoFap78YNFxSDmCYPbBL9sLu2RV7Yu+/1vKDGjUvB7RqdS13Cl3HfStv/6pMWiX2P1V/epYoYjrwKsi7EzC1W+h1feXw5GVlJjPsj7Bz9kz+z9gju6MbWJVX/SLNM6d/+NHIC70YNSj1vR0/wdp4MjWRHE9PJuY2wlZF0Y8hjFI/pjCHRSwjS/WPcI0b3CqjypKSVXL1VCUSanrxJZSdD5HLngo=
    Bi =
    0
    B
    B
    B
    @
    b11 b12
    · · · b1d
    b21 b22
    · · · b2d
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    bd1 bd2
    · · · bdd
    1
    C
    C
    C
    A
    AAADRXichVHNStxQGP0S+2PTWke7KXQzODjoZriJQqVQkLpx4cK/GS1eCcnNnZmLmSTc3BmYhrxAX6CLrhS6KD5At4KL9gW68BFKcaXYjWC/ZCKlDuoXknty7jlfzs3nRr6IFSEnmj7y4OGjx6NPjKfPxp6PlyYmG3HYlYzXWeiHctt1Yu6LgNeVUD7fjiR3Oq7Pt9y9pWx/q8dlLMJgU/UjvttxWoFoCuYopOzS0TtbvKU+b6oZg7q8JYLEkdLppwnDSg3XTkwzLVfLGbAyQJkXqrhgvJTSTGNda6whjZVraK9gbgODPt51H2+oj+elBuWBV+QzqBSttpq1SxVSI3mVh4FZgAoUtRqWvgEFD0Jg0IUOcAhAIfbBgRivHTCBQITcLiTISUQi3+eQgoHeLqo4Khxk9/DZwredgg3wPesZ526GX/HxlugswzT5Sb6SM/KDHJJf5PLWXkneI8vSx9UdeHlkj398ufHnXlcHVwXtf647MytowkKeVWD2KGeyU7CBv/fh09nGm/XppEoOyG/Mv09OyDGeIOidsy9rfP3zHXlczIJ/DAdk3hzHMGhYNXOuZq3NVxbfF6MahVcwBTM4j9ewCMuwCnVgWlVb0epaQ/+un+rn+sVAqmuF5wX8V/rVX3n50vg=
    fi =
    0
    B
    B
    B
    @
    br1
    br2
    .
    .
    .
    brd
    1
    C
    C
    C
    A
    AAAC2XichVExT9tAGH1xaaGmlLQsSCyoERUs0TkgtapUCYmlI0kIUMXIsp1LOOHY1vkSKbU8sCHWDh06UYkB8QMqdWXhDzCwdaUdQWJh6GfHUgWo9LPO9+7d9757d58TeiJSjJ0XtEcjj5+Mjj3Vx59NPJ8svni5HgU96fKGG3iB3HTsiHvC5w0llMc3Q8ntruPxDWdnJd3f6HMZicBfU4OQb3Xtji/awrUVUVZxrW2J96bH22peNx3eEX5sS2kPktilSHRHWrGRmGYGKikw+61ARTnTSnST+61co5tSdLbVglUssTLLYvY+MHJQQh6rQfE7TLQQwEUPXXD4UIQ92Ijoa8IAQ0jcFmLiJCGR7XMk0EnboyxOGTaxO/Tv0KqZsz6t05pRpnbpFI+GJOUs5tgZO2KX7JQdswt2889acVYj9TKg2RlqeWhN7k/Xr/+r6tKssP1X9aBnhTbeZl4FeQ8zJr2FO9T3P325rL+rzcWv2Tf2m/wfsHN2Qjfw+1fuYZXXvj7gxyEv9GLUIONuO+6D9UrZWCxXqkul5Y95q8Ywg1eYp368wTI+YBUNqv8DP3GBX1pT29X2tP1hqlbINVO4FdrnP/INsiY=
    fi = fi + br
    AAAClHichVHLSsNAFL2Nr1ofjQoiuCmWiiCUSRUUUSgUwZX0YR/SSknitA7NiyQt1NIf8AdcdKXYhfgBfoAbf8BFP0FcVnDjwps0IFqsd5jMnTP33DmTIxkKs2xCej5ubHxicso/HZiZnZsP8guLOUuvmzLNyrqimwVJtKjCNJq1ma3QgmFSUZUUmpdqCec836CmxXTtxG4a9EwVqxqrMFm0ESrzwUqZHeDcLElqS2qbZT5MosSN0HAieEkYvEjq/COU4Bx0kKEOKlDQwMZcAREsHEUQgICB2Bm0EDMxY+45hTYEkFvHKooVIqI1/FZxV/RQDfdOT8tly3iLgtNEZggi5IXckz55Jg/klXz+2avl9nC0NHGVBlxqlINXK5mPf1kqrjZcfLNGarahAruuVobaDRdxXiEP+I3L635mLx1prZNb8ob6b0iPPOELtMa73E3RdGeEHgm14B9Dg4TfdgwnuVhU2IrGUtvh+KlnlR9WYQ020I8diMMRJCHretaBO+hyy9w+l+AOB6Wcz+MswY/gjr8AyUGWCg==
    ✓i =
    0
    B
    B
    B
    @
    ✓1
    ✓2
    .
    .
    .
    ✓d
    1
    C
    C
    C
    A
    AAAC8XichVHNShxBEK4Z498YdWMuQi6SxaCXpWcTMAiCkItHf7JqcGToma3dbez5oad3YR3mBXwBkVyikEPwAfIAXjwHcvARxKMBPXhI7eyQmEhMDTP91df11Xzd5cVSJJqxC8MceDI4NDwyao09HZ+YLD2b2kyitvKx5kcyUtseT1CKEGtaaInbsUIeeBK3vL13vf2tDqpEROF73Y1xN+DNUDSEzzVRbkk6XpA6uoWaZ65YciQ29JzleNgUYcqV4t0s9Skyq1/kpnbmOL+Sap506pFO7rF1qsawXugtR4lmS89bllsqswrLY+YhsAtQhiJWo9JXcKAOEfjQhgAQQtCEJXBI6NkBGxjExO1CSpwiJPJ9hAws0rapCqmCE7tH3yZlOwUbUt7rmeRqn/4i6VWknIFZ9p19YdfsnJ2yS3b3z15p3qPnpUur19di7E4eTG/c/FcV0Kqh9Vv1qGcNDXibexXkPc6Z3in8vr6zf3i9sbg+m75iJ+yK/B+zC3ZGJwg7P/zPa7j+8RE/HnmhG6MB2X+P4yHYrFbs15Xq2pvy8odiVCPwAl7CHM1jAZZhBVahRv2/wa1hGKaZmEfmJ/OkX2oaheY5/BHm6U8qrLo1
    ✓i = B 1
    i
    fi
    AAACnnichVHLLgRBFD3a+z3YEJuJCbExqUZCJBIhERvxHEYMne5WQ0W/0l0zCZ2JvR+wsCKxEAlLH2DjByx8gliS2Fi409OJILid6jp16p5bp+oaniUCydhjlVJdU1tX39DY1NzS2tae6OhcC9yCb/KM6VqunzX0gFvC4RkppMWzns9127D4urE/U95fL3I/EK6zKg88vmXru47IC1OXRGmJnpxhhzm5x6Ve0sTktCa2wyG1lNeElkixNIsi+ROoMUghjkU3cYscduDCRAE2OBxIwhZ0BPRtQgWDR9wWQuJ8QiLa5yihibQFyuKUoRO7T/9dWm3GrEPrcs0gUpt0ikXDJ2US/eyBXbIXds+u2BN7/7VWGNUoezmg2ahouae1H3evvP2rsmmW2PtU/elZIo/xyKsg717ElG9hVvTFw5OXlYnl/nCAnbNn8n/GHtkd3cApvpoXS3z59A8/BnmhF6MGqd/b8ROsDafVkfTw0mhqaiNuVQN60YdB6scYpjCHRWSo/hEucI0bJanMKvPKQiVVqYo1XfgSSvYDonKaOQ==
    ର֯੒෼͸ͦͷཁҼͷࢼߦճ਺ɻ
    ͦͷଞͷ஋͸ɺର֯੒෼ͱͦͷཁҼͷ
    ૊Έ߹ΘͤʹΑΔࢼߦճ਺Λද͢ɻ
    ͦͷཁҼͷྦྷੵใुΛද͢ ͷٯߦྻͱ ͷੵΑΓཁҼ͝ͱͷઢ
    ܗύϥϝʔλΛਪఆ
    ʢཁҼ͕ഉଞͰ͋Ε͹ฏۉใुֹʣ
    B f
    

    View Slide

28. • ਪఆͨ͠ઢܗύϥϝʔλͷ஋ͷ࿨Λ༻͍ͯඇఆৗͳมԽΛݕग़͢Δ
    • → ཁҼ਺ʹΑΒͳ͍୯Ұͷ஋ͷΈ͔ΒมԽݕग़ΛࢼΈΔʢ ʣ
    • ADWINʹΑͬͯมԽΛݕग़ͨ͠৔߹ɺมԽલͷࢼߦճ਺ɺྦྷੵใुΛআڈ
    2d → d → 1
    28
    ઢܗύϥϝʔλͷมԽݕग़
    ✓i = B 1
    i
    fi
    AAACnnichVHLLgRBFD3a+z3YEJuJCbExqUZCJBIhERvxHEYMne5WQ0W/0l0zCZ2JvR+wsCKxEAlLH2DjByx8gliS2Fi409OJILid6jp16p5bp+oaniUCydhjlVJdU1tX39DY1NzS2tae6OhcC9yCb/KM6VqunzX0gFvC4RkppMWzns9127D4urE/U95fL3I/EK6zKg88vmXru47IC1OXRGmJnpxhhzm5x6Ve0sTktCa2wyG1lNeElkixNIsi+ROoMUghjkU3cYscduDCRAE2OBxIwhZ0BPRtQgWDR9wWQuJ8QiLa5yihibQFyuKUoRO7T/9dWm3GrEPrcs0gUpt0ikXDJ2US/eyBXbIXds+u2BN7/7VWGNUoezmg2ahouae1H3evvP2rsmmW2PtU/elZIo/xyKsg717ElG9hVvTFw5OXlYnl/nCAnbNn8n/GHtkd3cApvpoXS3z59A8/BnmhF6MGqd/b8ROsDafVkfTw0mhqaiNuVQN60YdB6scYpjCHRWSo/hEucI0bJanMKvPKQiVVqYo1XfgSSvYDonKaOQ==
    M =
    d
    X
    (B 1
    i
    fi)d
    AAACnXichVHLLgRBFD3T3uM12EgsTEwIC5NqJEQiESKxQGYwhnh0untqqOhXunsmYcIH+AELK8JChK0PsPEDFj5BLElsLNzp6UQQ3E51nTp1z61TdTXHEJ7P2GNEqqquqa2rb4g2NjW3tMba2lc8u+DqPKPbhu2uaqrHDWHxjC98g686LldNzeBZbXe6vJ8tctcTtrXs7zl801S3LZEXuuoTpcQ65yc2vIK5lYv3TyliqzQoH+QVMaDklFiCJVkQ8Z9ADkECYaTs2C02kIMNHQWY4LDgEzagwqNvHTIYHOI2USLOJSSCfY4DRElboCxOGSqxu/TfptV6yFq0Ltf0ArVOpxg0XFLG0cse2CV7Yffsij2x919rlYIaZS97NGsVLXeU1qPOpbd/VSbNPnY+VX969pHHWOBVkHcnYMq30Cv64v7xy9L4Ym+pj52xZ/J/yh7ZHd3AKr7qF2m+ePKHH4280ItRg+Tv7fgJVoaS8nByKD2SmFwLW1WPLvSgn/oxiknMIoUM1T/EOa5xI3VLM9KctFBJlSKhpgNfQsp+AGSOmMY=
    ᶄઢܗύϥϝʔλͷ૯࿨ ᶅ࣌ܥྻͱͯ͠஋Λอଘ
    ᶆ஋ ͷมԽΛݕग़
    M
    ᶇมԽલͷ஋Λআڈ
    ᶃઢܗύϥϝʔλͷਪఆ
    A = {Mt 100, Mt 99, · · · , Mt 1}
    AAACrnicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQtoOirYKsRU+8ZVl+gaGhjU6oBZlpa1OjHJKfklxRC+YW1MbbyAsoGeARgoYDIMoQxlBigIyBfYzhDDkMKQz5DMUMqQy5DKkMdQAmTnMCQyFANhNIMhgwFDAVAslqEaKFYEZGWC5VMZahm4gHpLgapSgSoSgaLZQDIdyIuGiuYB+SAzi8G6k4G25ABxEVCnAoOqwVWDlQafDU4YrDZ4afAHp1nVYDNAbqkE0kkQvakF8fxdEsHfCerKBdIlDBkIXXjdXMKQxmABdmsm0O0FYBGQL5Ih+suqpn8OtgpSrVYzWGTwGuj+hQY3DQ4DfZBX9iV5aWBq0Gw87kkCugUYYsAIMkSPDkxGmJGeobGeUaCJskMkNKo4GKQZlBg0gPFhzuDA4MEQwBAKNL+fYSvDPob9TAZMYUyxTPEQpUyMUD3CDCiAKQMAzZWevg==
    

    View Slide

29. ୳ࡧϋΠύʔύϥϝʔλͷ
    ಈతͳௐ੔
    

    View Slide

30. • มԽݕग़࣌ɺ࿹ͷࢼߦճ਺͕Ұ౓ݮগ͢Δ͜ͱʹண໨
    • → ࿹ͷࢼߦճ਺ʹ൓ͯ͠ɺ୳ࡧϋΠύʔύϥϝʔλΛ૿ݮ
    30
    ୳ࡧϋΠύʔύϥϝʔλͷಈతͳௐ੔
    τ =
    ∣A∣
    ∑
    i
    β
    ln ∣ Ai
    ∣ + 1
    ֤࿹ͷ΢Οϯυ΢αΠζࢼߦճ਺
    ͔ΒٻΊͨ஋ͷ૯࿨Λ༻͍Δ
    

    View Slide

31. ैདྷͷઢܗͳղ๏΁ͷద༻
    

    View Slide

32. • ઢܗͳଟ࿹όϯσΟοτ໰୊ͷղ๏Ͱ͋ΔLinear Thompson Samplingͱ
    LinUCBʹద༻
    • ैདྷͷ࿹બఆͷͨΊͷΞϧΰϦζϜΛมߋ͢Δ͜ͱͳ͘ɺਪఆͨ͠ઢܗύϥ
    ϝʔλͳΒͼʹ୳ࡧϋΠύʔύϥϝʔλΛར༻Մೳ
    32
    ैདྷͷઢܗͳղ๏΁ͷద༻
    ਪఆͨ͠ઢܗύϥϝʔλ
    ಈతͳ୳ࡧϋΠύʔύϥϝʔλ
    

    View Slide

33. 4.
    ධՁ
    

    View Slide

34. • ඇఆৗͰઢܗͳଟ࿹όϯσΟοτ໰୊ʹର͢ΔੑೳΛධՁ͢ΔγϛϡϨʔγϣ
    ϯΛߦ͏ɻ
    • ઢܗͳ؀ڥͱͯ͠ɺཁҼ਺8͔ΒͳΔ256௨Γͷจ຺ʢ͝ͱͷใु෼෍ʣΛ࣋
    ͭ࿹0ͱ࿹1Λ༻ҙ͠ɺ2000࣌఺ܦաޙͷྦྷੵใुͳΒͼʹྦྷੵϦάϨοτΛ
    ֤ղ๏Ͱൺֱɻ
    • ඇఆৗͳ؀ڥͱͯ͠ɺ࿹1ͷΈ500࣌఺Ͱใु෼෍ΛมԽͤ͞Δɻ
    • ൺֱ͢Δղ๏͸ɺThompson SamplingɺUCB1ͷΞϧΰϦζϜʹରͯͦ͠Ε
    ͧΕɺઢܗɺඇఆৗɺઢܗ͔ͭඇఆৗͷ֦ுΛࢪͨ͠΋ͷΛ༻͍Δɻ
    34
    ࣮ݧઃఆ
    

    View Slide

35. ࢼߦճ਺ͱྦྷੵϦάϨοτ
    35
    • มԽݕग़Λ༻͍ͳ͍ख๏Ͱ͸ྦྷੵϦ
    άϨοτͷ૿Ճ͕ऩଋ͍ͯ͠ͳ͍
    • → աڈσʔλͷӨڹ͕ݪҼ
    • มԽݕग़Λ༻͍ͯ΋จ຺͝ͱʹ؍ଌ
    ͢Δ৔߹͸ऩଋ·Ͱ͕͔͔࣌ؒΔ
    • → ਪఆͷͨΊͷσʔλ෼ࢄ͕ݪҼ
    • ఏҊख๏Λద༻ʹΑΓଟ࿹όϯσΟο
    τ໰୊ͷղ๏ͷ௥ैੑೳ͕޲্
    

    View Slide

36. ୳ࡧϋΠύʔύϥϝʔλͷਪҠ
    36
    • ఏҊख๏Ͱ͸ɺਅͷઢܗύϥϝʔλ
    ʹมԽ͕͋ͬͨ࣌఺Ҏ߱ͰҰ࣌తʹ
    ୳ࡧϋΠύʔύϥϝʔλͷ஋্͕ঢ
    • ఏҊख๏Ͱ͸ɺมԽ࣌ͷΈҰ࣌తʹ
    ্ঢͤ͞Δ͜ͱ͕Ͱ͖ΔͨΊɺมԽ
    ͷͳ͍৔߹͸୳ࡧϋΠύʔύϥϝʔλ
    ͷ஋Λ௿͘อͭ͜ͱ͕Մೳ
    • → ػձଛࣦͷ௿ݮ
    

    View Slide

37. ύϥϝʔλͷਪఆ஋ͱ΢Οϯυ΢αΠζͷਪҠ(࿹0)
    37
    • ࿹0ʹ͓͍ͯɺઢܗͳଟ࿹όϯσΟο
    τ໰୊ͷղ๏Β͸ɺ࠷ऴతʹઢܗύ
    ϥϝʔλͷ஋Λਖ਼͘͠ਪఆ͢Δ͜ͱ
    ͕Ͱ͖ͨ
    • ਅͷઢܗύϥϝʔλʹมԽ͕ͳ͍؀
    ڥͰ͕͋ͬͨɺఏҊख๏Ͱ͸΢Οϯ
    υ΢αΠζͷมԽ͕ݟΒΕͨɻ
    • → ϊΠζΛมԽͱͯ͠ݕग़ͯ͠͠
    ·ͬͨͨΊɺࠓޙͷରࡦ͕ඞཁ
    

    View Slide

38. ύϥϝʔλͷਪఆ஋ͱ΢Οϯυ΢αΠζͷਪҠ(࿹1)
    38
    • ࿹1ʹ͓͍ͯɺैདྷͷઢܗͳଟ࿹όϯ
    σΟοτ໰୊ͷղ๏͸ɺաڈσʔλ
    ͷӨڹʹΑΓਅͷ஋ʹରͯ͠ߴ͍ݟ
    ੵ΋Γ͕ଓ͍ͨ
    • ఏҊख๏Ͱ͸ɺ΢Οϯυ΢αΠζͷ
    มԽʹΑΓաڈσʔλΛऔΓআ͍ͨ
    ݁Ռɺਝ଎ʹ௥ै͢Δ͜ͱ͕Ͱ͖ͨ
    

    View Slide

39. ύϥϝʔλͷਪఆ஋ͱ΢Οϯυ΢αΠζͷਪҠ(࿹1)
    39
    • ҰํͰɺఏҊख๏Ͱ͸มԽޙͷਪఆ
    ஋͕ਅͷ஋ʹରͯ͠Լճͬͯ͠·ͬͯ
    ͍Δ
    • ݪҼᶃ: ࿹1͕ඞͣྼΔ࣮ݧઃఆ
    • → ࿹ͷબ୒ճ਺͕গͳ͍
    • ݪҼᶄ: ΢Οϯυ΢αΠζ͕গͳ͍
    • → มԽݕग़ʹར༻͢Δܥྻࣗମ
    Λॖখ͢ΔͨΊݕग़ͷ҆ఆੑ͕
    ௿Լ͍ͯ͠ΔՄೳੑ
    

    View Slide

40. 5.
    ·ͱΊ
    

    View Slide

41. • ਪનγεςϜΛ͸͡Ίͱ͢Δɺଟ༷͔ͭܧଓతʹมԽ͢Δ؀ڥΛ૝ఆͨ͠ଟ࿹
    όϯσΟοτ໰୊ͷղ๏ΛఏҊͨ͠
    • ैདྷͷઢܗɺඇఆৗͳղ๏Λ୯७ʹ૊Έ߹Θͤͨࡍʹൃੜ͢ΔҎԼͷ՝୊Λղ
    ܾ͢Δ൚༻తͳ֦ுͱͯ͠ఏҊ
    • ؍ଌ͢΂͖࣌ܥྻͷࢦ਺తͳ૿Ճ → ਪఆͨ͠ઢܗύϥϝʔλͷ࿨Λ༻͍Δ
    • ௥ैੑ → ಈతͳ୳ࡧϋΠύʔύϥϝʔλΛ༻͍Δ
    • ධՁͰ͸ɺඇఆৗ͔ͭઢܗͳ؀ڥʹ͓͍ͯैདྷͷղ๏ʹରͯ͠ྦྷੵใुɺྦྷੵ
    ϦάϨοτʹ͓͍ͯੑೳͷ޲্Λ֬ೝͨ͠
    41
    ·ͱΊ
    

    View Slide

42. • ਪఆͨ͠ઢܗύϥϝʔλͷ”࿨”͕ଟ࣍ݩͷ࣌ܥྻσʔλͷมԽݕग़ʹ༗ޮͰ͋
    Δཧ࿦తͳཪ෇͚͸·ͩߦ͍͑ͯͳ͍
    • → ଟมྔ࣌ܥྻʹؔ͢ΔมԽݕग़ͷߋͳΔௐ͕ࠪඞཁ
    • ADWINʹه࿥͢Δ஋ΛADWINͷ΢Οϯυ΢αΠζʹґଘͨ͠஋͔ΒٻΊͯ
    ͓ΓɺมԽݕग़ʹؔ͢Δ҆ఆੑͷ௿Լʹͭͳ͕͍ͬͯΔՄೳੑ
    • → ํࣜͷվળΛݕ౼
    42
    ٞ࿦
    

    View Slide

43. View Slide