Share
Embed
多様かつ継続的に変化する環境に適応する情報システム/thesis-defense-presentation
九州大学 大学院システム情報科学府 情報知能工学専攻
2024.08.22 学位論文公聴会
monochromegane
August 22, 2024
Transcript
152 slides
-
• ଟ༷͔ͭܧଓతʹมԽ͢ΔڥͷதͰɺใγεςϜ͕ܧଓతʹػೳ͢Δʹ ɺͦͷߏϩδοΫΛߋ৽͠มԽʹै͢Δඞཁ͕͋Δ • → ྫʣใγεςϜͷෛՙঢ়گɺར༻ऀͷߦಈͷมԽ • ͜Ε·Ͱɺ͜ͷैɺӡ༻ҡ࣋ۀͱͯ͠ӡ༻ऀ͕୲͖ͬͯͨ  7
ใγεςϜͱڥมԽ • ਓखʹΑΔڥͷมԽݕใγεςϜͷߋ৽ɺैͷ࣌ؒࠩΛ͏ • ݁Ռͱͯ͠ɺ҆ఆੑར༻ऀͷຬͷԼɺӡ༻ऀͷෛ୲ͷ૿ՃΛট͘ -
• ैདྷͷӡ༻ҡ࣋ͷऔΓΈͰɺใγεςϜΛڥͷมԽʹରԠͤ͞Δͨ ΊɺਓʹΑΔܦݧଇஅͱ෦తͳࣗಈԽʹཹ·͍ͬͯΔ • → ྫʣܦݧଇʹΑΔᮢઃఆɺԽͨ͠ར༻ऀͷߦಈୡʹΑΔஅ  8 ڥมԽʹࣗΒదԠ͢ΔใγεςϜʹ͚ͯ •
ਓʹΑΔஅߋ৽ͷఔΛࣗಈԽ͠ɺใγεςϜࣗମ͕ڥมԽΛଊ͑ม Խʹै͢ΔదԠతͳΈͷݚڀ • ͳΒͼʹ࣮ӡ༻ͷద༻ • ݚڀίϯηϓτ -
• ैདྷͷγεςϜ։ൃͰɺར༻ऀ͔Βͷೖྗ  ʹରͯ͠ग़ྗ  Λܾఆ͢Δؔ  Λઃܭ͍ͯ͠Δ x y
f  10 దԠతใγεςϜͷ࣮ݱʹ͚ͨΞϓϩʔν • ਓखʹΑΔڥͷมԽݕใγεςϜͷߋ৽ɺैͷ࣌ؒࠩΛ͏ • ݁Ռͱͯ͠ɺ҆ఆੑར༻ऀͷຬͷԼɺӡ༻ऀͷෛ୲ͷ૿ՃΛট͘ y = f(x) IUUQTJDPOTDPN -
• ػցֶशʮؔͷઃܭʯΛࣗಈԽ͢Δʢσʔλ͔ΒύϥϝʔλΛٻΊΔʣ  11 దԠతใγεςϜͷ࣮ݱʹ͚ͨΞϓϩʔν ᶃ ೖग़ྗͷରԠؔΛఆٛ ᶄ ༧ଌʹର͢ΔζϨΛఆٛ ᶅ
ֶशσʔλʹର͢Δ ༧ଌͷζϨΛ࠷খԽ͢Δ ᶆ ύϥϝʔλ͕ܾ·Δ ʢσʔλ͔Βؔͷઃܭ͕Ͱ͖ͨʣ y = f(x) IUUQTJDPOTDPN -
• ઃܭ͞Εͨؔɺ͋Δ࣌·Ͱͷσʔλʹجͮ͘ৼΔ͍Λߦ͏ʢૠੑʣ • αʔϏεΛऔΓר͘ڥܧଓతʹมԽ͍ͯ͠Δͱߟ͑ΒΕΔͨΊɺؔͷܧ ଓతͳ࠶ઃܭ͕ඞཁʢ֎ૠੑͷߟྀʣ  12 దԠతใγεςϜͷ࣮ݱʹ͚ͨΞϓϩʔν • ͋Δ࣌Ͱ༗ޮͳؔΛαʔϏεར༻ऀͷڥͷมԽʹԠͯ͡ܧଓతʹ࠶ઃ
ܭʢ͋Δ͍ผؔͷஔ͖͑ʣΛߦ͏ܥશମͷΈ IUUQTJDPOTDPN -
• ใγεςϜʹ͓͚ΔใաଟΛղܾ͢ΔɺਪનγεςϜͷಋೖ • ͳΜΒ͔ͷػցֶशϞσϧʢ=ਪનख๏ʣʹج͖ͮଟͷબࢶ͔Βར༻ऀ ͕ڵຯΛ࣋ͭͷΛఏҊ͢ΔγεςϜ • ଟ͘ͷਪનख๏͕ఏҊ͞Ε͍ͯΔ  15 6TFST
3FDPNNFOEBUJPO .FUIPET 1SPEVDUT IUUQTJDPOTDPN ۩ମྫʹݟΔదԠతใγεςϜͷ࣮ݱʹ͚ͨ՝ -
• ใγεςϜʹ͓͚ΔใաଟΛղܾ͢ΔɺਪનγεςϜͷಋೖ • ͳΜΒ͔ͷػցֶशϞσϧʢ=ਪનख๏ʣʹج͖ͮଟͷબࢶ͔Βར༻ऀ ͕ڵຯΛ࣋ͭͷΛఏҊ͢ΔγεςϜ • ଟ͘ͷਪનख๏͕ఏҊ͞Ε͍ͯΔ → ޮՌతͳʮਪનख๏ͷબఆʯ͕ॏཁ 
16 6TFST #FTU3FDPNNFOEBUJPO .FUIPE 1SPEVDUT IUUQTJDPOTDPN ۩ମྫʹݟΔదԠతใγεςϜͷ࣮ݱʹ͚ͨ՝ -
• ใγεςϜʹ͓͚ΔใաଟΛղܾ͢ΔɺਪનγεςϜͷಋೖ • ͳΜΒ͔ͷػցֶशϞσϧʢ=ਪનख๏ʣʹج͖ͮଟͷબࢶ͔Βར༻ऀ ͕ڵຯΛ࣋ͭͷΛఏҊ͢ΔγεςϜ • ଟ͘ͷਪનख๏͕ఏҊ͞Ε͍ͯΔ → ޮՌతͳʮਪનख๏ͷબఆʯ͕ॏཁ 
17 • ޮՌతͳਪનख๏ঢ়گʹΑͬͯҟͳΔ • ͔͠͠ͳ͕Βɺ࣮ڥͰͷܧଓతͳਪનख๏ͷධՁʹػձଛࣦ͕͏ ӡ༻্ͷ՝ ۩ମྫʹݟΔదԠతใγεςϜͷ࣮ݱʹ͚ͨ՝ -
• ʮʯͱݺΕΔෳͷީิ͔ΒಘΒΕΔใुΛ࠷େԽ͢Δ • ϓϨΠϠʔҰͷࢼߦͰ1ͭͷΛબ͠ɺใुΛಘΔ • ͦΕͧΕͷ͋Δใुʹै͍ใुΛੜ • ͨͩ͠ɺϓϨΠϠʔ͜ͷใुΛࢼߦͷ݁Ռ͔Βਪଌ͢Δඞཁ͕͋Δ  20
ଟόϯσΟοτ ʜ બ ใु ਪଌ ʮʯεϩοτϚγʔϯͷʮΞʔϜʢArmʣʯʹ༝དྷ IUUQTJDPOTDPN -
• ʮʯͱݺΕΔෳͷީิ͔ΒಘΒΕΔใुΛ࠷େԽ͢Δ • ϓϨΠϠʔҰͷࢼߦͰ1ͭͷΛબ͠ɺใुΛಘΔ • ͦΕͧΕͷ͋Δใुʹै͍ใुΛੜ • ͨͩ͠ɺϓϨΠϠʔ͜ͷใुΛࢼߦͷ݁Ռ͔Βਪଌ͢Δඞཁ͕͋Δ  21
ଟόϯσΟοτ • ϓϨΠϠʔ͋Δ࣌ͷͷධՁʹج͖ͮʮ׆༻ʯͱʮ୳ࡧʯΛฒߦͯ͠ߦ͏ • ͜ͷτϨʔυΦϑΛղফ͢ΔͨΊʹ༷ʑͳํࡦ͕ఏҊ͞Ε͍ͯΔ -
• ࠷୯७ͳํࡦͰ͋Δ  -Greedyɺൺ  ͰۉʹΛબʢ୳ࡧʣ͠ɺൺ  Ͱͦͷ࣌ͷฏۉใु͕࠷େ͖͍Λબʢ׆༻ʣ͢Δ ϵ ϵ
1 − ϵ  22 ଟόϯσΟοτํࡦ argmaxl=1,L ̂ y(l),1 − ϵ ∀a ∈ A, ϵ/L Bandit A/B testing ∀a ∈ A, ϵ/L -
ଟόϯσΟοτํࡦΛ༻͍ͨదԠతใγεςϜ  23 User(s) System Exploitation and Exploration using Multi-armed
bandits Feedback Estimation Truth t-1 t-1 t-1 • ଟόϯσΟοτํࡦʹΑΔػցֶशϞσϧͷબఆʹ͓͚Δػձଛࣦͷݮ • → దԠతใγεςϜͷ࣮ӡ༻ͷద༻ͷোนΛऔΓআ͘ IUUQTJDPOTDPN -
ଟόϯσΟοτํࡦΛ༻͍ͨదԠతใγεςϜ  24 User(s) System Exploitation and Exploration using Multi-armed
bandits Feedback Estimation Truth t-1 t-1 t-1 • ଟόϯσΟοτํࡦʹΑΔػցֶशϞσϧͷબఆʹ͓͚Δػձଛࣦͷݮ • → దԠతใγεςϜͷ࣮ӡ༻ͷద༻ͷোนΛऔΓআ͘ ػցֶशϞσϧͷಛੑΛ౿·͑ͨ ࣮༻తͳํࡦ IUUQTJDPOTDPN -
 25 ಛੑ(1): จ຺࣌ؒͷܦաʹΑΔ༗༻ੑͷมಈ શମ ΞΫηαϦΧςΰϦ ϑʔυΧςΰϦ • ػցֶशϞσϧͷ༗༻ੑɺจ຺࣌ؒͷܦաʹΑͬͯมಈ͢Δ •
ਪનख๏ͷ༗༻ੑ͕࣌ؒͷܦաʹ͍มԽ • ਪનख๏Λద༻͢ΔΧςΰϦ͝ͱʹ༗༻ੑ͕ҟͳΔ -
• ػցֶशϞσϧͷ࣮ߦ࣌ؒʹɺํࡦͷ࣮ߦ͕࣌ؒՃ͞ΕΔ • ػցֶशϞσϧͷԠ࣌ؒར༻ऀͷମݧʹతͳӨڹΛٴ΅͢ • ํࡦͷֶश͕͚࣌ؒΕར༻ऀͷᅂͷมԽʹैͰ͖ͳ͍  26 ಛੑ(2): దԠʹ͏͕࣌ؒٴ΅͢༗༻ੑͷӨڹ
up to100ms Request Response MAB Policy ML Model t=1 t=10 t=100 t=1000 x -
• ใγεςϜͰѻ͏σʔλͷछྨͱྔ͕૿Ճ • ਓޱ౷ܭֶతଐੑͷߏԽσʔλ͔Βɺը૾ɾࣗવݴޠͷඇߏԽσʔλ • ઢܗͳํࡦͰɺඇઢܗͷΑ͏ͳෳࡶͳؔੑΛॆʹѻ͏͜ͱ͕Ͱ͖ͳ͍  27 ಛੑ(3): ༗༻ੑͷਪఆʹ͓͚Δෳࡶͳؔੑ
• ίϯςΩετใ  ʹର͢ Δใुͷྫɻ • ࠨઢܗ  ɺӈඇઢܗΛԾ ఆ͢Δํࡦ͕ద͢Δͱߟ͑ΒΕΔ x = (x1 , x2 )⊤ ( ̂ y = x⊤w) -
ຊݚڀͷ၆ᛌਤ  29 "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU Context Non stationarity Context Non stationarity
Online performance Context (Non-linear) Non stationarity Online performance Multi-armed bandit polices 3ষ 4ষ 5ষ 6ষ -
 30 "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU Context Non stationarity Context Non stationarity Online
performance Context (Non-linear) Non stationarity Online performance Multi-armed bandit polices 3ষ 4ষ 5ষ 6ষ [48] ࡾ ༔հ, ็ ߃ݑ, Synapse: จ຺ʹԠͯ͡ܧଓతʹਪનख๏ͷબΛ࠷దԽ͢Δਪનγε ςϜ, ిࢠใ௨৴ֶձจࢽD, Vol.J103-D, No.11, pp.764-775, Nov 2020. -
 31 "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU Context Non stationarity Context Non stationarity Online
performance Context (Non-linear) Non stationarity Online performance Multi-armed bandit polices 3ষ 4ষ 5ষ 6ষ [49] ࡾ ༔հ, ็ ߃ݑ, Synapse: จ຺ͱ࣌ؒܦաʹԠͯ͡ਪનख๏ͷબΛ࠷దԽ͢Δϝλਪનγε ςϜ, ిࢠใ௨৴ֶձจࢽD, Vol.J105-D, No.11, pp.641-652, Nov. 2022. -
 32 "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU Context Non stationarity Context Non stationarity Online
performance Context (Non-linear) Non stationarity Online performance Multi-armed bandit polices 3ষ 4ষ 5ষ 6ষ "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU [50] Yusuke Miyake, Tsunenori Mine, Contextual and Nonstationary Multi-armed Bandits Using the Linear Gaussian State Space Model for the Meta-Recommender System, 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp.3138-3145, Oct 2023. -
 33 "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU Context Non stationarity Context Non stationarity Online
performance Context (Non-linear) Non stationarity Online performance Multi-armed bandit polices 3ষ 4ষ 5ষ 6ষ "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU [51] Yusuke Miyake, Ryuji Watanabe, Tsunenori Mine, Online Nonstationary and Nonlinear Bandits with Recursive Weighted Gaussian Process, The 48th IEEE International Conference on Computers, Software, and Applications (COMPSAC 2024), pp.11-20, Jul 2024. -
ຊݚڀͷ၆ᛌਤ  35 "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU Context Non stationarity Context Non stationarity
Online performance Context (Non-linear) Non stationarity Online performance Multi-armed bandit polices 3ষ 4ষ 5ষ 6ষ -
 38 AdaptEx [30] [W. Black 2023] • ଟόϯσΟοτํࡦΛ༻͍ͨར༻ऀମݧͷݸผԽϓϥοτϑΥʔϜ •
ΦϯϥΠϯཱྀߦαΠτʢExpediaʣͰར༻ऀͷମݧ ʢͱͯ͠ͷใͷ༰දࣔҐஔʣΛ࠷దԽ͢Δ • ମݧʢʣͷ։ൃऀɺઐࣝΛ࣋ͨͣʹଟόϯσΟοτʹΑΔൺֱධ ՁΛಋೖɾల։Մೳ • جຊత͘͠จ຺͖ͷଟόϯσΟοτํࡦ͔ΒબՄೳ • → ͷಛੑʹԠͨ͡ํࡦͷબఆʹΑΔػձଛࣦͷݮ͕ظͰ͖Δ -
 39 • ଟόϯσΟοτํࡦΛ༻͍ͨར༻ऀମݧͷݸผԽϓϥοτϑΥʔϜ • جຊత͘͠จ຺͖ͷଟόϯσΟοτํࡦ͔ΒબՄೳ • → ͷಛੑʹԠͨ͡ํࡦͷબఆʹΑΔػձଛࣦͷݮ͕ظͰ͖Δ •
͜ΕΒͷطଘͷํࡦͰɺػցֶशϞσϧͷ࣋ͭಛੑΛશͯߟྀͰ͖ͳ͍ • → ෳํࡦͷ͍͚ʹՃ͑ɺํࡦͦͷͷͷ։ൃ͕ٻΊΒΕΔ AdaptEx [30] [W. Black 2023] -
• ຊݚڀͰɺ࣮ڥͰͷධՁʹΑΔػձଛࣦͱ͍͏՝ʹର͠ɺਪનख๏ͷબ Λ࠷దԽ͢ΔͨΊɺҎԼͷཁ݅Λຬͨ͢ϝλਪનγεςϜΛઃܭ͢Δ  41 ܧଓతʹਪનख๏ͷબΛ࠷దԽ͢ΔϝλਪનγεςϜ 1. ਪનख๏ͷಛੑʹԠͨ͡બ 2. ҙͷਪનख๏Λൺֱରͱͯ͠ར༻
3. ܧଓతͳਪનख๏ͷ༗ޮੑͷධՁ 4. ධՁʹ·ͭΘΔػձଛࣦͷݮ -
 43 1. ਪનख๏ͷಛੑʹԠͨ͡બ 2. ҙͷਪનख๏Λൺֱରͱͯ͠ར༻ 3. ܧଓతͳਪનख๏ͷ༗ޮੑͷධՁ 4. ධՁʹ·ͭΘΔػձଛࣦͷݮ
ᶃ ਪનख๏ͷಛੑʹԠͯ͡ ଟόϯσΟοτํࡦΛܾఆ͢Δ ఏҊγεςϜʢSynapseʣ -
 44 1. ਪનख๏ͷಛੑʹԠͨ͡બ 2. ҙͷਪનख๏Λൺֱରͱͯ͠ར༻ 3. ܧଓతͳਪનख๏ͷ༗ޮੑͷධՁ 4. ධՁʹ·ͭΘΔػձଛࣦͷݮ
ᶄ ਪનख๏Λొ͢Δ ڞ௨ͷΠϯλʔϑΣʔεΛຬͨ͢ਪન ख๏Ͱ͋Εҙͷख๏Λར༻Մೳ ఏҊγεςϜʢSynapseʣ -
 45 1. ਪનख๏ͷಛੑʹԠͨ͡બ 2. ҙͷਪનख๏Λൺֱରͱͯ͠ར༻ 3. ܧଓతͳਪનख๏ͷ༗ޮੑͷධՁ 4. ධՁʹ·ͭΘΔػձଛࣦͷݮ
ᶅ ਪનख๏ΛબɾධՁ͢Δ ଟόϯσΟοτํࡦΛ༻͍ͯਪનख ๏Λબ͢Δɻ ར༻ऀͷߦಈ݁Ռอଘ͞ΕҰఆظؒ ͝ͱʹධՁ͕ߋ৽͞ΕΔɻ ఏҊγεςϜʢSynapseʣ -
• ఏҊγεςϜͰɺඪ४తͳํࡦͱͯ͠ɺैདྷͷจ຺͖ํࡦͰ͋Δɺ Linear Thompson SamplingʢLTSʣΛ࠾༻͢Δ  46 ܧଓతʹਪનख๏ͷબΛ࠷దԽ͢ΔϝλਪનγεςϜ  l*
t = argmaxl=1,L (x⊤ t ˜ w(l) N , ˜ w(l) N ∼ 𝒩 D (A−1 N bN , σ2 ϵ A−1 N )) ֶशʹΑΓੜ͞ΕΔཚ͕ɺίϯςΩετใʹ Ԡͨ͡ͱมԽ͍༷ͯ͘͠ࢠ • LTSʹ͓͚Δͷબఆʢ֬Ұக๏ʣ • ਪఆͨ͠ใुͷฏۉͱɺࢼߦճʹԠͨ͡ෆ࣮֬ੑͷදݱͰ͋Δڞࢄߦ ྻʹै͏ཚͱɺίϯςΩετใͷੵ͕࠷େ͖ͳΛબఆ͢Δ -
• બͨ͠ਪનख๏͔ΒಘΒΕΔΫϦοΫͷγϛϡϨʔγϣϯ • ํࡦʹΑΓબ͞Εͨਪનख๏ɺઃఆͨ͠ΫϦοΫͷϕϧψʔΠʹै͍ਪ ન݁Ռ͕ΫϦοΫ͞ΕΔͷͱ͢Δ • ֤ਪનख๏ΧςΰϦͱ͍͠18࣍ݩͷύϥϝʔλ  Λ࣋ͭ •
ΫϦοΫ  ͱίϯςΩετใ  ͷੵͰܭࢉ͞ΕΔ • ίϯςΩετใ  ɺ࣌  ʹ͓͍ͯར༻ऀ͕Ӿཡ͍ͯ͠ΔΧςΰϦͷ1-hot ϕΫτϧͱͯ͠දݱ͞ΕΔ • ࣮ࡍͷਪનγεςϜͷڍಈͱ߹ΘͤΔͨΊɺใु1࣌ؒ͝ͱʹ·ͱΊͯϑΟʔυ όοΫ͞ΕΔͷͱ͢Δ ˜ w(l) t ˜ w(l) t xt xt t  51 ධՁํ๏ʢ1/2ʣ -
• จ຺ͱ࣌ؒͷܦաͷߟྀͷͦΕͧΕͷد༩Λ໌Β͔ʹ͢Δ4άϧʔϓͷγ ϛϡϨʔγϣϯΛ࣮ࢪ  52 ධՁํ๏ʢ2/2ʣ ࣌ؒͷܦա º ˓ จ຺
º "ىटͷ࠷ળͳਪનख๏ΛશظؒҰ؏ ͯ͠༻͍Δ $࣌͝ͱʹධՁͷߴ͍ਪનख๏Λόϯ σΟοτΛ༻͍ͯબఆ ˓ #จ຺͝ͱʹ࠷ળͳਪનख๏Λશظؒ Ұ؏ͯ͠༻͍Δ %จ຺͝ͱ࣌͝ͱʹධՁͷߴ͍ਪનख ๏ΛόϯσΟοτΛ༻͍ͯબఆ • จ຺ʹɺਪન࣌ʹӾཡதͷΧςΰϦΛ༻͍Δ • ଟόϯσΟοτํࡦɺ -Greedy(B) ɺLinUCB(D) ɺLTS(D) ϵ -
• จ຺ͱ࣌ؒͷܦաͷߟྀͷͦΕͧΕͷد༩Λ໌Β͔ʹ͢Δ4άϧʔϓͷγ ϛϡϨʔγϣϯΛ࣮ࢪ  53 ධՁํ๏ʢ2/2ʣ ࣌ؒͷܦա º ˓ จ຺
º "ىटͷ࠷ળͳਪનख๏ΛશظؒҰ؏ ͯ͠༻͍Δ $࣌͝ͱʹධՁͷߴ͍ਪનख๏Λόϯ σΟοτΛ༻͍ͯબఆ ˓ #จ຺͝ͱʹ࠷ળͳਪનख๏Λશظؒ Ұ؏ͯ͠༻͍Δ %จ຺͝ͱ࣌͝ͱʹධՁͷߴ͍ਪનख ๏ΛόϯσΟοτΛ༻͍ͯબఆ ΦϯϥΠϯͰͷܧଓతͳධՁ͕ͦͦ༗ޮͰ͋Δ͔ ಛੑʹԠͨ͡ํࡦͷ͍͚͕༗ޮͰ͋Δ͔ -
 54 • Bʢจ຺ʣɿΧςΰϦ͝ͱͷظट࣌ ʹ͓͚Δ࠷దͳਪનख๏͕ҟͳΔͨΊɺ͜ ΕʹԠ͡Δ͜ͱͰվળ • Cʢ࣌ؒͷܦաʣɿશͯͷΧςΰϦΛ ௨ͯ͠ͷ࠷దͳਪનख๏ظؒதมԽ͠ͳ ͍ͨΊɺ݁Ռతʹ୳ࡧ͕ແବͱͳͬͨ
• → ࠷దͳਪનख๏͕໌Β͔ͰมԽ͕ͳ͍ͳΒํࡦ ͷಋೖ͕ରʹػձଛࣦΛੜΉ • Dʢจ຺ͱ࣌ؒͷܦաʣɿΧςΰϦ͝ ͱͷมԽʹैͨ͜͠ͱͰվળɻํࡦ͝ͱ ʹޮՌͷ͕ࠩݟΒΕͨ ධՁ݁Ռ: AάϧʔϓΛج४ͱͨ͠ྦྷੵใुͷࠩͷൺֱ จ຺ͷΈ ࣌ؒͷܦաͷΈ จ຺ͱ࣌ؒͷܦաΛߟྀ͢Δ ඪ४ํࡦʹΑͬͯ2%૿Ճ -
• Dάϧʔϓͷඪ४ํࡦʢLTSʣͷྦྷੵใु Λੳ͢ΔͱɺશͯͷΧςΰϦʹ͓͍ ͯɺ୳ࡧͷίετΛճऩͰ͖͍ͯΔΘ͚Ͱ ͳ͍ • ͨͩ͠ɺظؒதʹ࠷దͳਪનख๏͕Γସ ΘͬͨΧςΰϦʹ͓͍ͯେ͖͘վળ͢ ΔҰํͰɺͦ͏Ͱͳ͍߹ʹෆཁͳ୳ࡧ Λ͑ͨ͜ͱͰɺશମͱͯ͠ͷػձଛࣦΛ
ݮͨ͠ • → ۉͳ୳ࡧͷ  -Greedyɺใु͕ΕΔڥͰ ͕ݻఆ͞ΕΔLinUCBຊධՁઃఆͰෆརͱͳͬͨ ϵ  55 ධՁ݁Ռ: ΧςΰϦ͝ͱͷྦྷੵใुͷ༁ %άϧʔϓͷඪ४ํࡦʢ-54ʣͷྦྷੵใुͱɺ#άϧʔ ϓͷظटͷ࠷దͳਪનख๏ΛҰ؏ͨ͠߹ͱͷൺֱ ˎʢʣͦͷൺ -
 58 খ·ͱΊ • ࣮ڥͰͷධՁʹΑΔػձଛࣦͱ͍͏՝ʹର͠ɺਪનγεςϜΛࡐʹɺଟ όϯσΟοτํࡦΛ༻͍ͨϝλਪનγεςϜΛઃܭ • ධՁͰɺ࣮ࡍͷECαΠτ͔Βऔಘͨ͠ਪનख๏ͷΫϦοΫͷਪҠ࣮ σʔλΛ༻͍ͨγϛϡϨʔγϣϯͷ݁Ռɺจ຺࣌ؒͷܦաΛߟྀ͠ͳ͍ํࡦ ͱൺֱͯ͠ɺྦྷੵใुΛ࠷େ2%૿Ճͤ͞ΔޮՌΛ֬ೝͨ͠
• ਪનख๏ͷಛੑγεςϜཁ݅Λ౿·͑ͨదͳํࡦͷબఆ͕༗ޮ • ࠓޙͷݚڀ՝ͱͯ͠ɺ࣌ؒͷܦաΛ໌ࣔతʹѻ͏͜ͱͰղܾΛਤΔ -
ຊݚڀͷ၆ᛌਤ  60 "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU Context Non stationarity Context Non stationarity
Online performance Context (Non-linear) Non stationarity Online performance Multi-armed bandit polices 3ষ 4ষ 5ষ 6ষ -
 61 తͱఏҊͷࠎࢠ • ػցֶशϞσϧͷ༏ྼଟ͘ͷཁҼ͔ΒͳΔঢ়گʢ=จ຺ʣ࣌ؒͷܦաʹ Αͬͯࠨӈ͞ΕΔ • ޮՌతͳϞσϧΛػձଛࣦ͕ͳ͍Α͏จ຺ͱ࣌ؒͷܦաʹԠ͍͚͍ͯͨ͡ • จ຺ͱ࣌ؒͷܦաʹԠͨ͡બͷ࠷దԽΛɺඇఆৗ͔ͭจ຺͖ଟόϯ
σΟοτͱΈͳ͠ɺঢ়ଶۭؒϞσϧͱཻࢠϑΟϧλΛ༻͍ͨํࡦΛఏҊ -
• ͝ͱʹෳͷจ຺͕͋Γɺจ຺ʹԠͯ͡ใु͕ܾ·ΔଟόϯσΟοτ ͷઃఆ • ຊݚڀͰɺจ຺ɺෳͷཁҼͷύϥϝʔλͷΈ߹ΘͤͰදݱ͞Εͨ ঢ়ଶͷ͜ͱΛࢦ͢ • → ཁҼύϥϝʔλͷ͕{0,1}ͷ߹ɺจ຺ཁҼ 
ʹରͯ͠  ύλʔϯ d 2d  63 จ຺͖ଟόϯσΟοτ • จ຺͖ଟόϯσΟοτํࡦͰɺจ຺ͷ֬Ͱͳ͘ɺཁҼ͝ͱͷ ʢઢܗύϥϝʔλʣΛਪఆ͢Δ͜ͱͰ֤จ຺ʹ͓͚ΔใुΛ༧͢Δ -
• ಉ͡จ຺ʹ͓͍ͯ࣌ؒܦաʹΑͬͯใु͕มԽ͢ΔଟόϯσΟοτ ͷઃఆ • पظతͳมԽͰ͋ΕཁҼͷύϥϝʔλʹؚΊΔ͜ͱͰରԠͰ͖Δ͕ෆن ଇͳ߹ʢඇఆৗͳมԽͷ߹ʣ͜ͷݶΓͰͳ͍  64 ඇఆৗͳଟόϯσΟοτ •
ඇఆৗͳଟόϯσΟοτํࡦͰɺաڈʹ؍ଌͨ͠ใुʹଊΘΕͣͷධՁ Λਝʹߋ৽͢Δ͜ͱͰ֤จ຺ʹ͓͚ΔใुΛ༧͢Δ ٫ܕ εϥΠσΟϯά Οϯυܕ มԽݕग़ܕ ঢ়ଶۭؒϞσϧܕ -
ঢ়ଶۭؒϞσϧܕ • ঢ়ଶͷਪҠաఔΛঢ়ଶۭؒϞσϧͰѻ͍ɺஞ࣍ਪఆͨ͠ঢ়ଶΛར༻ • ใुܥྻͷ࣌ؒతͳมԽΛࣗવʹѻ͑Δɻಛʹ؇͔ͳมԽͷઃఆʹద͢Δ  65 • աఔͷදݱྗͱਪఆਫ਼ʹੑೳ͕ґଘɻ +
ଟมྔΛఆͨ͠ঢ়ଶͱ؍ଌͷؔੑදݱͰ͖ ΔͨΊɺจ຺͖ͷઃఆࣗવʹѻ͑Δ ঢ়ଶۭؒϞσϧܕ State space model State Context ඇఆৗ͔ͭจ຺͖ଟόϯσΟοτํࡦ IUUQTJDPOTDPN -
 68 ඇఆৗͳଟόϯσΟοτͷ՝ ʮऑ͍ʯ͕ڧ͘ͳΔঢ়گ ˠͷସ͕ΕΔ ྫᅂใͷੵʹ͏ਪન ΞϧΰϦζϜͷਫ਼্ͳͲ ຊݚڀͰͷ͜ͷঢ়گΛ ʮ൪ڰΘͤʯͱݺͿ •
ैདྷͷඇఆৗͳଟόϯσΟοτํࡦͷಘҙͳঢ়گͱෆಘҙͳঢ়گͷྫ -
• ଟόϯσΟοτͷํࡦɺ͋Δ࣌ͰධՁͷߴ͍Λ࠷ଟ͘׆༻͢Δ • ͷ༗༻ੑͷมԽͷஅʹɺมԽޙͷใु͔Βͷαϯϓϧ͕Ұఆඞཁ  69 ଟόϯσΟοτʹ͓͚Δ൪ڰΘͤͷ՝ ൪ڰΘͤͳΓସΘΓ • ैདྷͷඇఆৗ͔ͭจ຺͖ํࡦͰɺ໌ࣔత
ͳରԠ͕ಋೖ͞Ε͍ͯͳ͍ • ൪ڰΘͤͷݕग़ͷͨΊʹશͯͷͷ୳ࡧΛ Ұʹ૿͢[37,38]ͱมԽ͕ͳ͍ظؒจ຺Ͱ ͷػձଛࣦʹͭͳ͕Δ -
 70 Time-varying Thompson Sampling (TVTP) [42] [C. Zeng 2016]
• ঢ়ଶۭؒϞσϧܕͷඇఆৗ͔ͭจ຺͖ํࡦ • ͷ༗༻ੑͷมಈΛදݱ͢ΔͨΊͷDynamic Context drift ModelingΛఏҊ • ঢ়ଶͷෳࡶͳύϥϝʔλਪఆͷͨΊཻࢠϑΟϧλΛ༻͍Δ • → աఔͷදݱྗͱਪఆਫ਼ͷ՝Λղܾ • ҰํͰɺࢼߦճͷ૿Ճʹ͍୳ࡧͷׂ߹͕ٸܹʹԼ͢Δ • → ൪ڰΘͤͷै͕ेͰͳ͍ -
 72 <>'JH(SBQIJDBMNPEFMSFQSFTFOUBUJPOGPSCBOEJUQSPCMFN ίϯςΩετ ఆৗ߲ ඇఆৗ߲ ʢεέʔϧ߲ɺυϦϑτ߲ʣ ؍ଌޡࠩ Time-varying Thompson
Sampling (TVTP) [42] [C. Zeng 2016] y(l) t ∼ 𝒩 (x 𝖳 t (c(l) + θ(l) ⊙ η(l) t ), σ2(l) ϵ ) k → l, cwk → c(l), θk → θ(l), ηk,t → η(l) t , σ2 k → σ2(l) ϵ μc → μ(l) w , Σc → Σ(l) w , α → α(l) ϵ , β → β(l) ϵ , μθ → μ(l) θ , Σθ → Σ(l) θ ˎ ຊจͷදهͱͷ౷ҰͷͨΊݩจͷFig.2ʢӈ্ʣͷه๏ҎԼʹஔ͢Δ • ঢ়ଶۭؒϞσϧ • จ຺͖ɺ͔ͭɺඇఆৗͳใुͷมԽ Λѻ͏ͨΊใुͷมಈΛΈࠐΜͩϞ σϧΛ༻͍Δ -
 73 <>'JH(SBQIJDBMNPEFMSFQSFTFOUBUJPOGPSCBOEJUQSPCMFN Time-varying Thompson Sampling (TVTP) [42] [C. Zeng
2016] y(l) t ∼ 𝒩 (x 𝖳 t (c(l) + θ(l) ⊙ η(l) t ), σ2(l) ϵ ) ΧϧϚϯϑΟϧλ ཻࢠϑΟϧλ k → l, cwk → c(l), θk → θ(l), ηk,t → η(l) t , σ2 k → σ2(l) ϵ μc → μ(l) w , Σc → Σ(l) w , α → α(l) ϵ , β → β(l) ϵ , μθ → μ(l) θ , Σθ → Σ(l) θ ˎ ຊจͷදهͱͷ౷ҰͷͨΊݩจͷFig.2ʢӈ্ʣͷه๏ҎԼʹஔ͢Δ • ঢ়ଶͷਪఆ • ใुϞσϧͷύϥϝʔλͷࣄޙͱυ Ϧϑτ߲ͷજࡏঢ়ଶͷஞ࣍ਪఆʹཻࢠ ϑΟϧλͱΧϧϚϯϑΟϧλΛ༻͍Δ -
 74 <>'JH(SBQIJDBMNPEFMSFQSFTFOUBUJPOGPSCBOEJUQSPCMFN Time-varying Thompson Sampling (TVTP) [42] [C. Zeng
2016] y(l) t ∼ 𝒩 (x 𝖳 t (c(l) + θ(l) ⊙ η(l) t ), σ2(l) ϵ ) k → l, cwk → c(l), θk → θ(l), ηk,t → η(l) t , σ2 k → σ2(l) ϵ μc → μ(l) w , Σc → Σ(l) w , α → α(l) ϵ , β → β(l) ϵ , μθ → μ(l) θ , Σθ → Σ(l) θ ˎ ຊจͷදهͱͷ౷ҰͷͨΊݩจͷFig.2ʢӈ্ʣͷه๏ҎԼʹஔ͢Δ ࣄޙ ࣄޙ l* t = argmaxl=1,L x⊤ ¯ w(l) t−1 ¯ w(l) t−1 ∼ 𝒩 D ( ¯ μ(l) w , ¯ Σ(l) w ) • ֬Ұக๏ • ֤ͰٻΊͨύϥϝʔλͷࣄޙʹै ͍αϯϓϦϯάͨ݁͠ՌΛͷબఆʹ༻ ͍Δ͜ͱͰଟόϯσΟοτํࡦͱ౷߹ -
 75 TVTPͷ՝ • ࢼߦճͷ૿Ճʹ͏ͷબఆͷภΓ • ͋Δ࣌ͰධՁͷ͍Λ୳ࡧ͢Δػձ͕ۃʹԼ • ൪ڰΘͤͷঢ়گͷै͕ΕΔ ࢼߦճͷ૿Ճʹ͏ٸܹͳݮগʹΑΓ
͋Δ࣌ͷධՁʹج͍ͮͨબʹݻఆ͞ΕΔ l* t = argmaxl=1,L x⊤ ¯ w(l) t−1 ¯ w(l) t−1 ∼ 𝒩 D ( ¯ μ(l) w , ¯ Σ(l) w ) ¯ Σ(l) w = 1 p2 p ∑ i=1 σ2(l,i) ϵ Σ(l,i) w , XIFSF Q JT OVNCFS PG QBSUJDMFT -
¯ Σ(l) w = 1 p2 p ∑ i=1 σ2(l,i)
ϵ Σ(l,i) w , XIFSF Q JT OVNCFS PG QBSUJDMFT  76 ఏҊํࡦ: Aggressive Exploration TVTPʢAE-TVTPʣ • ࢼߦճͷ૿Ճʹ͏ͷબఆͷภΓΛղফ • ͋Δ࣌ͰධՁͷ͍Λੵۃతʹ୳ࡧ͢ΔػձΛઃ͚Δ • ൪ڰΘͤͷঢ়گʹ͓͚Δैੑͷ্ΛਤΔ ཻࢠͷฏۉ ཻ֤ࢠͰͷࢉͷΈ l* t = argmaxl=1,L x⊤ ¯ w(l) t−1 ¯ w(l) t−1 ∼ 𝒩 D ( ¯ μ(l) w , ¯ Σ(l) w ) -
• ࣮ࡍͷECαΠτ͔Β࠾औͨ͠4ͭͷਪનख๏ͷΧςΰϦ͝ͱͷΫϦοΫ ͷਪҠ࣮σʔλΛ༻͍ͯఏҊγεςϜͷ༗ޮੑΛධՁ͢Δ • 3ষͱಉҰͷσʔλ͕ͩɺҎԼͷมߋʹΑΓ࣌ؒܦաʹΑΔมಈΛڧௐͨ͠ • ΫϦοΫͷूܭΛʮ࣌·ͰͷྦྷܭʹΑΔฏۉʯ͔Βʮ͔࣌Β3લ· ͰͷࢦҠಈฏۉʯ • ूܭظؒதͷਪનճ͕ஶ͘͠গͳ͔ͬͨʮߪങཤྺʹΑΔڠௐܕਪનʯ
ͱʮςΩετʹΑΔ༰ϕʔεਪનʯͷೋͭͷਪનख๏Λআ͍ͨ • → ूܭํ๏ͷมߋʹΑΓΫϦοΫ͕҆ఆ͠ͳ͘ͳΓੳͷϊΠζͱͳΔ͜ͱ͕ݒ೦͞ΕͨͨΊ  78 ධՁσʔλͱਪનख๏ -
• બͨ͠ਪનख๏͔ΒಘΒΕΔΫϦοΫͷγϛϡϨʔγϣϯ • ํࡦʹΑΓબ͞Εͨਪનख๏ɺઃఆͨ͠ΫϦοΫͷϕϧψʔΠʹै͍ਪ ન݁Ռ͕ΫϦοΫ͞ΕΔͷͱ͢Δ • ֤ਪનख๏ΧςΰϦͱ͍͠18࣍ݩͷύϥϝʔλ  Λ࣋ͭ •
ΫϦοΫ  ͱίϯςΩετใ  ͷੵͰܭࢉ͞ΕΔ • ίϯςΩετใ  ɺ࣌  ʹ͓͍ͯར༻ऀ͕Ӿཡ͍ͯ͠ΔΧςΰϦͷ1-hot ϕΫτϧͱͯ͠දݱ͞ΕΔ • ࣮ࡍͷਪનγεςϜͷڍಈͱ߹ΘͤΔͨΊɺใु1࣌ؒ͝ͱʹ·ͱΊͯϑΟʔυ όοΫ͞ΕΔͷͱ͢Δ ˜ w(l) t ˜ w(l) t xt xt t  80 ධՁํ๏ʢ1/2ʣʢ3ষͱಉҰʣ -
• จ຺ͱ࣌ؒͷܦաͷߟྀͷͦΕͧΕͷد༩Λ໌Β͔ʹ͢Δ4άϧʔϓͷγ ϛϡϨʔγϣϯΛ࣮ࢪ  81 ධՁํ๏ʢ2/2ʣʢํࡦΛআ͖ɺ3ষͱಉҰʣ ࣌ؒͷܦա º ˓ จ຺
º "ىटͷ࠷ળͳਪનख๏ΛશظؒҰ؏ ͯ͠༻͍Δ $࣌͝ͱʹධՁͷߴ͍ਪનख๏Λόϯ σΟοτΛ༻͍ͯબఆ ˓ #จ຺͝ͱʹ࠷ળͳਪનख๏Λશظؒ Ұ؏ͯ͠༻͍Δ %จ຺͝ͱ࣌͝ͱʹධՁͷߴ͍ਪનख ๏ΛόϯσΟοτΛ༻͍ͯબఆ • จ຺ʹɺਪન࣌ʹӾཡதͷΧςΰϦΛ༻͍Δ • ଟόϯσΟοτํࡦɺLTS(จ຺) ɺTVTP(จ຺/࣌ؒͷܦա) ɺAE-TVTP(จ຺/࣌ؒͷܦա) -
• จ຺ͱ࣌ؒͷܦաͷߟྀͷͦΕͧΕͷد༩Λ໌Β͔ʹ͢Δ4άϧʔϓͷγ ϛϡϨʔγϣϯΛ࣮ࢪ  82 ࣌ؒͷܦա º ˓ จ຺ º
"ىटͷ࠷ળͳਪનख๏ΛશظؒҰ؏ ͯ͠༻͍Δ $࣌͝ͱʹධՁͷߴ͍ਪનख๏Λόϯ σΟοτΛ༻͍ͯબఆ ˓ #จ຺͝ͱʹ࠷ળͳਪનख๏Λશظؒ Ұ؏ͯ͠༻͍Δ %จ຺͝ͱ࣌͝ͱʹධՁͷߴ͍ਪનख ๏ΛόϯσΟοτΛ༻͍ͯબఆ ඇఆৗɾ൪ڰΘͤͷߟྀ͢Δํࡦ͕༗ޮͰ͋Δ͔ ධՁํ๏ʢ2/2ʣʢํࡦΛআ͖ɺ3ষͱಉҰʣ -
• Bάϧʔϓʢจ຺ʣظट࣌ʹ͓͍ͯAά ϧʔϓͱจ຺ʹΑΔࠩҟ͕΄΅ͳ͍ͨΊ݁ Ռࠩҟͳ͠ • Cάϧʔϓʢ࣌ؒͷܦաʣਪનख๏ͷ༗ ޮੑͷมԽʹैͨ͜͠ͱͰվળ͕ݟΒΕ Δ • Dάϧʔϓʢจ຺ͱ࣌ؒͷܦաʣΧ
ςΰϦ͝ͱͷมԽʹैͨ͜͠ͱͰߋͳΔ վળ͕ݟΒΕΔ  83 ධՁ݁Ռ: AάϧʔϓΛج४ͱͨ͠ྦྷੵใुͷࠩͷൺֱ จ຺ͱ࣌ؒͷܦաͷߟྀͳΒͼʹɺٯసڥͷ ैੑΛߴΊͨఏҊํࣜʹΑͬͯ૿Ճ จ຺ͷΈ ࣌ؒͷܦաͷΈ -
• Dάϧʔϓͷจ຺͝ͱͷվળ݁ՌΛੳ͢ Δͱɺਪનख๏ͷ༗ޮੑͷมಈͷগͳ͍จ ຺Ͱɺશͯͷํࡦʹ͓͍ͯ୳ࡧͷίετ ΛճऩͰ͖͍ͯͳ͍ • มಈͷେ͖͍จ຺Ͱੵۃతͳ୳ࡧʹΑΓ AE-TVTP͕େ͖͘վળͨ͠ɻTVTPਪન ख๏͕ݻఆ͞ΕվળʹࢸΒͳ͔ͬͨ 
84 ධՁ݁Ռ: ਪનख๏ͷ༗ޮੑͷมಈ߹͍ͱվળ • ԣ࣠ͷਪનख๏ͷ༗ޮੑͷมಈͷେ͖͞ɺظटʹ࠷ΫϦοΫ͕ߴ͔ͬͨਪનख๏ʹରͯ͠ɺ֤࣌Ͱ࠷େͷΫϦοΫͱͷࠩͷɺظؒ·Ͱͷ߹ܭ • ॎ࣠ͷྦྷੵใुͷվળɺDάϧʔϓͷ֤ํࡦͷྦྷੵใुͱɺBάϧʔϓͷ͏ͪਪનख๏ΛҰ؏ͯ͠༻͍ͨ݁Ռʹର͢Δൺ -
 85 ߟ • ਪનख๏ͷ༗ޮੑ͕ٯస͢Δࠨྻʹ͓͍ͯ ఏҊख๏͕༗ޮ • ӈྻʹ͓͍ͯɺఏҊख๏ͷੵۃతͳ୳ࡧ ʹىҼͯ͠ɺظؒதܧଓతʹྦྷੵϦάϨο τ͕૿Ճ͢Δɻ3ׂఔͷΧςΰϦͰ
ಉ༷ͷࣄΛ֬ೝɻ ਪનख๏ͷ༗ޮੑʹٯస͕͋ΔΧςΰϦ ࠨ ͱɺ ͳ͍ΧςΰϦ ӈ ʹ͓͚ΔྦྷੵϦάϨοτͷਪҠ ˎྦྷੵϦάϨοτਪનख๏ͷ͏ͪ࠷େͷظͱબͨ͠ਪનख ๏ͷظͷࠩΛظؒ·Ͱʹ߹ܭͨ͠ͷ • มԽͷͳ͍ظؒʹ͓͍ͯػձଛࣦΛݮ ͢ΔదԠతͳ୳ࡧख๏ͷݚڀ -
• ఏҊํࡦAE-TVTPͰ͋ͬͯɺظతʹڞࢄߦྻ  ͷཁૉখ͍͞ʹ ऩଋ͢Δ͜ͱɺ·ͨɺ୳ࡧଅਐɺબͨ͠ͷΈʹର͢ΔॲஔͰ͋Δ͜ͱ͔ Βɺظؒͷӡ༻࣌ʹ͓͚ΔैੑͷԼ͕ݒ೦͞ΕΔ • ঢ়ଶਪఆͰ༻͍ΔཻࢠϑΟϧλɺཻࢠ͝ͱͷෳճͷٯߦྻܭࢉΛؚΉͨ ΊɺίϯςΩετใͷ࣍ݩͱཻࢠͷ૿Ճʹ͍ܭࢉ͕࣌ؒ૿Ճ͢Δɻ •
→ ཻࢠɺࣄޙͷਪఆਫ਼ʹӨڹ͢ΔͨΊɺਫ਼ͱ࣮ߦ࣌ؒͷτϨʔυΦϑ͕ੜ͡Δ ¯ Σ(l) w  86 ఏҊʹؔ͢Δٞ ¯ Σ(l) w = 1 p p ∑ i=1 Σ(l,i) w l* t = argmaxl=1,L x⊤ ¯ w(l) t−1 ¯ w(l) t−1 ∼ 𝒩 D ( ¯ μ(l) w , ¯ Σ(l) w ) -
 88 খ·ͱΊ • จ຺࣌ؒͷܦաʹΑΔ༗༻ੑͷมಈͱ͍͏ಛੑʢ1ʣʹର͠ɺཻࢠϑΟϧλ Λ༻͍ͨඇఆৗ͔ͭจ຺͖ଟόϯσΟοτํࡦΛఏҊ • ൪ڰΘͤͷঢ়گͱ୳ࡧͷॏཁੑʹண͠ɺैདྷํࡦͷ֦ுʹΑͬͯվળ • ධՁͰɺಉECαΠτͰͷσʔλΛ༻͍ͨγϛϡϨʔγϣϯͷ݁Ռɺจ຺
࣌ؒͷܦաΛߟྀ͠ͳ͍ํࡦͱൺֱͯ͠ɺྦྷੵใुΛ࠷େ9.7%૿Ճͤ͞Δޮ ՌΛ֬ೝͨ͠ • ࠓޙͷݚڀ՝ͱͯ͠ɺػձଛࣦͷগͳ͍୳ࡧํࣜͷݕ౼͓Αͼํࡦͷ࣮ߦ࣌ ؒͷݮ͕ڍ͛ΒΕΔ -
ຊݚڀͷ၆ᛌਤ  90 "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU Context Non stationarity Context Non stationarity
Online performance Context (Non-linear) Non stationarity Online performance Multi-armed bandit polices 3ষ 4ষ 5ষ 6ষ -
 91 తͱఏҊͷࠎࢠ • ػցֶशϞσϧͷ༏ྼɺจ຺࣌ؒͷܦա͚ͩͰͳ͘ɺԠੑɾैੑʹΑͬ ͯࠨӈ͞ΕΔ • ޮՌతͳϞσϧΛػձଛࣦ͕ͳ͍Α͏จ຺ͱ࣌ؒͷܦաʹԠ͍͚ͯͨ͡ ͍ɻͦΕΛ࣮ߦ࣌ؒͷ؍Ͱҙࣝͤ͞Δ͜ͱͳ͘ߦ͍͍ͨ •
จ຺ͱ࣌ؒͷܦաʹԠͨ͡બͷ࠷దԽΛɺඇఆৗ͔ͭจ຺͖ଟόϯ σΟοτͱΈͳ͠ɺਝʹ͜ΕΛղ͘ɺઢܗΨεঢ়ଶۭؒϞσϧͱઢܗ ΧϧϚϯϑΟϧλΛ༻͍ͨํࡦΛఏҊ -
• ஞ࣍తʹಘΒΕΔใुɾίϯςΩετใʹର͢Δஞ࣍తͳֶशͷඞཁੑ • ஞ࣍తͳֶशʹ͓͚Δֶश࣌ؒͷओͨΔӨڹཁҼ  93 ଟόϯσΟοτํࡦͷ࣮ߦ࣌ؒʹؔ͢Δ՝ͱରࡦ 1. શͯͷֶशσʔλΛ༻͍ͨ࠶ܭࢉ →
 ·Ͱͷܭࢉ݁Ռͱ  Ͱͷ؍ଌͷΈΛ༻͍Δ࠶ؼతֶशػߏ 2. ٯߦྻͷܭࢉ → ্هͷ࠶ؼతֶशʹ͓͚Δ ɹ Woodburyͷ߃ࣜͷద༻ N N + 1 ஞ࣍࠷খೋ๏ -
 94 ํࣜ ࣜਤ ՝ ٫ܕ ٫͖ɺ͔ͭɺਖ਼ଇԽΛؚΉஞ࣍࠷খೋ๏ ൚Խੑೳ͕Լ [41] εϥΠσΟϯά
Οϯυܕ ैདྷํࡦͰஞ࣍࠷খೋ๏ͷద༻ํ๏ࣔ͞Εͳ͍ [69] มԽݕग़ܕ มԽݕग़Ϟσϧͷ࣮ߦ࣌ؒଟมྔʹରԠ͢ΔͨΊ૿Ճ [43,70,71] ঢ়ଶۭؒ Ϟσϧܕ ෳࡶͳաఔΛදݱͰ͖ΔϞσϧʹର͢Δਪఆख๏ ࣮ߦ͕͍࣌ؒ [42] Regression model Regression model Regression model Detection model State space model State γ0 γ1 γ2 γ3 w దԠͷߴԽͷ՝ • จ຺ͱඇఆৗੑΛಉ࣌ʹߟྀ͢Δࡍͷ࣮ߦ࣌ؒʹؔ͢Δ՝ IUUQTJDPOTDPN -
 95 Phased Initial Exploration of System [73] [J. Cornet
2022] • ঢ়ଶۭؒϞσϧܕͷඇఆৗ͔ͭจ຺͖ํࡦ • ͷ༗༻ੑͷมಈΛදݱ͢ΔͨΊɺঢ়ଶ؍ଌͷؔੑ͕ɺઢܗ͔ͭޡ ͕ࠩਖ਼نʹै͏ͱԾఆͨ͠ઢܗΨεঢ়ଶۭؒϞσϧΛ࠾༻ • ঢ়ଶͷύϥϝʔλਪఆͷͨΊܰྔͳઢܗΧϧϚϯϑΟϧλΛ༻͍Δ • → ঢ়ଶਪҠͷࣗ༝Λ੍ݶ͢Δ͜ͱͰ࣮ߦ࣌ؒͷॖΛਤΔ • ҰํͰɺίϯςΩετใ͕લͷ࣌ͷঢ়ଶʹґଘ͢Δͱ͍͏ԾఆΛ࣋ͭ • → ίϯςΩετใ͕֎෦͔Β༩͑ΒΕΔจ຺͖ͷΛѻ͑ͳ͍ Linear Gaussian state space (LGSS) -
 96 • ঢ়ଶۭؒϞσϧܕͷඇఆৗ͔ͭจ຺͖ํࡦ • ͷ༗༻ੑͷมಈΛදݱ͢ΔͨΊɺঢ়ଶ؍ଌͷؔੑ͕ɺઢܗ͔ͭޡ ͕ࠩਖ਼نʹै͏ͱԾఆͨ͠ઢܗΨεঢ়ଶۭؒϞσϧΛ࠾༻ • ঢ়ଶͷύϥϝʔλਪఆͷͨΊܰྔͳઢܗΧϧϚϯϑΟϧλΛ༻͍Δ •
→ ঢ়ଶਪҠͷࣗ༝Λ੍ݶ͢Δ͜ͱͰ࣮ߦ࣌ؒͷॖΛਤΔ • ·ͨɺ୳ࡧΛଅ͢ػߏΛඋ͍͑ͯͳ͍ • → ൪ڰΘͤͷঢ়گͷରԠ͕ॆͰͳ͍ Phased Initial Exploration of System [73] [J. Cornet 2022] -
 97 KF-MANB [40] [O. Granmo 2010] • ঢ়ଶۭؒϞσϧܕͷඇఆৗͳํࡦ •
ͷ༗༻ੑͷมಈΛදݱ͢ΔͨΊϩʔΧϧϨϕϧϞσϧΛ࠾༻ • ঢ়ଶͷύϥϝʔλਪఆͷͨΊܰྔͳΧϧϚϯϑΟϧλΛ༻͍Δ • ܽଌॲཧΛԉ༻ͨ͠ະબͷʹର͢Δ୳ࡧΛఏҊ • → ࣮ߦ࣌ؒͷॖͱɺ୳ࡧͷภΓͷղফΛ࣮ݱ • ҰํͰɺίϯςΩετใΛೖྗͰ͖ͳ͍ํࡦ • → จ຺͖ͷΛѻ͏͜ͱ͕Ͱ͖ͳ͍ ࢼߦ͝ͱʹશͯͷΛߋ৽͢Δͨ Ίɺܰྔͳਪఆख๏Ͱͷ࠾༻͕ ·͍͠ํࣜ -
• ఏҊํࡦͰใुϞσϧ͕ҎԼͷઢܗΨεঢ়ଶۭؒϞσϧʹै͏ͱԾఆ͢Δ ઢܗΨεঢ়ଶۭؒϞσϧΛ༻͍ͨใुϞσϧ  99 • ঢ়ଶ  ͕લͷ࣌ͷঢ়ଶ 
ʹใु  ͕ঢ়ଶ  ʹै͏͜ͱΛදݱ͍ͯ͠Δ αt αt−1 rt αt  Rt  t + 1  Zt  Tt  ηt  ϵt  rt  αt  αt+1  +  + rt = Zt αt + ϵt , ϵt ∼ 𝒩 (0,σ2 ϵ ), αt+1 = Tt αt + ηt Rt , ηt ∼ 𝒩 (0,σ2 η ), t = 1,…, τ α1 ∼ 𝒩 d (μ1 , Σ1 ), -
• Ұظઌ༧ଌͱ؍ଌ݁ՌʹΑΔϑΟϧλϦϯάΛ܁Γฦͯ͠ঢ়ଶΛܧଓతʹਪఆ • ঢ়ଶ  ฏۉ  ͱͦͷڞࢄߦྻ  Ͱදݱ͢Δ͜ͱ͕Ͱ͖Δ
α μ Σ  100 ઢܗΧϧϚϯϑΟϧλʹΑΔͷঢ়ଶਪఆ Ұظઌͷঢ়ଶ༧ଌ Ұظઌͷ؍ଌ༧ଌ ؍ଌͷޡࠩ ϑΟϧλϦϯά ઢܗΧϧϚϯϑΟϧλʹ͓͚Δঢ়ଶߋ৽ͷαΠΫϧ μt+1 = Tμt|t Σt+1 = TΣt|t T 𝖳 + RQR 𝖳 ̂ yt = Zμt vt = yt − ̂ yt Ft = ZΣt Z 𝖳 + H μt|t = μt + Gvt = μt + (Σt Z 𝖳 F−1 t )vt Σt|t = Σt − GFt G 𝖳 t = t + 1 -
 101 Ұظઌͷঢ়ଶ༧ଌ Ұظઌͷ؍ଌ༧ଌ ؍ଌͷޡࠩ ϑΟϧλϦϯά ઢܗΧϧϚϯϑΟϧλʹ͓͚Δঢ়ଶߋ৽ͷαΠΫϧ μt+1 = Tμt|t
Σt+1 = TΣt|t T 𝖳 + RQR 𝖳 ̂ yt = Zμt vt = yt − ̂ yt Ft = ZΣt Z 𝖳 + H μt|t = μt + Gvt = μt + (Σt Z 𝖳 F−1 t )vt Σt|t = Σt − GFt G 𝖳 t = t + 1 จ຺ʹԠͨ͡ͷධՁߋ৽ • ઢܗΧϧϚϯϑΟϧλʹ͓͚Δߦྻ   ϑΟϧλϦϯάҰظઌ༧ଌͷॲ ཧʹ͓͍ͯঢ়ଶͷฏۉࢄڞࢄͷͲͷཁૉΛߋ৽͢Δ͔ܾఆ͍ͯ͠Δ Z R ঢ়ଶਪҠ࣌ͷޡࠩΛ༩ ؍ଌ࣌ͷޡࠩΛ༩ -
 102 Ұظઌͷঢ়ଶ༧ଌ μt+1 = Tμt|t Σt+1 = TΣt|t T
𝖳 + RQR 𝖳 จ຺ʹԠͨ͡ͷධՁߋ৽ ঢ়ଶਪҠ࣌ͷޡࠩΛ༩ ഉଞతͳίϯςΩετͷ࣍ݩʹޡ͕ࠩੵͯ͠͠·͏ ίϯςΩετAͰऑ͍ɺܽଛॲཧʹΑΓଞͷίϯςΩετʹରͯ͠ޡࠩΛ༩͢Δɻ ίϯςΩετBͰ10ճఔͷ୳ࡧͰྑ͍ͷ͕20ճ୳ࡧ͞ΕΔ͔͠Εͳ͍ • ઢܗΧϧϚϯϑΟϧλʹ͓͚Δߦྻ   ϑΟϧλϦϯάҰظઌ༧ଌͷॲ ཧʹ͓͍ͯঢ়ଶͷฏۉࢄڞࢄͷͲͷཁૉΛߋ৽͢Δ͔ܾఆ͍ͯ͠Δ • → ଟ༷ͳจ຺Λѻ͏ઃఆͰৗʹಉ͡ߦྻΛ༻͍ͨߋ৽ద͞ͳ͍ Z R RQR 𝖳 = σ2 η σ2 η σ2 η σ2 η σ2 η σ2 η σ2 η σ2 η σ2 η R = [ 1 1 1 ] , Q = [σ2 η ] -
 103 Ұظઌͷঢ়ଶ༧ଌ μt+1 = Tμt|t Σt+1 = TΣt|t T
𝖳 + RQR 𝖳 จ຺ʹԠͨ͡ͷධՁߋ৽ ঢ়ଶਪҠ࣌ͷޡࠩΛ༩ Rt QR 𝖳 t = σ2 η 0 σ2 η 0 0 0 σ2 η 0 σ2 η Rt = [ 1 0 1 ] , Q = [σ2 η ] • ઢܗΧϧϚϯϑΟϧλʹ͓͚Δߦྻ   ϑΟϧλϦϯάҰظઌ༧ଌͷॲ ཧʹ͓͍ͯঢ়ଶͷฏۉڞࢄߦྻͷͲͷཁૉΛߋ৽͢Δ͔ܾఆ͍ͯ͠Δ • → จ຺ʹԠͨ࣌͡มͳߦྻΛ༻ҙ͢Δ Z R ίϯςΩετͷԠͨ͡  (=  )Λ༻ҙ͢Δʢ  ʹ͍ͭͯಉ༷ʣ R Rt Z Rt = xt = (1,0,1) 𝖳 (xt ∈ {0,1}m) -
 104 จ຺Λߟྀͨ͠ߦྻ • ఏҊͷߦྻͷઃܭߏ࣌ܥྻϞσϧʹجͮ͘ • ঢ়ଶͷਪҠʹ͓͍ͯਫ४ͷΈΛѻ͏߹ͷઃܭ • ঢ়ଶͷਪҠʹ͓͍ͯਫ४͚ͩͰͳ͘ΛؚΊΔΑ͏༰қʹ֦ுՄೳ Zt
= (x 𝖳 t , 01×D), Rt = [ xt 0D×1 0D×1 xt ] , Tt = [ ID diag(xt ) 0D×D ID ] xt ∈ {0,1}m αt [0 : D] = wt Zt = x 𝖳 t , Rt = xt , Tt = ID , xt ∈ {0,1}D -
 105 ఏҊํࡦ: LGSS bandits 1. ֤  ʹ͓͍ͯɺঢ়ଶ 
ͷฏۉ  ͱڞࢄߦྻ  Λਪఆ͢Δ 2. ਪఆΛ༻͍ͨଟมྔਖ਼نʹै͏ཚΛ֤ͷύϥϝʔλ  ͱͯ͠ಘΔ 3. ύϥϝʔλ  ͱίϯςΩετใ  ͷੵ͕࠷େ͖͍Λબఆ͢Δ l α(l) t ( = w(l) t ) μ(l) t Σ(l) t ˜ w(l) t ˜ w(l) t xt  l* t = argmaxl=1,L (x⊤ t ˜ w(l) t , ˜ w(l) t ∼ 𝒩 D (μ(l) t , Σ(l) t )) • ઢܗΧϧϚϯϑΟϧλͷঢ়ଶਪఆΛ༻͍ͨ֬Ұக๏ -
 106 ઢܗΧϧϚϯϑΟϧλʹΑΔͷঢ়ଶਪఆʢ࠶ܝʣ Ұظઌͷঢ়ଶ༧ଌ Ұظઌͷ؍ଌ༧ଌ ؍ଌͷޡࠩ ϑΟϧλϦϯά ઢܗΧϧϚϯϑΟϧλʹ͓͚Δঢ়ଶߋ৽ͷαΠΫϧ μt+1 =
Tμt|t Σt+1 = TΣt|t T 𝖳 + RQR 𝖳 ̂ yt = Zμt vt = yt − ̂ yt Ft = ZΣt Z 𝖳 + H μt|t = μt + Gvt = μt + (Σt Z 𝖳 F−1 t )vt Σt|t = Σt − GFt G 𝖳 t = t + 1 • Ұظઌ༧ଌͱ؍ଌ݁ՌʹΑΔϑΟϧλϦϯάΛ܁Γฦͯ͠ঢ়ଶΛܧଓతʹਪఆ • ঢ়ଶ  ฏۉ  ͱͦͷڞࢄߦྻ  Ͱදݱ͢Δ͜ͱ͕Ͱ͖Δ α μ Σ ࢄڞࢄΛখ͘͞ ࢄڞࢄΛେ͖͘ -
 107 Ұظઌͷঢ়ଶ༧ଌ ϑΟϧλϦϯά μt+1 = Tμt|t Σt+1 = TΣt|t
T 𝖳 + RQR 𝖳 μt|t = μt Σt|t = Σt t = t + 1 ࢄڞࢄΛେ͖͘ ܽଌΛ༻͍ͨԾతͳ୳ࡧ • ઢܗΧϧϚϯϑΟϧλͰ؍ଌ͕ಘΒΕͳ͍߹ܽଌͱͯ͠ѻ͑Δ • ͜ͷܽଌॲཧΛબఆ͞Εͳ͔ͬͨʹର͢Δߋ৽ૢ࡞ͱͯ͠औΓೖΕΔ ઢܗΧϧϚϯϑΟϧλʹ͓͚ΔܽଌॲཧͷαΠΫϧ • ظ͞ΕΔޮՌ • ֬Ұக๏ͷΈʹΑΓબఆػձͷগͳ͍ʹର͢Δ୳ࡧ͕ଅਐ͞ΕΔ -
• ࣮ࡍͷECαΠτ͔Β࠾औͨ͠4ͭͷਪનख๏ͷΧςΰϦ͝ͱͷΫϦοΫ ͷਪҠ࣮σʔλΛ༻͍ͯఏҊγεςϜͷ༗ޮੑΛධՁ͢Δ • 4ষͱಉҰͷσʔλ͕ͩɺධՁ࣌ؒͷ੍ݶʹΑΓରظؒΛॖͨ͠ • 2019/6/20ʙʮ8/4·Ͱͷ225ສճʯ͔Βʮ7/22·Ͱͷ149ສճʯͷਪ નσʔλ  109
ධՁσʔλͱਪનख๏ 0 100 200 300 400 500 600 Hours 0.05 0.10 0.15 Click-through rate Browsing path Demographic LLR Similar image ࠷ߴ͍ΫϦοΫͷΓସΘΓɻ ຊظؒʹ͓͍ͯɺਪનख๏ͷయܕతͳ ಛ͕ݱΕ͍ͯΔɻ -
• બͨ͠ਪનख๏͔ΒಘΒΕΔΫϦοΫͷγϛϡϨʔγϣϯ • ํࡦʹΑΓબ͞Εͨਪનख๏ɺઃఆͨ͠ΫϦοΫͷϕϧψʔΠʹै͍ਪ ન݁Ռ͕ΫϦοΫ͞ΕΔͷͱ͢Δ • ֤ਪનख๏ΧςΰϦͱ͍͠18࣍ݩͷύϥϝʔλ  Λ࣋ͭ •
ΫϦοΫ  ͱίϯςΩετใ  ͷੵͰܭࢉ͞ΕΔ • ίϯςΩετใ  ɺ࣌  ʹ͓͍ͯར༻ऀ͕Ӿཡ͍ͯ͠ΔΧςΰϦͷ1-hot ϕΫτϧͱͯ͠දݱ͞ΕΔ • ࣮ࡍͷਪનγεςϜͷڍಈͱ߹ΘͤΔͨΊɺใु1࣌ؒ͝ͱʹ·ͱΊͯϑΟʔυ όοΫ͞ΕΔͷͱ͢Δ ˜ w(l) t ˜ w(l) t xt xt t  110 ධՁํ๏ʢ1/2ʣʢ4ষͱಉҰʣ -
 111 ର ํࡦ උߟ ඇఆৗ ͔ͭ จ຺͖ LGSS banditsʢఏҊํࡦʣ
ঢ়ଶۭؒϞσϧܕ TVTP [42] ঢ়ଶۭؒϞσϧܕɻධՁ࣌ؒʢ24࣌ؒʣʹύϥϝʔλௐ ͱ࣮ݧͷ࣮ߦ͕ྃ͠ͳ͔ͬͨͨΊɺҰ෦ධՁͷΈ࣮ࢪ AdTS [43] มԽݕग़ܕɻׂͨ͠ܥྻͷฏۉͱڞࢄߦྻ͔ΒͷϚϋϥϊ Ϗεڑͷܥྻʹରͯ͠ϒʔτετϥοϓ๏ͰมԽݕग़ Decay LinUCB [68] ٫ܕ dLinUCB [70] มԽݕग़ܕɻใु༧ଌͷޡࠩΛݕग़͠ɺ৽ͨͳใु༻ͷ όϯσΟοτϞσϧΛՃɻᮢͷԼͰఆظతͳॳظԽ͋Γ จ຺͖ LTS [47] LinUCB [33] Neural Linear (+ LTS) [76] χϡʔϥϧωοτϫʔΫͷֶशִؒධՁ࣌ؒͷ੍ݶͱมԽ ͷैੑͷ؍͔ΒࢦؔతͳִؒΛ࠾༻ ධՁํ๏ʢ2/2ʣ • γϛϡϨʔγϣϯʹ༻͍Δํࡦ -
ධՁ݁Ռ: Ԡ࣌ؒͷӨڹ  112 0.0 0.2 0.4 ms 0.48 0.19
0.04 0.24 0.29 0.04 0.27 Select per 1 time 10°1 100 sec (Log) 1.49 0.04 0.06 0.09 0.04 0.05 1.82 Update per 2665 times adaptive thompson sampling decay linucb dynamic linucb LGSS bandits linear thompson sampling linucb neural linear Elapsed time for 2665 times for 18 dimension 4 arms • ͷબఆʹ͍ͭͯLGSS bandits0.24ϛϦඵ • ͯ͢ͷଌఆ 0.5 ϛϦඵະຬͰ͋Γɺ ॏେͳӨڹΛ༩͑ͳ͍ेͳύϑΥʔϚϯ εͰ͋ΔͱஅͰ͖Δɻ • ධՁͷߋ৽ʹ͍ͭͯ୯Ґ࣌ؒʢ1࣌ؒʣͱൺֱ ͯ͠ेʹ͍ • TVTPAdTSɺNeural LinearͰࢼߦճ ͷ૿Ճʹରͯ͠εέʔϥϏϦςΟ͕ͳ͍ ͷબఆͱධՁͷߋ৽ͷܦա࣌ؒͷଌఆ ͳ͓ɺTVTPʢཻࢠ5ʣͰɺ ͷબఆ1.32ϛϦඵɺ ධՁߋ৽13.6ඵ -
ධՁ݁Ռ: ඇఆৗ͔ͭจ຺͖ͷઃఆʹର͢Δ༗ޮੑ  113 0 5000 10000 15000 20000 Cumulative
regret adaptive thompson sampling decay linucb dynamic linucb LGSS bandits linear thompson sampling linucb neural linear select best arm at first select random arm 0 100 200 300 400 500 600 0.0 0.2 0.4 0.6 0.8 1.0 Best arm rate ֤ํࡦʹ͓͚ΔྦྷੵϦάϨοτͱ࠷దͷબఆׂ߹ • ఏҊํࡦLGSS bandits࠷ߴ͍ྦྷੵใु ʢ࠷গͳ͍ϦάϨοτʣΛୡ • มԽݕग़ܕͷํࡦAdTS͕࣍ • → ͜ΕΒʹ͍࣍ͰɺఆৗͳํࡦͷLTSɺLinUCBͷ݁ Ռ͕ྑ͔ͬͨ͜ͱ͔ΒɺຊධՁͰɺఆৗ࣌Ͱͷ୳ ࡧΛ͑ͨػձଛࣦͷݮॏཁͰ͋ͬͨɻ LGSS banditsͱAdTSɺ࠷దͷΓସ͚͑࣌ͩͰ ͳ͘ɺఆৗظؒதػձଛࣦΛݮͰ͖ͨɻ 0 100 200 300 400 500 600 Hours 0.05 0.10 0.15 Click-through rate Browsing path Demographic LLR Similar image ਪનख๏͝ͱͷΫϦοΫͷਪҠ • શͯͷΧςΰϦΛ௨ͯ͠ظटͷ࠷ద ΛҰ؏ͯ͠༻͍Δ߹ͱൺֱͯ͠ɺఏҊํ ࡦʹΑͬͯྦྷੵใु͕6.5%૿Ճ -
ධՁ݁Ռ: ඇఆৗ͔ͭจ຺͖ͷઃఆʹର͢Δ༗ޮੑ  114 0 5000 10000 15000 20000 Cumulative
regret adaptive thompson sampling decay linucb dynamic linucb LGSS bandits linear thompson sampling linucb neural linear select best arm at first select random arm 0 100 200 300 400 500 600 0.0 0.2 0.4 0.6 0.8 1.0 Best arm rate ֤ํࡦʹ͓͚ΔྦྷੵϦάϨοτͱ࠷దͷબఆׂ߹ • มԽݕग़ܕͷํࡦdLinUCBɺมԽݕग़ޙ ͷैੑೳʹ༏Ε͍͕ͯͨɺલͷϦά Ϩοτ૿ՃʹΑΓɺఆৗํࡦΑΓྦྷੵϦά Ϩοτ͕૿Ճͨ͠ • → ఆظతͳόϯσΟοτϞσϧՃͱ࠶ධՁʹىҼ • ٫ܕͷํࡦDecay LinUCBɺίϯςΩ ετใͷཁҼͷ؍ଌͷภΓʹΑΓɺ׆ ༻ͱ୳ࡧͷཱ͕྆ࠔͱͳͬͨ 0 100 200 300 400 500 600 Hours 0.05 0.10 0.15 Click-through rate Browsing path Demographic LLR Similar image ਪનख๏͝ͱͷΫϦοΫͷਪҠ -
ධՁ݁Ռ: ඇఆৗ͔ͭจ຺͖ͷઃఆʹର͢Δ༗ޮੑ  115 • ఏҊํࡦͰɺԾతͳ୳ࡧʹΑΓɺมԽ ݕग़ޙͷैΛ࣮ݱͭͭ͠ɺఆৗ࣌ͷػձ ଛࣦͷݮ͕࣮ݱͰ͖ͨ • →
Ծత୳ࡧ͕ͳ͍ɺAdTSdLinUCBͰɺఆৗ͠ ͘ඇఆৗͷ͍ͣΕ͔ʹ͔͠ରԠͰ͖͍ͯͳ͍ͨΊɺఏ Ҋํࡦʹൺͯ૯߹తͳੑೳͰѱԽͨ͠ 0.00 0.05 0.10 0.15 0.20 Change of click-through rate (Stationery) Browsing path Demographic LLR Similar image 0 100 200 300 400 500 600 0.0 0.2 0.4 0.6 0.8 1.0 Best arm rate (Stationery) adaptive thompson sampling dynamic linucb LGSS bandits linear thompson sampling 0 50 100 150 200 0.0 0.2 0.4 0.6 0.8 1.0 Period 1 240 260 280 300 Period 2 300 400 500 600 Period 3 0.00 0.05 0.10 0.15 0.20 Change of click-through rate (Plants) Browsing path Demographic LLR Similar image 0 100 200 300 400 500 600 0.0 0.2 0.4 0.6 0.8 1.0 Best arm rate (Plants) adaptive thompson sampling dynamic linucb LGSS bandits linear thompson sampling ֤ํࡦʹ͓͚ΔྦྷੵϦάϨοτͱ࠷దͷબఆׂ߹ Stationary Plants -
• ؍ଌࡶԻͷࢄ  ΧϧϚϯϑΟϧλͷ࠷ਪఆ݁ՌΑΓେ͖͍͕ɺঢ়ଶͷ ҆ఆͨ͠ਪఆʹ༗ޮͰ͋ͬͨɻ࣮ӡ༻Ͱͷల։࣌ʹཹҙ͍ͨ͠ • → ܽଌॲཧʹΑΓঢ়ଶͷڞࢄߦྻͷ͕ߴ͘อͨΕɺঢ়ଶ͕େ͖͘ิ ਖ਼͞ΕΔͨΊɻ •
ఏҊํࡦͰɺઢܗΨεঢ়ଶۭؒϞσϧΛԾఆ͢Δ͜ͱͰ࣮ߦ࣌ؒͷॖΛ ਤ͕ͬͨɺ͜ͷԾఆ͕ຬͨ͞Εͳ͍ඇઢܗͳจ຺͖ઃఆʹ͓͍ͯɺਪఆਫ਼ ͷԼͱػձଛࣦͷ૿Ճ͕ݒ೦͞ΕΔɻ σ2 ϵ  116 ఏҊʹؔ͢Δٞ -
ຊݚڀͷ၆ᛌਤ  120 "EBQUBCMF*OGPSNBUJPOTZTUFNTGPS%ZOBNJDBOE&WPMWJOH&OWJSPONFOU Context Non stationarity Context Non stationarity
Online performance Context (Non-linear) Non stationarity Online performance Multi-armed bandit polices 3ষ 4ষ 5ষ 6ষ -
 121 తͱఏҊͷࠎࢠ • ػցֶशϞσϧͷ༏ྼɺจ຺࣌ؒͷܦա͚ͩͰͳ͘ɺԠੑɾैੑʹΑͬ ͯࠨӈ͞ΕΔɻͳ͓ɺจ຺ͱใुͷؔੑ໌Β͔Ͱͳ͍ͱԾఆ͢Δ • ޮՌతͳϞσϧΛػձଛࣦ͕ͳ͍Α͏จ຺ͱ࣌ؒͷܦաʹԠ͍͚ͯͨ͡ ͍ɻͦΕΛ࣮ߦ࣌ؒɾίϯςΩετใઃܭͷ؍Ͱҙࣝͤͣ͞ʹߦ͍͍ͨ •
ෳࡶͳจ຺ͱ࣌ؒͷܦաʹԠͨ͡બͷ࠷దԽΛɺඇఆৗ͔ͭඇઢܗͳจ຺ ͖ଟόϯσΟοτͱΈͳ͠ɺਝʹ͜ΕΛղ͘ɺॏΈ͖ஞ࣍Ψεա ఔճؼΛ༻͍ͨํࡦΛఏҊ -
ίϯςΩετใ  ͱใु  ͷؒͷඇઢܗੑΛѻ͏ํࣜͷݕ౼ 1. ֶशࡁΈϞσϧʹΑΔඇઢܗม͞Εͨ  Λೖྗͱ͢Δઢܗͳํࡦ →
ҟͳΔλεΫྖҬͰಘͨ݁Ռ͕ɺద༻ઌͷใुਪఆʹ༗ޮ͔ෆ໌ 2. ඇઢܗํఔࣜΛಋೖͨ͠ঢ়ଶۭؒϞσϧΛ༻͍ͨํࡦ[42,74] → ঢ়ଶํఔࣜͱ؍ଌํఔࣜΛ໌ࣔత͔ͭݸผʹઃܭͯ͠༩͑Δ͜ͱࠔ 3. ඇઢܗճؼϞσϧΛ༻͍ͨํࡦ[76~82] → ෆ࣮֬ੑͱ࠶ؼతֶशػߏΛఏڙ͠ͳ͍NNํࡦόϯσΟοτʹదͣ͞ xt rt ˜ xt  123 จ຺͖ଟόϯσΟοτͱඇઢܗੑ -
ඇઢܗճؼϞσϧͱͯ͠ΨεաఔճؼʢGPʣΛ༻͍Δํࡦͷ՝ • GPɺσʔλ͔Βɺ͋ΔؔͷΛ֬աఔͱͯ͠ٻΊΔ • ଟͷجఈؔ  ʹΑΔߴ͍දݱྗ • Χʔωϧ๏ʹΑͬͯجఈؔͷ໌͕ࣔෆཁ ϕ:
ℝD → ℝ  124 จ຺͖ଟόϯσΟοτͱඇઢܗੑ k(p, q) ≜ (p 𝖳 q + c)m, m = 2, p, q ∈ ℝ2 ͜ͷଟ߲ࣜΧʔωϧؔͭͷجఈؔʹΑΔมͱੵΛऔͬͨ݁ՌͱҰக -
ඇઢܗճؼϞσϧͱͯ͠ΨεաఔճؼʢGPʣΛ༻͍Δํࡦͷ՝ • GPɺσʔλ͔Βɺ͋ΔؔͷΛ֬աఔͱͯ͠ٻΊΔ • ଟͷجఈؔ  ʹΑΔߴ͍දݱྗ • Χʔωϧ๏ʹΑͬͯجఈؔͷ໌͕ࣔෆཁ •
ඇઢܗੑͱෆ࣮֬ੑΛѻ͑ΔͨΊଟόϯσΟοτͷੑ͕ߴ͍ [88,89,90] • → Χʔωϧߦྻ  ͱͦͷٯߦྻͷܭࢉΛؚΉͨΊɺ ɹ ֶशσʔλ  ʹରֶͯ͠श͕࣌ؒࢦؔతʹ૿Ճ ϕ: ℝD → ℝ K ∈ ℝN×N N  125 จ຺͖ଟόϯσΟοτͱඇઢܗੑ  K−1  K ∈ ℝN×N  K−1 →  k(xi , xj ) -
 126 Weighted GP-UCB [95] [Y. Deng 2022] • ٫ܕͷඇఆৗ͔ͭඇઢܗͳจ຺͖ํࡦ
• ཚԽϑʔϦΤಛΛ༻͍ͯɺΨεաఔճؼͷ༧ଌͷܭࢉΛ  ͱͳΔ  ࣍ݩͷઢܗճؼͷܗࣜͰղ͘ • ೋͭͷॏΈߦྻʹΑͬͯඇఆৗੑΛॊೈʹѻ͏ • → σʔλ  ͷ૿Ճʹର͢Δֶश࣌ؒͷࢦత૿ՃΛճආ͢Δ R ⋘ N R N  K  ZZ⊤  ≃  K ∈ ℝN×N  Z ∈ ℝN×R  Z⊤Z  Z⊤Z ∈ ℝR×R  ⋙ -
 127 • ٫ܕͷඇఆৗ͔ͭඇઢܗͳจ຺͖ํࡦ • ཚԽϑʔϦΤಛΛ༻͍ͯɺΨεաఔճؼͷ༧ଌͷܭࢉΛ  ͱͳΔ  ࣍ݩͷઢܗճؼͷܗࣜͰղ͘
• → σʔλ  ͷ૿Ճʹର͢Δֶश࣌ؒͷࢦత૿ՃΛճආ͢Δ • ࠶ؼతֶशػߏఏڙ͞Εͳ͍ • → σʔλͷ૿Ճʹର͢Δֶश࣌ؒͷ૿ՃΛ෦తʹ͔͠ղܾ͠ͳ͍ R ⋘ N R N Weighted GP-UCB [95] [Y. Deng 2022]  K  ZZ⊤  ≃  K ∈ ℝN×N  Z ∈ ℝN×R  Z⊤Z  Z⊤Z ∈ ℝR×R  ⋙ -
 128 NysKRLS [41] [T. Zhang 2020] • ඇఆৗͳΧʔωϧஞ࣍࠷খೋ๏ •
NyströmۙࣅΛ༻͍ͯɺΨεաఔճؼͷ༧ଌͷܭࢉΛ  ͱͳΔ  ࣍ݩͷઢܗճؼͷܗࣜͰղ͘ • ஞ࣍࠷খೋ๏Λద༻͢Δ • ٫Λಋೖ͢Δ͜ͱͰඇఆৗੑΛѻ͏ • → σʔλ  ͷ૿Ճʹର͢Δஞֶ࣍शͷߴԽΛ࣮ݱ͢Δ R ⋘ N R N  (Z⊤ΓZ + γMΛ)−1 γ γ z xN -
 129 • ඇఆৗͳΧʔωϧஞ࣍࠷খೋ๏ • NyströmۙࣅΛ༻͍ͯɺΨεաఔճؼͷ༧ଌͷܭࢉΛ  ͱͳΔ  ࣍ݩͷઢܗճؼͷܗࣜͰղ͘
• → σʔλ  ͷ૿Ճʹର͢Δஞֶ࣍शͷߴԽΛ࣮ݱ͢Δ • ਖ਼ଇԽ߲ʹରͯ͠٫ॲཧ͕܁Γฦ͠ద༻͞ΕΔ • → ਖ਼ଇԽޮՌ͕ݮ͠աֶशʹΑΔਪఆਫ਼ͷྼԽ͕ੜͯ͡͠·͏ • ରࡦख๏Ͱ͋ͬͯ௨ৗͱൺͯܭࢉྔͷ૿Ճ࠷దʹऩଋ͠ͳ͍ [98, 46] R ⋘ N R N NysKRLS [41] [T. Zhang 2020]  (Z⊤ΓZ + γMΛ)−1 γ γ z xN -
 K  131 ཚԽϑʔϦΤಛΛ༻͍ͨΨεաఔճؼʢ1/2ʣ  K ∈ ℝN×N 
k(xi , xj ) • ཚԽϑʔϦΤಛɺ͋Δ֬  ͔Βͷ  ݸͷαϯϓϧΛ༻ ͍ͯΧʔωϧؔ  Λ  ͱۙࣅ͢Δख๏ • ͳ͓ɺ  • ֬  ΧʔωϧؔͷछྨʹΑܾͬͯ·Δ p(ω) R′  = R/2 k(xi , xj ) ̂ k(xi , xj ) = z(xi )⊤z(xj ) z(xi ) = 1/R′  (cos(ω⊤ 1 xi ), sin(ω⊤ 1 xi ), …, cos(ω⊤ R′  xi ), sin(ω⊤ R′  xi )) p(ω) ݸผͷΧʔωϧؔʹରͯ͠ܭࢉίετ͕ ૿Ճ͢Δ͕ɺجఈؔͷద༻ͱੵͷࠞ߹ૢ ࡞ͷ݁ՌΛղͨ͠ͱݟΔ͜ͱ͕Ͱ͖Δ → ύϥϝʔλͷ࣍ݩΛ  ࣍ݩʹݻఆͰ͖Δ R ϕ(x) = (ϕ1 (x), …, ϕ∞ (x))⊤ ∈ ℝ∞ ϕ(xi )⊤ϕ(xj ) = k(xi , xj ) ≈ z(xi )⊤z(xj ) -
 132 ཚԽϑʔϦΤಛΛ༻͍ͨΨεաఔճؼʢ2/2ʣ  Λ = σ−2 w IR •
ద༻ʹΑΓɺݻఆͷ  ࣍ݩͷϕΠζઢܗճؼϞσϧ૬ͷܗ͕ࣜಘΒΕΔ • → ઢܗͳஞ࣍࠷খೋ๏ͷద༻͕Մೳ R  K  ZZ⊤  ≃  K ∈ ℝN×N  Z ∈ ℝN×R  Z⊤Z  Z⊤Z ∈ ℝR×R  ⋙ k(xi , xj ) ≃ z(xi )⊤z(xj ) • ݩͷΨεաఔճؼϞσϧͷ༧ଌͱͷൺֱ -
• ΨεաఔճؼϞσϧͷ٫ػߏͷಋೖ • ؍ଌޡࠩͷै͏֬ͷࢄ͕ɺաڈʹ ḪΔ΄Ͳେ͖͘ͳΔΑ͏ఆࣜԽ  133 ॏΈ͖ஞ࣍ΨεաఔճؼϞσϧʢ1/3ʣ  Γ−1
= diag ((g(n))1≤n≤N) ∈ ℝN×N • ॏΈ͖Ψεաఔճؼͷ༧ଌ ಛʹ  ͷͱ͖ɺ  ͱ٫ ͖࠷খೋ๏ʹΑΔઢܗճؼͷ ఆࣜԽ͕Ұக σ2 ϵ = 1 ̂ μ′  ′  -
• ॏΈ͖ΨεաఔճؼϞσϧͷஞ࣍࠷খೋ๏ͷద༻ • ٯߦྻܭࢉͷճආʹΑΔܭࢉྔͷݮͷͨΊ  ͱ  Λ݁Ϳߋ৽ࣜΛಘΔ • ஞ࣍ܭࢉ
 ͷ૿Ճʹ͍ɺ  ͷޡ͕ࠩྦྷੵ͠ɺਖ਼ଇԽޮՌ͕ݮগ • → ޡࠩΛิਖ਼͢Δʹɺஞ࣍࠷খೋ๏ͷద༻͕ෆՄೳͱͳΔ PN PN+1 N (γN − 1)Λ  134 ॏΈ͖ஞ࣍ΨεաఔճؼϞσϧʢ2/3ʣ  P ˎ  ·ͨ  ͷ߹ʹ߲֘ফڈ γ = 1 Λ = diag(0) -
 135 =  Z⊤ΓZ + Λ z ⋮ xN
x1 xN−1  (Z⊤ΓZ + Λ)−1 Inv  (Z⊤ΓZ + γ0Λ)−1 γ γ z ⋮ xN x1 xN−1 ޡࠩิਖ਼ 1. ɹͷٯߦྻͷܭࢉ 2. ɹɹͱͷޡࠩͷղফ 3. ɹɹͷٯߦྻͷܭࢉ −1 −1 ॏΈ͖ஞ࣍ΨεաఔճؼϞσϧʢ3/3ʣ • ॏΈ͖ΨεաఔճؼϞσϧͷஞ࣍࠷খೋ๏ͷద༻ • ޡࠩͷൃੜճΛѻ͏  Λಋೖͨ͠࠶ ఆࣜԽʹΑΓɺ࠷খೋ๏ͱޡࠩิਖ਼ Λஞ࣍Խ • ఏҊख๏ͰɺਪఆޡࠩΛڐ༰্ͨ͠ ͰɺຖճͰͳ͘ҙͷִؒͰิਖ਼ M ͜ͷૢ࡞  ߦྻʹର͢ΔٯߦྻܭࢉΛ2ճؚΉ ͨΊܭࢉίετ͕ߴ͍ɻ͕ͨͬͯ͠ɺڐ༰Ͱ͖Δਪ ఆਫ਼Λ౿·͑ɺҙͷִؒͰૢ࡞Λ࣮ߦ͢Δɻ ˎޡࠩิਖ਼ͷΛܾΊΔͷաڈͷֶशσʔλͰ ͳ͘Ͱ͋ΔͨΊɺ͜ͷޡࠩิਖ਼๏࠶ؼతֶशͰ͋ Δ R × R -
 136 ॏΈ͖ஞ࣍ΨεաఔճؼϞσϧΛ༻͍ͨํࡦ z P N−2,M−2 Q
N−2,M−2 x N−1 y N−1 (M = 0) γ P N−1,M−1 Q N−1,M−1 z x N y N γ P N,M Q N,M z x * p(y N−1 ∣ x N−1 , X, y) p(y N ∣ x N , X, y) p(y * ∣ x * , X, y) (M = 0) ᶃ ͝ͱʹॏΈ͖ஞ࣍ΨεաఔճؼʢRW-GPRʣ ϞσϧΛ༻ҙ ᶄ ͝ͱͷRW-GPRϞσϧͷ༧ଌͷฏۉͱࢄύϥ ɹ ϝʔλΛ༻͍ͨUCB1ํࡦʹΑͬͯΛબఆ ᶅ બ͔ͨ͠ΒͷใुΛ༻͍ͯRW-GPRϞσϧΛ٫ ɹ ͖ͷ࠶ؼֶशʹΑͬͯߋ৽ ᶅ’ ִؒ  ͝ͱʹޡࠩΛิਖ਼ τ -
ճؼϞσϧͷੑೳ͓Αͼޡࠩͱิਖ਼๏ͷಛੑධՁ • ඇఆৗ͔ͭඇઢܗͳճؼͱͯ͠2ͭͷඇઢܗ ؔΛมԽલޙʢ  ͱ  ʣͱݟཱͯσʔλΛੜ • ൚ԽੑೳͷධՁͷͨΊֶशσʔλൣғΛ
 ͱ্ͨ͠Ͱɺൣғ֎ͷ  ·Ͱ༧ଌͨ͠ • ఏҊճؼϞσϧʢ  ʣ৽͍͠ํͷσʔλʹ ্ख͘ద߹ͨ͠ fA fB [−3,3] [−4,4] M = 0  138 ճؼϞσϧͷੳ °4 °3 °2 °1 0 1 2 3 4 x °4 °2 0 2 4 y Predictive distribution (ˆ µ00 and 1æ confidence based on ˆ ß00) yA fA (x) yB fB (x) ˆ µ00 1æ confidence -
ճؼϞσϧͷੑೳ͓Αͼޡࠩͱิਖ਼๏ͷಛੑධՁ • ༧ଌͷฏۉͱࢄύϥϝʔλͷ࠷దͱ࠶ ؼతֶशͷࠩɺ  ͷͱ͖Ұக͠ɺਖ਼͘͠ ػೳ͢Δ͜ͱΛ֬ೝͰ͖ͨ •  ͕େ͖͍ͱ͖͕ࠩΓɺֶशσʔλൣғ֎
ͰಛʹݦஶʹͳΔ • → ޡࠩͷੵ͕ճؼϞσϧʹ͓͚ΔॏΈύϥϝʔλͷࢄʹ૬ͳେ͖ ͞ΛԾఆ͢Δ͜ͱʹ͍ͨ͠Ίɻ࣮ࡍʹɺ൚ԽੑೳͷѱԽ֬ೝͨ͠ɻ M = 0 M  139 ճؼϞσϧͷੳ °0.10 °0.05 0.00 0.05 0.10 Error of ˆ µ00 Estimation error of ˆ µ00 for each M M=0 M=200 M=400 M=600 °4 °3 °2 °1 0 1 2 3 4 x 0.0000 0.0005 0.0010 0.0015 0.0020 Error of ˆ ß00 Estimation error of ˆ ß00 for each M M=0 M=200 M=400 M=600 -
ඇఆৗ͔ͭඇઢܗͳจ຺͖ଟόϯσΟοτͷγϛϡϨʔγϣϯ • Wheel bandits[71]Λ֦ுͨ͠ɺඇఆৗWheel banditsͰධՁ • ͷใुਖ਼ن  ʹै͏͕ɺฏۉ 
ίϯςΩετใ  ʹରͯ͠ԼਤͷΑ͏ʹରԠ͢Δ • ଳͷஔ࣌ؒਐలʹରͯ͠ࠨճΓͰҠಈ͢Δʢ4000ճͷࢼߦதʹ1ճసʣ 𝒩 (μ, σ2) μ xt = (xt,d )1≤d≤2  140 ํࡦͷධՁख๏  μ1 = 0.1,μ2 = 0.0,μ3 = 1.0,σ2 = 0.01,δ = 0.8,ρ = 4000  ͷใुͷฏۉίϯςΩε τใͱ࣌ؒʹؔΘΒͣৗʹಉ͡ a(1) ฏۉͷؔ  Ͱ͋ ΓɺଳͷൣғͰରԠ͢ΔΛɺ ͦΕҎ֎Ͱ  Λબ͢Δ͜ ͱ͕ظใुͷ࠷େԽʹͭͳ͕Δ μ2 < μ1 ≪ μ3 a(1) -
 141 ํࡦͷධՁख๏ ඇઢܗ ඇఆৗ ࠶ؼతֶश ํࡦ ऍ ✓ ✓
✓ RW-GPB (Proposal) ޡࠩิਖ਼ͷޮՌΛൺֱ͢ΔͨΊෳͷิਖ਼ִؒτͰධՁ ✓ ✓ GP+UCB (Weighted, RFF) [95] - ࠷ઌํࡦ - ཚԽϑʔϦΤಛͷ࠾༻ ✓ ✓ GP+UCB (Weighted) [95] ✓ ✓ GP+UCB (Sliding Window) [93] ✓ ✓ GP+UCB (Restarting) [93] ✓ GP+UCB [88~90] ✓ ✓ Decay LinUCB [68] ֤ํࡦͷճؼϞσϧͷ༗ޮੑ͕༰қʹൺֱͰ͖ΔΑ͏ɺ୳ࡧεέʔϦϯά߲  ʹἧ͑ͨ β = 1 • γϛϡϨʔγϣϯʹ༻͍Δํࡦ -
750 800 850 900 950 Cumulative rewards 102 103 Trials
per second ø = 1600 ø = 800 ø = 400 ø = 100 ø = 1 ø = 40 ø = 4 Cumulative rewards - Trials per second trade-oÆ RW-GPB GP+UCB (Sliding Window) GP+UCB (Weighted) GP+UCB (Weighted, RFF) ྦྷੵใुͱ࣮ߦ࣌ؒͷτϨʔυΦϑͷൺֱ • ఏҊํࡦ͕ɺ࠷ઌํࡦʹରͯ͠ߴ͍ྦྷੵใुͱ ͍࣮ߦ࣌ؒΛୡͨ͠ • GP+UCBʢWeightedʣྦྷੵใु͕࠷ߴ͘ ࠷࣮ߦ͕͍࣌ؒ • ۙࣅਫ਼ͷ্͕ํࡦʹॏཁ • ଞͷඇఆৗํࣜ୳ࡧ͕ຫੑతʹ૿Ճ • ઢܗͳํࡦਖ਼͘͠ਪఆͰ͖ͳ͔ͬͨ  142 ํࡦͷධՁ݁Ռ -
࣮ߦ࣌ؒͷੳͱޡࠩิਖ਼๏ͷ༗ޮੑ • ఏҊํࡦɺ࠶ؼతֶशʹΑ࣮ͬͯߦ࣌ؒ ΛҰఆʹอͭ • RFFΛར༻͠ͳ͍ํࡦɺࢦతʹ࣮ߦ࣌ؒ ͕૿Ճɺར༻͢Δ߹ֶश͕࣌ؒ૿Ճ • ྦྷੵޡࠩͱͦͷิਖ਼ճ࣮ߦ࣌ؒΑΓ ྦྷੵใुͷӨڹ͕େ͖͍
• ੵۃతͳޡࠩิਖ਼͕༗ޮ  143 ํࡦͷධՁ݁Ռ 500 1000 1500 2000 2500 3000 3500 4000 Number of trials 0 20 40 60 80 Cumulative execution time (Sec) Cumulative execution time RW-GPB (ø = 4) GP+UCB (Sliding Window) GP+UCB (Weighted) GP+UCB (Weighted, RFF) 750 800 850 900 950 Cumulative rewards 102 103 Trials per second ø = 1600 ø = 800 ø = 400 ø = 100 ø = 1 ø = 40 ø = 4 Cumulative rewards - Trials per second trade-oÆ RW-GPB GP+UCB (Sliding Window) GP+UCB (Weighted) GP+UCB (Weighted, RFF) -
• ఏҊํࡦɺطʹͨ͡ɺ٫ܕͷํࡦʹ͓͚ΔҎԼͷ՝Λ౿ऻ͢Δɻ • i. ճؼϞσϧϕʔεͰ͋ΔͨΊঢ়ଶۭؒϞσϧͷܽଌॲཧʹجͮ͘൪ڰΘ ͤͷରࡦ͕࠾༻Ͱ͖ͳ͍ • ii. มಈͷ͕۠ؒෆنଇͳঢ়گʹ͓͚Δ٫ͷύϥϝʔλௐͷࠔ͞ •
iii. ίϯςΩετใͷཁҼ͝ͱͷ؍ଌͷภΓ͕͋Δ߹ͷiiͷқ্ • ຊධՁͰɺ͜ΕΒ͕Өڹ͠ͳ͍γϛϡϨʔγϣϯΛઃఆ͕ͨ͠ɺ࣮ӡ༻ͷ ద༻ʹ͚ɺ͜ΕΒͷରࡦ͕·ΕΔɻ • → iʹ͍ͭͯະબͷʹର͢Δ٫ૢ࡞ͷΈΛద༻͢ΔํࣜΛݕ౼த  144 ఏҊʹؔ͢Δٞ -
• ಛੑʢ1ʣͱʢ2ʣʹՃ͑ɺ༗༻ੑͷਪఆʹ͓͚Δෳࡶͳؔੑͱ͍͏ಛੑ ʢ3ʣʹରͯ͠ɺॏΈ͖ஞ࣍ΨεաఔճؼϞσϧΛ༻͍ͨํࡦΛఏҊ • ఏҊճؼϞσϧʹର͢ΔධՁͰɺඇఆৗɾඇઢܗͳճؼʹରͯ͠ɺ༧ଌ Λਫ਼ྑ͘ਪఆͰ͖Δ͜ͱΛ֬ೝ • ఏҊํࡦʹର͢ΔධՁͰɺඇఆৗ͔ͭඇઢܗͳଟόϯσΟοτʹର ͠ɺ࠷ઌํࡦͱൺֱͯ͠ྦྷੵใु0.3%վળͱ࣮ߦ࣌ؒ92%ॖΛ࣮ݱ •
ࠓޙͷݚڀ՝ͱͯ͠ɺิਖ਼ִؒΛ͡Ίͱ͢ΔదԠతͳύϥϝʔλௐػߏ ͓ΑͼܽଌΛ༻͍ͨԾత୳ࡧػߏͷ࣮ݱΛਐΊ͍ͨ  146 খ·ͱΊ -
• ຊݚڀͰɺଟ༷͔ͭܧଓతʹมԽ͢ΔڥࣗΒదԠ͢ΔใγεςϜͷ࣮ݱʹ͚ͯɺ దԠʹ͏ػձଛࣦͷղফΛଟόϯσΟοτͱΈͳͯ͠ղ͘ɺదԠతใγεςϜͱ ͦͷํࡦΛఏҊͨ͠ • ຊݚڀͰɺଟόϯσΟοτʹରͯ͠ɺ࣮༻্ͷཁͰ͋Δࡾͭͷಛੑͷߟྀ͕Մೳ ͳํࡦΛఏҊͨ͠ɻ ͜ΕΒʹର͢Δؔ࿈ݚڀɺݸผʹղܾ͕ՄೳͰ͕͋ͬͨɺઃఆͷΈ߹ΘͤʹΑͬͯੜ ͡Δ৽ͨͳ՝ʹରͯ͠ɺैདྷख๏ͷ୯७ͳΈ߹ΘͤͰेʹରԠͰ͖ͳ͔ͬͨɻ ຊݚڀͰɺ͜ͷΑ͏ͳΈ߹ΘͤʹΑͬͯੜ͡Δ՝Λಉʹର͢Δݚڀ՝ͱͯ͠
ཧ্ͨ͠Ͱɺ͜ΕΛղܾ͢Δํࡦͷྫࣔͨ͠ʹ͓͍ͯɺֶज़తͳߩݙ͕Ͱ͖ͨͱߟ͑Δ  148 ·ͱΊͱຊݚڀͷҙٛʢ1/2ʣ -
ݚڀ֓ཁ: ଟ༷͔ͭܧଓతʹมԽ͢ΔڥʹదԠ͢ΔใγεςϜ എܠͱత ՝ Ռ [1] ࡾ ༔հ, ็ ߃ݑ,
Synapse: จ຺ʹԠͯ͡ܧଓతʹਪનख๏ͷબΛ࠷దԽ͢Δਪન γεςϜ, ిࢠใ௨৴ֶձจࢽD, Vol.J103-D, No.11, pp.764-775, Nov 2020. [2] ࡾ ༔հ, ็ ߃ݑ, Synapse: จ຺ͱ࣌ؒܦաʹԠͯ͡ਪનख๏ͷબΛ࠷దԽ͢Δϝ λਪનγεςϜ, ిࢠใ௨৴ֶձจࢽD, Vol.J105-D, No.11, pp.641-652, Nov. 2022. [3] Yusuke Miyake, Tsunenori Mine, Contextual and Nonstationary Multi-armed Bandits Using the Linear Gaussian State Space Model for the Meta-Recommender System, 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp.3138-3145, Oct 2023. [4] Yusuke Miyake, Ryuji Watanabe, Tsunenori Mine, Online Nonstationary and Nonlinear Bandits with Recursive Weighted Gaussian Process, The 48th IEEE International Conference on Computers, Software, and Applications (COMPSAC 2024) (to appear) 1. ܧଓతʹબΛ࠷దԽ͢ΔใγεςϜͷઃܭ [1] ՝1ʹ͍ͭͯɺਪનγεςϜΛࡐʹɺҎԼͰఏҊ͢ΔػցֶशϞσϧͷಛੑΛߟྀͨ͠ଟόϯσΟοτͷํࡦʹΑͬͯɺࣗಈత͔ͭܧଓతʹબΛ ࠷దԽ͢ΔదԠܕใγεςϜج൫ΛઃܭɾධՁͨ͠ɻ 2. ଟ༷͔ͭܧଓతʹมԽ͢ΔڥͷదԠ [2] ՝2ʹ͍ͭͯɺཻࢠϑΟϧλΛ༻͍ͨจ຺͖͔ͭඇఆৗͳଟόϯσΟοτํࡦΛఏҊ͠ɺଟ༷͔ͭܧଓతʹมԽ͢ΔڥͷదԠྗ্Λ࣮ݱͨ͠ɻ 3. దԠͷߴԽ [3] ՝2ɾ3ʹ͍ͭͯɺઢܗΧϧϚϯϑΟϧλΛ༻͍ͨจ຺͖͔ͭඇఆৗͳଟόϯσΟοτํࡦͷఏҊ͠ɺదԠͷߴԽΛ࣮ݱͨ͠ɻ 4. ඇઢܗੑͷରԠ [4] ՝2ɾ3ɾ4ʹ͍ͭͯɺॏΈ͖ஞ࣍ΨεաఔճؼΛ༻͍ͨඇఆৗ͔ͭඇઢܗͳจ຺͖ଟόϯσΟοτํࡦΛఏҊ͠ɺඇઢܗͳઃఆʹ͓͚Δඇఆৗ ੑͷରԠͱॲཧͷ্Λ࣮ݱͨ͠ɻ 1. ࣮ڥͰͷධՁʹΑΔػձଛࣦ ैདྷͷଟόϯσΟοτͷํࡦΛ༻͍ͯϞσϧબͷ࠷దԽΛਤΔใγεςϜͰɺػցֶश ϞσϧͷಛੑΛߟྀͰ͖ͣɺػձଛࣦΛेʹ͑ΒΕͳ͍ɻ 2. จ຺࣌ؒͷܦաʹΑΔ༗༻ੑͷมಈ ػցֶशϞσϧͷ༗༻ੑɺར༻ऀγεςϜͷঢ়گʹΑͬͯɺ·ͨಉ͡ঢ়گͰ͋ͬͯ࣌ؒͷܦա ʹΑͬͯมಈ͢Δɻ 3. దԠʹ͏͕࣌ؒٴ΅͢༗༻ੑͷӨڹ จ຺࣌ؒͷܦաΛߟྀͨ͠ܧଓతͳൺֱධՁͷΈͷಋೖɺԠʹΕΛҾ͖ى͜͢ɻ 4. ༗༻ੑͷਪఆʹ͓͚Δෳࡶͳؔੑ จ຺ͱػցֶशϞσϧͷ༗༻ੑͷؒʹɺඇઢܗͳؔੑ͋ΓಘΔɻ 1. ใγεςϜͱڥมԽ ଟ༷͔ͭܧଓతʹมԽ͢ΔڥͷதͰɺใγεςϜ͕ܧଓతʹػೳ͢ΔʹɺैདྷͷਓखʹΑΔӡ༻ Ͱͳ͘ɺࣗಈԽ͞ΕͨదԠػߏͷ࣮ݱ͕՝ʹͳΔɻ 2. ڥมԽʹࣗΒదԠ͢ΔใγεςϜ దԠܕใγεςϜͷ࣮ݱʹɺσʔλ͔Βಈతʹಈ࡞Λઃܭ͢ΔػցֶशϞσϧͱͷ౷߹͕ෆՄܽͰ ͋Δ͕ɺͲͷػցֶशϞσϧ͕ਅʹޮՌతͰ͋Δ͔Λ༧ΊΔ͜ͱ͍͠ɻ 3. બͷ࠷దԽ ࣮ڥͰͷධՁʹΑΔػցֶशϞσϧͷબͰɺظతͳධՁʹΑΔػձଛࣦ࠷దͳϞσϧΛݟಀ ͢ϦεΫ͕͏ͨΊɺ͜ͷબաఔΛ࠷దԽ͠ػձଛࣦΛܰݮ͢ΔΈ͕ٻΊΒΕΔɻ બͷ࠷దԽͷ՝ దԠܕใγεςϜͷ࣮ݱʹ͚ͨબ ͷ࠷దԽ બͷ࠷దԽͷ՝ͷղܾ IUUQTJDPOTDPN