不完全情報ゲームAIを支えるゲーム理論的技術の進展

2026年度人工知能学会全国大会 (JSAI2026) の企画セッション「学習理論におけるゲーム理論のフロンティア」での発表資料です．
https://conf.ai-gakkai.or.jp/jsai2026/ks/#ks-6

Transcript

1. JSAI2026 اըηογϣϯ KS-6 ʗ 2026.6.12 Ѩ෦ ݓ೭ʢαΠόʔΤʔδΣϯτ AI Lab / Game AI labʣ ෆ׬શ৘ใήʔϜAIΛࢧ͑Δ ήʔϜཧ࿦తٕज़ͷਐల
2. 2 • ໊લ o Ѩ෦ ݓ೭ʢ͋΂ ͚Μ͠ʣ o @bakanaouji • ܦྺ o αΠόʔΤʔδΣϯτ AI Labʢ2018೥ʙʣ q γχΞϦαʔναΠΤϯςΟετ o ಉࣾ Game AI Lab ݉೚ʢ2025೥ʙʣ • ઐ໳ྖҬ o Learning in Games o ڧԽֶश o ΦϯϥΠϯֶश ࣗݾ঺հ 01 OVERVIEW
3. 3 • ڊେͳෆ׬શ৘ใήʔϜΛѻͬͨ̏ͭͷ୅දతͳࣄྫͱɺͦͷத֩ͱͳٕͬͨज़Λ঺հ ຊ೔ͷൃද 01 OVERVIEW ᶃ Libratus (2017) ೋਓରઓͷϙʔΧʔ ᶄ Pluribus (2019) ࿡ਓରઓͷϙʔΧʔ ᶅ DeepNash (2022) ೋਓϓϨΠϘʔυήʔϜ ʰStrategoʱ
4. 4 ෆ׬શ৘ใήʔϜͱ͸ʁ
5. 5 ෆ׬શ৘ใήʔϜ͸ήʔϜ໦ʢల։ܕήʔϜʣͰهड़͞ΕΔɿ • 𝑁 ∪ 𝑐 ɿϓϨΠϠʔͷू߹ o ϓϨΠϠʔ𝑐ɿϥϯμϜੑΛૢ࡞͢ΔԾ૝తͳϓϨΠϠʔ • ℎ ∈ 𝐻ɿϓϨΠϠʔͷߦಈͷཤྺ • 𝑧 ∈ 𝑍 ⊆ 𝐻ɿऴ୺ཤྺ • 𝑥ɿ৘ใू߹ • 𝐴 𝑥 ɿ৘ใू߹𝑥Ͱબ΂Δߦಈͷू߹ • 𝑢! 𝑧 ɿऴ୺ཤྺͰಘΒΕΔϓϨΠϠʔ𝑖ͷརಘ ల։ܕήʔϜ 02 INTRODUCTION ల։ܕήʔϜʢήʔϜ໦ʣ ߦಈ a ֤ϓϨΠϠʔ͕ήʔϜ໦ͷࠓݱࡏͲͷϊʔυʢཤྺʣʹ͍Δ͔͕ਖ਼֬ʹ͸෼͔Βͳ͍ CHALLENGE ߦಈ b ཤྺ0 ཤྺ1 ཤྺ3 ཤྺ4 ཤྺ3 ཤྺ4 ཤྺ2 ৘ใू߹𝑥ɿϓϨΠϠʔ͕۠ผͰ͖ͳ͍2ͭͷཤྺ
6. 6 ֤ϓϨΠϠʔ͸֬཰෼෍ʹैͬͯߦಈΛબ୒͢Δɿ • 𝜋! (⋅∣ 𝑥) ∈ Δ(𝐴(𝑥))ɿϓϨΠϠʔ𝑖ͷઓུʢ֤৘ใू߹𝑥Ͱ֤ߦಈΛબͿ֬཰ʣ • 𝜋 = (𝜋! , 𝜋"! )ɿશϓϨΠϠʔͷઓུͷ૊ • 𝜌#(ℎ)ɿઓུͷ૊𝜋ͷ΋ͱͰཤྺℎʹ౸ୡ͢Δ֬཰ ֤ϓϨΠϠʔ𝑖͸ࣗ਎ͷརಘͷظ଴஋Λ࠷େԽ͢Δ͜ͱΛ໨ࢦ͢ɿ 𝑢! (𝜋) = 7 $∈& 𝜌# (𝑧) 𝑢! (𝑧) ઓུͱظ଴རಘ 02 INTRODUCTION
7. 7 φογϡۉߧ𝜋⋆ɿ֤ϓϨΠϠʔ͕ࣗ਎ͷརಘΛ࠷େԽͰ͖͍ͯΔΑ͏ͳઓུͷ૊ɿ Exploitabilityɿ༩͑ΒΕͨઓུ𝜋͕φογϡۉߧͱͲͷఔ౓͍͔ۙΛଌΔࢦඪɿ φογϡۉߧͱ exploitability 02 INTRODUCTION 𝑢! (𝜋! ⋆, 𝜋#! ⋆ ) ≥ 𝑢! (𝜋! , 𝜋#! ⋆ ) ∀𝑖, ∀𝜋! expl(𝜋) = 1 𝑛 0 !∈ % max &! " 𝑢! 𝜋! ', 𝜋#! − 𝑢! (𝜋) ࠷ѱͷ૬खʹͲΕ͚ͩࡡऔ͞ΕΔ͔ɻexpl(𝜋) = 0 ⇔ φογϡۉߧɻ ࣗ෼͚ͩઓུΛม͑Δ͜ͱͷϝϦοτ͕୭΋ͳ͍Α͏ͳঢ়گ ಛʹೋਓྵ࿨ήʔϜʹ͓͍ͯɼφογϡۉߧΛ࠷దղͱֶͯ͠श͢Δ͜ͱ͕̍ͭͷେ͖ͳ໨త POINT ೋਓྵ࿨ήʔϜʹ͓͍ͯɺφογϡۉߧઓུ͸࠷ѱͷ૬खʹର͢Δظ଴རಘΛ࠷େԽ͢Δ THEOREM
8. 8 ෆ׬શ৘ใήʔϜ͸ͳͥ೉͍͔͠ 02 INTRODUCTION ׬શ৘ใήʔϜ Ͳͷཤྺʹ͍Δ͔͕ਖ਼֬ʹ෼͔Δ ⇛ ෦෼໦Λղ͚ͩ͘Ͱ࠷దͳઓུΛܭࢉͰ͖Δ ྫɿνΣεɺғޟɺকع ෆ׬શ৘ใήʔϜ ཤྺͷ৘ใ͕͢΂ͯ͸؍ଌͰ͖ͳ͍ ⇛ ෦෼໦ʹ੾Γग़ͯ͠ղ͘͜ͱ͕Ͱ͖ͳ͍ ྫɿϙʔΧʔɺຑ੃ɺަবήʔϜ vs
9. 9 Libratus
10. 10 • ϓϨΠϠʔ͕ೋਓͷϊʔϦϛοτɾςΩαεϗʔϧσϜʹ͓͍ͯॳͷ superhuman Λୡ੒ • ๲େͳ৘ใू߹਺ʢ໿10()(ʣΛѻ͏ͨΊʹʰࣄલʹૈ͘ղ͖ɺ࣮ઓͰਫ਼៛Խʱͱ͍͏ํ๏Λ࠾༻ Libratus ͱ͸ʁ 03 LIBRATUS ڊେͳήʔϜ ந৅Խ (abstraction) ந৅ήʔϜͷۉߧΛܭࢉ ࣮ઓதʹ౸ୡہ໘Λղ͖௚͢ ৘ใू߹਺͸໿10¹⁶¹ ࣅͨঢ়ଶɾߦಈΛ ·ͱΊͯήʔϜ໦Λѹॖ ૈ͍ઓུ ʢblueprint ઓུʣΛ࡞Δ ॏཁͳہ໘ʹ౸ୡͨ͠Β ࡉ͔͘ղ͖௚͢ 𝜋*+ 0.7 0.3 Nested Safe Subgame Solving coarse fine Brown & Sandholm, lSuperhuman AI for heads-up no-limit poker: Libratus beats top professionalsz, Science 2018
11. 11 • ࣅͨ؍ଌ΍ߦಈΛ·ͱΊͯɺήʔϜ໦Λѹॖ͢Δ • ࠃࡍձٞͰίϯϖ͕ఆظతʹ։࠵͞Ε͖ͯͨྺ࢙తͳܦҢ͔ΒɺϙʔΧʔ͸υϝΠϯ஌ࣝΛ૊Έ ࠐΜͩ abstraction ͕ඇৗʹൃల Abstraction 03 LIBRATUS K ― K ‖ Q — K – K ‖ Q — … 7 – 6 — 5 ‖ 7 ― 6 ‖ 5 — … Bucket 1 Bucket k φογϡۉߧ͕ܭࢉՄೳͳαΠζʹڊେͳήʔϜ໦ΛѹॖͰ͖Δʢͨͩ͠৘ใͷૈ͞ʹޡࠩͷݪҼʹ΋ͳΔʣ POINT
12. 12 CFR (counterfactual regret minimization) Ͱந৅Խͨ͠ήʔϜͷઓུΛֶश ந৅ήʔϜͷۉߧΛܭࢉ 03 LIBRATUS 1 ৘ใू߹͝ͱͷ regret Λܭࢉ Regret =ʮผͷߦಈΛબΜͰ͍Ε͹ͲΕ͚ͩಘ͔ͩͬͨʯ 2 regret ʹԠͯ͡ઓུΛߋ৽ ಘͩͬͨ͸ͣͷߦಈͷ֬཰Λ૿΍͢ 3 ൓෮ؒͷ࣌ؒฏۉΛऔͬͨઓུʢฏۉઓུʣΛ جຊઓུʢblueprint ઓུʣͱͯ͠ग़ྗ ݩͷήʔϜʹ͓͍ͯ΋ɺ͋Δఔ౓ڧ͍ϓϨΠ͕Ͱ͖Δ͜ͱ͕ظ଴͞ΕΔ ೋਓྵ࿨ήʔϜʹ͓͍ͯɺʰregret ͷ࣌ؒฏۉ͕0ʹऩଋ ⇛ ฏۉઓུ͕φογϡۉߧ΁ऩଋʱ THEOREM Zinkevich et al., lRegret Minimization in Games with Incomplete Informationz, NeurIPS 2007 ※ CFR ͷৄࡉ͕ؾʹͳΔํ͸ ڈ೥ͷൃදࢿྉ Λ͝ཡ͍ͩ͘͞ʂ
13. 13 • blueprint ઓུ͸͋͘·Ͱந৅ήʔϜͷղ • abstraction ͕ૈ͍ͱɺݩͷήʔϜΛϓϨΠͨ͠ͱ͖ʹ͸ΠϚΠν͔΋͠Εͳ͍⋯ o ಛʹऴ൫΍ϕοτֹ͕େ͖͍ͱ͖ʹ͸ɺΑΓଟ༷ͳߦಈΛબ΂Δ͜ͱ͕ॏཁ blueprint ઓུͷݶք 03 LIBRATUS ૈ͍ abstraction ͷ΋ͱͰऔΕΔߦಈ ॏཁͳہ໘Ͱ͸ɺଟ༷ͳߦಈΛऔΕΔඞཁ͕͋Δ 0.5x pot 1x pot all-in 1/3 pot 2/3 pot pot 1.5x pot 2x pot all-in ࣮ࡍʹϓϨΠͨ͠ͱ͖ʹॏཁͳہ໘ʹ౸ୡͨ͠৔߹ɺabstraction Λࡉ͔ͨ͘͠ઓུΛܭࢉ͠ͳ͓͍ͨ͠ʂ POINT
14. 14 • ׬શ৘ใήʔϜͰ͸ɺࠓ͍Δཤྺℎ͕ਖ਼֬ʹ෼͔Δ o ℎҎԼͷ subgame Λղ͚͹ྑ͍ • ෆ׬શ৘ใήʔϜͰ͸ɺཤྺ͕ਖ਼֬ʹ͸Θ͔Βͣɺ৘ใू߹͚͕ͩ؍ଌͰ͖Δ o φΠʔϒʹ subgame ͷղ͖௚͠Ͱ͸ɺ ಛఆͷ৘ใू߹Ͱ૬खͷظ଴རಘ্͕ঢͯ͠͠· ͏͜ͱ͕͋Δ φΠʔϒʹ subgame Λղ͘͜ͱͷ໰୊఺ 03 LIBRATUS ׬શ৘ใɿℎ ͕෼͔Δ ℎ subgame ℎ ҎԼͷ subgame Λղ͚͹ྑ͍ ෆ׬શ৘ใɿ৘ใू߹ 𝑥 ͔͠෼͔Βͳ͍ ℎ ℎ′ ৘ใू߹ 𝑥 Ͳͷཤྺ͔͕෼͔Βͳ͍
15. 15 Safe subgame solving 03 LIBRATUS 1 શһʹݟ͍͑ͯΔঢ়ଶ𝑆ͱ੔߹͢Δཤྺͷू߹Λ𝐻! ͱ͢Δ 2 𝐻! Λ૬ख͔Βݟ͍͑ͯΔঢ়گ𝑥͝ͱʹ෼ׂ͠ɺͦΕͧΕʹରͯ͠ blueprint ઓུͷ΋ͱͰͷظ଴རಘ𝑣"## $% 𝑥 Λݟੵ΋Δ 3 ֤𝑥ʹʮsubgame ʹਐΉʯ͔ʮ𝑣"## $% 𝑥 Λड͚औͬͯ߱ΓΔ (opt-out)ʯ͔Λ ૬ख͕બ΂Δ augmented subgame Λߏங͢Δ 4 augmented subgame Λղ͖ɺsubgame ಺ͷ৽͍͠ઓུ𝜋$%&ͱͯ͠ग़ྗ ૬खͷ৘ใू߹ x x opt-outʢԾ૝ߦಈʣ continueʢsubgame ΁ʣ 𝑣!"" #$ (𝑥) Λड͚औΔ ʢ͜͜Ͱऴྃʣ THEOREM ʢΔɿݟੵ΋Γޡࠩʣ opt-out ʹΑΓɺ૬ख͸֤ঢ়گ𝑥Ͱ blueprint ઓུͰಘΒΕͨ͸ͣͷظ଴རಘ𝑣122 34 𝑥 Λ͍ͭͰ΋֫ಘͰ͖Δ ⇛ ͋Δঢ়گ𝑥Ͱ૬खͷظ଴རಘΛ𝑣'(( )* 𝑥 ΑΓ΋্͛ͯ͠·͏ͱɺͦͷ෼Λଞͷঢ়گͰԼ͛ͯิঈ͢Δ͜ͱ͕Ͱ͖ͳ͍ ⇛ exploitability ΛѱԽͤ͞ͳ͍ (safe) POINT ೋਓྵ࿨ήʔϜʹ͓͍ͯɺ৽͍͠ઓུ𝝅𝐧𝐞𝐰͸𝐞𝐱𝐩𝐥(𝝅𝐧𝐞𝐰) ≤ 𝐞𝐱𝐩𝐥(𝝅𝑩𝑷) + 𝟐𝚫Λຬͨ͢
16. 16 • ॏཁͳہ໘ʹ౸ୡ͢Δͨͼʹ subgame solving Λ܁Γฦ͢͜ͱͰɺઓུΛஈ֊తʹਫ਼៛Խ Nested subgame solving 03 LIBRATUS blueprint ૈ͍શମઓུ ॏཁͳہ໘ʹ౸ୡ subgame Λղ͖௚͢ ͞Βʹਂ͍ہ໘ʹ౸ୡ ࠶౓ subgame Λղ͖௚͢ ஈ֊తʹઓུվળ … Brown & Sandholm, lSafe and Nested Subgame Solving for Imperfect-Information Gamesz, NeurIPS 2017
17. 17 Pluribus
18. 18 • ࿡ਓϓϨΠϠʔͷ no-limit ςΩαεϗʔϧσϜͰ superhuman Λୡ੒ • ઃܭࢥ૝͸ Libratus ͱಉ͡ʢࣄલʹૈ͘ղ͖ɺ࣮ઓͰਫ਼៛Խʣ Pluribus ͱ͸ʁ 04 PLURIBUS ϓ ϩ ϓ ϩ ϓ ϩ AI ϓ ϩ ϓ ϩ AI ʴ ਓؒτοϓϓϩ5ਓͰରઓ Brown & Sandholm, lSuperhuman AI for multiplayer pokerz, Science 2019 ଟਓ਺ήʔϜͰ͸ೋਓྵ࿨Ͱͷཧ࿦తอূ͕ऑ·ΔͨΊɺ࣮ઓͰ่Εʹ͍͘ઃܭΛॏࢹ POINT
19. 19 • ೋਓྵ࿨Λࢧ͍͑ͯͨཧ࿦ͷ౔୆͕ɺ3ਓҎ্Ͱ͸่ΕΔ ଟਓ਺ήʔϜͰ͸Կ͕յΕΔ͔ 04 PLURIBUS ᶃ ۉߧ͕࠷దͱ͸ݶΒͳ͍ ᶄ ܭࢉ͕ࠔ೉ ೋਓྵ࿨ 㾎 ෛ͚ͳ͍อূ 3ਓҎ্ ✗ อূͳ͠ ۉߧઓུΛ࢖ͬͯ΋େ͖͘ෛ͚ಘΔ PPAD-complete ೋਓྵ࿨Λӽ͑ͨήʔϜͰͷφογϡۉߧͷ ۙࣅܭࢉ͸Ұൠʹࠔ೉ɻ CFR ͷऩଋอূ΋ࣦΘΕΔ
20. 20 • Subgame solving ࣌ͷঢ়ଶ਺രൃΛආ͚ΔͨΊɺ subgame ΛҰఆͷਂ͞·Ͱల։͢Δ • ଧͪ੾ͬͨϊʔυʹ͓͍ͯɺ༧Ίܭࢉ͓͍ͯͨ͠𝑘 = 4ݸͷઓུ͔Β֤ϓϨΠϠʔ͕ҰͭΛબͿɿ o ௨ৗͷ blueprint ઓུ o ߱ΓΔ͜ͱʹಛԽͨ͠ blueprint ઓུ o ίʔϧ͢Δ͜ͱʹಛԽͨ͠ blueprint ઓུ o ϨΠζ͢Δ͜ͱʹಛԽͨ͠ blueprint ઓུ • બ͹Εͨઓུʹैͬͯ rollout ͠ɺϊʔυͷظ଴རಘ ΛϞϯςΧϧϩਪఆ Depth-limited search 04 PLURIBUS depth limit π₁ π₂ π₃ π₄ k = 4 Brown & Sandholm, lSuperhuman AI for multiplayer pokerz, Science 2019 ୯ҰͷઓུͰظ଴རಘΛਪఆ͠ͳ͍͜ͱͰɺಛఆͷະདྷ΁ͷա৒ద߹Λආ͚Δ POINT
21. 21 DeepNash
22. 22 • ೋਓϓϨΠϘʔυήʔϜʰStrategoʱͰ superhuman Λୡ੒ o ޓ͍ͷۨͷਖ਼ମ͕Ӆ͞Εͨ··ਐΉɺ௕ظઓུܕͷෆ׬શ৘ใήʔϜ • ޓ͍ͷࢲతͳ৘ใͷଟ͞ͱήʔϜ͕௕ظԽ͠΍͍͢ੑ࣭ͷͨΊɼsubgame ͕ґવͱͯ͠ڊେ ⇛ ϙʔΧʔܕͷ abstraction + subgame solving ͕ద༻͠ʹ͍͘ DeepNash ͱ͸ʁ 05 DEEPNASH ہॴతʹղ͖௚͢ͷͰ͸ͳ͘ɺશہ໘Ͱ࢖͑Δ NN ํࡦΛ model-free self-play RL Ͱ௚઀ֶश POINT Perolat et al., lMastering the game of Stratego with model-free multiagent reinforcement learningz, Science 2022
23. 23 • regularization policy 𝜋234͔Β཭Ε͗͢ͳ͍Α͏ʹརಘΛม׵ͨ͠ήʔϜΛ self-play RL Ͱղ͘ 𝑢 ! 234(𝑎! , 𝑎"! ; 𝜋) = 𝑢! (𝑎! , 𝑎"! ) − 𝜂log 𝜋! (𝑎! ) 𝜋 ! 234(𝑎! ) + 𝜂log 𝜋"! (𝑎"! ) 𝜋 "! 234(𝑎"! ) • े෼ʹֶशͨ͠ޙʹɺͦͷઓུΛ৽͍͠ regularization policy ʹͯ͠࠶౓ֶश R-NaD 05 DEEPNASH 1 regularization policy Λݻఆ 2 KL ਖ਼ଇԽͨ͠ใुͷ΋ͱͰ self-play RL 3 ֶशͤͨ͞ઓུΛɺ৽͍͠ regularization policy ͱͯ͠ݻఆ regularization policy Λఆظతʹߋ৽ͯ͠܁Γฦ͢ རಘม׵ʹΑΓɺʮ૬खʹউͭઓུʯΛޓ͍ʹ௥͍͔͚ଓ͚Δࣗݾରઓͷෆ҆ఆੑΛ཈͑ɺֶशΛ҆ఆԽ͢Δɻ POINT
24. 24 • R-NaD ͷ֩৺ɿརಘม׵ͨ͠ήʔϜΛղ͖ɺͦͷղΛ࣍ͷ regularization policy ͱͯ͠൓෮ ͢Δ͜ͱ • ͜ͷ൓෮ʹΑΓɺֶश͞ΕΔઓུ͕φογϡۉߧ΁޲͔͏͜ͱ͕ཧ࿦తʹ΋ࣔ͞Ε͍ͯΔ R-NaD Λࢧ͑Δཧ࿦อূ 05 DEEPNASH Perolat et al., lFrom Poincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationz, ICML 2021 Abe et al., lAdaptively Perturbed Mirror Descent for Learning in Gamesz, ICML 2024 R-NaDͷརಘม׵ + regularization policy ͷߋ৽ʹΑֶͬͯश͞ΕΔઓུ͸ɺφογϡۉߧ΁ऩଋ͢Δ THEOREM (Perolat et al., 2021) རಘม׵ΛΑΓҰൠͷܗʹஔ͖׵͑ͯ΋ɺֶश͞ΕΔઓུ͸φογϡۉߧ΁ऩଋ͢Δ THEOREM (Abe et al., 2024)
25. 25 ·ͱΊ Libratus Abstraction + CFR Ͱࣄલʹૈ͘ղ͖ɺϓϨΠதʹ safe subgame solving Ͱਫ਼៛Խ Pluribus ଟਓ਺ԽͰཧ࿦อূ͸ऑ·Δ͕ɺdepth-limited search + ෳ਺ ͷϩʔϧΞ΢τઓུͰ࣮ઓͰͷ่Εʹ͘͞Λ࣮ݱ DeepNash Regularization policy ʹΑΔརಘม׵ + self-play RL Ͱ NN ํ ࡦΛֶश
26. 26 • N. Brown & T. Sandholm, “Superhuman AI for heads-up no-limit poker: Libratus beats top professionals”, Science, 2018. • N. Brown & T. Sandholm, “Safe and Nested Subgame Solving for Imperfect-Information Games”, NeurIPS 2017. • N. Burch, M. Johanson & M. Bowling, “Solving Imperfect Information Games Using Decomposition”, AAAI 2014. • M. Zinkevich, M. Johanson, M. Bowling & C. Piccione, “Regret Minimization in Games with Incomplete Information”, NeurIPS 2007. • N. Brown & T. Sandholm, “Superhuman AI for multiplayer poker”, Science, 2019. • J. Perolat et al., “Mastering the game of Stratego with model-free multiagent reinforcement learning”, Science, 2022. • J. Perolat et al., “From Poincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization”, ICML 2021. • K. Abe et al., “Adaptively Perturbed Mirror Descent for Learning in Games”, ICML 2024. ࢀߟจݙ 06 REFERENCES