iOSと（深層）強化学習

by yuky_az

Slide 1

Slide 1 text

J04ͱʢਂ૚ʣڧԽֶश :VLJOBHB"[VNB !ZVLZ@B[ J04%$+BQBO

Slide 2

Slide 2 text

ࣗݾ঺հ :VLJOBHB"[VNB  ʢզ࠺޾௕ʣ 4"*-BCגࣜձࣾ୅දऔక໾ !ZVLZ@B[ ʮώτͱ"*ͷڞੜʯ͕ϛογϣϯ ࠷ۙΑ͘ݴΘΕΔ͜ͱ 6EFNZͰ"*ؔ࿈ͷߨ࠲Λෳ਺ల։ ສਓۙ͘ͷडߨੜ ࿩͠ํ͕"*ʹࣅ͖ͯͨͶ

Slide 3

Slide 3 text

J04%$ʹొஃͨ݁͠Ռʜ 4#ΫϦΤΠςΟϒΑΓൃചʂ

Slide 4

Slide 4 text

ൃදͷྲྀΕ 4FDڧԽֶशͱ͸ʁ 4FD4XJGUͰڧԽֶश 4FD"DDFMFSBUF'SBNFXPSLʹΑΔχϡʔϥϧωοτϫʔΫ 4FD4XJGUͰਂ૚ڧԽֶश

Slide 5

Slide 5 text

ൃදͷςʔϚ J04ʹਂ૚ڧԽֶश͸࣮૷Ͱ͖Δͷ͔ʁ

Slide 6

Slide 6 text

Sec. 1 ڧԽֶशͱ͸ʁ wڧԽֶशͷ֓ཁ wڧԽֶशʹඞཁͳ֤֓೦ w2ֶश wਂ૚ڧԽֶश

Slide 7

Slide 7 text

ਓ޻஌ೳ(AI)ɺػցֶशɺڧԽֶश ਓ޻஌ೳ "* ػցֶश σΟʔϓϥʔχϯά ڧԽֶश

Slide 8

Slide 8 text

ڧԽֶशͱ͸ʁ ػցֶशͷҰछ ࢼߦࡨޡΛ௨ͯ͡ʮ؀ڥʹ͓͍ͯ࠷΋ใु͕ಘΒΕ΍͍͢ߦಈʯΛ ʮΤʔδΣϯτʯֶ͕श͢Δ ߦಈ ใु ΤʔδΣϯτ ؀ڥ ֶश

Slide 9

Slide 9 text

ڧԽֶशͷԠ༻ྫ ήʔϜͷ߈ུ ˠ"MQIB(PɺϒϩοΫ่͠ͷ߈ུɺϧʔϏοΫΩϡʔϒͷ߈ུFUDʜ ϩϘοτͷ੍ޚ ˠೋ଍าߦϩϘοτɺ࢈ۀ༻ϩϘοτɺFUDʜ σʔληϯλʔͷిྗ࡟ݮ ˠIUUQTCMPHHPPHMFPVUSFBDIJOJUJBUJWFTFOWJSPONFOUEFFQNJOEBJSFEVDFTFOFSHZVTFEGPS Ϗϧͷ஍਒ରࡦ ˠIUUQTJOGPSJVNOUUEBUBDPNGPSFTJHIUBJWJCSBUJPODPOUSPMIUNM  FUDʜ

Slide 10

Slide 10 text

ڧԽֶशʹඞཁͳ֓೦ ߦಈʢBDUJPOʣ ঢ়ଶʢTUBUFʣ ใुʢSFXBSEʣ 4UBSU (PBM ΤʔδΣϯτ ؀ڥ

Slide 11

Slide 11 text

1. ߦಈʢactionʣ ΤʔδΣϯτ͕؀ڥʹಇ͖͔͚Δ͜ͱ ໎࿏ͷྫͰݴ͑͹ɺ ΤʔδΣϯτ͕໎࿏಺ΛҠಈ͢Δ͜ͱ ෳ਺ͷߦಈͷத͔ΒͭΛબ୒͢Δ 4UBSU (PBM

Slide 12

Slide 12 text

2. ঢ়ଶʢstateʣ ΤʔδΣϯτ͕؀ڥʹ͓͍ͯஔ͔Εͨঢ়ଶ ໎࿏ͷྫͰݴ͑͹ɺ ΤʔδΣϯτͷҐஔ44͕ঢ়ଶ ߦಈʹΑͬͯঢ়ଶ͸มԽ͢Δ 4 4 4 4 4 4 4 4 4

Slide 13

Slide 13 text

3. ใुʢrewardʣ ΤʔδΣϯτ͕ड͚औΔใु ໎࿏ͷྫͰݴ͑͹ɺ ΤʔδΣϯτ͕ΰʔϧʹ౸ୡ͢Ε͹ͷใु ΤʔδΣϯτ͕᠘ʹ౸ୡ͢Ε͹ͷใु ใुΛ΋ͱʹɺ࠷దͳߦಈΛֶश͍ͯ͘͠ 4UBSU (PBM ! "

Slide 14

Slide 14 text

Qֶश 2ֶश͸ɺڧԽֶशͷҰछͰɺ֤ঢ়ଶͱߦಈͷ૊Έ߹Θͤʹ2஋Λઃఆ ΤʔδΣϯτ͸࠷΋2஋ͷߴ͍ߦಈΛબ୒͢Δ 25BCMFͷ֤஋͕࠷దԽ͞ΕΔ͜ͱͰֶश͢Δ ˢ ˣ ˡ ˠ 4 4 4 ʜ ʜ ʜ ʜ ʜ 4 4 4 4 4 4 4 4 4 25BCMF

Slide 15

Slide 15 text

Q஋ͷߋ৽ ߦಈͷ݁ՌɺಘΒΕͨใुͱ࣍ͷঢ়ଶͰ࠷େͷ 2஋ʢׂΓҾ͘ʣ͔Βݱࡏͷ2஋Λࠩ͠Ҿ͘ ͜Εʹֶश܎਺ʢͳͲʣΛ͔͚ͯ2஋ͷߋ৽ ྔͱ͢Δ 4UBSU (PBM ! " 2஋ͷߋ৽ྔֶश܎਺Y ใुׂҾ཰Y࣍ͷঢ়ଶͰ࠷େͷ2஋ݱࡏͷ2஋ ߦಈʹΑΔ2஋ͷߋ৽

Slide 16

Slide 16 text

Qֶशͷ໰୊఺ ѻ͏ঢ়ଶͷ਺͕ଟ͍ͱ25BCMF͕ڊେʹͳΓ ֶश͕͏·͘ਐ·ͳ͘ͳͬͯ͠·͏ ˢ ˣ ˡ ˠ 4 4 ʜ ʜ ʜ ʜ ʜ 4 25BCMF

Slide 17

Slide 17 text

ਂ૚ڧԽֶश ڧԽֶशʹਂ૚ֶशʢσΟʔϓϥʔχϯάʣΛऔΓೖΕͨͷ͕ਂ૚ڧԽֶश  %FFQ2/FUXPSL %2/ ͸ਂ૚ڧԽֶशͷҰछͰɺ 25BCMFͷ୅ΘΓʹχϡʔϥϧωοτϫʔΫΛ࢖༻͢Δ ˢ ˣ ˡ ˠ 4 4 4 ʜ ʜ ʜ ʜ ʜ

Slide 18

Slide 18 text

ʜ ঢ়ଶ st ϓϨʔϠʔͷҐஔ ϓϨʔϠʔͷ଎౓ ఢΩϟϥͷҐஔ ఢΩϟϥͷ଎౓ ɹ2 st, ߦಈ ɹ2 st, ߦಈ ɹ2 st, ߦಈ Deep Q-NetworkʢDQNʣ χϡʔϥϧωοτϫʔΫͰɺঢ়ଶ͔Β֤ߦಈͷ2஋ΛٻΊΔ

Slide 19

Slide 19 text

ʜ ঢ়ଶ st ϓϨʔϠʔͷҐஔ ϓϨʔϠʔͷ଎౓ ఢΩϟϥͷҐஔ ఢΩϟϥͷ଎౓ ɹ2 st, ߦಈ ɹ2 st, ߦಈ ɹ2 st, ߦಈ Deep Q-Networkͷֶश ޡࠩͷٯ఻೻ ޡࠩ ใुׂҾ཰Y࣍ͷঢ়ଶͰ࠷େͷ2஋ݱࡏͷ2஋

Slide 20

Slide 20 text

Sec. 2 SwiftͰڧԽֶश w$BSU1PMF໰୊ w2ֶशͷ࣮૷ w$BSU1PMF໰୊ͷσϞ

Slide 21

Slide 21 text

Cart Pole໰୊ $BSU1PMF໰୊͸ڧԽֶशͷݹయతͳ໰୊ $BSUΛࠨӈʹҠಈͤͯ͞ɺ্ʹ৐ͬͨ1PMF͕౗Εͳ͍Α͏ʹ͢Δ $BSU 1PMF ঢ়ଶ $BSUͷҐஔ $BSUͷ଎౓ 1PMFͷ֯౓ 1PMFͷ֯଎౓ ߦಈ $BSUΛࠨʹಈ͔͢ $BSUΛӈʹಈ͔͢

Slide 22

Slide 22 text

؀ڥͷߏங 4QSJUF,JUΛ࢖ͬͯ%ͷ$BSUͱ1PMFΛ࡞੒ $BSUͱ1PMFΛKPJOUͰ઀ଓ͠ɺॏྗ͸1PMFʹͷΈʹద༻ cart = SKSpriteNode(imageNamed: "robot_normal.png") cart.size = cartSize cart.physicsBody = SKPhysicsBody(rectangleOf: cartSize) cart.physicsBody?.affectedByGravity = false pole = SKShapeNode(rectOf: poleSize) pole.physicsBody = SKPhysicsBody(rectangleOf: poleSize) let joint = SKPhysicsJointPin.joint(… self.addChild(pole) self.addChild(cart) self.physicsWorld.add(joint)

Slide 23

Slide 23 text

Q-Tableͷ࣮૷ 2ֶशΛ༻͍Δ ؆୯ʹ͢ΔͨΊʹɺ1PMFͷ֯౓ͱ֯଎౓ͷΈ ͔Βঢ়ଶΛܾΊΔ 1PMFͷ֯౓ɺ֯଎౓ΛͦΕͧΕʹ෼͚ͯ σδλϧԽ ঢ়ଶͷ਺͸Y௨Γ ˡ ˠ 4 4 ʜ ʜ ʜ 4 25BCMF

Slide 24

Slide 24 text

Q-Tableͷ࣮૷ 25BCMF var qTable = [[CGFloat]]() 25BCMFͷߋ৽ qTable[state][action] += eta * (reward + gamma*maxQNext - qTable[state][action])

Slide 25

Slide 25 text

ใुʹ͍ͭͯ $BSU 1PMF $BSU 1PMF ϑϨʔϜΩʔϓͰ͖ͨΒใु ܏͍ͨΒใु

Slide 26

Slide 26 text

Demo: Cart Pole໰୊ -Qֶश-

Slide 27

Slide 27 text

Sec. 3 Accelerate FrameworkʹΑΔ χϡʔϥϧωοτϫʔΫ wχϡʔϥϧωοτϫʔΫͷ࣮૷ํ๏Λબఆ w"DDFMFSBUF'SBNFXPSL w#-"4ʢ#BTJD-JOFBS"MHFCSB4VCQSPHSBNTʣ wߦྻͷૢ࡞

Slide 28

Slide 28 text

ਂ૚ڧԽֶशΛ࣮૷͢ΔͨΊʹ… ༧ଌ͚ͩͰ͸ͳֶ͘श͕Ͱ͖ΔχϡʔϥϧωοτϫʔΫ͕ඞཁ ॱ఻೻ʹΑΔ༧ଌ ٯ఻೻ʹΑΔֶश ೖྗ ग़ྗ

Slide 29

Slide 29 text

iOS ػցֶशؔ࿈Frameworkͷߏ੒ $PSF.- :PVSBQQ Vision "DDFSBSBUFBOE#//4 ϋʔυ΢ΣΞଆ Ϣʔβʔଆ .FUBM1FSGPSNBODF4IBEFST Natural Language Processing GameplayKit IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPODPSFNM

Slide 30

Slide 30 text

ֶशՄೳͳϑϨʔϜϫʔΫ͸…? $PSF.- ˠ܇࿅ʢֶशʣࡁΈͷϞσϧΛΞϓϦʹಋೖ͢Δ ˠػցֶशͷ༧ଌʹಛԽ .14 .FUBM1FSGPSNBODF4IBEFST ˠ(16Λ༻͍ͨ.FUBMͷߴ͍ԋࢉػೳΛΞϓϦʹಋೖ ˠػցֶशͷ༧ଌʹಛԽ #//4 #BTJD/FVSBM/FUXPSL4VCSPVUJOFT   ˠ$16ͷੑೳΛϑϧʹҾ͖ग़ͯ͠ԋࢉΛߦ͏  ˠ༧ଌʹಛԽ # $ %

Slide 31

Slide 31 text

χϡʔϥϧωοτϫʔΫͷࣗ࡞ χϡʔϥϧωοτϫʔΫΛ࣮૷͢Δͷʹॏཁͳߦྻԋࢉ ߦྻੵ సஔ

Slide 32

Slide 32 text

ߦྻੵ º ߦྻੵ ߦ ྻ YYYY

Slide 33

Slide 33 text

సஔ ߦྻͷɺߦͱྻΛೖΕସ͑Δ

Slide 34

Slide 34 text

Ͳ͏΍ͬͯߦྻੵͱసஔΛ࣮૷͢Δ͔ʁ 1VSF4XJGUʹΑΔ࣮૷ ˠߦྻ΍ϕΫτϧΛࣗ෼Ͱఆٛ͢Δඞཁ ˠશͯΛࣗલͰ࣮૷͢Δඞཁ͕͋ΔͷͰख͕͔͔ؒΔ "DDFMFSBUF'SBNFXPSLʹΑΔ࣮૷ ˠઢܗ୅਺ϥΠϒϥϦ#-"4ΛؚΉ ˠߦྻੵɺసஔͷͨΊͷؔ਺͕͋Δ .FUBMʹΑΔ࣮૷ ˠ4QSJUF,JUͱͳ͔ͥׯবͯ͠ը໘͕ͪΒͭ͘ ˠ(16΁සൟʹΞΫηε͢ΔͱύϑΥʔϚϯε͕མͪΔ & ' (

Slide 35

Slide 35 text

"DDFMFSBUF'SBNFXPSLͱ͸ʁ ˠେن໛ͳ਺ֶܭࢉɺը૾ԋࢉʹ༻͍ΒΕΔ  ˠ$16ͷੑೳΛϑϧʹҾ͖ग़ͯ͠ԋࢉΛߦ͏ ˠߴ͍ύϑΥʔϚϯεɺলిྗ ؚΉϥΠϒϥϦ܈  ˠը૾ॲཧ༻ͷW*NBHF ˠχϡʔϥϧωοτϫʔΫ༻ͷ#//4 ˠઢܗ୅਺༻ͷ#-"4 ˠFUDʜ

Slide 36

Slide 36 text

#-"4ʢ#BTJD-JOFBS"MHFCSB4VCQSPHSBNTʣ ߦྻϕΫτϧΛද͢ܕɺMB@PCKFDU@U ഑ྻ͔ΒߦྻΛੜ੒ let mat = la_matrix_from_double_buffer(array, rows, cols, cols, la_hint_t(LA_NO_HINT), la_attribute_t(LA_DEFAULT_ATTRIBUTES)) ߦྻੵ la_matrix_product(leftMat, rightMat) సஔ la_transpose(mat)

Slide 37

Slide 37 text

Sec. 4 SwiftͰਂ૚ڧԽֶश w%FFQ2/FUXPSLʢ%2/ʣ wχϡʔϥϧωοτϫʔΫͷߏங w%FFQ2/FUXPSLͷσϞ

Slide 38

Slide 38 text

ʜ ঢ়ଶ st ϓϨʔϠʔͷҐஔ ϓϨʔϠʔͷ଎౓ ఢΩϟϥͷҐஔ ఢΩϟϥͷ଎౓ ɹ2 st , ߦಈ ɹ2 st , ߦಈ ɹ2 st , ߦಈ ٯ఻೻ʹΑΔֶश Deep Q-NetworkʢDQNʣ ૚

Slide 39

Slide 39 text

χϡʔϥϧωοτϫʔΫͷߏங ߦྻੵͷԋࢉࢠΛఆٛ public func *(left: la_object_t, right: la_object_t) -> la_object_t { return la_matrix_product(left, right) } ֤૚ΛΫϥεͱ࣮ͯ͠૷ class MiddleLayer: BaseLayer{ … } సஔͷ࣮૷ extension la_object_t { var trans : la_object_t { return la_transpose(self) } }

Slide 40

Slide 40 text

χϡʔϥϧωοτϫʔΫͷߏங ॱ఻೻ͷϝιου func forward(x: la_object_t) -> la_object_t { let u_mat = x_mat * self.w + self.b let y = sigmoid(u: u) return y } ٯ఻೻ͷϝιου func backward(t: la_object_t) -> la_object_t { let delta = self.y - t self.dW = self.x.trans * delta self.db = delta let dx = delta * self.w.trans return dx }

Slide 41

Slide 41 text

χϡʔϥϧωοτϫʔΫͷߏ੒ ೖྗ૚ தؒ૚ தؒ૚ ग़ྗ૚ O O 1PMFͷ֯౓ 1PMFͷ֯଎౓ 2஋ʢࠨʹಈ͘ʣ 2஋ʢӈʹಈ͘ʣ

Slide 42

Slide 42 text

Demo: Cart Pole໰୊ -Deep Q-Network-

Slide 43

Slide 43 text

ΤʔδΣϯτͷઓུ ͿΔͿΔҠಈ࡞ઓ

Slide 44

Slide 44 text

ΤʔδΣϯτͷઓུ εΠον࡞ઓ

Slide 45

Slide 45 text

ΤʔδΣϯτͷઓུ Χ΢ϯλʔΞλοΫ

Slide 46

Slide 46 text

·ͱΊ ֶशՄೳͳχϡʔϥϧωοτϫʔΫ ˠ"DDFMFSBUF'SBNFXPSLͷ#-"4Λ࢖͑͹ൺֱత༰қ গͳ͘ͱ΋ɺൺֱతγϯϓϧͳ$BSU1PMF໰୊ͷΑ͏ͳ΋ͷͰ͋Ε͹ J04ʹਂ૚ڧԽֶश͸࣮૷Ͱ͖Δ $BSU1PMF໰୊ ˠ2ֶशɺ%FFQ2/FUXPSLΛ࣮૷ ˠ"*͕ࢥΘ͵૑଄ੑʢʁʣΛൃش͢Δ

Slide 47

Slide 47 text

͝੩ௌɺ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ ຊ೔࢖༻ͨ͠ίʔυ IUUQTHJUIVCDPNZVLJOBHB$BSU1PMF4XJGU IUUQTHJUIVCDPNZVLJOBHB$BSU1PMF%FFQ4XJGU