iOSと（深層）強化学習

J04ͱʢਂ૚ʣڧԽֶश :VLJOBHB"[VNB !ZVLZ@B[ J04%$+BQBO

ࣗݾ঺հ :VLJOBHB"[VNB  ʢզ࠺޾௕ʣ 4"*-BCגࣜձࣾ୅දऔక໾ !ZVLZ@B[ ʮώτͱ"*ͷڞੜʯ͕ϛογϣϯ ࠷ۙΑ͘ݴΘΕΔ͜ͱ 6EFNZͰ"*ؔ࿈ͷߨ࠲Λෳ਺ల։ ສਓۙ͘ͷडߨੜ ࿩͠ํ͕"*ʹࣅ͖ͯͨͶ

J04%$ʹొஃͨ݁͠Ռʜ 4#ΫϦΤΠςΟϒΑΓൃചʂ

ൃදͷྲྀΕ 4FDڧԽֶशͱ͸ʁ 4FD4XJGUͰڧԽֶश 4FD"DDFMFSBUF'SBNFXPSLʹΑΔχϡʔϥϧωοτϫʔΫ 4FD4XJGUͰਂ૚ڧԽֶश

ൃදͷςʔϚ J04ʹਂ૚ڧԽֶश͸࣮૷Ͱ͖Δͷ͔ʁ

Sec. 1 ڧԽֶशͱ͸ʁ wڧԽֶशͷ֓ཁ wڧԽֶशʹඞཁͳ֤֓೦ w2ֶश wਂ૚ڧԽֶश

ਓ޻஌ೳ(AI)ɺػցֶशɺڧԽֶश ਓ޻஌ೳ "* ػցֶश σΟʔϓϥʔχϯά ڧԽֶश

ڧԽֶशͱ͸ʁ ػցֶशͷҰछ ࢼߦࡨޡΛ௨ͯ͡ʮ؀ڥʹ͓͍ͯ࠷΋ใु͕ಘΒΕ΍͍͢ߦಈʯΛ ʮΤʔδΣϯτʯֶ͕श͢Δ ߦಈ ใु ΤʔδΣϯτ ؀ڥ ֶश

ڧԽֶशͷԠ༻ྫ ήʔϜͷ߈ུ ˠ"MQIB(PɺϒϩοΫ่͠ͷ߈ུɺϧʔϏοΫΩϡʔϒͷ߈ུFUDʜ ϩϘοτͷ੍ޚ ˠೋ଍าߦϩϘοτɺ࢈ۀ༻ϩϘοτɺFUDʜ σʔληϯλʔͷిྗ࡟ݮ ˠIUUQTCMPHHPPHMFPVUSFBDIJOJUJBUJWFTFOWJSPONFOUEFFQNJOEBJSFEVDFTFOFSHZVTFEGPS Ϗϧͷ஍਒ରࡦ ˠIUUQTJOGPSJVNOUUEBUBDPNGPSFTJHIUBJWJCSBUJPODPOUSPMIUNM 
FUDʜ

ڧԽֶशʹඞཁͳ֓೦ ߦಈʢBDUJPOʣ ঢ়ଶʢTUBUFʣ ใुʢSFXBSEʣ 4UBSU (PBM ΤʔδΣϯτ ؀ڥ

1. ߦಈʢactionʣ ΤʔδΣϯτ͕؀ڥʹಇ͖͔͚Δ͜ͱ ໎࿏ͷྫͰݴ͑͹ɺ ΤʔδΣϯτ͕໎࿏಺ΛҠಈ͢Δ͜ͱ ෳ਺ͷߦಈͷத͔ΒͭΛબ୒͢Δ 4UBSU (PBM

2. ঢ়ଶʢstateʣ ΤʔδΣϯτ͕؀ڥʹ͓͍ͯஔ͔Εͨঢ়ଶ ໎࿏ͷྫͰݴ͑͹ɺ ΤʔδΣϯτͷҐஔ44͕ঢ়ଶ ߦಈʹΑͬͯঢ়ଶ͸มԽ͢Δ 4 4 4 4
4 4 4 4 4

3. ใुʢrewardʣ ΤʔδΣϯτ͕ड͚औΔใु ໎࿏ͷྫͰݴ͑͹ɺ ΤʔδΣϯτ͕ΰʔϧʹ౸ୡ͢Ε͹ ͷใु ΤʔδΣϯτ͕᠘ʹ౸ୡ͢Ε͹ͷใु ใुΛ΋ͱʹɺ࠷దͳߦಈΛֶश͍ͯ͘͠ 4UBSU (PBM
! "

Qֶश 2ֶश͸ɺڧԽֶशͷҰछͰɺ֤ঢ়ଶͱߦಈͷ૊Έ߹Θͤʹ2஋Λઃఆ ΤʔδΣϯτ͸࠷΋2஋ͷߴ͍ߦಈΛબ୒͢Δ 25BCMFͷ֤஋͕࠷దԽ͞ΕΔ͜ͱͰֶश͢Δ ˢ ˣ ˡ ˠ 4
4 4 ʜ ʜ ʜ ʜ ʜ 4 4 4 4 4 4 4 4 4 25BCMF

Q஋ͷߋ৽ ߦಈͷ݁ՌɺಘΒΕͨใुͱ࣍ͷঢ়ଶͰ࠷େͷ 2஋ʢׂΓҾ͘ʣ͔Βݱࡏͷ2஋Λࠩ͠Ҿ͘ ͜Εʹֶश܎਺ʢͳͲʣΛ͔͚ͯ2஋ͷߋ৽ ྔͱ͢Δ 4UBSU (PBM ! "
2஋ͷߋ৽ྔֶश܎਺Y ใु ׂҾ཰Y࣍ͷঢ়ଶͰ࠷େͷ2஋ݱࡏͷ2஋ ߦಈʹΑΔ2஋ͷߋ৽

Qֶशͷ໰୊఺ ѻ͏ঢ়ଶͷ਺͕ଟ͍ͱ25BCMF͕ڊେʹͳΓ ֶश͕͏·͘ਐ·ͳ͘ͳͬͯ͠·͏ ˢ ˣ ˡ ˠ 4
4 ʜ ʜ ʜ ʜ ʜ 4 25BCMF

ਂ૚ڧԽֶश ڧԽֶशʹਂ૚ֶशʢσΟʔϓϥʔχϯάʣΛऔΓೖΕͨͷ͕ਂ૚ڧԽֶश  %FFQ2/FUXPSL %2/ ͸ਂ૚ڧԽֶशͷҰछͰɺ 25BCMFͷ୅ΘΓʹχϡʔϥϧωοτϫʔΫΛ࢖༻͢Δ ˢ ˣ ˡ ˠ
4 4 4 ʜ ʜ ʜ ʜ ʜ

ʜ ঢ়ଶ st ϓϨʔϠʔͷҐஔ ϓϨʔϠʔͷ଎౓ ఢΩϟϥͷҐஔ ఢΩϟϥͷ଎౓ ɹ2 st, ߦಈ
ɹ2 st, ߦಈ ɹ2 st, ߦಈ Deep Q-NetworkʢDQNʣ χϡʔϥϧωοτϫʔΫͰɺঢ়ଶ͔Β֤ߦಈͷ2஋ΛٻΊΔ

ʜ ঢ়ଶ st ϓϨʔϠʔͷҐஔ ϓϨʔϠʔͷ଎౓ ఢΩϟϥͷҐஔ ఢΩϟϥͷ଎౓ ɹ2 st, ߦಈ
ɹ2 st, ߦಈ ɹ2 st, ߦಈ Deep Q-Networkͷֶश ޡࠩͷٯ఻೻ ޡࠩ ใु ׂҾ཰Y࣍ͷঢ়ଶͰ࠷େͷ2஋ݱࡏͷ2஋

Sec. 2 SwiftͰڧԽֶश w$BSU1PMF໰୊ w2ֶशͷ࣮૷ w$BSU1PMF໰୊ͷσϞ

Cart Pole໰୊ $BSU1PMF໰୊͸ڧԽֶशͷݹయతͳ໰୊ $BSUΛࠨӈʹҠಈͤͯ͞ɺ্ʹ৐ͬͨ1PMF͕౗Εͳ͍Α͏ʹ͢Δ $BSU 1PMF ঢ়ଶ $BSUͷҐஔ $BSUͷ଎౓ 1PMFͷ֯౓
1PMFͷ֯଎౓ ߦಈ $BSUΛࠨʹಈ͔͢ $BSUΛӈʹಈ͔͢

؀ڥͷߏங 4QSJUF,JUΛ࢖ͬͯ%ͷ$BSUͱ1PMFΛ࡞੒ $BSUͱ1PMFΛKPJOUͰ઀ଓ͠ɺॏྗ͸1PMFʹͷΈʹద༻ cart = SKSpriteNode(imageNamed: "robot_normal.png") cart.size = cartSize
cart.physicsBody = SKPhysicsBody(rectangleOf: cartSize) cart.physicsBody?.affectedByGravity = false pole = SKShapeNode(rectOf: poleSize) pole.physicsBody = SKPhysicsBody(rectangleOf: poleSize) let joint = SKPhysicsJointPin.joint(… self.addChild(pole) self.addChild(cart) self.physicsWorld.add(joint)

Q-Tableͷ࣮૷ 2ֶशΛ༻͍Δ ؆୯ʹ͢ΔͨΊʹɺ1PMFͷ֯౓ͱ֯଎౓ͷΈ ͔Βঢ়ଶΛܾΊΔ 1PMFͷ֯౓ɺ֯଎౓ΛͦΕͧΕʹ෼͚ͯ σδλϧԽ ঢ়ଶͷ਺͸Y௨Γ ˡ ˠ 4
4 ʜ ʜ ʜ 4 25BCMF

Q-Tableͷ࣮૷ 25BCMF var qTable = [[CGFloat]]() 25BCMFͷߋ৽ qTable[state][action] += eta
* (reward + gamma*maxQNext - qTable[state][action])

ใुʹ͍ͭͯ $BSU 1PMF $BSU 1PMF ϑϨʔϜΩʔϓͰ͖ͨΒใु ܏͍ͨΒใु

Demo: Cart Pole໰୊ -Qֶश-

Sec. 3 Accelerate FrameworkʹΑΔ χϡʔϥϧωοτϫʔΫ wχϡʔϥϧωοτϫʔΫͷ࣮૷ํ๏Λબఆ w"DDFMFSBUF'SBNFXPSL w#-"4ʢ#BTJD-JOFBS"MHFCSB4VCQSPHSBNTʣ wߦྻͷૢ࡞

ਂ૚ڧԽֶशΛ࣮૷͢ΔͨΊʹ… ༧ଌ͚ͩͰ͸ͳֶ͘श͕Ͱ͖ΔχϡʔϥϧωοτϫʔΫ͕ඞཁ ॱ఻೻ʹΑΔ༧ଌ ٯ఻೻ʹΑΔֶश ೖྗ ग़ྗ

iOS ػցֶशؔ࿈Frameworkͷߏ੒ $PSF.- :PVSBQQ Vision "DDFSBSBUFBOE#//4 ϋʔυ΢ΣΞଆ Ϣʔβʔଆ .FUBM1FSGPSNBODF4IBEFST Natural
Language Processing GameplayKit IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPODPSFNM

ֶशՄೳͳϑϨʔϜϫʔΫ͸…? $PSF.- ˠ܇࿅ʢֶशʣࡁΈͷϞσϧΛΞϓϦʹಋೖ͢Δ ˠػցֶशͷ༧ଌʹಛԽ .14 .FUBM1FSGPSNBODF4IBEFST ˠ(16Λ༻͍ͨ.FUBMͷߴ͍ԋࢉػೳΛΞϓϦʹಋೖ ˠػցֶशͷ༧ଌʹಛԽ #//4
#BTJD/FVSBM/FUXPSL4VCSPVUJOFT   ˠ$16ͷੑೳΛϑϧʹҾ͖ग़ͯ͠ԋࢉΛߦ͏  ˠ༧ଌʹಛԽ # $ %

χϡʔϥϧωοτϫʔΫͷࣗ࡞ χϡʔϥϧωοτϫʔΫΛ࣮૷͢Δͷʹॏཁͳߦྻԋࢉ ߦྻੵ సஔ

ߦྻੵ º
ߦྻੵ ߦ ྻ Y Y Y Y

సஔ ߦྻͷɺߦͱྻΛೖΕସ͑Δ

Ͳ͏΍ͬͯߦྻੵͱసஔΛ࣮૷͢Δ͔ʁ 1VSF4XJGUʹΑΔ࣮૷ ˠߦྻ΍ϕΫτϧΛࣗ෼Ͱఆٛ͢Δඞཁ ˠશͯΛࣗલͰ࣮૷͢Δඞཁ͕͋ΔͷͰख͕͔͔ؒΔ "DDFMFSBUF'SBNFXPSLʹΑΔ࣮૷ ˠઢܗ୅਺ϥΠϒϥϦ#-"4ΛؚΉ ˠߦྻੵɺసஔͷͨΊͷؔ਺͕͋Δ .FUBMʹΑΔ࣮૷ ˠ4QSJUF,JUͱͳ͔ͥׯবͯ͠ը໘͕ͪΒͭ͘ ˠ(16΁සൟʹΞΫηε͢ΔͱύϑΥʔϚϯε͕མͪΔ
& ' (

"DDFMFSBUF'SBNFXPSLͱ͸ʁ ˠେن໛ͳ਺ֶܭࢉɺը૾ԋࢉʹ༻͍ΒΕΔ  ˠ$16ͷੑೳΛϑϧʹҾ͖ग़ͯ͠ԋࢉΛߦ͏ ˠߴ͍ύϑΥʔϚϯεɺলిྗ ؚΉϥΠϒϥϦ܈  ˠը૾ॲཧ༻ͷW*NBHF ˠχϡʔϥϧωοτϫʔΫ༻ͷ#//4 ˠઢܗ୅਺༻ͷ#-"4 ˠFUDʜ

#-"4ʢ#BTJD-JOFBS"MHFCSB4VCQSPHSBNTʣ ߦྻϕΫτϧΛද͢ܕɺMB@PCKFDU@U ഑ྻ͔ΒߦྻΛੜ੒ let mat = la_matrix_from_double_buffer(array, rows, cols, cols,
la_hint_t(LA_NO_HINT), la_attribute_t(LA_DEFAULT_ATTRIBUTES)) ߦྻੵ la_matrix_product(leftMat, rightMat) సஔ la_transpose(mat)

Sec. 4 SwiftͰਂ૚ڧԽֶश w%FFQ2/FUXPSLʢ%2/ʣ wχϡʔϥϧωοτϫʔΫͷߏங w%FFQ2/FUXPSLͷσϞ

ʜ ঢ়ଶ st ϓϨʔϠʔͷҐஔ ϓϨʔϠʔͷ଎౓ ఢΩϟϥͷҐஔ ఢΩϟϥͷ଎౓ ɹ2 st
, ߦಈ ɹ2 st , ߦಈ ɹ2 st , ߦಈ ٯ఻೻ʹΑΔֶश Deep Q-NetworkʢDQNʣ ૚

χϡʔϥϧωοτϫʔΫͷߏங ߦྻੵͷԋࢉࢠΛఆٛ public func *(left: la_object_t, right: la_object_t) -> la_object_t
{ return la_matrix_product(left, right) } ֤૚ΛΫϥεͱ࣮ͯ͠૷ class MiddleLayer: BaseLayer{ … } సஔͷ࣮૷ extension la_object_t { var trans : la_object_t { return la_transpose(self) } }

χϡʔϥϧωοτϫʔΫͷߏங ॱ఻೻ͷϝιου func forward(x: la_object_t) -> la_object_t { let u_mat
= x_mat * self.w + self.b let y = sigmoid(u: u) return y } ٯ఻೻ͷϝιου func backward(t: la_object_t) -> la_object_t { let delta = self.y - t self.dW = self.x.trans * delta self.db = delta let dx = delta * self.w.trans return dx }

χϡʔϥϧωοτϫʔΫͷߏ੒ ೖྗ૚ தؒ૚ தؒ૚ ग़ྗ૚ O O 1PMFͷ֯౓ 1PMFͷ֯଎౓ 2஋ʢࠨʹಈ͘ʣ
2஋ʢӈʹಈ͘ʣ

Demo: Cart Pole໰୊ -Deep Q-Network-

ΤʔδΣϯτͷઓུ ͿΔͿΔҠಈ࡞ઓ

ΤʔδΣϯτͷઓུ εΠον࡞ઓ

ΤʔδΣϯτͷઓུ Χ΢ϯλʔΞλοΫ

·ͱΊ ֶशՄೳͳχϡʔϥϧωοτϫʔΫ ˠ"DDFMFSBUF'SBNFXPSLͷ#-"4Λ࢖͑͹ൺֱత༰қ গͳ͘ͱ΋ɺൺֱతγϯϓϧͳ$BSU1PMF໰୊ͷΑ͏ͳ΋ͷͰ͋Ε͹ J04ʹਂ૚ڧԽֶश͸࣮૷Ͱ͖Δ $BSU1PMF໰୊ ˠ2ֶशɺ%FFQ2/FUXPSLΛ࣮૷ ˠ"*͕ࢥΘ͵૑଄ੑʢʁʣΛൃش͢Δ

͝੩ௌɺ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ ຊ೔࢖༻ͨ͠ίʔυ IUUQTHJUIVCDPNZVLJOBHB$BSU1PMF4XJGU IUUQTHJUIVCDPNZVLJOBHB$BSU1PMF%FFQ4XJGU

iOSと（深層）強化学習

iOSと（深層）強化学習

More Decks by yuky_az

Other Decks in Programming

Featured

Transcript