iOSと（深層）強化学習

J04ͱʢਂ૚ʣڧԽֶश
:VLJOBHB"[VNB
!ZVLZ@B[
J04%$+BQBO

View Slide

ࣗݾ঺հ
:VLJOBHB"[VNB 
ʢզ࠺޾௕ʣ
4"*-BCגࣜձࣾ୅දऔక໾
!ZVLZ@B[
ʮώτͱ"*ͷڞੜʯ͕ϛογϣϯ
࠷ۙΑ͘ݴΘΕΔ͜ͱ
6EFNZͰ"*ؔ࿈ͷߨ࠲Λෳ਺ల։
ສਓۙ͘ͷडߨੜ
࿩͠ํ͕"*ʹࣅ͖ͯͨͶ

View Slide

J04%$ʹొஃͨ݁͠Ռʜ
4#ΫϦΤΠςΟϒΑΓൃചʂ

View Slide

ൃදͷྲྀΕ
4FDڧԽֶशͱ͸ʁ
4FD4XJGUͰڧԽֶश
4FD"DDFMFSBUF'SBNFXPSLʹΑΔχϡʔϥϧωοτϫʔΫ
4FD4XJGUͰਂ૚ڧԽֶश

View Slide

ൃදͷςʔϚ
J04ʹਂ૚ڧԽֶश͸࣮૷Ͱ͖Δͷ͔ʁ

View Slide

Sec. 1 ڧԽֶशͱ͸ʁ
wڧԽֶशͷ֓ཁ
wڧԽֶशʹඞཁͳ֤֓೦
w2ֶश
wਂ૚ڧԽֶश

View Slide

ਓ޻஌ೳ(AI)ɺػցֶशɺڧԽֶश
ਓ޻஌ೳ "*

ػցֶश
σΟʔϓϥʔχϯά
ڧԽֶश

View Slide

ڧԽֶशͱ͸ʁ
ػցֶशͷҰछ
ࢼߦࡨޡΛ௨ͯ͡ʮ؀ڥʹ͓͍ͯ࠷΋ใु͕ಘΒΕ΍͍͢ߦಈʯΛ
ʮΤʔδΣϯτʯֶ͕श͢Δ
ߦಈ
ใु
ΤʔδΣϯτ
؀ڥ
ֶश

View Slide

ڧԽֶशͷԠ༻ྫ
ήʔϜͷ߈ུ
ˠ"MQIB(PɺϒϩοΫ่͠ͷ߈ུɺϧʔϏοΫΩϡʔϒͷ߈ུFUDʜ
ϩϘοτͷ੍ޚ
ˠೋ଍าߦϩϘοτɺ࢈ۀ༻ϩϘοτɺFUDʜ
σʔληϯλʔͷిྗ࡟ݮ
ˠIUUQTCMPHHPPHMFPVUSFBDIJOJUJBUJWFTFOWJSPONFOUEFFQNJOEBJSFEVDFTFOFSHZVTFEGPS

Ϗϧͷ஍਒ରࡦ
ˠIUUQTJOGPSJVNOUUEBUBDPNGPSFTJHIUBJWJCSBUJPODPOUSPMIUNM 
FUDʜ

View Slide

ڧԽֶशʹඞཁͳ֓೦
ߦಈʢBDUJPOʣ
ঢ়ଶʢTUBUFʣ
ใुʢSFXBSEʣ
4UBSU
(PBM
ΤʔδΣϯτ ؀ڥ

View Slide

1. ߦಈʢactionʣ
ΤʔδΣϯτ͕؀ڥʹಇ͖͔͚Δ͜ͱ
໎࿏ͷྫͰݴ͑͹ɺ
ΤʔδΣϯτ͕໎࿏಺ΛҠಈ͢Δ͜ͱ
ෳ਺ͷߦಈͷத͔ΒͭΛબ୒͢Δ
4UBSU
(PBM

View Slide

2. ঢ়ଶʢstateʣ
ΤʔδΣϯτ͕؀ڥʹ͓͍ͯஔ͔Εͨঢ়ଶ
ΤʔδΣϯτͷҐஔ44͕ঢ়ଶ
ߦಈʹΑͬͯঢ়ଶ͸มԽ͢Δ
4 4 4
4 4 4
4 4 4

View Slide

3. ใुʢrewardʣ
ΤʔδΣϯτ͕ड͚औΔใु
ΤʔδΣϯτ͕ΰʔϧʹ౸ୡ͢Ε͹ͷใु
ΤʔδΣϯτ͕᠘ʹ౸ୡ͢Ε͹ͷใु
ใुΛ΋ͱʹɺ࠷దͳߦಈΛֶश͍ͯ͘͠
4UBSU
(PBM
!
"

View Slide

Qֶश
2ֶश͸ɺڧԽֶशͷҰछͰɺ֤ঢ়ଶͱߦಈͷ૊Έ߹Θͤʹ2஋Λઃఆ
ΤʔδΣϯτ͸࠷΋2஋ͷߴ͍ߦಈΛબ୒͢Δ
25BCMFͷ֤஋͕࠷దԽ͞ΕΔ͜ͱͰֶश͢Δ
ˢ ˣ ˡ ˠ
4
4
4
ʜ ʜ ʜ ʜ ʜ
4 4 4
4 4 4
4 4 4
25BCMF

View Slide

Q஋ͷߋ৽
ߦಈͷ݁ՌɺಘΒΕͨใुͱ࣍ͷঢ়ଶͰ࠷େͷ
2஋ʢׂΓҾ͘ʣ͔Βݱࡏͷ2஋Λࠩ͠Ҿ͘
͜Εʹֶश܎਺ʢͳͲʣΛ͔͚ͯ2஋ͷߋ৽
ྔͱ͢Δ
4UBSU
(PBM
!
"

2஋ͷߋ৽ྔֶश܎਺Y ใुׂҾ཰Y࣍ͷঢ়ଶͰ࠷େͷ2஋ݱࡏͷ2஋

ߦಈʹΑΔ2஋ͷߋ৽

View Slide

Qֶशͷ໰୊఺
ѻ͏ঢ়ଶͷ਺͕ଟ͍ͱ25BCMF͕ڊେʹͳΓ
ֶश͕͏·͘ਐ·ͳ͘ͳͬͯ͠·͏
ˢ ˣ ˡ ˠ
4
4
ʜ ʜ ʜ ʜ ʜ
4
25BCMF

View Slide

ਂ૚ڧԽֶश
ڧԽֶशʹਂ૚ֶशʢσΟʔϓϥʔχϯάʣΛऔΓೖΕͨͷ͕ਂ૚ڧԽֶश 
%FFQ2/FUXPSL %2/
͸ਂ૚ڧԽֶशͷҰछͰɺ
25BCMFͷ୅ΘΓʹχϡʔϥϧωοτϫʔΫΛ࢖༻͢Δ
ˢ ˣ ˡ ˠ
4
4
4
ʜ ʜ ʜ ʜ ʜ

View Slide

ʜ
ঢ়ଶ
st
ϓϨʔϠʔͷҐஔ
ϓϨʔϠʔͷ଎౓
ఢΩϟϥͷҐஔ
ఢΩϟϥͷ଎౓
ɹ2 st, ߦಈ

ɹ2 st, ߦಈ

ɹ2 st, ߦಈ

Deep Q-NetworkʢDQNʣ
χϡʔϥϧωοτϫʔΫͰɺঢ়ଶ͔Β֤ߦಈͷ2஋ΛٻΊΔ

View Slide

ʜ
ঢ়ଶ
st
ϓϨʔϠʔͷҐஔ
ϓϨʔϠʔͷ଎౓
ఢΩϟϥͷҐஔ
ఢΩϟϥͷ଎౓
ɹ2 st, ߦಈ

ɹ2 st, ߦಈ

ɹ2 st, ߦಈ

Deep Q-Networkͷֶश
ޡࠩͷٯ఻೻
ޡࠩ ใुׂҾ཰Y࣍ͷঢ়ଶͰ࠷େͷ2஋ݱࡏͷ2஋

View Slide

Sec. 2 SwiftͰڧԽֶश
w$BSU1PMF໰୊
w2ֶशͷ࣮૷
w$BSU1PMF໰୊ͷσϞ

View Slide

Cart Pole໰୊
$BSU1PMF໰୊͸ڧԽֶशͷݹయతͳ໰୊
$BSUΛࠨӈʹҠಈͤͯ͞ɺ্ʹ৐ͬͨ1PMF͕౗Εͳ͍Α͏ʹ͢Δ
$BSU
1PMF
ঢ়ଶ
$BSUͷҐஔ
$BSUͷ଎౓
1PMFͷ֯౓
1PMFͷ֯଎౓
ߦಈ
$BSUΛࠨʹಈ͔͢
$BSUΛӈʹಈ͔͢

View Slide

؀ڥͷߏங
4QSJUF,JUΛ࢖ͬͯ%ͷ$BSUͱ1PMFΛ࡞੒
$BSUͱ1PMFΛKPJOUͰ઀ଓ͠ɺॏྗ͸1PMFʹͷΈʹద༻
cart = SKSpriteNode(imageNamed: "robot_normal.png")
cart.size = cartSize
cart.physicsBody = SKPhysicsBody(rectangleOf: cartSize)
cart.physicsBody?.affectedByGravity = false
pole = SKShapeNode(rectOf: poleSize)
pole.physicsBody = SKPhysicsBody(rectangleOf: poleSize)
let joint = SKPhysicsJointPin.joint(…
self.addChild(pole)
self.addChild(cart)
self.physicsWorld.add(joint)

View Slide

Q-Tableͷ࣮૷
2ֶशΛ༻͍Δ
؆୯ʹ͢ΔͨΊʹɺ1PMFͷ֯౓ͱ֯଎౓ͷΈ
͔Βঢ়ଶΛܾΊΔ
1PMFͷ֯౓ɺ֯଎౓ΛͦΕͧΕʹ෼͚ͯ
σδλϧԽ
ঢ়ଶͷ਺͸Y௨Γ
ˡ ˠ
4
4
ʜ ʜ ʜ
4
25BCMF

View Slide

Q-Tableͷ࣮૷
25BCMF
var qTable = [[CGFloat]]()
25BCMFͷߋ৽
qTable[state][action] += eta * (reward + gamma*maxQNext - qTable[state][action])

View Slide

ใुʹ͍ͭͯ
$BSU
1PMF
$BSU
1PMF
ϑϨʔϜΩʔϓͰ͖ͨΒใु ܏͍ͨΒใु

View Slide

Demo: Cart Pole໰୊ -Qֶश-

View Slide

Sec. 3 Accelerate FrameworkʹΑΔ
χϡʔϥϧωοτϫʔΫ
wχϡʔϥϧωοτϫʔΫͷ࣮૷ํ๏Λબఆ
w"DDFMFSBUF'SBNFXPSL
w#-"4ʢ#BTJD-JOFBS"MHFCSB4VCQSPHSBNTʣ
wߦྻͷૢ࡞

View Slide

ਂ૚ڧԽֶशΛ࣮૷͢ΔͨΊʹ…
༧ଌ͚ͩͰ͸ͳֶ͘श͕Ͱ͖ΔχϡʔϥϧωοτϫʔΫ͕ඞཁ
ॱ఻೻ʹΑΔ༧ଌ
ٯ఻೻ʹΑΔֶश
ೖྗ ग़ྗ

View Slide

iOS ػցֶशؔ࿈Frameworkͷߏ੒
$PSF.-
:PVSBQQ
Vision
"DDFSBSBUFBOE#//4
ϋʔυ΢ΣΞଆ
Ϣʔβʔଆ
.FUBM1FSGPSNBODF4IBEFST
Natural Language Processing GameplayKit
IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPODPSFNM

View Slide

ֶशՄೳͳϑϨʔϜϫʔΫ͸…?
$PSF.-
ˠ܇࿅ʢֶशʣࡁΈͷϞσϧΛΞϓϦʹಋೖ͢Δ
ˠػցֶशͷ༧ଌʹಛԽ
.14 .FUBM1FSGPSNBODF4IBEFST

ˠ(16Λ༻͍ͨ.FUBMͷߴ͍ԋࢉػೳΛΞϓϦʹಋೖ
ˠػցֶशͷ༧ଌʹಛԽ
#//4 #BTJD/FVSBM/FUXPSL4VCSPVUJOFT
 
ˠ$16ͷੑೳΛϑϧʹҾ͖ग़ͯ͠ԋࢉΛߦ͏ 
ˠ༧ଌʹಛԽ
#
$
%

View Slide

χϡʔϥϧωοτϫʔΫͷࣗ࡞
χϡʔϥϧωοτϫʔΫΛ࣮૷͢Δͷʹॏཁͳߦྻԋࢉ
ߦྻੵ
సஔ

View Slide

ߦྻੵ

º
ߦྻੵ

ߦ
ྻ
YYYY

View Slide

సஔ
ߦྻͷɺߦͱྻΛೖΕସ͑Δ

View Slide

Ͳ͏΍ͬͯߦྻੵͱసஔΛ࣮૷͢Δ͔ʁ
1VSF4XJGUʹΑΔ࣮૷
ˠߦྻ΍ϕΫτϧΛࣗ෼Ͱఆٛ͢Δඞཁ
ˠશͯΛࣗલͰ࣮૷͢Δඞཁ͕͋ΔͷͰख͕͔͔ؒΔ
"DDFMFSBUF'SBNFXPSLʹΑΔ࣮૷
ˠઢܗ୅਺ϥΠϒϥϦ#-"4ΛؚΉ
ˠߦྻੵɺసஔͷͨΊͷؔ਺͕͋Δ
.FUBMʹΑΔ࣮૷
ˠ4QSJUF,JUͱͳ͔ͥׯবͯ͠ը໘͕ͪΒͭ͘
ˠ(16΁සൟʹΞΫηε͢ΔͱύϑΥʔϚϯε͕མͪΔ
&
'
(

View Slide

"DDFMFSBUF'SBNFXPSLͱ͸ʁ
ˠେن໛ͳ਺ֶܭࢉɺը૾ԋࢉʹ༻͍ΒΕΔ 
ˠ$16ͷੑೳΛϑϧʹҾ͖ग़ͯ͠ԋࢉΛߦ͏
ˠߴ͍ύϑΥʔϚϯεɺলిྗ
ؚΉϥΠϒϥϦ܈ 
ˠը૾ॲཧ༻ͷW*NBHF
ˠχϡʔϥϧωοτϫʔΫ༻ͷ#//4
ˠઢܗ୅਺༻ͷ#-"4
ˠFUDʜ

View Slide

#-"4ʢ#BTJD-JOFBS"MHFCSB4VCQSPHSBNTʣ
ߦྻϕΫτϧΛද͢ܕɺMB@PCKFDU@U
഑ྻ͔ΒߦྻΛੜ੒
let mat = la_matrix_from_double_buffer(array, rows, cols, cols,
la_hint_t(LA_NO_HINT), la_attribute_t(LA_DEFAULT_ATTRIBUTES))
ߦྻੵ
la_matrix_product(leftMat, rightMat)
సஔ
la_transpose(mat)

View Slide

Sec. 4 SwiftͰਂ૚ڧԽֶश
w%FFQ2/FUXPSLʢ%2/ʣ
wχϡʔϥϧωοτϫʔΫͷߏங
w%FFQ2/FUXPSLͷσϞ

View Slide

ʜ
ঢ়ଶ
st
ϓϨʔϠʔͷҐஔ
ϓϨʔϠʔͷ଎౓
ఢΩϟϥͷҐஔ
ఢΩϟϥͷ଎౓
ɹ2
st
, ߦಈ

ɹ2
st
, ߦಈ

ɹ2
st
, ߦಈ

ٯ఻೻ʹΑΔֶश
Deep Q-NetworkʢDQNʣ
૚

View Slide

χϡʔϥϧωοτϫʔΫͷߏங
ߦྻੵͷԋࢉࢠΛఆٛ
public func *(left: la_object_t, right: la_object_t) -> la_object_t {
return la_matrix_product(left, right)
}
֤૚ΛΫϥεͱ࣮ͯ͠૷
class MiddleLayer: BaseLayer{
…
}
సஔͷ࣮૷
extension la_object_t {
var trans : la_object_t {
return la_transpose(self)
}
}

View Slide

χϡʔϥϧωοτϫʔΫͷߏங
ॱ఻೻ͷϝιου
func forward(x: la_object_t) -> la_object_t {
let u_mat = x_mat * self.w + self.b
let y = sigmoid(u: u)
return y
}
ٯ఻೻ͷϝιου
func backward(t: la_object_t) -> la_object_t {
let delta = self.y - t
self.dW = self.x.trans * delta
self.db = delta
let dx = delta * self.w.trans
return dx
}

View Slide

χϡʔϥϧωοτϫʔΫͷߏ੒
ೖྗ૚ தؒ૚ தؒ૚ ग़ྗ૚
O O
1PMFͷ֯౓
1PMFͷ֯଎౓
2஋ʢࠨʹಈ͘ʣ
2஋ʢӈʹಈ͘ʣ

View Slide

Demo: Cart Pole໰୊ -Deep Q-Network-

View Slide

ΤʔδΣϯτͷઓུ
ͿΔͿΔҠಈ࡞ઓ

View Slide

εΠον࡞ઓ

View Slide

Χ΢ϯλʔΞλοΫ

View Slide

·ͱΊ
ֶशՄೳͳχϡʔϥϧωοτϫʔΫ
ˠ"DDFMFSBUF'SBNFXPSLͷ#-"4Λ࢖͑͹ൺֱత༰қ
গͳ͘ͱ΋ɺൺֱతγϯϓϧͳ$BSU1PMF໰୊ͷΑ͏ͳ΋ͷͰ͋Ε͹
J04ʹਂ૚ڧԽֶश͸࣮૷Ͱ͖Δ
$BSU1PMF໰୊
ˠ2ֶशɺ%FFQ2/FUXPSLΛ࣮૷
ˠ"*͕ࢥΘ͵૑଄ੑʢʁʣΛൃش͢Δ

View Slide

͝੩ௌɺ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ
ຊ೔࢖༻ͨ͠ίʔυ
IUUQTHJUIVCDPNZVLJOBHB$BSU1PMF4XJGU
IUUQTHJUIVCDPNZVLJOBHB$BSU1PMF%FFQ4XJGU

View Slide

iOSと（深層）強化学習

iOSと（深層）強化学習

yuky_az

More Decks by yuky_az

Other Decks in Programming

Featured

Transcript