Super Resolution with CoreML (Long Version)

Super Resolution with CoreML (Long Version)

This is the slide presented at try! Swift NewYork 2018.
https://www.tryswift.co/events/2018/nyc/#coreml

The ‘Super Resolution' technique is used for converting low resolution images into high resolution, which reduces the amount of image data that needs to be transferred. In this talk, I'd like to show you the implementation of super resolution with CoreML and Swift, and compare the results with conventional methods. I’ll also talk about how to train your own model using your own data step by step. In addition, I’d like to introduce recent topics such as Turi Create, Swift for TensorFlow, CoreML2 and CreateML, which will be enhanced in iOS 12 and I’ll look at how iOS will take advantage of machine learning technology in the future.

D2f212ce418f3daa29c23914c9b6892b?s=128

Kentaro Matsumae

September 05, 2018
Tweet

Transcript

  1. Super Resolution with CoreML Kentaro Matsumae Mercari #tryswiftnyc Sep 5th

    2018 try! Swift NYC
  2. +1 64 6, 64

  3. .BOHB"QQ 1SFWJPVTXPSL ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack

    SHUHO SATO
  4. .BOHB"QQ 1SFWJPVTXPSL ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack

    SHUHO SATO
  5. .BOHB"QQ 1SFWJPVTXPSL ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack

    SHUHO SATO
  6. 3FTPMVUJPOWT'JMF4J[F ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO

    SATO
  7. Y Y 3FTPMVUJPOWT'JMF4J[F ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black

    Jack SHUHO SATO
  8. Y Y 3FTPMVUJPOWT'JMF4J[F ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black

    Jack SHUHO SATO
  9. Y Y 3FTPMVUJPOWT'JMF4J[F ,# ,# ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards

    to Black Jack SHUHO SATO
  10. Y Y 3FTPMVUJPOWT'JMF4J[F ,# ,# ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards

    to Black Jack SHUHO SATO
  11. 4VQFS3FTPMVUJPO -PX3FTPMVUJPO )JHI3FTPMVUJPO ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black

    Jack SHUHO SATO
  12. $PSF.-

  13. 4VQFS3FTPMVUJPOXJUI$PSF.- 4VQFS 3FTPMVUJPO     ,# Y ϒϥοΫδϟοΫʹΑΖ͘͠

    ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  14. %FNP

  15. 4VQFS3FTPMVUJPO.FDIBOJTN

  16. *NBHF4VQFS3FTPMVUJPO6TJOH %FFQ$POWPMVUJPOBM/FUXPSLT $IBP%POHFUBM

  17. 43$// 4VQFS3FTPMVUJPOVTJOH$POWPMVUJPOBM /FVSBM/FUXPSL

  18. 43$// 4VQFS3FTPMVUJPOVTJOH$POWPMVUJPOBM /FVSBM/FUXPSL

  19. ,FSOFMBOE$POWPMVUJPO ,FSOFM ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack

    SHUHO SATO
  20. ,FSOFMBOE$POWPMVUJPO ,FSOFM *OQVU*NBHF ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black

    Jack SHUHO SATO
  21. ,FSOFMBOE$POWPMVUJPO ,FSOFM *OQVU*NBHF ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black

    Jack SHUHO SATO
  22. ,FSOFMBOE$POWPMVUJPO ,FSOFM *OQVU*NBHF 0VUQVU*NBHF $POWPMVUJPO ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards

    to Black Jack SHUHO SATO
  23. /FVSBM/FUXPSL *OQVU ʜ ʜ 0VUQVU

  24. $POWPMVUJPOBM/FVSBM/FUXPSL $// *OQVU ʜ ʜ 0VUQVU

  25. $POW-BZFS $POW-BZFS $POW-BZFS 4VQFS3FTPMVUJPOXJUI$// 43$// ʜ ʜ ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give

    My Regards to Black Jack SHUHO SATO
  26. $POW-BZFS $POW-BZFS $POW-BZFS 4VQFS3FTPMVUJPOXJUI$// 43$// ʜ ʜ ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give

    My Regards to Black Jack SHUHO SATO
  27. $POW-BZFS $POW-BZFS $POW-BZFS 4VQFS3FTPMVUJPOXJUI$// 43$// ʜ ʜ ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give

    My Regards to Black Jack SHUHO SATO *OQVU -PX3FTPMVUJPO
  28. $POW-BZFS $POW-BZFS $POW-BZFS 4VQFS3FTPMVUJPOXJUI$// 43$// ʜ ʜ ʜ ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ

    Give My Regards to Black Jack SHUHO SATO *OQVU -PX3FTPMVUJPO
  29. $POW-BZFS $POW-BZFS $POW-BZFS 4VQFS3FTPMVUJPOXJUI$// 43$// ʜ ʜ ʜ ʜ ϒϥοΫδϟοΫʹΑΖ͘͠

    ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO *OQVU -PX3FTPMVUJPO
  30. 0VUQVU )JHI3FTPMVUJPO $POW-BZFS $POW-BZFS $POW-BZFS 4VQFS3FTPMVUJPOXJUI$// 43$// ʜ ʜ ʜ

    ʜ ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO *OQVU -PX3FTPMVUJPO
  31. 0VUQVU )JHI3FTPMVUJPO *NBHF $POW-BZFS $POW-BZFS $POW-BZFS 4VQFS3FTPMVUJPOXJUI$// 43$// ʜ ʜ

    ʜ ʜ ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  32. 0VUQVU )JHI3FTPMVUJPO *NBHF $POW-BZFS $POW-BZFS $POW-BZFS 4VQFS3FTPMVUJPOXJUI$// 43$// ʜ ʜ

    ʜ ʜ ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  33. 0VUQVU )JHI3FTPMVUJPO *NBHF $POW-BZFS $POW-BZFS $POW-BZFS 4VQFS3FTPMVUJPOXJUI$// 43$// ʜ ʜ

    ʜ ʜ ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  34. 0VUQVU )JHI3FTPMVUJPO *NBHF $POW-BZFS $POW-BZFS $POW-BZFS 4VQFS3FTPMVUJPOXJUI$// 43$// ʜ ʜ

    ʜ ʜ ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  35. 5SBJOJOH

  36. )BSEUPDPMMFDUUSBJOJOHEBUB

  37. &BTZUPDPMMFDUUSBJOJOHEBUBGPS43$// )3*NBHFT ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack

    SHUHO SATO
  38. &BTZUPDPMMFDUUSBJOJOHEBUBGPS43$// )3*NBHFT ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack

    SHUHO SATO
  39. &BTZUPDPMMFDUUSBJOJOHEBUBGPS43$// )3*NBHFT )31BUDIFT ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black

    Jack SHUHO SATO
  40. &BTZUPDPMMFDUUSBJOJOHEBUBGPS43$// )3*NBHFT -31BUDIFT )31BUDIFT Y #JDVCJD *OUFSQPMBUJPO ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give

    My Regards to Black Jack SHUHO SATO
  41. 5SBJOJOH4FU ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO

    SATO
  42. ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO

  43. 5SBJOJOH43$// )31BUDI -31BUDI *OQVU ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to

    Black Jack SHUHO SATO
  44. 5SBJOJOH43$// 0VUQVU )31BUDI -31BUDI *OQVU ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards

    to Black Jack SHUHO SATO
  45. 5SBJOJOH43$// 0VUQVU -PTT 'VODUJPO )31BUDI -31BUDI *OQVU ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give

    My Regards to Black Jack SHUHO SATO
  46. 5SBJOJOH43$// 0VUQVU -PTT 'VODUJPO -PTT )31BUDI -31BUDI *OQVU ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ

    Give My Regards to Black Jack SHUHO SATO
  47. 5SBJOJOH43$// 0VUQVU -PTT 'VODUJPO 6QEBUFLFSOFMUPNJOJNJ[FMPTT -PTT )31BUDI -31BUDI *OQVU ϒϥοΫδϟοΫʹΑΖ͘͠

    ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  48. 5SBJOJOH43$// 0VUQVU -PTT 'VODUJPO 6QEBUFLFSOFMUPNJOJNJ[FMPTT )31BUDI -31BUDI *OQVU ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ

    Give My Regards to Black Jack SHUHO SATO
  49. ,FSOFM8FJHIU CFGPSFUSBJOJOH

  50. ,FSOFM8FJHIU BGUFSUSBJOJOH

  51. ,FSOFM8FJHIU BGUFSUSBJOJOH (BVTTJBO'JMUFS

  52. ,FSOFM8FJHIU BGUFSUSBJOJOH (BVTTJBO'JMUFS -BQMBDJBO'JMUFS -BQMBDJBOpMUFS

  53. ,FSOFM8FJHIU BGUFSUSBJOJOH (BVTTJBO'JMUFS -BQMBDJBO'JMUFS -BQMBDJBOpMUFS &EHF%FUFDUPS'JMUFS

  54. *NQMFNFOUBUJPO43$//

  55. 43$//.PEFM4USVDUVSF ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO

    SATO
  56. 43$//.PEFM4USVDUVSF <  > <  > <  

    > <   > <   > ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  57. 43$//.PEFM4USVDUVSF <  > <  > <  

    > <   > <   >  QBSBNFUFST ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  58. 5SBJOJOHDPEF ch = 1 SRCNN = Sequential(input_shape=(200, 200, ch)) SRCNN.add(Conv2D(128,

    9, 9, activation='relu')) SRCNN.add(Conv2D(64, 3, 3, activation='relu')) SRCNN.add(Conv2D(ch, 5, 5, activation='linear')) adam = Adam(lr=0.0003) SRCNN.compile(optimizer=adam, loss='mean_squared_error') SRCNN.fit(..) SRCNN.save(model_dir, 'model.h5')
  59. $POWFSUUP$PSF.-.PEFM from coremltools.converters.keras import convert model = convert(‘model.h5’, …) model.save('SRCNN.mlmodel')

    SRCNN.mlmodel
  60. $PSF.-.PEFMpMFTJ[F SRCNN.mlmodel 440 KB IUUQTEFWFMPQFSBQQMFDPNNBDIJOFMFBSOJOHCVJMESVONPEFMT  .PCJMF/FU .# 4RVFF[F/FU .#

    (PPH-F/FU1MBDFT .# 3FTOFU .# *ODFQUJPO7 .# 7(( .#
  61. &NCFE$PSF.-.PEFMJOUP"QQ

  62. &NCFE$PSF.-.PEFMJOUP"QQ let model = SRCNN() let highImage = try! model.predict(image:

    lowImage)
  63. .-.PEFM"DDVSBDZ *NQSPWFNFOU

  64. "DDFMFSBUJOHUIF4VQFS3FTPMVUJPO $POWPMVUJPOBM/FVSBM/FUXPSL $IBP%POHFUBM

  65. 'BTU4VQFS3FTPMVUJPOCZ$// '43$//

  66. %FNP

  67. '43$//QBSBNFUFST ʜ *OQVU -3*NBHF 0VUQVU )3*NBHF ʜ ʜ ʜ ʜ

    ʜ ʜ <   > <   > <   > <   > <   > <   > <   > <   > ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  68. 43$// QBSBNFUFST '43$//QBSBNFUFST ʜ *OQVU -3*NBHF 0VUQVU )3*NBHF ʜ ʜ

    ʜ ʜ ʜ ʜ <   > <   > <   > <   > <   > <   > <   > <   > ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  69. 43$// QBSBNFUFST '43$//QBSBNFUFST ʜ *OQVU -3*NBHF 0VUQVU )3*NBHF ʜ ʜ

    ʜ ʜ ʜ ʜ <   > <   > <   > <   > <   > <   > <   > <   > '43$// QBSBNFUFST ϒϥοΫδϟοΫʹΑΖ͘͠ ࠤ౻लๆ Give My Regards to Black Jack SHUHO SATO
  70. Performance #JDVCJD 43$// '43$// 1SFEJDUJPO 5JNFQBHF TFD TFD TFD 5SBJOJOH

    5JNF  IPVST IPVST .-.PEFM 4J[F  ,# ,# 14/3 EC EC EC
  71. 3FDFOUHSFBUQBQFST 1IPUP3FBMJTUJD4JOHMF*NBHF4VQFS3FTPMVUJPO6TJOHB(FOFSBUJWF "EWFSTBSJBM/FUXPSL $ISJTUJBO-FEJHFUBM 3FBM5JNF4JOHMF*NBHFBOE7JEFP4VQFS3FTPMVUJPO6TJOHBO&⒏DJFOU4VC 1JYFM$POWPMVUJPOBM/FVSBM/FUXPSL 8FO[IF4IJFUBM "DDVSBUF*NBHF4VQFS3FTPMVUJPO6TJOH7FSZ%FFQ$POWPMVUJPOBM/FUXPSLT +JXPO,JNFUBM 1JYFM3FDVSTJWF4VQFS3FTPMVUJPO

    3ZBO%BIMFUBM
  72. This is useful for any type of image. Not only

    manga.
  73. This is useful for any type of image. Not only

    manga. Give it a try!
  74. https://github.com/kenmaz/SuperResolutionKit

  75. Open Source https://github.com/kenmaz/SuperResolutionKit

  76. Easy to use let imageView: UIImageView = … let image:

    UIImage = … imageView.setSRImage(image) //Super Resolution⭐ pod “SuperResolutionKit”
  77. SuperResolutionKit

  78. SuperResolutionKit •UIImageView extension

  79. SuperResolutionKit •UIImageView extension •SRConverter (UIImage to UIImage)

  80. SuperResolutionKit •UIImageView extension •SRConverter (UIImage to UIImage) •Include pre-trained model

    (FSRCNN, SRCNN)
  81. SuperResolutionKit •UIImageView extension •SRConverter (UIImage to UIImage) •Include pre-trained model

    (FSRCNN, SRCNN) •Include Python script to train your own model
  82. -JNJUBUJPO

  83. -JNJUBUJPO w1SFUSBJOFENPEFMQFSGPSNBODF

  84. -JNJUBUJPO w1SFUSBJOFENPEFMQFSGPSNBODF w2VBMJUZEFQFOETPOBUZQFTPGJNBHF

  85. )PXUPMFWFSBHF.-JO .PCJMF"QQEFWFMPQNFOU

  86. .PUJWBUJPOGPS.-JO.PCJMF"QQT

  87. .PUJWBUJPOGPS.-JO.PCJMF"QQT 8IZOPUKVTUSVOBMMUIJTPOUIFTFSWFSTJEF #

  88. .PUJWBUJPOGPS.-JO.PCJMF"QQT w2VJDLSFTQPOTF 8IZOPUKVTUSVOBMMUIJTPOUIFTFSWFSTJEF #

  89. .PUJWBUJPOGPS.-JO.PCJMF"QQT w2VJDLSFTQPOTF w0⒐JOFTVQQPSU 8IZOPUKVTUSVOBMMUIJTPOUIFTFSWFSTJEF #

  90. .PUJWBUJPOGPS.-JO.PCJMF"QQT w2VJDLSFTQPOTF w0⒐JOFTVQQPSU w3FEVDJOHTFSWFSBOEOFUXPSLDPTU 8IZOPUKVTUSVOBMMUIJTPOUIFTFSWFSTJEF #

  91. .PUJWBUJPOGPS.-JO.PCJMF"QQT w2VJDLSFTQPOTF w0⒐JOFTVQQPSU w3FEVDJOHTFSWFSBOEOFUXPSLDPTU w4FDVSJUZ 8IZOPUKVTUSVOBMMUIJTPOUIFTFSWFSTJEF #

  92. $IPPTFUIFCFTUXBZ 5FDIOPMPHZ (PBMT -FWFM 7JTJPOGSBNFXPSL 'BDF5FYU%FUFDUJPO  &BTZ $SFBUF.- $VTUPN*NBHF5FYU$MBTTJpDBUJPO

     &BTZ 5VSJ$SFBUF $VTUPN0CKFDU%FUFDUJPO 4UZMF5SBOTGFS  .JE $PSF.- :PVSPXONPEFM 0UIFST $VUUJOHFEHFNFUIPET )BSE
  93. 4IPVME*MFBSO1ZUIPO

  94. 4IPVME*MFBSO1ZUIPO w:FT GPSOPX

  95. 4IPVME*MFBSO1ZUIPO w:FT GPSOPX w1ZUIPO /VN1Z ,FSBT 5FOTPS'MPXʜ

  96. 4IPVME*MFBSO1ZUIPO w:FT GPSOPX w1ZUIPO /VN1Z ,FSBT 5FOTPS'MPXʜ w1ZUIPOBMUFSOBUJWF

  97. 4IPVME*MFBSO1ZUIPO w:FT GPSOPX w1ZUIPO /VN1Z ,FSBT 5FOTPS'MPXʜ w1ZUIPOBMUFSOBUJWF w6TF1ZUIPO"1*TXJUI%ZOBNJD.FNCFS-PPLVQ

  98. from keras.models import Sequential from keras.layers import Conv2D from keras.optimizers

    import Adam ch = 1 SRCNN = Sequential(input_shape=(200, 200, ch)) SRCNN.add(Conv2D(128, 9, 9, activation='relu')) SRCNN.add(Conv2D(64, 3, 3, activation='relu')) SRCNN.add(Conv2D(ch, 5, 5, activation='linear')) adam = Adam(lr=0.0003) SRCNN.compile(optimizer=adam, loss='mean_squared_error')
  99. let Sequential = Python.import("keras.models.Sequential") let Conv2D = Python.import("keras.layers.Conv2D") let Adam

    = Python.import("keras.optimizers.Adam") let ch = 1 let SRCNN = Sequential(input_shape: (200, 200, ch)) SRCNN.add(Conv2D(128, (9, 9), activation: 'relu')) SRCNN.add(Conv2D(64, (3, 3), activation: 'relu')) SRCNN.add(Conv2D(ch, (5, 5), activation: 'linear')) let adam = Adam(lr: 0.0003) SRCNN.compile(optimizer: adam, loss: 'mean_squared_error') %ZOBNJD.FNCFS-PPLVQ
  100. 4IPVME*MFBSO1ZUIPO w:FT GPSOPX w1ZUIPO /VN1Z ,FSBT 5FOTPS'MPXʜ w1ZUIPOBMUFSOBUJWF w6TF1ZUIPO"1*TXJUI%ZOBNJD.FNCFS-PPLVQ

  101. 4IPVME*MFBSO1ZUIPO w:FT GPSOPX w1ZUIPO /VN1Z ,FSBT 5FOTPS'MPXʜ w1ZUIPOBMUFSOBUJWF w6TF1ZUIPO"1*TXJUI%ZOBNJD.FNCFS-PPLVQ w4XJGUGPS5FOTPS'MPX

  102. 3FDBQ w*NBHF4VQFS3FTPMVUJPOEFUBJMT w)PXUPJNQMFNFOU*NBHF4VQFS 3FTPMVUJPOXJUI1ZUIPOBOE$PSF.- w4VQFS3FTPMVUJPO,JU 1MFBTFDPOUSJCVUF

  103. 5IBOL:PV Twitter / github @kenmaz https://github.com/kenmaz/SuperResolutionKit