Slide 1
Slide 1 text
Super Resolution with
CoreML
Kentaro Matsumae
@DeNA
#tryswiftconf
2018.03.02 try! Swift
Slide 2
Slide 2 text
Super Resolution (SR)
SR
Slide 3
Slide 3 text
Super Resolution (SR)
SR
Slide 4
Slide 4 text
SR Method
• SRCNN
• SR method with Deep Learning technology
• Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang,
’Image Super-Resolution Using Deep Convolutional
Networks’ (2015)
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
Reduce Manga image data size
with SRCNN + CoreML
Slide 7
Slide 7 text
Demo
Slide 8
Slide 8 text
400x600 WebP
50 KB Display
SR on CoreML
Overview
800x1200 WebP
200 KB
Client
Server
Data Size
1/4
Resize
Slide 9
Slide 9 text
Client
Server
?
How to get MLModel
Slide 10
Slide 10 text
How to get MLModel
(A) Use an open source MLModel
(B) Train your own model
Slide 11
Slide 11 text
• https://github.com/imxieyi/waifu2x-ios
• Waifu2x iOS version
Try, but…
waifu2x-ios
Slide 12
Slide 12 text
#1. Illegible Serif characters
waifu2x-ios
Original
Slide 13
Slide 13 text
#2. Lost screen tone texture
waifu2x-ios
Original
Slide 14
Slide 14 text
Waifu2x is trained using Anime Style images, not
Manga style.
Why?
Slide 15
Slide 15 text
How to prepare MLModel file
(A) Use public MLModel
(B) Train your own model (with Manga images)
Slide 16
Slide 16 text
Training
Slide 17
Slide 17 text
Implement SRCNN
Slide 18
Slide 18 text
Training Environment
• Training Data are Manga image files.
• 340,000 patch images
• AWS EC2 GPU instance (p3.2xlarge )
Slide 19
Slide 19 text
About 24hr later… ($25)
PSNR: 45.9 db
Slide 20
Slide 20 text
Results
Slide 21
Slide 21 text
Waifu2x-ios Our model
Original
Illegible Serif characters
Slide 22
Slide 22 text
Waifu2x-ios Our model
Original
Lost screen tone texture
Slide 23
Slide 23 text
Import a trained model into
your app
Slide 24
Slide 24 text
Convert to MLModel
from coremltools.converters.keras import convert
model = convert(‘model.h5’, …)
coreml_model.save('SRCNN.mlmodel')
SRCNN.mlmodel
400 KB
Slide 25
Slide 25 text
Run Super Resolution process
let model = SRCNN()
for patch in patches {
let res = try! model.prediction(image: patch.buff)
outs.append(res)
}
Slide 26
Slide 26 text
Performance
Patch size Device Time
32x32 iPhone X 10.89 sec
112x112 iPhone X 2.39 sec
200x200 iPhone X 1.04 sec
200x200 iPhone 7 1.21 sec
Slide 27
Slide 27 text
This is useful for any types of images.
Not only Manga.
Slide 28
Slide 28 text
Open Source
https://github.com/DeNA/SRCNNKit
Slide 29
Slide 29 text
let imageView: UIImageView = …
let image: UIImage = …
imageView.setSRImage(image) //Super Resolution
Slide 30
Slide 30 text
SRCNNKit
•UIImageView+SRCNN extension
•SRCNNConverter (UIImage to UIImage)
•Include pre-trained model
•Include python script to train your own model
Slide 31
Slide 31 text
SRCNNKit
•UIImageView+SRCNN extension
•SRCNNConverter (UIImage to UIImage)
•Include pre-trained model
•Include python script to train your own model
Coming Soon
Slide 32
Slide 32 text
Recap
• Reduced image file size with CoreML + SRCNN
• You need only Swift skill ( if you have a model )
• CoreML is a good parts for building apps
• I feel the CoreML have great potential for the future
Slide 33
Slide 33 text
Thank You
Twitter / github @kenmaz
https://github.com/DeNA/SRCNNKit