Super Resolution with CoreML @ try! Swift Tokyo 2018

The 'super resolution' technique is used for converting low resolution image into high resolution, which reduces the amount of image data that needs to be transfered. In this talk, I'd like to show you the implementation of super resolution with CoreML and Swift, and compare the results with conventional methods.

Video:
https://www.youtube.com/watch?v=E65lXzau_0Y

https://www.tryswift.co/events/2018/tokyo/en/#coreml
https://github.com/DeNA/SRCNNKit

Transcript

1. Super Resolution with
   CoreML
   Kentaro Matsumae
   
   @DeNA
   #tryswiftconf
   2018.03.02 try! Swift
   

   View Slide

2. Super Resolution (SR)
   SR
   

   View Slide

3. Super Resolution (SR)
   SR
   

   View Slide

4. SR Method
   • SRCNN
   
   • SR method with Deep Learning technology
   
   • Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang,
   ’Image Super-Resolution Using Deep Convolutional
   Networks’ (2015)
   

   View Slide

5. View Slide

6. Reduce Manga image data size
   with SRCNN + CoreML
   

   View Slide

7. Demo
   

   View Slide

8. 400x600 WebP
   
   50 KB Display
   SR on CoreML
   Overview
   800x1200 WebP
   
   200 KB
   Client
   Server
   Data Size
   1/4
   Resize
   

   View Slide

9. Client
   Server
   ?
   How to get MLModel
   

   View Slide

10. How to get MLModel
    (A) Use an open source MLModel
    
    (B) Train your own model
    

    View Slide

11. • https://github.com/imxieyi/waifu2x-ios
    
    • Waifu2x iOS version
    
    Try, but…
    waifu2x-ios
    

    View Slide

12. #1. Illegible Serif characters
    waifu2x-ios
    Original
    

    View Slide

13. #2. Lost screen tone texture
    waifu2x-ios
    Original
    

    View Slide

14. Waifu2x is trained using Anime Style images, not
    Manga style.
    Why?
    

    View Slide

15. How to prepare MLModel file
    (A) Use public MLModel
    
    (B) Train your own model (with Manga images)
    

    View Slide

16. Training
    

    View Slide

17. Implement SRCNN
    

    View Slide

18. Training Environment
    • Training Data are Manga image files.
    • 340,000 patch images
    
    • AWS EC2 GPU instance (p3.2xlarge )
    

    View Slide

19. About 24hr later… ($25)
    PSNR: 45.9 db
    

    View Slide

20. Results
    

    View Slide

21. Waifu2x-ios Our model
    Original
    Illegible Serif characters
    

    View Slide

22. Waifu2x-ios Our model
    Original
    Lost screen tone texture
    

    View Slide

23. Import a trained model into
    your app
    

    View Slide

24. Convert to MLModel
    from coremltools.converters.keras import convert
    model = convert(‘model.h5’, …)
    coreml_model.save('SRCNN.mlmodel')
    SRCNN.mlmodel
    400 KB
    

    View Slide

25. Run Super Resolution process
    let model = SRCNN()
    for patch in patches {
    let res = try! model.prediction(image: patch.buff)
    outs.append(res)
    }
    

    View Slide

26. Performance
    Patch size Device Time
    32x32 iPhone X 10.89 sec
    112x112 iPhone X 2.39 sec
    200x200 iPhone X 1.04 sec
    200x200 iPhone 7 1.21 sec
    

    View Slide

27. This is useful for any types of images.
    Not only Manga.
    

    View Slide

28. Open Source
    https://github.com/DeNA/SRCNNKit
    

    View Slide

29. let imageView: UIImageView = …
    let image: UIImage = …
    imageView.setSRImage(image) //Super Resolution
    

    View Slide

30. SRCNNKit
    •UIImageView+SRCNN extension
    •SRCNNConverter (UIImage to UIImage)
    •Include pre-trained model
    •Include python script to train your own model
    

    View Slide

31. SRCNNKit
    •UIImageView+SRCNN extension
    •SRCNNConverter (UIImage to UIImage)
    •Include pre-trained model
    •Include python script to train your own model
    Coming Soon
    

    View Slide

32. Recap
    • Reduced image file size with CoreML + SRCNN
    
    • You need only Swift skill ( if you have a model )
    
    • CoreML is a good parts for building apps
    
    • I feel the CoreML have great potential for the future
    

    View Slide

33. Thank You
    Twitter / github @kenmaz
    https://github.com/DeNA/SRCNNKit
    

    View Slide