$30 off During Our Annual Pro Sale. View Details »

Super Resolution with CoreML @ try! Swift Tokyo 2018

kenmaz
March 02, 2018

Super Resolution with CoreML @ try! Swift Tokyo 2018

The 'super resolution' technique is used for converting low resolution image into high resolution, which reduces the amount of image data that needs to be transfered. In this talk, I'd like to show you the implementation of super resolution with CoreML and Swift, and compare the results with conventional methods.

Video:
https://www.youtube.com/watch?v=E65lXzau_0Y

https://www.tryswift.co/events/2018/tokyo/en/#coreml
https://github.com/DeNA/SRCNNKit

kenmaz

March 02, 2018
Tweet

More Decks by kenmaz

Other Decks in Programming

Transcript

  1. Super Resolution with
    CoreML
    Kentaro Matsumae

    @DeNA
    #tryswiftconf
    2018.03.02 try! Swift

    View Slide

  2. Super Resolution (SR)
    SR

    View Slide

  3. Super Resolution (SR)
    SR

    View Slide

  4. SR Method
    • SRCNN

    • SR method with Deep Learning technology

    • Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang,
    ’Image Super-Resolution Using Deep Convolutional
    Networks’ (2015)

    View Slide

  5. View Slide

  6. Reduce Manga image data size
    with SRCNN + CoreML

    View Slide

  7. Demo

    View Slide

  8. 400x600 WebP

    50 KB Display
    SR on CoreML
    Overview
    800x1200 WebP

    200 KB
    Client
    Server
    Data Size
    1/4
    Resize

    View Slide

  9. Client
    Server
    ?
    How to get MLModel

    View Slide

  10. How to get MLModel
    (A) Use an open source MLModel

    (B) Train your own model

    View Slide

  11. • https://github.com/imxieyi/waifu2x-ios

    • Waifu2x iOS version

    Try, but…
    waifu2x-ios

    View Slide

  12. #1. Illegible Serif characters
    waifu2x-ios
    Original

    View Slide

  13. #2. Lost screen tone texture
    waifu2x-ios
    Original

    View Slide

  14. Waifu2x is trained using Anime Style images, not
    Manga style.
    Why?

    View Slide

  15. How to prepare MLModel file
    (A) Use public MLModel

    (B) Train your own model (with Manga images)

    View Slide

  16. Training

    View Slide

  17. Implement SRCNN

    View Slide

  18. Training Environment
    • Training Data are Manga image files.
    • 340,000 patch images

    • AWS EC2 GPU instance (p3.2xlarge )

    View Slide

  19. About 24hr later… ($25)
    PSNR: 45.9 db

    View Slide

  20. Results

    View Slide

  21. Waifu2x-ios Our model
    Original
    Illegible Serif characters

    View Slide

  22. Waifu2x-ios Our model
    Original
    Lost screen tone texture

    View Slide

  23. Import a trained model into
    your app

    View Slide

  24. Convert to MLModel
    from coremltools.converters.keras import convert
    model = convert(‘model.h5’, …)
    coreml_model.save('SRCNN.mlmodel')
    SRCNN.mlmodel
    400 KB

    View Slide

  25. Run Super Resolution process
    let model = SRCNN()
    for patch in patches {
    let res = try! model.prediction(image: patch.buff)
    outs.append(res)
    }

    View Slide

  26. Performance
    Patch size Device Time
    32x32 iPhone X 10.89 sec
    112x112 iPhone X 2.39 sec
    200x200 iPhone X 1.04 sec
    200x200 iPhone 7 1.21 sec

    View Slide

  27. This is useful for any types of images.
    Not only Manga.

    View Slide

  28. Open Source
    https://github.com/DeNA/SRCNNKit

    View Slide

  29. let imageView: UIImageView = …
    let image: UIImage = …
    imageView.setSRImage(image) //Super Resolution

    View Slide

  30. SRCNNKit
    •UIImageView+SRCNN extension
    •SRCNNConverter (UIImage to UIImage)
    •Include pre-trained model
    •Include python script to train your own model

    View Slide

  31. SRCNNKit
    •UIImageView+SRCNN extension
    •SRCNNConverter (UIImage to UIImage)
    •Include pre-trained model
    •Include python script to train your own model
    Coming Soon

    View Slide

  32. Recap
    • Reduced image file size with CoreML + SRCNN

    • You need only Swift skill ( if you have a model )

    • CoreML is a good parts for building apps

    • I feel the CoreML have great potential for the future

    View Slide

  33. Thank You
    Twitter / github @kenmaz
    https://github.com/DeNA/SRCNNKit

    View Slide