IMF and tensor interchange system

Slide 1

Slide 1 text

IMF and tensor interchange system Kenta Murata Tokyo Ruby Kaigi 11 2016.05.28

Slide 2

Slide 2 text

self.inspect • name: Kenta Murata • aﬃliation: ‣ Ruby committer ‣ Julia Tokyo ‣ Recruit Holdings, MTL

Slide 3

Slide 3 text

self.gems • bigdecimal • daru-td • enumerable-statistics

Slide 4

Slide 4 text

Today's summary • 「Ruby でも機械学習やりたい！」 • Everything Tensor • My approach • Demonstration

Slide 5

Slide 5 text

Ruby でも機械学習やりたい！

Slide 6

Slide 6 text

SciRuby http://gems.sciruby.com

Slide 7

Slide 7 text

OMG

Slide 8

Slide 8 text

OK, use scikit-learn この流れを止めたい

Slide 9

Slide 9 text

Image Processing on Ruby • RMagick • ruby-opencv • hornetseye

Slide 10

Slide 10 text

My case • Image Classiﬁcation • Deep Learning ‣ Convolutional Neural Network • How to do it by Ruby?

Slide 11

Slide 11 text

Answer • Use Python • Use Pillow for processing images • Use Chainer for building CNN • NumPy bridges Pillow and Chainer

Slide 12

Slide 12 text

Use Python !! この流れを止めたい

Slide 13

Slide 13 text

What are missing?

Slide 14

Slide 14 text

What are missing? • Processing images ‣ Loading / saving ‣ Resizing ‣ Horizontal ﬂipping ‣ Average image calculation ‣ Multi-dimensional array conversion RMagick ruby-opencv hornetseye

Slide 15

Slide 15 text

What are missing? • Processing images ‣ Loading / saving ‣ Resizing ‣ Horizontal ﬂipping ‣ Average image calculation ‣ Multi-dimensional array conversion hornetseye  (MultiArray) ruby-opencv  (CvMat) NArray (Numo) NMatrix

Slide 16

Slide 16 text

What are missing? • Building CNN ‣ Tensor calculation ‣ Convolutional ﬁlter calculation ‣ GPGPU ‣ Neural network builder ‣ Gradient descent optimizer hornetseye  (MultiArray) ruby-opencv  (CvMat) NArray (Numo) NMatrix

Slide 17

Slide 17 text

What are missing? • Building CNN ‣ Tensor calculation ‣ Convolutional ﬁlter calculation ‣ GPGPU ‣ Neural network builder ‣ Gradient descent optimizer

Slide 18

Slide 18 text

What are missing? • Building CNN ‣ Tensor calculation ‣ Convolutional ﬁlter calculation ‣ GPGPU ‣ Neural network builder ‣ Gradient descent optimizer

Slide 19

Slide 19 text

Library relationship Image RMagick ruby-opencv hornetseye Tensor NArray (Numo) NMatrix MultiArray  (hornetseye) CvMat  (ruby-opencv) Others GPGPU DL OpenCV ML  (ruby-opencv) decisiontree liblinear gsl_matrix  (rb-gsl)

Slide 20

Slide 20 text

A lot of multi-dimensional array data types • RMagick, OpenCV, multiarray, ... • NArray, NMatrix, StatSample, ... • gsl, tensorﬂow, ... • Julia, Python, R, ...

Slide 21

Slide 21 text

Everything tensor • 1-tensor (vector) ‣ e.g. 1ch sound, text, time series scalar data • 2-tensor (matrix) ‣ e.g. 2ch sound, grayscale image, time series vector data • 3-tensor ‣ e.g. RGB image, grayscale animation • 4-tensor ‣ e.g. RGB animation

Slide 22

Slide 22 text

Image Animation Sound Tensor Function Statistical  Model Neural  Network

Slide 23

Slide 23 text

What is best for me? • RMagick • ruby-opencv • hornetseye

Slide 24

Slide 24 text

Diﬀerent gems have diﬀerent types for the same purpose • RMagick: Magick::Image • ruby-opencv: CvMat • horneteyes: MultiArray

Slide 25

Slide 25 text

What data type is best? • RMagick: Magick::Image • ruby-opencv: CvMat • horneteyes: MultiArray

Slide 26

Slide 26 text

Generic Tensor • 4 parameters ‣ component type (size) ‣ dimension ‣ shape ‣ steps / strides

Slide 27

Slide 27 text

A B G R component_size shape[2] = pixel_channels A B G R A B G R A B G R shape[1] = image_width ... ... ... ... shape[0] = im age_height shape = (image_height, image_width, pixel_channels)

Slide 28

Slide 28 text

B G R component_size shape[2] = pixel_channels B G R B G R B G R shape[1] = image_width ... ... ... shape[0] = im age_height shape = (pixel_channels, image_width, image_height)

Slide 29

Slide 29 text

0 B G R shape[2] = pixel_channels = 3 0 B G R 0 B G R 0 B G R shape[1] = image_width = 17 ... ... ... ... shape = (19, 17, 3) steps[1] = 4 shape[0] = im age_height = 19 0 0 0 0 steps = (18, 4) steps[0] = 18

Slide 30

Slide 30 text

Generic Tensor • 4 parameters ‣ component type (size) ‣ dimension ‣ shape ‣ steps / strides

Slide 31

Slide 31 text

Image Animation Sound Tensor Function Statistical  Model Neural  Network

Slide 32

Slide 32 text

With Standard Tensor Representation • We can separate to develop tensor serializer and tensor operators ‣ Image, Audio, DB I/O ‣ Tensor transformation • and easy to integrate with each other

Slide 33

Slide 33 text

I tried to make it • IMF -- Image I/O • NumBuﬀer -- Tensor data type

Slide 34

Slide 34 text

IMF • Image Manipulation Framework • https://github.com/mrkn/IMF • I've aimed it to reproduce PIL • Pluggable design

Slide 35

Slide 35 text

Current status Detect Load Save JPEG ○ ○ PNG ○ ○ △ GIF ○ WEBP ○

Slide 36

Slide 36 text

NumBuﬀer • Tensor allocation and manipulation • Data type conversion • I/F for Ruby and C • I want support both dense and sparse tensors • I will release a gem until RubyKaigi

Slide 37

Slide 37 text

Demonstration • Integrating with tensorﬂow

Slide 38

Slide 38 text

tf-waifu2x • Waifu2x on tensorﬂow • It works, but it's incomplete

Slide 39

Slide 39 text

IMF tf-wife2x tensorﬂow MPBEJNBHF XSBQX  /VN#V⒎FS $POWFSUUPqPBUUFOTPS  GPSUFOTPSqPXGSPN /VN#V⒎FS MPBEUFOTPSqPX  NPEFM BQQMZNPEFM $POWFSUUP  VJOU/VN#V⒎FS  GPSPVUQVU NBLFJNBHF  GSPN/VN#V⒎FS TBWFJNBHF

Slide 40

Slide 40 text

waifu2x 7-step  CNN Magnify x2  Nearest  Neighbor Pixel value  adjustment 7-pixel padding insertion ↑ prepared ↑ omitted ↑ incomplete ↑ OK https://github.com/mrkn/tf-waifu2x/blob/master/convert_waifu2x.py

Slide 41

Slide 41 text

Demonstration

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Machine Learning on Ruby • We need standard tensor interchanging system like numpy • It's enough to provide the common data type for dense tensors ‣ I can't select one library

Slide 44

Slide 44 text

If you're interested in • SciRuby-jp ‣ https://groups.google.com/forum/#! forum/sciruby-jp ‣ Japanese de OK • Slack ‣ https://sciruby-jp.herokuapp.com

Slide 45

Slide 45 text

No content