JPEG streaming in Go

pixiv Night #02 - 画像処理技術（go, blender, C++ライブラリ等）
2017-03-14
https://pixiv.connpass.com/event/50284/

Transcript

1. JPEG streaming in Go
   !IBSVLBTBO4IVOTVLF.JDIJJ
   ͋Δ͍͸+1&(ͷϒϩοΫϊΠζͱ͸ͳʹ͔
   

   View Slide

2. Harukasan
   Shunsuke MICHII
   Lead engineer of pixiv Inc.,

   Product manager of ImageFlux,

   Infrastructure engineer, Kibana, Gopher
   

   View Slide

3. View Slide

4. © pixiv Inc., SAKURA Internet Inc.
   ImageFlux service
   ͓٬༷ετϨʔδ
   ίϯςϯπΩϟογϡ
   ը૾ม׵ॲཧ
   ΦϦδφϧը૾Λऔಘ
   ม׵ޙΩϟογϡΛ഑৴
   ΤϯυϢʔβ
   ։ൃऀ
   ഑৴URLͷઃఆ
   ഑৴ݩΦϦδϯͷઃఆ

   ϨϙʔτͷӾཡ
   طଘετϨʔδΛͦͷ··ར༻Մೳ
   ը૾σʔλΛطଘͷετϨʔδ΍αʔό͔
   ΒҠಈ͢Δඞཁ͸͋Γ·ͤΜɻ؅ཧը໘͔
   ΒΦϦδϯΛઃఆ͢Δ͚ͩͰ͙͢ʹը૾ม
   ׵͕ར༻ՄೳͰ͢ɻ
   URLΛม͑Δ͚ͩͰը૾ม׵
   ഑৴URLʹύϥϝʔλΛ௥Ճ͢Δ͚ͩͰ೚
   ҙͷը૾ม׵Λߦ͑·͢ɻ
   εϚʔτϑΥϯɺPCͰผʑͷURLΛ࢖͏͜
   ͱͰɺ1ຕͷը૾͔Β༷ʑͳαΠζͷը૾
   Λ഑৴͢Δ͜ͱ͕ՄೳͰ͢ɻ
   ը૾ม׵ॲཧ
   

   View Slide

5. © pixiv Inc., SAKURA Internet Inc.
   ImageFlux service
   ͓٬༷ετϨʔδ
   ίϯςϯπΩϟογϡ
   ը૾ม׵ॲཧ
   ΦϦδφϧը૾Λऔಘ
   ม׵ޙΩϟογϡΛ഑৴
   ΤϯυϢʔβ
   ։ൃऀ
   ഑৴URLͷઃఆ
   ഑৴ݩΦϦδϯͷઃఆ

   ϨϙʔτͷӾཡ
   طଘετϨʔδΛͦͷ··ར༻Մೳ
   ը૾σʔλΛطଘͷετϨʔδ΍αʔό͔
   ΒҠಈ͢Δඞཁ͸͋Γ·ͤΜɻ؅ཧը໘͔
   ΒΦϦδϯΛઃఆ͢Δ͚ͩͰ͙͢ʹը૾ม
   ׵͕ར༻ՄೳͰ͢ɻ
   URLΛม͑Δ͚ͩͰը૾ม׵
   ഑৴URLʹύϥϝʔλΛ௥Ճ͢Δ͚ͩͰ೚
   ҙͷը૾ม׵Λߦ͑·͢ɻ
   εϚʔτϑΥϯɺPCͰผʑͷURLΛ࢖͏͜
   ͱͰɺ1ຕͷը૾͔Β༷ʑͳαΠζͷը૾
   Λ഑৴͢Δ͜ͱ͕ՄೳͰ͢ɻ
   ը૾ม׵ॲཧ
   

   View Slide

6. Decoding a JPEG image
   Resizing
   Encoding into a new JPEG image
   Fetching from an origin
   Writing content into a client
   http.Transport
   libjpeg-turbo
   libswscale from ffmpeg
   http.ResponseWriter
   libjpeg-turbo
   

   View Slide

7. streaming in Go
   • io.Reader: reading content into the given buffer
   • io.Writer: writing content from the given buffer
   typedef Reader interface {
   func Read(p []byte) (n int, err error)
   }
   typedef Writer interface {
   func Write(p []byte) (n int, err error)
   }
   

   View Slide

8. • Copying content from a reader into a writer
   io.Copy(dst, src)
   32KB buffer
   io.Reader
   io.Writer
   Read(buffer)
   Write(buffer)
   : 32KBͷόοϑΝ͸಺෦Ͱຖճmake
   

   View Slide

9. • Copying content from a reader into a writer
   • To avoid allocating a buffer, use io.CopyBuffer:
   io.Copy(dst, src)
   32KB buffer
   io.Reader
   io.Writer
   Read(buffer)
   Write(buffer)
   io.CopyBuffer(dst, src, buf)
   : 32KBͷόοϑΝ͸಺෦Ͱຖճmake
   

   View Slide

10. func ServeHTTP(w *http.ResponseWriter, r *http.Request) {
    req, _ := http.NewRequest("GET", "http://www.example.com", nil)
    resp, _ := http.DefaultTransport.RoundTrip(req)
    

    io.CopyBuffer(w, resp.Body, buf)
    }
    HTTP΋ίϐʔ͢Δ͚ͩͰετϦʔϛϯάͰ͖ΔʢΤϥʔॲཧ͸……ʣ
    

    View Slide

11. Decoding a JPEG image
    Resizing
    Encoding into a new JPEG image
    Fetching from an origin
    Writing content into a client
    http.Transport
    libjpeg-turbo
    libswscale from ffmpeg
    http.ResponseWriter
    libjpeg-turbo
    

    View Slide

12. Streaming image scanlines
    • ScanlineReader interface
    typedef ScanlineReader interface {
    func Config() *Config → plane sizes, pixel format,...
    func ReadScanLine(p [][]uint8) (n int, err error)
    → reading scanlines of planes into each buffer
    → PNG: JPEG:
    }
    R
    G
    B
    Y
    Cb
    Cr
    *blue difference chroma
    *red difference chroma
    

    View Slide

13. Decoding a JPEG image
    Resizing
    Encoding into a new JPEG image
    Fetching from an origin
    Writing content into a client
    http.Transport
    libjpeg-turbo
    libswscale from ffmpeg
    http.ResponseWriter
    libjpeg-turbo
    

    View Slide

14. Decoding a JPEG image
    Resizing
    Encoding into a new JPEG image
    Fetching from an origin
    Writing content into a client
    http.Transport
    Decoder.ReadScanLines
    Scaler.ReadScanLines
    http.ResponseWriter
    Encoder.ReadScanLines
    

    View Slide

15. func ServeHTTP(w *http.ResponseWriter, r *http.Request) {
    req, _ := http.NewRequest("GET", "http://www.example.com", nil)
    resp, _ := http.DefaultTransport.RoundTrip(req)
    dec := jpeg.NewDecoder(resp.Body io.Reader) → ScanlineReader
    dec := scale.NewResize(dec ScanlineReader) → ScanlineReader
    enc := jpeg.NewEncoder()
    enc.Write(w io.Writer, dec ScanlineReader)
    }
    

    View Slide

16. streaming image scanlines
    • ScanlineReader interfaceΛ࢖͏͜ͱͰը૾σʔλ΋ετϦʔϜͱ͠
    ͯѻ͏͜ͱ͕Ͱ͖Δ
    • interfaceΛ࢖͏͜ͱͰ֤ॲཧͷར༻ํ๏Λ౷ҰͰ͖Δ
    • ඪ४ϥΠϒϥϦͷΠϯλʔϑΣʔε͸Α͘ߟ͑ΒΕͯΔͷͰϚω͢Δ
    ͱྑ͍ײ͡ʹͳΔ
    

    View Slide

17. ͳͥετϦʔϛϯά͢Δඞཁ͕͋Δͷ͔
    • ը૾σʔλΛ͍ͬ΃Μʹॲཧ͢Δͱશըૉ෼ͷϝϞϦ͕ඞཁ͕ͩ

    ετϦʔϜॲཧ͢Ε͹όοϑΝΛҰ౓֬อ͢Δ͚ͩͰࡁΉ
    • ֤৭ۭؒຖʹॲཧΛهड़Ͱ͖ɺແବͳ৭ม׵͕ൃੜ͠ͳ͍

    JPEGಉ࢜ͷϦαΠζͷ৔߹͸Y'CbCr৭ۭؒͷ··ѻ͏͜ͱͰɺ
    RGB΁ͷม׵ʹΑΔϩε͕ൃੜ͠ͳ͍
    • cgo಺ͰϒϩοΫ͢Δ࣌ؒΛ୹͘͢Δ͜ͱ͕Ͱ͖Δ
    

    View Slide

18. Streaming JPEG scanlines
    typedef ScanlineReader interface {
    func Config() *Config → plane sizes, pixel format,...
    func ReadScanLine(p [][]uint8) (n int, err error)
    → JPEG:
    }
    Y
    Cb
    Cr
    *blue difference chroma
    *red difference chroma
    ͭ·Γɺ͜ͷΠϯλʔϑΣʔεͰJPEGΛσίʔυ͢Ε͹JPEGΛετϦʔϜॲཧͰ͖Δ
    

    View Slide

19. color conversion
    chroma subsampling
    huffman coding
    quantization
    DCT transformation
    ReadScanline
    io.Reader
    RGB→Y'CbCrͷ৭ۭؒม׵
    ΫϩϚαϒαϯϓϦϯάॲཧ
    DCTม׵
    ྔࢠԽ
    ϋϑϚϯѹॖ
    

    View Slide

20. 8x8ϐΫηϧ͝ͱʹDCTʢ཭ࢄίαΠϯม׵ʣͰ

    ௚ྲྀ੒෼ͱप೾਺͝ͱͷަྲྀ੒෼ʹม׵

    →ߴप೾੒෼΄Ͳߥ͘ͳΔΑ͏ʹྔࢠԽ͢Δ͜ͱͰ৘ใྔΛ࡟ݮ

    →ϋϑϚϯූ߸Խͯ͠ѹॖ
    ΊͬͪΌࡶʹJPEGͷѹॖΛઆ໌͢Δͱ
    ߴप೾੒෼
    ௿प೾੒෼
    

    View Slide

21. 8x8ϐΫηϧ͝ͱʹDCTʢ཭ࢄίαΠϯม׵ʣͰ

    ௚ྲྀ੒෼ͱप೾਺͝ͱͷަྲྀ੒෼ʹม׵

    →ߴप೾੒෼΄Ͳߥ͘ͳΔΑ͏ʹྔࢠԽ͢Δ͜ͱͰ৘ใྔΛ࡟ݮ

    →ϋϑϚϯූ߸Խͯ͠ѹॖ
    ΊͬͪΌࡶʹJPEGͷѹॖΛઆ໌͢Δͱ
    ߴप೾੒෼
    ௿प೾੒෼
    

    View Slide

22. • JPEGͰ͸8x8ϐΫηϧʢ1ϒϩοΫʣ͝ͱʹDCTɺྔࢠԽΛߦ͏
    • MCU: 8x8ըૉ͝ͱʹY'CbCrΛฒ΂ͨ΋ͷ
    • ͜ͷ8x8ըૉ͝ͱʹDCT/ྔࢠԽͰ৘ใྔΛམͱ͢ͷͰϒϩοΫڥք
    ʹϊΠζ͕ൃੜˠϒϩοΫϊΠζ
    MCU
    

    View Slide

23. color conversion
    chroma subsampling
    huffman coding
    quantization
    DCT transformation
    ReadScanline
    io.Reader
    8x8ըૉ͝ͱॲཧ͢Δ

    →ετϦʔϛϯάॲཧͰ͖Δʂ
    1ըૉ͝ͱʹॲཧͰ͖Δ
    

    View Slide

24. /*
    ...
    static JDIMENSION read_scanlines(j_decompress_ptr dinfo, unsigned char *buf, int stride, int height) {
    JSAMPROW *rows = alloca(sizeof(JSAMPROW) * height);
    int i;
    for (i = 0; i < height; i++) {
    rows[i] = &buf[i * stride];
    }
    return jpeg_read_scanlines(dinfo, rows, height);
    }
    */
    import "c"
    func (dec *Decoder) ReadScanline(p [][]uint8) (n int, err error) {
    n := C.read_scanline(dec.dinfo, p, dec.Stride, dec.Height);
    return n, nil
    }
    libjpegͷ৔߹͸jpeg_read_scanlinesΛݺͿ͚ͩ
    

    View Slide

25. Conclusion
    • GoͰ͸ඇৗʹ؆୯ʹετϦʔϜॲཧΛ࣮ݱͰ͖Δ
    • JPEGͷ৔߹8ߦͣͭσίʔυ/Τϯίʔυ͢Δ͜ͱ͕Ͱ͖Δ

    →ετϦʔϜͰ͖ΔΠϯλʔϑΣʔε͢Ε͹ͪΐͬͱͣͭॲཧͰ͖Δ

    →libjpeg͸ετϦʔϜͰ͖ΔAPIΛ࣋ͬͯΔͷͰcgoͰݺͼग़͚ͩ͢
    • ϒϩοΫϊΠζͱ͸8x8ըૉ͝ͱʹॲཧ͢ΔͷͰڥքʹग़ΔϊΠζ
    

    View Slide