Upgrade to Pro — share decks privately, control downloads, hide ads and more …

サムネイルマスタとgo-thumber / Use thumbnail master with go-thumber

Harukasan
February 18, 2015

サムネイルマスタとgo-thumber / Use thumbnail master with go-thumber

画像変換Night / 2015-02-18

Harukasan

February 18, 2015
Tweet

More Decks by Harukasan

Other Decks in Technology

Transcript

  1. αϜωΠϧϚελͱgo-thumber
    ͸Δ͔͞Μ / Shunsuke Michii
    αϜωΠϧม׵Night
    2015-02-18

    View full-size slide

  2. ͸Δ͔͞Μ / Shunsuke Michii
    • ͍ͭ΋͸ը૾഑৴ͱ͔ը૾पΓͱ͔΍ͬͯΔ
    • Fluentdͱ͔Kibanaͱ͔σʔλղੳ·ΘΓ΋΍ͬͯΔ
    • ࡢ೔͔ΒSwiftΛॻ͖࢝Ίͨʢۀ຿ʣ
    Infrastructure team at pixiv Inc., since 2012

    View full-size slide

  3. αϜωΠϧϚελͱgo-thumber
    • WebαʔϏεʹ͓͚Δը૾౤ߘػೳࣄྫ
    • ϦΫΤετ࣌ͷαϜωΠϧੜ੒ʹ͓͚Δ໰୊఺
    • αϜωΠϧϚελ
    • JPEGʹ͓͚Δ໰୊
    • go-thumber

    View full-size slide

  4. WebαʔϏεʹ͓͚Δ

    ը૾౤ߘػೳࣄྫ
    • ҰൠతͳCGMʹ͓͚Δ࿩
    • ͨͿΜΑ͋͘Δߏ੒

    View full-size slide

  5. Application
    Creator Storage
    CDN
    Users
    • ը૾σʔλग़ྗ • ೖྗը૾ݕূ
    • อଘ
    • ഑৴

    View full-size slide

  6. Creator EC2 S3
    CloudFront
    Users
    • ը૾σʔλग़ྗ • ೖྗը૾ݕূ
    • อଘ
    • ഑৴

    View full-size slide

  7. ࣄྫ:

    ΦϦδφϧը૾Λͦͷ··഑৴
    • సૹྔ͕ΊͬͪΌ͓͓͍
    • ը૾ͷಡΈࠐΈ͕஗͍
    • σβΠϯΛ͋ͯΔͷ͕ͪΐͬͱେม
    • ͋ͱ͔ΒαϜωΠϧͭ͘Δͷ͕େม
    • αʔϏε͕େ͖͘ͳΔʹͭΕ͖ͯͭ͘ͳΔ

    View full-size slide

  8. Creator Application Storage
    CDN
    Users
    • ը૾σʔλग़ྗ • ೖྗը૾ݕূ
    • αϜωΠϧੜ੒
    • ޙ͔Βੜ੒

    View full-size slide

  9. ࣄྫ:

    Ξοϓϩʔυ࣌ʹαϜωΠϧੜ੒
    • Ξοϓϩʔυॲཧ͕େม
    • αΠζมߋ͢Δͷ͕େม
    • Ξοϓϩʔυ͢ΔϢʔβΛ଴ͨͤͳ͍Α͏ʹ

    ඇಉظॲཧʹͨ͠Γ͢Δඞཁ͕͋Δ

    View full-size slide

  10. Creator Application Storage
    CDN
    Users
    • ը૾σʔλग़ྗ • ೖྗը૾ݕূ
    Converter
    • ಈతੜ੒

    View full-size slide

  11. ࣄྫ:

    ϦΫΤετ࣌ʹαϜωΠϧੜ੒
    • ฐࣾͰ͸ಈతม׵ͱݺ͹Ε͍ͯΔ
    • σβΠϯมߋ͕؆୯
    • Ωϟογϡͯ͘͠ΕΕ͹ͦΜͳʹϦΫΤετ͸ͳ͍
    • SMALLLIGHTɺTOFUͳͲ
    • ͍͍ͩͨΑͦ͞͏

    View full-size slide

  12. ϦΫΤετ࣌ͷ

    αϜωΠϧੜ੒ͰࠔΔ͜ͱ
    • ੜ੒ݩը૾ʹґଘ
    • ॲཧ͕໘౗

    View full-size slide

  13. ੜ੒ݩը૾ʹґଘ
    • PNGͩͬͨΓJPEGͩͬͨΓGIFͩͬͨΓ͢Δ
    • αΠζ͕ἧ͍ͬͯͳ͍ͷͰॲཧ࣌ؒ΍࢖༻ϝϞϦྔ͕

    όϥόϥ

    View full-size slide

  14. ॲཧ͕໘౗
    • ͍ΖΜͳը૾ϑΥʔϚοτʹରԠ͠ͳ͍ͱ͍͚ͳ͍
    • ൚༻తͳॲཧʹ͢Δͱॲཧ࣌ؒ΍ϝϞϦޮ཰ʹ໰୊
    • εέʔϦϯάҎ֎ʹ΋Ϋϩοϐϯάͱ͔͋Δ
    • ImageMagickͰ͍͍ͩͨͰ͖Δ͚Ͳɺ

    ͦͷͭΒ͞͸͜ͷޙͩΕ͔ޠͬͯ͘ΕΔ͸ͣ

    View full-size slide

  15. αϜωΠϧϚελ
    • αϜωΠϧੜ੒ݩͱͳΔϚελը૾
    • ϦΫΤετ࣌αϜωΠϧੜ੒ͷ໰୊Λ؇࿨͢Δ
    • ଞͷ͢΂ͯͷαϜωΠϧΑΓ΋Ͱ͔͍ʢ1200x1200ʣ
    • Ϋϩοϓͨ͠ը૾Λ༻ҙ͓ͯ͘͠

    View full-size slide

  16. ௕೥ഓΘΕͨΫϩοϓٕज़
    ԣ௕ͷը૾͸ը૾தԝΛΫϩοϓ ॎ௕ͷը૾͸ը૾্෦ΛΫϩοϓ
    ϝΠϯͱͳΔ෦෼͸

    ͍͍ͩͨਅΜதʹ͋Δ
    ॎ௕ͩͱ

    إͱ͔ϩΰͱ͔͸্ͷํʹ͋Δ

    View full-size slide

  17. ௕೥ഓΘΕͨΫϩοϓٕज़
    ਅΜதͰ੾ΓऔΔͱ͜͏ͳΔ ԣ௕ͷը૾͸ը૾্෦ΛΫϩοϓ
    ॎ௕ͩͱ

    إͱ͔ϩΰͱ͔͸্ͷํʹ͋Δ

    View full-size slide

  18. Creator Application Storage
    CDN
    Users
    • ը૾σʔλग़ྗ • ೖྗը૾ݕূ
    Converter
    • ಈతੜ੒
    Thumbnail
    Master
    Thumbnail
    Master

    View full-size slide

  19. αϜωΠϧϚελΛجʹͨ͠

    αϜωΠϧը૾ੜ੒
    • Ϛελը૾ʹ͸JPEGΛ࠾༻
    • ༨ܭͳσʔλΛམͱ͓ͯ͘͠
    • εέʔϦϯάσίʔυͰ͖ΔͷͰCPU/ϝϞϦ࢖༻ྔΛ࡟ݮ
    • ·͋JPEGʹ΋͍Ζ͍Ζͱ໰୊͕͋Δ

    View full-size slide

  20. JPEGʹ͓͚Δ໰୊
    • ૬ޓޓ׵༻ϑΥʔϚοτͰ͋ΔJIF͕ඪ४Խ͞Ε͕ͨ

    ৭ʑͱൈ͚͍ͯͨ
    • ΧϥʔϞσϧͱ͔αϒαϯϓϦϯάͱ͔ॏཁͳࣄ͕

    ܾ·͍ͬͯͳ͍
    • த਎͕RGBͰ΋ྑ͍͠ɺCMYKͰ΋ྑ͍͠ɺ

    ΞϧϑΝνϟϯωϧ͕͋ͬͯ΋ྑ͍͠ԿͰ΋͋Γ

    View full-size slide

  21. JPEG/JFIF
    • APP0-JFIFϔομʹඞཁͳ৘ใΛೖΕΔ͜ͱʹܾΊͨ
    • ΧϥʔϞσϧΛY’CbCrʹܾΊͨ

    (ITU-R BT.601Λ0-255ʹม׵ͨ͠΍ͭ)
    • ίϯϙʔωϯτ͸3ͭͰͦΕͧΕYɺCbɺCrͱܾΊͨ
    • αϒαϯϓϦϯάൺΛ4छྨʹܾΊͨ

    View full-size slide

  22. JPEG/EXIF
    • ೔ຊͷσδΧϝ԰͞Μ͕ඞཁ֦ͩͬͨுΛߦͬͨ
    • JFIFΛϕʔεʹΧϥʔϞσϧ͚ͩͰͳ͘৭ۭؒΛѻ͏

    ͨΊΧϥʔϓϩϑΝΠϧΛ࢖͑ΔΑ͏ʹͨ͠
    • APP1-EXIFϔομʹTIFFϑΥʔϚοτͰ֨ೲͯ͋͠Δ:

    ଏʹEXIF৘ใͱݺ͹ΕΔͷ͸͜Ε

    View full-size slide

  23. JPEG/EXIFʹ͓͚Δ໰୊
    • ϒϥ΢β͕ICCϓϩϑΝΠϧΛղऍͨ͠Γ

    ICCϓϩϑΝΠϧ͸ղऍ͢ΔͷʹϞχλͷϓϩϑΝΠϧ͸

    ແࢹͨ͠Γ͢Δ
    • Ϣʔβ͕sRGBͰӾཡͯ͠Δͷʹ

    ग़ྗ࣌ʹAdobe RGBϓϩϑΝΠϧΛೖΕͨΓ͢Δ

    ʢը૾͕৭᧙ͤͯݟ͑ΔݪҼʣ
    • ݁ہͲ͏ѻͬͨΒྑ͍͔Α͘Θ͔Βͳ͍

    View full-size slide

  24. CMYK JPEG

    දࣔग़དྷͳ͍໰୊
    • App1 EXIFʹؚ·Ε͍ͯΔTIFF৘ใͷColorSpace͕CMYK
    ʹͳ͍ͬͯΔ΋ͷʢଏʹCMYK JPEGͱݺͿʣ
    • Ұ෦ϒϥ΢βͰදࣔͰ͖ͳ͔ͬͨʢࠓ͸େৎ෉ʁʣ
    • ฐࣾͰ͸Ұ୴PNGʹॻ͖ग़͢ṖίʔυͰ͜ͷ৘ใΛ࡟আ

    ʢϫʔΫΞϥ΢ϯυʣ

    View full-size slide

  25. ṖίʔυͲ͏ʹ͔͍ͨ͠
    // =========================================
    // CMYK形式の画像をRGB形式に変換する
    // 劣化およびディスクI/Oによる負荷を最小限にするため
    // 一旦PNGに変換してからJPEGに変換する
    // =========================================
    public static function cmyk2rgb($image, $tmp_name)
    {
    // Workaround for TransformImageColorspace() bug
    //
    // It means same below:
    //
    // $image->setImageColorspace(Imagick::COLORSPACE_RGB);
    //
    $image->setImageFormat('PNG');
    $image->setImageCompressionQuality(75);
    $image->readImageBlob($image->getImageBlob());
    $image->setImageFormat('JPEG');
    $image->setImageCompressionQuality(100);
    $image->writeImage($tmp_name);
    }

    View full-size slide

  26. JPEGʹ͓͚Δ৭ۭؒ
    • ࠞಉ͞Ε͕͚ͪͩͲEXIF৘ใͷColorSpace͸

    JPEGͷූ߸Խʹ༻͍ΒΕ͍ͯΔ৭ۭؒʢΧϥʔϞσϧʣ

    ͱ͸ҧ͏
    • JPEG/JFIFɺJPEG/EXIFͱ΋ʹΧϥʔϞσϧͱͯ͠࢖༻Ͱ͖
    Δ৭ۭؒ͸YCbCr͚ͩ
    • APP12-AdobeϔομͰRGBͱ͔CMYKͱ͔YCCKͷ

    ΧϥʔϞσϧΛ࢖͑Δ͚Ͳࠓ͸ܹϨΞ

    View full-size slide

  27. Ͳ͏͢Ε͹Α͍ͷ͔
    • ΧϥʔϚωδϝϯτͰ͖ΔΤϯδχΞ͕ෆࡏ
    • ϓϩϑΝΠϧΛਖ਼͘͠ѻ͍͍͚ͨͲ

    Ϣʔβ؀ڥ͕ͦ͏ͳͬͯͳ͍
    • ImageMagick͕Ͳ͏ͳͬͯΔ͔Α͘Θ͔Βͳ͍ʢఘΊʣ

    View full-size slide

  28. αϜωΠϧϚελͰ

    ͜ΕΒͷ໰୊Λ؇࿨͢Δ
    • αϜωΠϧੜ੒࣌ʹ͍ΖΜͳ͜ͱߟ͑Δͷ͸໘౗
    • JPEG/JFIFޓ׵ͷJPEGΛ͓ͭͬͯ͘͘
    • EXIF৘ใΛ࡟আ
    • ΧϥʔϞσϧ͸sRGB
    • ICCϓϩϑΝΠϧ͸ߟྀͯ͠ͳ͍͚ͲͲ͏ʹ͔͍ͨ͠

    View full-size slide

  29. αϜωΠϧϚελΛ

    ࢖͏͜ͱͰॲཧΛޮ཰Խ͢Δ
    • αϜωΠϧੜ੒࣌ʹ͍Ζ͍ΖͳࣄΛߟ͑ͳͯ͘ྑ͍
    • ೖྗը૾͸JPEG͚ͩ
    • ग़ྗ΋ͱΓ͋͑ͣJPEGͰྑ͍
    • ෆཁͳ৘ใ͸͢Ͱʹআڈͯ͠JPEG/JFIFޓ׵

    View full-size slide

  30. go-thumber
    • αϜωΠϧϚελ͔ΒͷαϜωΠϧੜ੒͚ͩΛߟ͑ͯ
    ઃܭ͞ΕͨαϜωΠϧੜ੒ϓϩΩγ
    • @marcan42͕਺िؒͰͭͬͨ͘
    • YCbCrσʔλҎ֎͸͢΂ͯύεεϧʔ͢ΔͷͰ

    EXIFͰ΋େৎ෉
    https://github.com/pixiv/go-thumber

    View full-size slide

  31. ϓϩΩγʹͳͬͯΔͷͰ؆୯
    GET http://localhost:8000/w=128,h=128,q=95/upstream.example.com/example.jpg
    w: ੜ੒͢ΔαϜωΠϧͷ࠷େ෯ (required)
    h: ੜ੒͢ΔαϜωΠϧͷ࠷େߴ͞ (required)
    q: JPEG quality (default 90)
    u: ࢦఆαΠζΑΓখ͔ͬͨ͞ΒΞοϓεέʔϧ(Ҿ͖Ԇ͹͠)͢Δ͔Ͳ͏͔ (default 1)
    a: ΞεϖΫτΛΩʔϓ͢Δ͔Ͳ͏͔ (default 1)
    o: JPEGͷoptimizeΛߦ͏ (default 0)
    p: μ΢ϯαϯϓϦϯάϑΝΫλ (default 2)

    View full-size slide

  32. go-thumber͸΍͍
    • LibJPEGv6 APIΛੜͰୟ͍ͯYCbCrσʔλΛऔಘɺॻ͖ࠐΈ

    ʢlibjpeg-turboΛ࢖༻ʣ
    • LibswscaleʹΑΔImageMagickͱಉਫ਼౓͔ͭߴ଎ͳ

    εέʔϦϯάॲཧ

    View full-size slide

  33. ϕϯνϚʔΫ
    • Tecnick’s public test imagesͷը૾Λ256x256pxʹॖখ͢Δ
    • ຊ౰͸Πϥετը૾ͷςετηοτ͕ग़དྷΔͱྑ͍͚Ͳ

    ݖརपΓ͕໘౗
    • mod_smalllight(ImageMagickɺImlib2)ͱൺֱ
    • Apacheͷνϡʔχϯά͸͋Μ·͕Μ͹ͬͯͳ͍
    • εέʔϦϯάσίʔυ͸༗ޮ

    View full-size slide

  34. Resize to 256x256px Benchmark (quality=99)
    Requests per second
    0
    25
    50
    75
    100
    125
    Numbers of Connections
    0 1 2 3 4 5 6 7 8 9 10
    smallight(imlib2)
    go-thumber
    smalllight(magick)
    εϧʔϓοτͷൺֱ

    View full-size slide

  35. Mean of Latency (size=256x256, quality=99)
    Mean of Latency (ms)
    0
    50
    100
    150
    200
    Numbers of Connections
    0 1 2 3 4 5 6 7 8 9 10
    smalllight(magick)
    go-thumber
    smallight(imlib2)
    ฏۉॲཧ஗Ԇͷൺֱ

    View full-size slide

  36. go-thumber͸΍͍
    • imlib2Λ࢖ͬͨ৔߹ʹඖఢ͢Δ଎͞
    • ͡Ό͋៉ྷͳͷʁ

    View full-size slide

  37. ϒϥΠϯυςετΛ΍ͬͨ
    • Ή͔͠ࢹ֮৺ཧ࣮ݧͱ͔΍ͬͨ͜ͱ͋ͬͨͷͰͳΜͱͳ͘
    ΍ͬͯΈͨ
    • ࣾ಺νϟοτʹ౤͛ͨͱ͜ΖՋͦ͏ͳΤϯδχΞ͕ࢀՃ
    • ࣮ݧ৚݅ͳͲ͸͋·Γߟ͑ͯͳ͍

    View full-size slide

  38. ݁Ռ
    දࣔճ਺ બ୒͞Εͨճ਺ બ୒͞Εׂͨ߹
    SMALLLIGHT(imlib2) 467 58 8.2%
    SMALLLIGHT(ImageMagick) 466 275 39.3%
    go-thumber 467 367 52.4%
    • Ϣʔβʔ͕៉ྷͰ͋Δͱબ୒ͨ͠ճ਺
    • go-thumberͰੜ੒͞ΕͨαϜωΠϧ͕࠷΋ଟ͘બ୒͞Εͨ

    View full-size slide

  39. go-thumberͷѹউ
    • ImageMagickͱಉ͡ϑΟϧλΛ༻͍͍ͯΔͷʹԿނ͔
    • ΋ͪΖΜΧϥΫϦ͕͋Δ

    View full-size slide

  40. • LibJPEGͰ͸1/1ʙ1/8ͷ޷͖ͳεέʔϧͰσίʔυग़དྷΔ
    • go-thumberͰ͸ੜ੒͢ΔαϜωΠϧαΠζͷ2ഒʢp=2ʣ
    • ImageMagickͰ΋αϜωΠϧαΠζͷ2ഒΛࢦఆ͢Ε͹

    ಉ͘͡Β͍៉ྷʹͳΔ͸ͣ
    • ৄ͘͠͸go-thumberͷREADME.mdʹॻ͍ͯ͋Δ
    εέʔϦϯάσίʔυͷ

    αΠζ࠷దԽ

    View full-size slide

  41. αϜωΠϧϚελͱgo-thumber
    • αϜωΠϧϚελʹ͍Ζ͍Ζͳ໰୊Λԡ͠ࠐΉ͜ͱͰ
    ϦΫΤετ࣌ͷαϜωΠϧੜ੒Λ؆୯ʹͨ͠
    • go-thumber͕҆ఆͯ͠ΔͷͰ͏·͍ͬͯ͘Δ
    • ࣍͸: webp-thumberʁ

    View full-size slide