サムネイルマスタとgo-thumber / Use thumbnail master with go-thumber

by Harukasan

Slide 1

Slide 1 text

αϜωΠϧϚελͱgo-thumber ͸Δ͔͞Μ / Shunsuke Michii αϜωΠϧม׵Night 2015-02-18

Slide 2

Slide 2 text

͸Δ͔͞Μ / Shunsuke Michii • ͍ͭ΋͸ը૾഑৴ͱ͔ը૾पΓͱ͔΍ͬͯΔ • Fluentdͱ͔Kibanaͱ͔σʔλղੳ·ΘΓ΋΍ͬͯΔ • ࡢ೔͔ΒSwiftΛॻ͖࢝Ίͨʢۀ຿ʣ Infrastructure team at pixiv Inc., since 2012

Slide 3

Slide 3 text

αϜωΠϧϚελͱgo-thumber • WebαʔϏεʹ͓͚Δը૾౤ߘػೳࣄྫ • ϦΫΤετ࣌ͷαϜωΠϧੜ੒ʹ͓͚Δ໰୊఺ • αϜωΠϧϚελ • JPEGʹ͓͚Δ໰୊ • go-thumber

Slide 4

Slide 4 text

WebαʔϏεʹ͓͚Δ  ը૾౤ߘػೳࣄྫ • ҰൠతͳCGMʹ͓͚Δ࿩ • ͨͿΜΑ͋͘Δߏ੒

Slide 5

Slide 5 text

Application Creator Storage CDN Users • ը૾σʔλग़ྗ • ೖྗը૾ݕূ • อଘ • ഑৴

Slide 6

Slide 6 text

Creator EC2 S3 CloudFront Users • ը૾σʔλग़ྗ • ೖྗը૾ݕূ • อଘ • ഑৴

Slide 7

Slide 7 text

ࣄྫ:  ΦϦδφϧը૾Λͦͷ··഑৴ • సૹྔ͕ΊͬͪΌ͓͓͍ • ը૾ͷಡΈࠐΈ͕஗͍ • σβΠϯΛ͋ͯΔͷ͕ͪΐͬͱେม • ͋ͱ͔ΒαϜωΠϧͭ͘Δͷ͕େม • αʔϏε͕େ͖͘ͳΔʹͭΕ͖ͯͭ͘ͳΔ

Slide 8

Slide 8 text

Creator Application Storage CDN Users • ը૾σʔλग़ྗ • ೖྗը૾ݕূ • αϜωΠϧੜ੒ • ޙ͔Βੜ੒

Slide 9

Slide 9 text

ࣄྫ:  Ξοϓϩʔυ࣌ʹαϜωΠϧੜ੒ • Ξοϓϩʔυॲཧ͕େม • αΠζมߋ͢Δͷ͕େม • Ξοϓϩʔυ͢ΔϢʔβΛ଴ͨͤͳ͍Α͏ʹ  ඇಉظॲཧʹͨ͠Γ͢Δඞཁ͕͋Δ

Slide 10

Slide 10 text

Creator Application Storage CDN Users • ը૾σʔλग़ྗ • ೖྗը૾ݕূ Converter • ಈతੜ੒

Slide 11

Slide 11 text

ࣄྫ:  ϦΫΤετ࣌ʹαϜωΠϧੜ੒ • ฐࣾͰ͸ಈతม׵ͱݺ͹Ε͍ͯΔ • σβΠϯมߋ͕؆୯ • Ωϟογϡͯ͘͠ΕΕ͹ͦΜͳʹϦΫΤετ͸ͳ͍ • SMALLLIGHTɺTOFUͳͲ • ͍͍ͩͨΑͦ͞͏

Slide 12

Slide 12 text

ϦΫΤετ࣌ͷ  αϜωΠϧੜ੒ͰࠔΔ͜ͱ • ੜ੒ݩը૾ʹґଘ • ॲཧ͕໘౗

Slide 13

Slide 13 text

ੜ੒ݩը૾ʹґଘ • PNGͩͬͨΓJPEGͩͬͨΓGIFͩͬͨΓ͢Δ • αΠζ͕ἧ͍ͬͯͳ͍ͷͰॲཧ࣌ؒ΍࢖༻ϝϞϦྔ͕  όϥόϥ

Slide 14

Slide 14 text

ॲཧ͕໘౗ • ͍ΖΜͳը૾ϑΥʔϚοτʹରԠ͠ͳ͍ͱ͍͚ͳ͍ • ൚༻తͳॲཧʹ͢Δͱॲཧ࣌ؒ΍ϝϞϦޮ཰ʹ໰୊ • εέʔϦϯάҎ֎ʹ΋Ϋϩοϐϯάͱ͔͋Δ • ImageMagickͰ͍͍ͩͨͰ͖Δ͚Ͳɺ  ͦͷͭΒ͞͸͜ͷޙͩΕ͔ޠͬͯ͘ΕΔ͸ͣ

Slide 15

Slide 15 text

αϜωΠϧϚελ • αϜωΠϧੜ੒ݩͱͳΔϚελը૾ • ϦΫΤετ࣌αϜωΠϧੜ੒ͷ໰୊Λ؇࿨͢Δ • ଞͷ͢΂ͯͷαϜωΠϧΑΓ΋Ͱ͔͍ʢ1200x1200ʣ • Ϋϩοϓͨ͠ը૾Λ༻ҙ͓ͯ͘͠

Slide 16

Slide 16 text

௕೥ഓΘΕͨΫϩοϓٕज़ ԣ௕ͷը૾͸ը૾தԝΛΫϩοϓ ॎ௕ͷը૾͸ը૾্෦ΛΫϩοϓ ϝΠϯͱͳΔ෦෼͸  ͍͍ͩͨਅΜதʹ͋Δ ॎ௕ͩͱ  إͱ͔ϩΰͱ͔͸্ͷํʹ͋Δ

Slide 17

Slide 17 text

௕೥ഓΘΕͨΫϩοϓٕज़ ਅΜதͰ੾ΓऔΔͱ͜͏ͳΔ ԣ௕ͷը૾͸ը૾্෦ΛΫϩοϓ ॎ௕ͩͱ  إͱ͔ϩΰͱ͔͸্ͷํʹ͋Δ

Slide 18

Slide 18 text

Creator Application Storage CDN Users • ը૾σʔλग़ྗ • ೖྗը૾ݕূ Converter • ಈతੜ੒ Thumbnail Master Thumbnail Master

Slide 19

Slide 19 text

αϜωΠϧϚελΛجʹͨ͠  αϜωΠϧը૾ੜ੒ • Ϛελը૾ʹ͸JPEGΛ࠾༻ • ༨ܭͳσʔλΛམͱ͓ͯ͘͠ • εέʔϦϯάσίʔυͰ͖ΔͷͰCPU/ϝϞϦ࢖༻ྔΛ࡟ݮ • ·͋JPEGʹ΋͍Ζ͍Ζͱ໰୊͕͋Δ

Slide 20

Slide 20 text

JPEGʹ͓͚Δ໰୊ • ૬ޓޓ׵༻ϑΥʔϚοτͰ͋ΔJIF͕ඪ४Խ͞Ε͕ͨ  ৭ʑͱൈ͚͍ͯͨ • ΧϥʔϞσϧͱ͔αϒαϯϓϦϯάͱ͔ॏཁͳࣄ͕  ܾ·͍ͬͯͳ͍ • த਎͕RGBͰ΋ྑ͍͠ɺCMYKͰ΋ྑ͍͠ɺ  ΞϧϑΝνϟϯωϧ͕͋ͬͯ΋ྑ͍͠ԿͰ΋͋Γ

Slide 21

Slide 21 text

JPEG/JFIF • APP0-JFIFϔομʹඞཁͳ৘ใΛೖΕΔ͜ͱʹܾΊͨ • ΧϥʔϞσϧΛY’CbCrʹܾΊͨ  (ITU-R BT.601Λ0-255ʹม׵ͨ͠΍ͭ) • ίϯϙʔωϯτ͸3ͭͰͦΕͧΕYɺCbɺCrͱܾΊͨ • αϒαϯϓϦϯάൺΛ4छྨʹܾΊͨ

Slide 22

Slide 22 text

JPEG/EXIF • ೔ຊͷσδΧϝ԰͞Μ͕ඞཁ֦ͩͬͨுΛߦͬͨ • JFIFΛϕʔεʹΧϥʔϞσϧ͚ͩͰͳ͘৭ۭؒΛѻ͏  ͨΊΧϥʔϓϩϑΝΠϧΛ࢖͑ΔΑ͏ʹͨ͠ • APP1-EXIFϔομʹTIFFϑΥʔϚοτͰ֨ೲͯ͋͠Δ:  ଏʹEXIF৘ใͱݺ͹ΕΔͷ͸͜Ε

Slide 23

Slide 23 text

JPEG/EXIFʹ͓͚Δ໰୊ • ϒϥ΢β͕ICCϓϩϑΝΠϧΛղऍͨ͠Γ  ICCϓϩϑΝΠϧ͸ղऍ͢ΔͷʹϞχλͷϓϩϑΝΠϧ͸  ແࢹͨ͠Γ͢Δ • Ϣʔβ͕sRGBͰӾཡͯ͠Δͷʹ  ग़ྗ࣌ʹAdobe RGBϓϩϑΝΠϧΛೖΕͨΓ͢Δ  ʢը૾͕৭᧙ͤͯݟ͑ΔݪҼʣ • ݁ہͲ͏ѻͬͨΒྑ͍͔Α͘Θ͔Βͳ͍

Slide 24

Slide 24 text

CMYK JPEG  දࣔग़དྷͳ͍໰୊ • App1 EXIFʹؚ·Ε͍ͯΔTIFF৘ใͷColorSpace͕CMYK ʹͳ͍ͬͯΔ΋ͷʢଏʹCMYK JPEGͱݺͿʣ • Ұ෦ϒϥ΢βͰදࣔͰ͖ͳ͔ͬͨʢࠓ͸େৎ෉ʁʣ • ฐࣾͰ͸Ұ୴PNGʹॻ͖ग़͢ṖίʔυͰ͜ͷ৘ใΛ࡟আ  ʢϫʔΫΞϥ΢ϯυʣ

Slide 25

Slide 25 text

ṖίʔυͲ͏ʹ͔͍ͨ͠ // ========================================= // CMYK形式の画像をRGB形式に変換する // 劣化およびディスクI/Oによる負荷を最小限にするため // 一旦PNGに変換してからJPEGに変換する // ========================================= public static function cmyk2rgb($image, $tmp_name) { // Workaround for TransformImageColorspace() bug // // It means same below: // // $image->setImageColorspace(Imagick::COLORSPACE_RGB); // $image->setImageFormat('PNG'); $image->setImageCompressionQuality(75); $image->readImageBlob($image->getImageBlob()); $image->setImageFormat('JPEG'); $image->setImageCompressionQuality(100); $image->writeImage($tmp_name); }

Slide 26

Slide 26 text

JPEGʹ͓͚Δ৭ۭؒ • ࠞಉ͞Ε͕͚ͪͩͲEXIF৘ใͷColorSpace͸  JPEGͷූ߸Խʹ༻͍ΒΕ͍ͯΔ৭ۭؒʢΧϥʔϞσϧʣ  ͱ͸ҧ͏ • JPEG/JFIFɺJPEG/EXIFͱ΋ʹΧϥʔϞσϧͱͯ͠࢖༻Ͱ͖ Δ৭ۭؒ͸YCbCr͚ͩ • APP12-AdobeϔομͰRGBͱ͔CMYKͱ͔YCCKͷ  ΧϥʔϞσϧΛ࢖͑Δ͚Ͳࠓ͸ܹϨΞ

Slide 27

Slide 27 text

Ͳ͏͢Ε͹Α͍ͷ͔ • ΧϥʔϚωδϝϯτͰ͖ΔΤϯδχΞ͕ෆࡏ • ϓϩϑΝΠϧΛਖ਼͘͠ѻ͍͍͚ͨͲ  Ϣʔβ؀ڥ͕ͦ͏ͳͬͯͳ͍ • ImageMagick͕Ͳ͏ͳͬͯΔ͔Α͘Θ͔Βͳ͍ʢఘΊʣ

Slide 28

Slide 28 text

αϜωΠϧϚελͰ  ͜ΕΒͷ໰୊Λ؇࿨͢Δ • αϜωΠϧੜ੒࣌ʹ͍ΖΜͳ͜ͱߟ͑Δͷ͸໘౗ • JPEG/JFIFޓ׵ͷJPEGΛ͓ͭͬͯ͘͘ • EXIF৘ใΛ࡟আ • ΧϥʔϞσϧ͸sRGB • ICCϓϩϑΝΠϧ͸ߟྀͯ͠ͳ͍͚ͲͲ͏ʹ͔͍ͨ͠

Slide 29

Slide 29 text

αϜωΠϧϚελΛ  ࢖͏͜ͱͰॲཧΛޮ཰Խ͢Δ • αϜωΠϧੜ੒࣌ʹ͍Ζ͍ΖͳࣄΛߟ͑ͳͯ͘ྑ͍ • ೖྗը૾͸JPEG͚ͩ • ग़ྗ΋ͱΓ͋͑ͣJPEGͰྑ͍ • ෆཁͳ৘ใ͸͢Ͱʹআڈͯ͠JPEG/JFIFޓ׵

Slide 30

Slide 30 text

go-thumber • αϜωΠϧϚελ͔ΒͷαϜωΠϧੜ੒͚ͩΛߟ͑ͯ ઃܭ͞ΕͨαϜωΠϧੜ੒ϓϩΩγ • @marcan42͕਺िؒͰͭͬͨ͘ • YCbCrσʔλҎ֎͸͢΂ͯύεεϧʔ͢ΔͷͰ  EXIFͰ΋େৎ෉ https://github.com/pixiv/go-thumber

Slide 31

Slide 31 text

ϓϩΩγʹͳͬͯΔͷͰ؆୯ GET http://localhost:8000/w=128,h=128,q=95/upstream.example.com/example.jpg w: ੜ੒͢ΔαϜωΠϧͷ࠷େ෯ (required) h: ੜ੒͢ΔαϜωΠϧͷ࠷େߴ͞ (required) q: JPEG quality (default 90) u: ࢦఆαΠζΑΓখ͔ͬͨ͞ΒΞοϓεέʔϧ(Ҿ͖Ԇ͹͠)͢Δ͔Ͳ͏͔ (default 1) a: ΞεϖΫτΛΩʔϓ͢Δ͔Ͳ͏͔ (default 1) o: JPEGͷoptimizeΛߦ͏ (default 0) p: μ΢ϯαϯϓϦϯάϑΝΫλ (default 2)

Slide 32

Slide 32 text

go-thumber͸΍͍ • LibJPEGv6 APIΛੜͰୟ͍ͯYCbCrσʔλΛऔಘɺॻ͖ࠐΈ  ʢlibjpeg-turboΛ࢖༻ʣ • LibswscaleʹΑΔImageMagickͱಉਫ਼౓͔ͭߴ଎ͳ  εέʔϦϯάॲཧ

Slide 33

Slide 33 text

ϕϯνϚʔΫ • Tecnick’s public test imagesͷը૾Λ256x256pxʹॖখ͢Δ • ຊ౰͸Πϥετը૾ͷςετηοτ͕ग़དྷΔͱྑ͍͚Ͳ  ݖརपΓ͕໘౗ • mod_smalllight(ImageMagickɺImlib2)ͱൺֱ • Apacheͷνϡʔχϯά͸͋Μ·͕Μ͹ͬͯͳ͍ • εέʔϦϯάσίʔυ͸༗ޮ

Slide 34

Slide 34 text

Resize to 256x256px Benchmark (quality=99) Requests per second 0 25 50 75 100 125 Numbers of Connections 0 1 2 3 4 5 6 7 8 9 10 smallight(imlib2) go-thumber smalllight(magick) εϧʔϓοτͷൺֱ

Slide 35

Slide 35 text

Mean of Latency (size=256x256, quality=99) Mean of Latency (ms) 0 50 100 150 200 Numbers of Connections 0 1 2 3 4 5 6 7 8 9 10 smalllight(magick) go-thumber smallight(imlib2) ฏۉॲཧ஗Ԇͷൺֱ

Slide 36

Slide 36 text

go-thumber͸΍͍ • imlib2Λ࢖ͬͨ৔߹ʹඖఢ͢Δ଎͞ • ͡Ό͋៉ྷͳͷʁ

Slide 37

Slide 37 text

ϒϥΠϯυςετΛ΍ͬͨ • Ή͔͠ࢹ֮৺ཧ࣮ݧͱ͔΍ͬͨ͜ͱ͋ͬͨͷͰͳΜͱͳ͘ ΍ͬͯΈͨ • ࣾ಺νϟοτʹ౤͛ͨͱ͜ΖՋͦ͏ͳΤϯδχΞ͕ࢀՃ • ࣮ݧ৚݅ͳͲ͸͋·Γߟ͑ͯͳ͍

Slide 38

Slide 38 text

͜Μͳͷ

Slide 39

Slide 39 text

݁Ռ දࣔճ਺ બ୒͞Εͨճ਺ બ୒͞Εׂͨ߹ SMALLLIGHT(imlib2) 467 58 8.2% SMALLLIGHT(ImageMagick) 466 275 39.3% go-thumber 467 367 52.4% • Ϣʔβʔ͕៉ྷͰ͋Δͱબ୒ͨ͠ճ਺ • go-thumberͰੜ੒͞ΕͨαϜωΠϧ͕࠷΋ଟ͘બ୒͞Εͨ

Slide 40

Slide 40 text

go-thumberͷѹউ • ImageMagickͱಉ͡ϑΟϧλΛ༻͍͍ͯΔͷʹԿނ͔ • ΋ͪΖΜΧϥΫϦ͕͋Δ

Slide 41

Slide 41 text

• LibJPEGͰ͸1/1ʙ1/8ͷ޷͖ͳεέʔϧͰσίʔυग़དྷΔ • go-thumberͰ͸ੜ੒͢ΔαϜωΠϧαΠζͷ2ഒʢp=2ʣ • ImageMagickͰ΋αϜωΠϧαΠζͷ2ഒΛࢦఆ͢Ε͹  ಉ͘͡Β͍៉ྷʹͳΔ͸ͣ • ৄ͘͠͸go-thumberͷREADME.mdʹॻ͍ͯ͋Δ εέʔϦϯάσίʔυͷ  αΠζ࠷దԽ

Slide 42

Slide 42 text

αϜωΠϧϚελͱgo-thumber • αϜωΠϧϚελʹ͍Ζ͍Ζͳ໰୊Λԡ͠ࠐΉ͜ͱͰ ϦΫΤετ࣌ͷαϜωΠϧੜ੒Λ؆୯ʹͨ͠ • go-thumber͕҆ఆͯ͠ΔͷͰ͏·͍ͬͯ͘Δ • ࣍͸: webp-thumberʁ