画像変換Night / 2015-02-18
αϜωΠϧϚελͱgo-thumberΔ͔͞Μ / Shunsuke MichiiαϜωΠϧมNight2015-02-18
View Slide
Δ͔͞Μ / Shunsuke Michii• ͍ͭը૾৴ͱ͔ը૾पΓͱ͔ͬͯΔ• Fluentdͱ͔Kibanaͱ͔σʔλղੳ·ΘΓͬͯΔ• ࡢ͔ΒSwiftΛॻ͖࢝ΊͨʢۀʣInfrastructure team at pixiv Inc., since 2012
αϜωΠϧϚελͱgo-thumber• WebαʔϏεʹ͓͚Δը૾ߘػೳࣄྫ• ϦΫΤετ࣌ͷαϜωΠϧੜʹ͓͚Δ• αϜωΠϧϚελ• JPEGʹ͓͚Δ• go-thumber
WebαʔϏεʹ͓͚Δ ը૾ߘػೳࣄྫ• ҰൠతͳCGMʹ͓͚Δ• ͨͿΜΑ͋͘Δߏ
ApplicationCreator StorageCDNUsers• ը૾σʔλग़ྗ • ೖྗը૾ݕূ• อଘ• ৴
Creator EC2 S3CloudFrontUsers• ը૾σʔλग़ྗ • ೖྗը૾ݕূ• อଘ• ৴
ࣄྫ: ΦϦδφϧը૾Λͦͷ··৴• సૹྔ͕ΊͬͪΌ͓͓͍• ը૾ͷಡΈࠐΈ͕͍• σβΠϯΛ͋ͯΔͷ͕ͪΐͬͱେม• ͋ͱ͔ΒαϜωΠϧͭ͘Δͷ͕େม• αʔϏε͕େ͖͘ͳΔʹͭΕ͖ͯͭ͘ͳΔ
Creator Application StorageCDNUsers• ը૾σʔλग़ྗ • ೖྗը૾ݕূ• αϜωΠϧੜ• ޙ͔Βੜ
ࣄྫ: Ξοϓϩʔυ࣌ʹαϜωΠϧੜ• Ξοϓϩʔυॲཧ͕େม• αΠζมߋ͢Δͷ͕େม• Ξοϓϩʔυ͢ΔϢʔβΛͨͤͳ͍Α͏ʹ ඇಉظॲཧʹͨ͠Γ͢Δඞཁ͕͋Δ
Creator Application StorageCDNUsers• ը૾σʔλग़ྗ • ೖྗը૾ݕূConverter• ಈతੜ
ࣄྫ: ϦΫΤετ࣌ʹαϜωΠϧੜ• ฐࣾͰಈతมͱݺΕ͍ͯΔ• σβΠϯมߋ͕؆୯• Ωϟογϡͯ͘͠ΕΕͦΜͳʹϦΫΤετͳ͍• SMALLLIGHTɺTOFUͳͲ• ͍͍ͩͨΑͦ͞͏
ϦΫΤετ࣌ͷ αϜωΠϧੜͰࠔΔ͜ͱ• ੜݩը૾ʹґଘ• ॲཧ͕໘
ੜݩը૾ʹґଘ• PNGͩͬͨΓJPEGͩͬͨΓGIFͩͬͨΓ͢Δ• αΠζ͕ἧ͍ͬͯͳ͍ͷͰॲཧ࣌ؒ༻ϝϞϦྔ͕ όϥόϥ
ॲཧ͕໘• ͍ΖΜͳը૾ϑΥʔϚοτʹରԠ͠ͳ͍ͱ͍͚ͳ͍• ൚༻తͳॲཧʹ͢Δͱॲཧ࣌ؒϝϞϦޮʹ• εέʔϦϯάҎ֎ʹΫϩοϐϯάͱ͔͋Δ• ImageMagickͰ͍͍ͩͨͰ͖Δ͚Ͳɺ ͦͷͭΒ͜͞ͷޙͩΕ͔ޠͬͯ͘ΕΔͣ
αϜωΠϧϚελ• αϜωΠϧੜݩͱͳΔϚελը૾• ϦΫΤετ࣌αϜωΠϧੜͷΛ؇͢Δ• ଞͷͯ͢ͷαϜωΠϧΑΓͰ͔͍ʢ1200x1200ʣ• Ϋϩοϓͨ͠ը૾Λ༻ҙ͓ͯ͘͠
ഓΘΕͨΫϩοϓٕज़ԣͷը૾ը૾தԝΛΫϩοϓ ॎͷը૾ը૾্෦ΛΫϩοϓϝΠϯͱͳΔ෦ ͍͍ͩͨਅΜதʹ͋Δॎͩͱ إͱ͔ϩΰͱ্͔ͷํʹ͋Δ
ഓΘΕͨΫϩοϓٕज़ਅΜதͰΓऔΔͱ͜͏ͳΔ ԣͷը૾ը૾্෦ΛΫϩοϓॎͩͱ إͱ͔ϩΰͱ্͔ͷํʹ͋Δ
Creator Application StorageCDNUsers• ը૾σʔλग़ྗ • ೖྗը૾ݕূConverter• ಈతੜThumbnailMasterThumbnailMaster
αϜωΠϧϚελΛجʹͨ͠ αϜωΠϧը૾ੜ• Ϛελը૾ʹJPEGΛ࠾༻• ༨ܭͳσʔλΛམͱ͓ͯ͘͠• εέʔϦϯάσίʔυͰ͖ΔͷͰCPU/ϝϞϦ༻ྔΛݮ• ·͋JPEGʹ͍Ζ͍Ζͱ͕͋Δ
JPEGʹ͓͚Δ• ૬ޓޓ༻ϑΥʔϚοτͰ͋ΔJIF͕ඪ४Խ͞Ε͕ͨ ৭ʑͱൈ͚͍ͯͨ• ΧϥʔϞσϧͱ͔αϒαϯϓϦϯάͱ͔ॏཁͳࣄ͕ ܾ·͍ͬͯͳ͍• த͕RGBͰྑ͍͠ɺCMYKͰྑ͍͠ɺ ΞϧϑΝνϟϯωϧ͕͋ͬͯྑ͍͠ԿͰ͋Γ
JPEG/JFIF• APP0-JFIFϔομʹඞཁͳใΛೖΕΔ͜ͱʹܾΊͨ• ΧϥʔϞσϧΛY’CbCrʹܾΊͨ (ITU-R BT.601Λ0-255ʹมͨͭ͠)• ίϯϙʔωϯτ3ͭͰͦΕͧΕYɺCbɺCrͱܾΊͨ• αϒαϯϓϦϯάൺΛ4छྨʹܾΊͨ
JPEG/EXIF• ຊͷσδΧϝ͞Μ͕ඞཁ֦ͩͬͨுΛߦͬͨ• JFIFΛϕʔεʹΧϥʔϞσϧ͚ͩͰͳ͘৭ۭؒΛѻ͏ ͨΊΧϥʔϓϩϑΝΠϧΛ͑ΔΑ͏ʹͨ͠• APP1-EXIFϔομʹTIFFϑΥʔϚοτͰ֨ೲͯ͋͠Δ: ଏʹEXIFใͱݺΕΔͷ͜Ε
JPEG/EXIFʹ͓͚Δ• ϒϥβ͕ICCϓϩϑΝΠϧΛղऍͨ͠Γ ICCϓϩϑΝΠϧղऍ͢ΔͷʹϞχλͷϓϩϑΝΠϧ ແࢹͨ͠Γ͢Δ• Ϣʔβ͕sRGBͰӾཡͯ͠Δͷʹ ग़ྗ࣌ʹAdobe RGBϓϩϑΝΠϧΛೖΕͨΓ͢Δ ʢը૾͕৭᧙ͤͯݟ͑ΔݪҼʣ• ݁ہͲ͏ѻͬͨΒྑ͍͔Α͘Θ͔Βͳ͍
CMYK JPEG දࣔग़དྷͳ͍• App1 EXIFʹؚ·Ε͍ͯΔTIFFใͷColorSpace͕CMYKʹͳ͍ͬͯΔͷʢଏʹCMYK JPEGͱݺͿʣ• Ұ෦ϒϥβͰදࣔͰ͖ͳ͔ͬͨʢࠓେৎʁʣ• ฐࣾͰҰ୴PNGʹॻ͖ग़͢ṖίʔυͰ͜ͷใΛআ ʢϫʔΫΞϥϯυʣ
ṖίʔυͲ͏ʹ͔͍ͨ͠// =========================================// CMYK形式の画像をRGB形式に変換する// 劣化およびディスクI/Oによる負荷を最小限にするため// 一旦PNGに変換してからJPEGに変換する// =========================================public static function cmyk2rgb($image, $tmp_name){// Workaround for TransformImageColorspace() bug//// It means same below://// $image->setImageColorspace(Imagick::COLORSPACE_RGB);//$image->setImageFormat('PNG');$image->setImageCompressionQuality(75);$image->readImageBlob($image->getImageBlob());$image->setImageFormat('JPEG');$image->setImageCompressionQuality(100);$image->writeImage($tmp_name);}
JPEGʹ͓͚Δ৭ۭؒ• ࠞಉ͞Ε͕͚ͪͩͲEXIFใͷColorSpace JPEGͷූ߸Խʹ༻͍ΒΕ͍ͯΔ৭ۭؒʢΧϥʔϞσϧʣ ͱҧ͏• JPEG/JFIFɺJPEG/EXIFͱʹΧϥʔϞσϧͱͯ͠༻Ͱ͖Δ৭ۭؒYCbCr͚ͩ• APP12-AdobeϔομͰRGBͱ͔CMYKͱ͔YCCKͷ ΧϥʔϞσϧΛ͑Δ͚ͲࠓܹϨΞ
Ͳ͏͢ΕΑ͍ͷ͔• ΧϥʔϚωδϝϯτͰ͖ΔΤϯδχΞ͕ෆࡏ• ϓϩϑΝΠϧΛਖ਼͘͠ѻ͍͍͚ͨͲ Ϣʔβڥ͕ͦ͏ͳͬͯͳ͍• ImageMagick͕Ͳ͏ͳͬͯΔ͔Α͘Θ͔Βͳ͍ʢఘΊʣ
αϜωΠϧϚελͰ ͜ΕΒͷΛ؇͢Δ• αϜωΠϧੜ࣌ʹ͍ΖΜͳ͜ͱߟ͑Δͷ໘• JPEG/JFIFޓͷJPEGΛ͓ͭͬͯ͘͘• EXIFใΛআ• ΧϥʔϞσϧsRGB• ICCϓϩϑΝΠϧߟྀͯ͠ͳ͍͚ͲͲ͏ʹ͔͍ͨ͠
αϜωΠϧϚελΛ ͏͜ͱͰॲཧΛޮԽ͢Δ• αϜωΠϧੜ࣌ʹ͍Ζ͍ΖͳࣄΛߟ͑ͳͯ͘ྑ͍• ೖྗը૾JPEG͚ͩ• ग़ྗͱΓ͋͑ͣJPEGͰྑ͍• ෆཁͳใ͢Ͱʹআڈͯ͠JPEG/JFIFޓ
go-thumber• αϜωΠϧϚελ͔ΒͷαϜωΠϧੜ͚ͩΛߟ͑ͯઃܭ͞ΕͨαϜωΠϧੜϓϩΩγ• @marcan42͕िؒͰͭͬͨ͘• YCbCrσʔλҎ֎ͯ͢ύεεϧʔ͢ΔͷͰ EXIFͰେৎhttps://github.com/pixiv/go-thumber
ϓϩΩγʹͳͬͯΔͷͰ؆୯GET http://localhost:8000/w=128,h=128,q=95/upstream.example.com/example.jpgw: ੜ͢ΔαϜωΠϧͷ࠷େ෯ (required)h: ੜ͢ΔαϜωΠϧͷ࠷େߴ͞ (required)q: JPEG quality (default 90)u: ࢦఆαΠζΑΓখ͔ͬͨ͞ΒΞοϓεέʔϧ(Ҿ͖Ԇ͠)͢Δ͔Ͳ͏͔ (default 1)a: ΞεϖΫτΛΩʔϓ͢Δ͔Ͳ͏͔ (default 1)o: JPEGͷoptimizeΛߦ͏ (default 0)p: μϯαϯϓϦϯάϑΝΫλ (default 2)
go-thumber͍• LibJPEGv6 APIΛੜͰୟ͍ͯYCbCrσʔλΛऔಘɺॻ͖ࠐΈ ʢlibjpeg-turboΛ༻ʣ• LibswscaleʹΑΔImageMagickͱಉਫ਼͔ͭߴͳ εέʔϦϯάॲཧ
ϕϯνϚʔΫ• Tecnick’s public test imagesͷը૾Λ256x256pxʹॖখ͢Δ• ຊΠϥετը૾ͷςετηοτ͕ग़དྷΔͱྑ͍͚Ͳ ݖརपΓ͕໘• mod_smalllight(ImageMagickɺImlib2)ͱൺֱ• Apacheͷνϡʔχϯά͋Μ·͕Μͬͯͳ͍• εέʔϦϯάσίʔυ༗ޮ
Resize to 256x256px Benchmark (quality=99)Requests per second0255075100125Numbers of Connections0 1 2 3 4 5 6 7 8 9 10smallight(imlib2)go-thumbersmalllight(magick)εϧʔϓοτͷൺֱ
Mean of Latency (size=256x256, quality=99)Mean of Latency (ms)050100150200Numbers of Connections0 1 2 3 4 5 6 7 8 9 10smalllight(magick)go-thumbersmallight(imlib2)ฏۉॲཧԆͷൺֱ
go-thumber͍• imlib2Λͬͨ߹ʹඖఢ͢Δ͞• ͡Ό͋៉ྷͳͷʁ
ϒϥΠϯυςετΛͬͨ• Ή͔͠ࢹ֮৺ཧ࣮ݧͱ͔ͬͨ͜ͱ͋ͬͨͷͰͳΜͱͳͬͯ͘Έͨ• ࣾνϟοτʹ͛ͨͱ͜ΖՋͦ͏ͳΤϯδχΞ͕ࢀՃ• ࣮ݧ݅ͳͲ͋·Γߟ͑ͯͳ͍
͜Μͳͷ
݁Ռදࣔճ બ͞Εͨճ બ͞Εׂͨ߹SMALLLIGHT(imlib2) 467 58 8.2%SMALLLIGHT(ImageMagick) 466 275 39.3%go-thumber 467 367 52.4%• Ϣʔβʔ͕៉ྷͰ͋Δͱબͨ͠ճ• go-thumberͰੜ͞ΕͨαϜωΠϧ͕࠷ଟ͘બ͞Εͨ
go-thumberͷѹউ• ImageMagickͱಉ͡ϑΟϧλΛ༻͍͍ͯΔͷʹԿނ͔• ͪΖΜΧϥΫϦ͕͋Δ
• LibJPEGͰ1/1ʙ1/8ͷ͖ͳεέʔϧͰσίʔυग़དྷΔ• go-thumberͰੜ͢ΔαϜωΠϧαΠζͷ2ഒʢp=2ʣ• ImageMagickͰαϜωΠϧαΠζͷ2ഒΛࢦఆ͢Ε ಉ͘͡Β͍៉ྷʹͳΔͣ• ৄ͘͠go-thumberͷREADME.mdʹॻ͍ͯ͋ΔεέʔϦϯάσίʔυͷ αΠζ࠷దԽ
αϜωΠϧϚελͱgo-thumber• αϜωΠϧϚελʹ͍Ζ͍ΖͳΛԡ͠ࠐΉ͜ͱͰϦΫΤετ࣌ͷαϜωΠϧੜΛ؆୯ʹͨ͠• go-thumber͕҆ఆͯ͠ΔͷͰ͏·͍ͬͯ͘Δ• ࣍: webp-thumberʁ
harukasan
yellowshippo
line_developers
yud0uhu
yanzm
mtsmfm
murahigeas
igsr5
sansantech
yokawasa
joere
mhidaka
marcelosomers
eileencodes
lemiorhan
danielanewman
gr2m
frogandcode
phodgson
stephaniewalter
rasmusluckow
malarkey
ammeep
sonatard