Java のデータ圧縮ライブラリを極める #jjug_ccc #ccc_c7

Slide 1

Slide 1 text

Java ͷ σʔλѹॖϥΠϒϥϦΛۃΊΔ JJUG CCC 2018 Spring / 2018-05-26 KOMIYA Atsushi

Slide 2

Slide 2 text

@komiya_atsushi / খٶ ಞ࢙

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

σʔλѹॖͷجૅ஌ࣝ

Slide 5

Slide 5 text

ଛࣦͷͳ͍ ൚༻తͳ σʔλѹॖΞϧΰϦζϜ

Slide 6

Slide 6 text

ଛࣦͷ༗ແ • ѹॖ͞Εͨঢ়ଶͷσʔλ͔ΒɺѹॖલͷσʔλΛ 1 bit ΋ ҧΘͣ෮ݩͰ͖Δ͔൱͔ • 1 bit Ͱ΋ҟͳΔͷͰ͋Ε͹ʮଛࣦ͋Γʯ • ଛࣦͷͳ͍ѹॖΞϧΰϦζϜͷྫ • Deﬂate (ZIP, gzip, PNG), LZW (GIF, compress) • ଛࣦͷ͋ΔѹॖΞϧΰϦζϜͷྫ • JPEG (ISO/IEC 10918-1:1994), MPEG-1

Slide 7

Slide 7 text

൚༻తͳѹॖΞϧΰϦζϜ • ςΩετɾը૾ɾԻ੠ͳͲͷσʔλͷछྨΛ ໰Θͣʹద༻Ͱ͖ΔѹॖΞϧΰϦζϜͷ͜ͱ • ઌͷྫͩͱ Deﬂate ΍ LZW ͳͲ͕֘౰ • JPEG / MPEG ͷΑ͏ʹɺͦΕͧΕͷσʔλʹ ಛԽͨ͠ѹॖΞϧΰϦζϜͷํ͕Ұൠతʹ͸ ੑೳ͕Α͍

Slide 8

Slide 8 text

ѹॖΞϧΰϦζϜͷධՁ • ѹॖ཰ • ΋ͱͷσʔλαΠζ (όΠτ਺) ʹରͯ͠ɺͲΕͩ ͚খ͘͞Ͱ͖͔ͨʁ • খ͚͞Ε͹খ͍͞΄ͲΑ͍ • ʮѹॖલͷαΠζ / ѹॖޙͷαΠζʯͰද͞ΕΔ ͜ͱ΋͋ΔͷͰ஫ҙ͕ඞཁ • ͜ͷ৔߹͸ɺ஋͕େ͖͍ํ͕Α͍ੑೳͱͳΔ

Slide 9

Slide 9 text

ѹॖΞϧΰϦζϜͷධՁ • ॲཧ଎౓ • ୯Ґ࣌ؒ͋ͨΓʹѹॖɾ෮ݩͰ͖Δσʔλͷྔ (୯Ґ͸ MB/ඵͳͲ) • ΞϧΰϦζϜʹΑͬͯ͸ɺѹॖͱ෮ݩͦΕͧΕʹ͔͔Δ ͕࣌ؒେ͖͘ҟͳΔ͜ͱ΋͋Δ (ѹॖ͸஗͍͕෮ݩ͸଎ ͍ɺͳͲ) • ѹॖɾ෮ݩʹ͔͔Δ࣌ؒʹՃ͑ͯɺѹॖσʔλΛ఻ૹ͢ Δͷʹ͔͔Δ࣌ؒ΋ؚΊͨεϧʔϓοτΛग़͢͜ͱ΋͋ Δ

Slide 10

Slide 10 text

ѹॖΞϧΰϦζϜͷධՁ • ѹॖ཰ͱॲཧ଎౓ͷؔ܎ • τϨʔυΦϑͷؔ܎ʹͳΔ • ߴ͍ѹॖ཰ΛಘΔͨΊʹίϯϐϡʔςΟϯάϦ ιʔεΛࠅ࢖ͯ͠೉͍͠ॲཧΛ͢Δ → ॲཧ଎ ౓͸஗͘ͳΔ • ଎͍ॲཧ଎౓ΛٻΊΔͱɺ೉͍͠ॲཧ͸Ͱ͖ͳ ͘ͳΔ → ѹॖ཰͸௿͘ͳΔ

Slide 11

Slide 11 text

ѹॖΞϧΰϦζϜͷධՁ • ϝϞϦɾϑοτϓϦϯτ • ѹॖ͓Αͼ෮ݩͦΕͧΕͷॲཧʹ͓͍ͯඞཁͱͳΔ࠷େͷ ϫʔΩϯάϝϞϦྔ • ѹॖΞϧΰϦζϜ΍ύϥϝʔλʹΑͬͯେ͖͘ࠨӈ͞ΕΔ • ॲཧ଎౓ͱಉ͘͡ɺѹॖɾ෮ݩͦΕͧΕͷϝϞϦɾϑοτ ϓϦϯτ͕ඇରশʹͳΔ͜ͱ΋͋Δ • ෮ݩॲཧͰඞཁͳϝϞϦɾϑοτϓϦϯτ͕খ͍͞έʔ ε͕ଟ͍

Slide 12

Slide 12 text

࠷ۙͷ σʔλѹॖΞϧΰϦζϜࣄ৘

Slide 13

Slide 13 text

2010 ೥୅ʹೖΔ·Ͱ • Deﬂate ͕σϑΝΫτελϯμʔυͱͯ͠܅ྟ • ͦͦ͜͜ͷ଎౓Ͱɺ·ͣ·ͣͷѹॖ཰Λୡ੒Ͱ͖Δ • ϝϞϦɾϑοτϓϦϯτ͕ѹॖ࣌Ͱ΋਺ඦ KB ͱ  (ࠓͱͳͬͯ͸ͱͯ΋) খͯ͘͞ࡁΉ • ଎౓ੑೳ͕ٻΊΒΕΔ৔໘Ͱ͸ LZO ͱ͍͏બ୒ࢶ͕͋ͬͨ • ͨͩ͠ϥΠηϯε͕ GPLv2 ͳͷͰѻ͍ͮΒ͍

Slide 14

Slide 14 text

࠷ۙ (2010೥Ҏ߱) ͷτϨϯυ • طଘΞϧΰϦζϜͷѹॖ཰ΛྗٕͰ࠷దԽ • Zopﬂi (Deﬂate), Guetzli (JPEG) • ॲཧ଎౓ͷ଎͞ʹಛԽ • Snappy, LZ4 • ॲཧ଎౓ͱѹॖ཰ͷόϥϯεΛॏࢹ • Zstandard, Brotli

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Snappy • Google ੡ • LZ77 Λϕʔεͱͨ͠ΞϧΰϦζϜ • όΠτ୯Ґͷ I/O • ѹॖ͓Αͼ෮ݩͷॲཧ଎౓͕଎͍ • ͦͷ෼ɺѹॖ཰͸΄Ͳ΄ͲͰܾͯ͠ྑ͘͸ͳ͍ • BSD-type ϥΠηϯε

Slide 17

Slide 17 text

LZ4 • Snappy ͱಉ༷ʹɺLZ77 ϕʔε & όΠτ୯Ґͷ I/O Λ࠾༻ • ಛʹ෮ݩॲཧͷ଎౓Λॏࢹ͍ͯ͠Δ • ѹॖ཰͸ Snappy ͱಉఔ౓͔ͪΐͬͱѱ͍͙Β͍ʁ • ѹॖॲཧʹ͔͔Δ࣌ؒΛ٘ਜ਼ʹͭͭ͠ɺߴ͍ѹॖ཰ΛಘΔ HC (high compression) Φϓγϣϯ΋ఏڙ͍ͯ͠Δ • Deﬂate ΞϧΰϦζϜʹഭΔѹॖ཰Λୡ੒Ͱ͖Δ • BSD ϥΠηϯε

Slide 18

Slide 18 text

LZ4 ར༻࣮੷͕ͱͯ΋๛෋

Slide 19

Slide 19 text

Zstandard • LZ77 ͱ ANS (Asymmetric numeral system) ͷҰ࣮૷Ͱ͋Δ FSE (Finite state entropy) Λ૊Έ߹ΘͤͨΞϧΰϦζϜ • FSE ͷ୅ΘΓʹ Huffman ූ߸Λ࢖͏͜ͱ΋Ͱ͖Δ • Deﬂate ͱಉ౳͔ɺͦΕҎ্ͷѹॖ཰͓Αͼॲཧ଎౓Λୡ੒͢Δ • ෮ݩ଎౓͸ Snappy, LZ4 ΄ͲͰ͸ͳ͍ʹͯ͠΋ Deﬂate ΑΓ଎͍ • σʔλʹಛԽͨࣙ͠ॻΛߏங͠ɺͦΕΛ༻͍ͯѹॖ཰Λ޲্ͤ͞Δ࢓૊ΈΛඋ͑ͯ ͍Δ • LZ4 ͱಉ͡։ൃऀ (ݱࡏ͸ Facebook ʹॴଐ) • BSD / GPLv2 ͷσϡΞϧϥΠηϯε • Ҏલ͸͋ͷ Facebook BSD + Patents ϥΠηϯεͩͬͨ

Slide 20

Slide 20 text

Brotli • Google ੡ • LZ77 ͱ 2 ࣍ͷ౷ܭతϞσϦϯάΛར༻ͨ͠ Huffman ූ߸ͷ૊Έ ߹Θͤ • Zstandard ͱಉ༷ʹɺDeﬂate ͱಉఔ౓Ҏ্ͷੑೳΛୡ੒͢Δ • HTTP ѹॖʹ͓͚ΔΤϯίʔσΟϯάͷҰͭͱͯ͠࠾༻͞Ε͍ͯΔ • ࣄલఆٛ͞Ε Brotli ʹ૊Έࠐ·ΕͨࣙॻΛ༻͍ͯɺѹॖ཰Λ޲্ ͤ͞Δ࢓૊Έ͕උΘ͍ͬͯΔ • MIT ϥΠηϯε

Slide 21

Slide 21 text

Brotli Chrome Ͱ www.google.com ʹΞΫηεͯ͠ΈΔͱ…

Slide 22

Slide 22 text

Java ʹ͓͚Δ  σʔλѹॖϥΠϒϥϦ

Slide 23

Slide 23 text

ϥΠϒϥϦʹ๬·ΕΔཁૉɾಛੑ • ѹॖΞϧΰϦζϜͷ࣮૷ (JNI binding vs pure Java) • σʔλѹॖ͸ಘͯͯ͠ CPU-intensive ͳॲཧʹͳΔ • ϦϑΝϨϯε࣮૷ͷωΠςΟϒϥΠϒϥϦΛ JNI binding ͢Δͷ͕଎౓ੑೳతʹ๬·͍͠ • ֤छ OS / ΞʔΩςΫνϟ޲͚ͷϏϧυࡁΈωΠςΟϒ ϥΠϒϥϦΛ༻ҙ͠ͳ͚Ε͹ͳΒͳ͍͕ ☹ • Pure Java ࣮૷͸ɺ଎౓ੑೳҎ֎ʹ΋ࡉ͔ͳڍಈͷҧ͍΍ ΞϧΰϦζϜͦͷ΋ͷͷਐԽͱ͍ͬͨ఺ͰϦϑΝϨϯε࣮ ૷ͱဃ཭͕ੜ͡΍͍͢

Slide 24

Slide 24 text

ϥΠϒϥϦʹ๬·ΕΔཁૉɾಛੑ • ఏڙ͞ΕΔΠϯλϑΣʔε • ϦϑΝϨϯε࣮૷ͷϥΠϒϥϦ͕ఏڙ͢ΔΠϯλϑΣʔεʹՃ ͑ͯɺjava.io ύοέʔδͷ InputStream / OutputStream ʹैͬͨΠϯλϑΣʔεΛఏڙ͍ͯ͠Δͷ͕๬·͍͠ • ଞݴޠͷόΠϯσΟϯάͱͷ૬ޓӡ༻ੑ • ྫ͑͹ɺJava ͷϥΠϒϥϦͰѹॖͨ͠σʔλΛ Ruby ͷόΠϯ σΟϯάͰ෮ݩͰ͖Δ͔ʁ • LZ4 ͷΑ͏ʹɺඪ४ͷϑϨʔϜϑΥʔϚοτ͕ଘࡏ͠ͳ͔ͬͨ ѹॖΞϧΰϦζϜ͸ಛʹ஫ҙ͕ඞཁ

Slide 25

Slide 25 text

Java ޲͚ͷσʔλѹॖϥΠϒϥϦ • Snappy • 'org.xerial.snappy:snappy-java' • LZ4 • 'org.lz4:lz4-java' • Zstandard • 'com.github.luben:zstd-jni' • Brotli • 'org.meteogroup.jbrotli:jbrotli'

Slide 26

Slide 26 text

snappy-java • JNI binding ʹΑΔ࣮૷ • ଟ༷ͳ OS / ΞʔΩςΫνϟΛαϙʔτ • ਺஋ྻσʔλͷѹॖ཰ΛߴΊΔ BitShufﬂe ͷ࣮૷Λఏڙ • ૬ޓӡ༻ੑͷ͋ΔΠϯλϑΣʔε • SnappyFramedInputStream • SnappyFramedOutputStream

Slide 27

Slide 27 text

lz4-java • JNI binding, Unsafe API, pure Java ͷ 3 ͭͷ࣮૷Λఏڙ • JNI binding ͸ओཁͳ OS / ΞʔΩςΫνϟΛαϙʔτ • ͳΔ΂͘ߴ଎ͳ࣮૷͕࢖ΘΕΔ࢓૊Έʹͳ͍ͬͯΔ • JNI biding → Unsafe API → pure Java ͷ༏ઌॱҐ • ૬ޓӡ༻ੑͷ͋ΔΠϯλϑΣʔε • LZ4FrameInputStream • LZ4FrameOutputStream • ࠷৽όʔδϣϯ (1.4.1) Ͱ͸ High compression Φϓγϣϯ͕ࢦఆͰ͖ͳ ͍ͳͲͷ੍ݶ͕͋Δ

Slide 28

Slide 28 text

zstd-jni • JNI binding ʹΑΔ࣮૷ • ओཁͳ OS / ΞʔΩςΫνϟΛαϙʔτ • ૬ޓӡ༻ੑͷ͋ΔΠϯλϑΣʔε • ZstdInputStream • ZstdOutputStream • όοϑΝϦϯάͷ࢓૊Έ͕࣮૷͞Ε͍ͯͳ͍ͷͰɺ ZstdOutputStream#write(int) Λසൟʹݺͼग़͢ͳͲͷ࢖ ͍ํΛ͢Δͱ JNI ༝དྷͷΦʔόʔϔουͰ஗͘ͳΔՄೳੑ͕͋Δ

Slide 29

Slide 29 text

jbrotli • JNI binding ʹΑΔ࣮૷ • ωΠςΟϒϥΠϒϥϦ͸ຊମίʔυͷ jar ϑΝΠϧ ͱ͸ผʹఏڙ͞Ε͍ͯΔ • OS / ΞʔΩςΫνϟ͝ͱʹ jar ϑΝΠϧ͕༻ҙ͞ Ε͍ͯΔͷͰɺ։ൃ؀ڥ / ࣮ߦ؀ڥʹ߹ͬͨϥΠ ϒϥϦ͚ͩґଘؔ܎ʹ௥Ճ͢Ε͹Α͍ • BrotliLibraryLoader.loadBrotli() ͷ͓· ͡ͳ͍͕ඞཁ

Slide 30

Slide 30 text

jbrotli • ૬ޓӡ༻ੑͷ͋ΔΠϯλϑΣʔε • BrotliInputStream • BrotliOutputStream • ࠷৽όʔδϣϯ (0.5.0) Ͱ͸ෆ۩߹͕͋ΔΑ͏Ͱɺ BrotliOutputStream Ͱѹॖͨ͠σʔλΛ BrotliInputStream Ͱ෮ݩͰ͖ͳ͍͜ͱ͕͋Δ

Slide 31

Slide 31 text

Java ΞϓϦέʔγϣϯʹ͓͚Δ ѹॖΞϧΰϦζϜͷબఆج४

Slide 32

Slide 32 text

ͲͷѹॖΞϧΰϦζϜΛબ୒͢΂͖͔ʁ • ॲཧ଎౓ͱѹॖ཰ɺͲͪΒΛॏࢹ͢Δͷ͔ʁ • ॲཧ଎౓Λॏࢹ: LZ4, Snappy • ѹॖ཰Λॏࢹ: Zstandard (, Brotli) • ରԠ͍ͯ͠Δ OS / ΞʔΩςΫνϟͷଟ༷͞ • Deﬂate (JDK built-in), LZ4 (pure Java) • ࣍఺: Snappy

Slide 33

Slide 33 text

ࢀߟ: Docker ίϯςφ಺Ͱͷར༻ • Alpine Linux ͱ JNI binding ͳσʔλѹॖϥΠϒϥϦ ͷ૊Έ߹Θͤ͸໰୊͕ൃੜ͠΍͍͢ͷͰ஫ҙ͕ඞཁ • https://github.com/xerial/snappy-java/issues/181 • https://github.com/luben/zstd-jni/issues/38 • https://github.com/luben/zstd-jni/issues/43 • (ͦΕͧΕ࠷৽όʔδϣϯͰ͸ղফࡁΈ)

Slide 34

Slide 34 text

ࢀߟ: ѹॖ଎౓ɾѹॖ཰ͷൺֱ ϥΠϒϥϦ ύϥϝʔλ ѹॖॲཧ଎౓<.#T> ѹॖ཰<> EFqBUF EFqBUF EFqBUF M[KBWB ,# M[KBWB .# TOBQQZKBWB ,# [TUEKOJ [TUEKOJ [TUEKOJ -BSHFUFYUDPNQSFTTJPOCFODINBSLͷϑΝΠϧ FOXJLΑΓઌ಄.#Λ੾Γग़ͯ͠ධՁʹར༻

Slide 35

Slide 35 text

ࢀߟ: ෮ݩ଎౓ɾѹॖ཰ͷൺֱ ϥΠϒϥϦ ύϥϝʔλ ෮ݩॲཧ଎౓<.#T> ѹॖ཰<> EFqBUF M[KBWB ,# TOBQQZKBWB ,# [TUEKOJ

Slide 36

Slide 36 text

·ͱΊ

Slide 37

Slide 37 text

·ͱΊ • ѹॖ཰ͱॲཧ଎౓ɺͲͪΒΛॏࢹ͢Δ͔Ͱɺద੾ͳѹॖΞ ϧΰϦζϜ͸มΘͬͯ͘Δ • ݸਓతͳݟղͱͯ͠͸… • ѹॖ཰ (ͱॲཧ଎౓ͷόϥϯε) ॏࢹ: Zstandard • ॲཧ଎౓ॏࢹ: LZ4 • ѹॖΞϧΰϦζϜͱσʔλͱͷʮ૬ੑʯ͕͋Γ͏ΔͷͰɺ  ࣮ࡍʹѻ͏σʔλͰϕϯνϚʔΫΛऔͬͯΈΔ͜ͱ͕େࣄ

Slide 38

Slide 38 text

Thank you!

Slide 39

Slide 39 text

We’re hiring! • ͢΂ͯͷืू৬छ • bit.ly/SmartNews-Hiring • Φʔϓϯϙδγϣϯ / ϙδγϣϯαʔν • bit.ly/SmartNews-OpenPosition