Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Index Compression Using Byte-Aligned ANS Coding and Two-Dimensional Contexts

chie8842
April 15, 2018

Index Compression Using Byte-Aligned ANS Coding and Two-Dimensional Contexts

WSDM 2018 論文読み会の資料
https://atnd.org/events/95510

chie8842

April 15, 2018
Tweet

More Decks by chie8842

Other Decks in Technology

Transcript

  1. *OEFY$PNQSFTTJPO6TJOH#ZUF"MJHOFE"/4
    $PEJOHBOE5XP%JNFOTJPOBM$POUFYU
    84%.ಡΈձ
    $IJF)BZBTIJEB

    View full-size slide

  2. Θͨ͠ͷ͠ΐ͏͔͍
    $IJF)BZBTIJEB !DIJF


    ΫοΫύουɹػցֶशνʔϜ
    $POUSJCVUFUP"QBDIF4QBSL )BEPPQ FUD

    ൃചͷ8%#13&44ͷ
    1ZUIPOͱՊֶٕज़ܭࢉͷষ
    ॻ͍ͨɻ

    View full-size slide

  3. "#453"$5
    •  #MPDLCBTFEJOWFSUFEJOEFYDPNQSFTTJPOͷ࣮ݧΛߦͬͨɻ
    •  "TZNNFUSJD/VNFSBM4ZTUFNT "/4
    ΤϯτϩϐʔίʔσΟϯ
    άʹରͯ͠ҎԼͷ֦ுΛߦ͏͜ͱʹΑΔվળ͕ಘΒΕͨɻ
    –  ͭͷू໿ύϥϝʔλʹΑΔ࣍ݩͷ৚݅ίϯςΩετʹΑͬͯɺγ
    ϯϘϧͷΧςΰϥΠζΛߦ͏ɻ
    –  #ZUFGSJFOEMZͳϚοϐϯάΛ࢖ͬͯγϯϘϧΛ"/4ͷDPEFXPSE
    CVDLFUʹΘ͚Δ
    –  ίϯςΩετϚʔδͷϓϩηεʹΑΓɺෳ਺ͷEJTUSJCVUJPOΛͭͱ
    Έͳ͞ΕΔɻ
    •  ͜ͷख๏ʹΑΓɺJOUFSQʹΑΔΠϯσοΫεͷѹॖͷੑೳج
    ४఺Λ্ճͬͨɻ
    •  ࣮ݧ͸(#ͷHPWίʔύεͱ৽͍͠χϡʔεهࣄΛ࢖͍ɺ
    ΫΤϦධՁ͸ผͷCMPDLCBTFEͷख๏ͱൺֱͨ͠ɻ

    View full-size slide

  4. */530%6$5*0/
    •  *OWFSUFE సஔ
    ΠϯσοΫε͸ςΩετσʔλʹର͢Δ
    ߴ଎ͳΫΤϦ࣮ߦͷॏཁͳߏ଄Ͱ͋Δɻ
    term Posting list
    J1IPOF
    Ϗοάσʔλ
    Χό
    ϝϩϯ




    posting
    ͦΕͧΕͷυΩϡϝϯτ*%ʹରͯ͠୯ޠͷग़ݱස౓΍ϙδ
    γϣϯͷ৘ใ΋࣋ͭɻ
    ίϯύΫτͳQPTUJOHMJTUTʹ͸ɺCZUFBMJHOFEDPEFT XPSE
    BMJHOFEDPEFT CJOBSZQBDLFEΞϓϩʔν͕͋Δɻ

    View full-size slide

  5. */530%6$5*0/
    •  ϋϑϚϯූ߸ԽͷΑ͏ͳΤϯτϩϐʔϕʔεͷख๏΋ఏҊ͞Ε͍ͯ
    ͕ͨɺҎલ͸͋·ΓސΈΒΕΔ͜ͱ͕ͳ͔ͬͨɻ೥લʹ+FSFL
    %VEB͕"/4ϝΧχζϜΛ։ൃͨ͜͠ͱʹΑΓɺΤϯτϩϐʔίʔ
    σΟϯά͕࠶ͼ஫໨͞ΕΔΑ͏ʹͳͬͨɻ
    •  7#ZUF 4JNQMF 1BDLFEͷख๏Λ༻͍ͯɺͦΕͧΕѹॖޮ཰͕վળ͞
    Εͨɻ
    •  1BDLFE"/4ΞϓϩʔνͷఏҊ
    –  ݻఆ௕ϒϩοΫͷγϯϘϧ͕ηϨΫλʹΑͬͯಛఆ͞Εͨ֬཰෼෍ͱؔ࿈
    ͯ͠ූ߸Խ͞ΕΔɻ
    –  1BDLFE"/4Ξϓϩʔνͷ঺հʹ͓͍ͯ͸ɺηϨΫλ͸ϒϩοΫͷ࠷্Ґ
    ͕ར༻͞ΕΔɻ
    –  QBDLFEGSBNFPGSFGFSFODF 1'03
    ϝΧχζϜͷΑ͏ͳར఺͕͋Δɻ
    –  ΤϯτϩϐʔίʔσΟϯάͷϝϦοτ
    •  ΤϯτϩϐʔίʔσΟϯάͷॊೈੑʹΑΓɺࣅ௨ͬͨූ߸ͷ֬཰ٴͼූ
    ߸௕ͷ໘Ͱѹॖޮ཰্͕͕ͬͨɻ
    –  ϒϩοΫͷू໿ʹ͓͍ͯ࣍ݩͷηϨΫλ͕ར༻Ͱ͖ͨɻ
    •  γϯϘϧͷࣝผࢠ͔Β"/4ͷ஋΁ͱϚοϐϯάΛม͑ΔলϝϞϦͰ
    CZUFGSJFOEMZͳग़ྗ
    •  σίʔυͷࡍͷϝϞϦ࢖༻ྔΛ࡟ݮ͢ΔͨΊͷ$POUFYU.FSHJOH

    View full-size slide

  6. #"$,(306/%ɿ*OEFY$PNQSFTTJPO
    ˙$PNQSFTTJPO"QQSPBDI
    •  طଘͷํࣜ
    –  ఻౷తͳख๏ɹ&MJBTDPEFT
    –  7#ZUF΍ͦͷมܗ
    –  ϫʔυϕʔεͷγϯϓϧͳΞϓϩʔν
    –  *OUFSQNFDIBOJTN
    •  εέʔϧͨ͠ͱ͖ʹྑ͍ѹॖޮ཰͕ಘΒΕ͕ͨɺEFDPEF͕
    &MJBTΑΓ஗͔ͬͨɻ

    View full-size slide

  7. ิ଍ɿ&MJBT(BNNB
    https://en.wikipedia.org/wiki/Elias_gamma_coding

    View full-size slide

  8. ิ଍ɿ7#ZUF
    1 0 0 0 0 0 1 0
    value
    End marker
    1000000110000010
    整数自体を符号化するのでなく、逐次差分を符号化する方法

    View full-size slide

  9. #"$,(306/%ɿQBDLFEDPEFT
    ˙1BDLFE$PEFT
    •  ݻఆ௕ͷQPTUJOHTʹର͢ΔQBDLFEDPEFT
    •  ൺֱత࠷ۙͷٕज़
    •  ΩʔΞΠσΞ͸ɺϩʔΧϧͳҰ؏ੑΛར༻͢Δํ๏Ͱ͋
    Δɻ
    •  ݻఆ௕ʢେ఍ʣͷϒϩοΫ͝ͱͷ஋ΛͭͷϢχο
    τͱͯ͠ॲཧ͢Δɻ
    •  ྫ͑͹ɺ

    ͔ΒηϨΫλΛબͿ͜ͱͰɺ͋Δ஋Λ

    ͱ͍ͬͨॻ͖ํͰදͤΔɻ

    View full-size slide

  10. #"$,(306/%ɿ1BUDIFE'SBNFPG3FGFSFODF
    2 1 31 1 0 … 2
    10 01 01 01 00 … 10
    b=5 b’=2 1exception position3
    •  ϒϩοΫຖʹͦͷϒϩοΫ಺ͷ਺ͷ΄ͱΜͲΛද͢͜ͱ͕Ͱ͖ΔϏοτ਺Λ
    ܾΊͯ
    Τϯίʔυ͢Δɻ
    •  খ͍͞Ϗοτ਺ͰදͤΔͨΊσʔλαΠζ͕খ͘͞ͳΔɻ
    •  1'03ͷมܗͰ͋Δ0QU1'03Ͱ͸ɺ࠷΋શମͷύοέʔδ͕͜Μͺ͘ͱʹͳ
    ΔΑ͏ʹηϨΫλΛબͿΑ͏ʹͳ͍ͬͯΔɻ
    b
    b’
    metadata

    View full-size slide

  11. #"$,(306/%ɿ7BSJBCMF-FOHUI#SPDLT
    •  Մม௕ͷϒϩοΫʹύʔςΟγϣχϯάͨ͠৔߹ͷख๏
    Λ0UUBWJBOPFUBM͕ఏҊ͍ͯ͠Δɻ
    •  ಉҰͷσʔλ͕ग़ݱͨ͠ͱ͖ʹେ͖ͳΠϯϓοτϢχο
    τΛ࣋ͭηϨΫλετϨʔδΛঈ٫͢ΔΑ͏ʹγʔΫ͢
    Δɻ஋͕ہॴతʹόʔετ͢Δ৔߹ʹ༗ޮͳํ๏Ͱ͋Δɻ

    View full-size slide

  12. #"$,(306/%ɿ"TZNNFUSJD/VNFSBM
    4ZTUFNTʢ"/4ʣ
    •  +BSFL%VEBʹΑͬͯ։ൃ͞Εͨ৽͍͠Τϯτϩϐʔූ߸
    Խํࣜ
    •  ϋϑϚϯූ߸Խ΍ࢉज़ූ߸ͱಉ͡໾ׂΛՌͨ͢ɻ
    •  ग़ݱස౓ͷߴ͍γϯϘϧʹϏοτ਺ͷখ͍͞ූ߸ɺग़ݱ
    ස౓ͷ௿͍γϯϘϧʹϏοτ਺ͷେ͖͍ූ߸ΛׂΓ౰ͯ
    Δɻ

    View full-size slide

  13. "/4Λ༻͍ͨΠϯσοΫεѹॖͷྫ
    •  γϯϘϧ1 2 3͸ͦΕͧΕग़ݱ֬཰ɺɺͱ͢Δ
    •  ϋϑϚϯූ߸ԽͰ͸ɺ1͕̍Ϗοτɺ2ͱ3͕̎ϏοτͱͳΔɻ
    •  "/4ූ߸ԽͰ͸ɺ.ݸͷཁૉͷ͕1ɺ͕2ɺ͕3ʹϥϕϦϯά͞Εɺ
    ͦͷΞϧϑΝϕοτ্ͷه߸ͷγʔέϯε͸ɺରԠ͢ΔϑϨʔϜͷঢ়ଶ஋ͱ͠
    ͯදݱ͞ΕΔɻ
    •  ଞͷϑϨʔϜΛ࡞Δ৔߹͸ɺଞͷॱྻ΋͘͠͸.ΑΓେ͖͍஋Λ࢖༻͢Δɻ
    •  ϑϨʔϜͷූ߸Խ͸ɺભҠؔ਺5 ɾ ɾ
    Λ༻͍ͯߦ͏ɻ5্ͷΧϥϜ͸ɺεςʔ
    τม਺ͰΠϯσοΫεԽ͞ΕΔɻ
    •  શମΛූ߸Խ͢Δ৔߹͸ɺεςʔτม਺͕େ͖͘ͳΔɻ
    •  σίʔυ͢Δͱ͖͸ɺɹɹɹ͕࢖ΘΕΔɻ

    View full-size slide

  14. "/4Λ༻͍ͨΠϯσοΫεѹॖͷྫʢଓ͖ʣ
    •  TFRVFODF2222222222͸ݸͷεςʔτΛ
    ࣋ͪɺϏοτͰද͞ΕΔɻϋϑϚϯූ߸Խͩͱ
    ϏοτͰ͋Δɻ
    •  ରͯ͠ɺTRVFODF3333333333͸εςʔτ͕
    ݸͱͳΓɺϏοτʹͳΔɻ
    •  ͜ͷख๏͸ɺ4LFXͷ͋Δσʔλʹ༗ޮͰ͋Δɻ
    39364
    118096
    ….

    View full-size slide

  15. "/4$PNQBSFE
    •  ϋϑϚϯූ߸ٴͼ"SJUINFUJDDPEJOHͱͷൺֱ
    –  "/4͸ཧ૝తͳූ߸Խ௕͕"SJUINFUSJDDPEJOHͱۙ͘ͳΔɻ
    –  ߴ͍ग़ݱස౓ͷγϯϘϧΛൃੜ͢ΔͨΊʹɺϏοτͷίʔυ͢
    Βൃੜ͠ͳ͍͍ͯ͘৔߹͕͋ΔɻʢϋϑϚϯූ߸ͩͱৗʹϏο
    τҎ্͕ඞཁɻʣ
    –  "SJUINFUSJDDPEJOHΑΓ΋͸΍͘ɺߴ౓ʹઃܭ͞ΕͨϋϑϚϯූ
    ߸ͷ࣮૷ʹ͍ۙεϧʔϓοτϨʔτΛग़͢͜ͱ͕Ͱ͖Δɻ
    –  "/4͸"SJUINFUJDDPEJOHͱҟͳΓ੩తϝΧχζϜͰ͋Γɺҟͳ
    ΔίϯςΩετͷ΋ͱͰγϯϘϧΛੜ੒͢Δ͜ͱ͕Ͱ͖ͳ͍ɻ
    ͨͩ͜͠ͷ͜ͱ͸ΠϯσοΫεѹॖʹ͸Өڹ͠ͳ͍ɻ

    View full-size slide

  16. "/4GPS*OEFY$PNQSFTTJPO
    •  Ҏલͷݚڀ
    –  "/4Λ̏ͭͷҟͳΔΠϯσοΫεѹॖํࣜͱ૊Έ߹Θͤͨɻ
    –  ಛʹ7#ZUFͱ૊Έ߹ΘͤΔΞϓϩʔνΛQPTUJOHMJTUͷѹॖʹ࢖༻
    ͨ͠ɻ
    –  σίʔυͷεϧʔϓοτ͸গ͠௿Լ͢Δ͚ͩͰɺετϨʔδΛ
    ͔ͳΓઅ໿Ͱ͖Δ͜ͱΛ֬ೝͨ͠ɻ
    •  ࠓճͷख๏ɿʮ1BDLFE"/4ʯϝΧχζϜ
    –  ʮͭͷύϥϝʔλʯΛ࢖͏
    –  গͳ͍ϝϞϦ࢖༻ྔͷίϯύΫτͳ࣮૷
    –  ίϯςΩετϚʔδ
    •  ͜ΕΒͷख๏ͷ༗༻ੑΛ࣮ূ͢ΔͨΊʹɺͭͷେ͖ͳ
    ςΩετίϨΫγϣϯΛ࢖༻ͯ͠ɺѹॖͷ༗ޮੑͱΫΤ
    Ϧͷεϧʔϓοτʹؔ͢Δৄࡉͳ࣮ݧ݁ՌΛఏڙ͢Δɻ

    View full-size slide

  17. 1BDLFE"/4
    •  ඪ४తͳ1BDLFEBSSBOHFNFOUͰ͸ɺίϯςΩετΛಛఆ
    ͢ΔηϨΫλ͕֤ϒϩοΫͷූ߸Խͷௐ੔ʹར༻͞ΕΔɻ
    •  ූ߸Խεςοϓ͕όΠφϦίʔυΛ࢖༻͢Δ৔߹ɺηϨ
    Ϋλ͸ɺϒϩοΫ಺ͷ࠷େ஋Λݟग़͠ɺόΠφϦͷେ͖
    ͞Λܾఆ͢Δɻ
    •  ࣍ʹɹɹɹɹͱͳΔΑ͏ͳୈͷ஋ɹΛٻΊΔɻैདྷͷ
    1BDLFEߏ੒Ͱ͸ɺM൪໨ͷίϯςΩετΛ࢖༻ͯ͠ɹɹɹɹɹ
    ஋ΛؚΉϖΠϩʔυ͕ੜ੒͞ΕΔɻ

    View full-size slide

  18. *.1307&%1"$,&%$0.13&44*0/
    1BDLFE"/4
    •  1BDLFE"/4ϝΧχζϜͰ͸ɺ͜ΕΒͷ͞·͟·ͳཁૉ͕ີ
    ઀ʹ݁ͼ͔ͭͳ͍ɻ
    •  ηϨΫλ͸ϒϩοΫ಺ͷ࠷େ஋ͱ͠ɺ"/4ϑϨʔϜϓʔϧ
    Λࣝผ͢Δɻ
    •  ֤ϑϨʔϜʹ͸ɺ૊ͷϙεςΟϯάΛॲཧ͢Δ࠷ॳͷύεத
    ʹ౷ܭΛ஝ੵ͠ɺ֬཰෼෍Λর߹͢ΔͨΊʹߏங͞Εͨҟ
    ͳΔγϯϘϧ֬཰ηοτΛؚΉɻ
    •  γϯϘϧͷස౓͸ਐ਺ͷόέοτʹ஝ੵ͞ΕɺͦͷΑ͏ͳ֤
    όέοτ಺ͷ͢΂ͯͷγϯϘϧʢͨͱ͑͹ʣʹͳΔɻ
    •  ྫ͑͹ɺʜ ʜͱʜ͸౳֬཰Ͱ͋ΔͱԾఆ
    ͞ΕΔɻ
    •  ΋͕͠γʔέϯεɹͷ൪໨ͷγϯϘϧ͕ϒϩοΫͷηο
    τͱͯ͠ූ߸Խ͞Ε͍ͯΔͳΒ͹ɺ͸ͱಉ͡ϒϩοΫͷ
    γϯϘϧʹΘͨΔ࠷େਐ਺͔Βܭࢉ͞ΕͨηϨΫλ஋Ͱ͋Δɻ

    View full-size slide

  19. 5XP%JNFOUJPOBM$POUFYUT
    •  ίϯςΩετΛ֦ு͢Δ͜ͱͰɺηϨΫλΛ֨ೲ͢Δί
    ετ͸૿Ճ͢Δ͕ɺਖ਼֬ʹλʔήοτͷ֬཰෼෍Λ࢖༻
    ͢Δ͜ͱ͕Ͱ͖Δ
    •  ίϯςΩετͷ௥Ճํ๏ͱͯ͠͸ɺҎԼ͕ڍ͛ΒΕΔ
    –  ̎ͷྦྷ৐ʹΑΓηϨΫλϕΫτϧΛऔಘ͢Δ
    –  όέοτ͕খ͘͞ͳΔΑ͏ʹର਺ͷఈΛมߋ͢Δ
    •  ্ͨͩ͠ه͸྆ํڞϒϩοΫ಺ͷ࠷େ஋ʹΑܾͬͯ·ͬ
    ͯ͠·͏ͷͰɺΤϯτϩϐʔίʔσΟϯάͷར఺Λ׆͔
    ͤͳ͍

    View full-size slide

  20. 5XP%JNFOUJPOBM$POUFYUT
    •  ࣍ݩͷηϨΫλΛར༻͍ͨ͠ɻ
    –  ୈҰ࣍ݩ͕ϒϩοΫͷ࠷େ஋ɺୈೋ࣍ݩ͕ϒϩοΫͷ࠷খ஋ͱ
    ͢Δɻ
    •  ͜ͷ্ݶ஋ΑΓେ͖͍஋ɺԼݶ஋ΑΓখ͍͞஋͸ྫ֎ͱͯ͠
    ѻ͏͜ͱ͕Ͱ͖Δɻ
    –  ϒϩοΫͷதԝ஋Λར༻͢Δ͜ͱ΋Ͱ͖Δɻ

    View full-size slide

  21. 5XP%JNFOUJPOBM$POUFYUT
    •  ࣍ݩ͔Βೋ࣍ݩͷίϯςΩετʹ੾Γସ͑ͨ৔߹ͷ༷ࢠɻ

    View full-size slide

  22. 3FEVDJOHUIF'SBNFTJ[F
    •  1BDLFE"/4Ͱ͸ɺσίʔυʹ࢖༻͞ΕΔςʔϒϧͷ
    ҡ࣋ίετ͕͔͔Δͱ͍͏σϝϦοτ͕͋Δɻ
    •  "/4ΛγϯϘϧͷਐ਺ͷൣғͰ࢖༻͠ɺ"/4ετϦʔ
    ϜͱิॿϏοτετϦʔϜΛฒྻʹॲཧ͢Δ͜ͱͰ஋Λ
    ࠶ߏ੒͢Δͱ͍͏खஈ͕͋Δɻ
    •  ͜ͷ৔߹௥ՃͷΞϥΠϝϯτʹΑΔίετ͕͔͔Δ
    ͜ͷͨΊɺ1'03΍0QU1'03Ͱར༻͞ΕΔྫ֎Λఆٛ͠
    ͯγϯϘϧ஋ͱ"/4λʔήοτͷؒͷϚοϐϯάΛఆٛ
    ͢Δɻ
    •  ͜ΕʹΑΓѹॖͷ༗ޮੑΛߴΊΔ͜ͱ͕Ͱ͖Δɻ

    View full-size slide

  23. 1BDLFE"/4ͷσίʔυ

    View full-size slide

  24. $POUFYU.FSHJOH
    •  ίϯςΩετΛ௥Ճ͢ΔͱɺηϨΫλ͕௥ՃϏοτΛඞ
    ཁͱ͢ΔͨΊɺίϯςΩετ͕ଟ͗͢Δͱޮ཰͕ѱ͘ͳ
    ΔՄೳੑ͕͋ΔɻίϯςΩετ਺͕ͷྦྷ৐Ͱ͋Δ͜ͱ
    Λอূ͢Δ΂͖ɻ
    •  ίϯςΩετͷϖΞΛϚʔδ͢Δ͜ͱͰɺ୯Ұͷίϯς
    Ωετʹஔ͖׵͑ΔҊɻ
    γϯϘϧT͕ίϯςΩετB্ͰOB T
    ճൃੜ͢Δͱ͢Δͱɺ
    τʔλϧͷPDDVSSFODF͸
    ΤϯτϩϐʔίʔσΟϯά͸ՄೳͰɺͦͷίϯςΩετ
    ͷτʔλϧίετ͸

    View full-size slide

  25. &YQFSJNFOUT
    &YQFSJNFOUBM$POUFYU
    •  ࣍ݩͷηϨΫλΛར༻ͨ͠ࡍͷOFXTHBQͷ֬཰෼෍ͷ

    View full-size slide

  26. γϯϘϧग़ݱස౓ʹجͮ͘ѹॖ༗ޮੑ

    View full-size slide

  27. .FBTVSFE*NQSPWFNFOU

    View full-size slide

  28. ࣮ߦ࣌ؒɺϝϞϦ࢖༻ྔɺΫΤϦεϧʔϓοτ
    •  *OUFSQΑΓ΋ΫΤϦϨεϙϯε͕ૣ͘ɺѹॖ཰͸࠷΋ྑ
    ͍ͱ͍͏݁Ռ͕ಘΒΕͨɻ

    View full-size slide

  29. $0/$-64*0/4"/%'6563&803,
    ˙$PODMVTJPOT
    •  1BDLFE"/4ϝΧχζϜʹΑΓɺͭͷҟͳΔख๏Λվળͨ͠ɻ
    –  ࣍ݩίϯςΩετ
    –  ࠷্ҐόΠτϚοϐϯά
    –  ίϯςΩετϚʔδϯά
    •  ೥લͷ*OUFSQͷϕϯνϚʔΫʹউͬͨɻ
    •  1BDLFE"/4ͷ࣮૷͸0QU1'03΍&'PQUʹൺ΂ͯΫΤϦεϐʔ
    υʹ͓͍ͯ͸ྼ͍ͬͯΔ͕ɺ"/4ʹ͸·ͩվળͷ༨஍͕͋Δɻ
    •  ͢Ͱʹ΄ͱΜͲͷQPTUJOHT͕খ͍͞αΠζʹූ߸ԽͰ͖Δ͜ͱΛ֬
    ೝͨ͠ɻ
    ˙'VUVSFXPSL
    •  ෳ਺ͷঢ়ଶΛฒߦͯ͠෮߸͢ΔͨΊʹ֦ு͞Εͨ෮߸Խςʔϒϧ͓
    Αͼ4*.%ػߏΛ࢖༻͢Δͱɺ෮߸Խ଎౓ΛഒҎ্ʹ૿Ճͤ͞ΔՄ
    ೳੑ͕͋Δɻ
    •  Մม௕ϒϩοΫͷ࢖༻
    •  ΫΤϦͷ౷ܭ৘ใΛར༻ͯ͠ҟͳΔѹॖϝΧχζϜΛมߋ͢Δ

    View full-size slide