Share
Embed
第9回 全日本コンピュータビジョン勉強会 発表資料
ICCV 2021
DocFormer: End-to-End Transformer for Document Understanding
Yamato.OKAMOTO
December 12, 2021
Transcript
25 slides
-
ຊɺհ͢Δจ w %PDVNFOU6OEFSTUBOEJOHΛλʔήοτͱͨ͠5SBOTGPSNFSϞσϧͷఏҊ ͍ͭͰʹࢲͷհ w Ԭຊେʢ͓͔ͱ·ͱʣ w ژେඒೱݚˠΦϜϩϯˠ-*/&גࣜձࣾ ,ZPUP0GGJDF
w ӉҰָ͍͠$7-BCΛژʹ࡞Δ͘νʔϜϏϧυத w ࠷ۙͷڵຯυΩϡϝϯτղੳٕज़ → LINE DevDay2021Ͱొஃͯ͠ɺυΩϡϝϯτղੳʹ͍ͭͯ͠·ͨ͠ʙ̇ ɹɹhttps://www.youtube.com/watch?v=D45wIMN6R_I -
-*/&BU*$$7 https://linecorp.com/ja/pr/news/ja/2021/3843 https://iccv2021.thecvf.com/sponsors ฐࣾͷΤʔε͕ຊ௨͠·ͨ͠ εϙϯαʔͱͯ͠ϒʔεΛग़͖ͤͯ͞·ͨ͠ ɾ(FOFSBMJ[FE4IV ffl FE-JOFBS3FHSFTTJPOʢ0SBMͰ͢ʣ ɹ'FJSBO-J ,FOU'VKJXBSB
'VNJP0LVSB BOE:BTVZVLJ.BUTVTIJUB ɾ"$MPTFS-PPLBU3PUBUJPOJOWBSJBOU%FFQ1PJOU$MPVE"OBMZTJT ɹ'FJSBO-J ,FOU'VKJXBSB 'VNJP0LVSB BOE:BTVZVLJ.BUTVTIJUB -
4FRVFODF-BCFMJOH )FBEFS PSBOHF 2VFTUJPO CMVF "OTXFS HSFFO ʹྨ ̍ͭ̍ͭͷςΩετΛʮ2VFTUJPOʯʮ"OTXFSʯͳͲͷΠϯςϯτΛϕʔεʹɺάϧʔϓ͚͢ΔλεΫɻ '6/4%
GPSNVOEFSTUBOEJOHJOOPJTZTDBOOFEEPDVNFOUT %BUBTFU͕༗໊ɻ https://arxiv.org/abs/1905.13538 จࣈྻɺ࠲ඪΛڭࢣϥϕϧͱͯ͠༩ ςΩετϦϯΫͷڭࢣϥϕϧ͋ΔͷͰɺ ,FZ7BMVFΛநग़͢ΔΑ͏ͳλεΫՄೳɻ -
&OUJUZ&YUSBDUJPO<> • $03%"$POTPMJEBUFE3FDFJQU%BUBTFUGPS1PTU0$31BSTJOH • ςΩετɺ࠲ඪɺΫϥεʢNFOV UPUBM ͳͲछྨʣͷڭࢣϥϕϧ͕༩͞Ε͍ͯΔ υΩϡϝϯτ͔Βಛఆใʢ&YNFOV QSJDFʣΛநग़͢ΔλεΫɻ Ϩγʔτը૾ͷ$03%%BUBTFUɺܖॻը૾ͷ,MFJTUFS/%"͕͋Δɻ
https://github.com/clovaai/cordɹɹhttps://arxiv.org/pdf/2003.02356.pdf -
ɿೖྗΛ.VMUJNPEBMʹ͢Δ • $//Ͱநग़ͨ͠7JTJPOಛྔʹɺ-BOHVBHFͷಛྔΛԣ͔Βࠩ͠ࠐΜͰɺߏղੳʹνϟϨϯδͨ͠ݚڀɻ • -BOHVBHFͷಛྔநग़ʹɺςΩετͷྖҬݕग़ͱೝࣝΛ͢Δ0$3෦͕ඞཁͱͳΔɻ • ͜ͷจͰɺطଘͷ0$3Τϯδϯʢ5FTTFSBDUʣΛར༻͍ͯ͠Δɻ೦ͳ͕Β͜ͷ෦&&Ͱֶश͞Εͳ͍ɻ υΩϡϝϯτʹը૾ݴޠؚ·Ε͍ͯͯɺͦΕͧΕ͕ώϯτʹͳΔͣʂ ͦ͜Ͱɺ྆ํͷಛྔΛநग़ͯ͠ར༻͢Δ.VMUJNPEBMͳݚڀ͕ਐలதɻ Learning
to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network (CVPR'17) https://arxiv.org/pdf/1706.02337.pdf -
ɿڭࢣͳֶ͠शͰ5SBOTGPSNFSߏΛQSFUSBJO ෳͷڭࢣͳ͠λεΫͷఏҊʹΑΓ5SBOTGPSNFSߏͷֶशʹޭɻ 1.UFYUUPLFOͷҰ෦ΛӅͯ͠ɺͦΕΛ෮ݩͤ͞Δɻ 2.Ұఆׂ߹ͷςΩετΛࠇ͘ృΓͭͿ͢ɻͦͯ͠ɺృΓ௵͞Ε͍ͯΔ͔ $PWFSFE ɺృΓ௵͞Εͯͳ͍͔ /PU$PWFSFE Λ༧ଌͤ͞Δɻ 3.ͱ͖͓Γɺը૾ͱςΩετͷϖΞΛ͢Γସ͑Δɻ͜ͷ݅ԼͰɺ
υΩϡϝϯτը૾ͱςΩετͷϖΞ͕ద͔ʢ.BUDIʣɺ ͢Γସ͑ΒΕ͍ͯΔ͔ʢ/PU.BUDIʣΛ༧ଌͤ͞Δɻ ҖΛฃ͍ͬͯΔ5SBOTGPSNFSʂʂ େྔσʔλͰֶशͰ͖ΔΑ͏ʹڭࢣͳ͠λεΫΛઃఆ͢Δɻ LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding (ACL'21) https://arxiv.org/pdf/2012.14740.pdf -
ɿ5SBOTGPSNFSͷΞʔΩςΫνϟʔΛվળ B 7JTJPOͱ8PSEͷ5PLFOΛྻʹฒͯॱ൪ʹೖྗ͢Δɻ ҟͳΔ.PEBMಉ࢜ͷσʔλͰదʹ"UUFOUJPO͕͔͔Δ͔ٙɻ C 7JTJPOͱ8PSEΛҟͳΔ5SBOTGPSNFSͰॲཧ͔ͯ͠Β'VTJPO͢Δɻ "UUFOUJPO͔͔Γͦ͏͕ͩɺ.VMUJNPEBMͷԸܙ͕ݮΓͦ͏ɻ D
5SBOTGPSNFSʹೖྗ͢Δલͷ5PLFOஈ֊Ͱ'VTJPOͤ͞Δɻ DPODBUͳͲͰγϯϓϧͳख๏Ͱ'VTJPOͤͯ͞Α͍ͷ͔ո͍͠ɻ ओு͍ͯ͠Δརͱܽਖ਼͍͠ͷ͔ٙΘ͍͠ɺɺɺɺ ͩΜͩΜͲΕ͕࠷దͳͷ͔͔Βͳ͘ͳ͖ͬͯͨɻɻɻɻ 7JTJPO-BOHVBHFͳͲੑ࣭ͷҟͳΔಛྔΛ͍͔ʹ'VTJPOͤ͞Δ͔ʁ ΞʔΩςΫνϟઃܭͰݚڀ͕നதʂʂ https://arxiv.org/abs/2106.11539 -
7JTJPO -BOHVBHF 4QBUJBMͷछྨͷಛྔΛೖྗ 7JTJPO'FBUVSF • 3FT/FUͰ֫ಘͨ͠'FBUVSF.BQΛɺY@$POWͰҙͷDI E ʹมޙɺMJOFBSUSBOTGPSNFSMBZFSͰҙͷαΠζ E /
ʹม -BOHVBHF'FBUVSF • 0$3ΤϯδϯͰςΩετΛݕग़ʴೝࣝʴಛྔԽͨ͠5PLFOΛɺ&NCFEEJOHMBZFSʹ͔͚Δɻʢ˞0$3෦ֶशෆՄʣ • நग़ͨ͠ςΩετͷݸ͕ɺ/ݸʹຬͨͳ͍߹<1"%>Ͱ݀ຒΊ͢Δɻ/ݸΛ͑Δ߹5PLFOΛഁغ͢Δɻ 4QBUJBM'FBUVSF • 0$3Τϯδϯ͕ݕग़֤ͨ͠ςΩετͷCCPY࠲ඪΛՃͯ͠4QBUJBM'FBUVSFͱ͢Δɻ • CCPYͷࠨ্࠲ඪɺӈԼ࠲ඪɺ෯ɺߴ͞ɺʢը૾্Ͱࠨ্͔ΒӈԼʹ͔͚ͯCCPYΛφϯόϦϯάͨ͠ͱԾఆͯ͠ʣྡͷ#PYͱͷڑΛɺ&NCFEEJOHMBZFS USBJOBCMF ʹ͔͚Δɻ • 7JTJPOઐ༻ͷ&NCFEEJOHMBZFSͱɺ-BOHVBHFઐ༻ͷ&NCFEEJOHMBZFSΛઃ͚ͯɺͦΕͧΕผʑͷ4QBUJBM'FBUVSFΛੜ͢Δɻ • ͜ͷͱ͖ɺ͋Θͤͯ7JTJPO-BOHVBHFͷ%1PTJUJPOʢԿ൪ͷಛྔ͔ʁʣɺ͋Θͤͯ&NCFEEJOHMBZFSʹೖྗ͢Δɻ ը૾σʔλ͔ΒҟͳΔϞμϦςΟͷಛྔΛநग़ͯ͠ɺೖྗΛΠΠײ͡ʹͨ͠ ຊจͷΞʔΩςΫνϟͰɺ֤ϞμϦςΟͰ ಉ͡ݸ ͜ͷਤͰ/ ͷಛྔΛ४උ͍ͯ͠Δɻ CCPYͰUFYUಛྔͷҐஔใΛ࣋ͨͤΔͱͱʹɺ %1PTJUJPOʹΑͬͯɺWJTJPOಛྔͷҐஔใ࣋ͨͤΒΕΔͱ༧ɻ -
3FDPOTUSVDUJPOΛؚΉͭͷڭࢣͳ͠λεΫΛఆٛ .BTLFE-BOHVBHF.PEFMJOH • ैདྷͷ#&35ͱಉ༷ʹɺ&ODPEF͞Εͨಛྔ͔Β.BTLFE5FYUΛ༧ଌ͢Δɻ • ͨͩ͠ɺ.VMUJNPEBMʹͨ͠ಛྔ͔Β෮ݩͤ͞Δ͕ಛతͰ͋Δɻ 3FDPOTUSVDU • &ODPEF͞ΕͨશͯͷಛྔΛར༻ͯ͠ɺ"VUP&ODPEFSͱಉ༷ʹը૾Λ3FDPOTUSVDUJPO͢Δɺ-PTTTNPPUI-Λ࠾༻ɻ •
ग़ྗը૾ͷΈ͕ͩɺจࣈը૾Λਖ਼֬ʹ3FDPOTUSVDUJPO͘͢ɺ7JTJPOʹՃ͑ͯ-BOHVBHFͷ׆༻๏ֶशͯ͘͠ΕΔͱظɻ 5FYU%FTDSJCF*NBHF • Ұఆׂ߹Ͱը૾ͱ5FYUͷΈ߹ΘͤΛγϟοϑϧͯ͠ɺϖΞ͕ద͔Ͳ͏͔ PS Λ༧ଌ͢Δɻ • ্ड़ͷͭͷλεΫ͕ओʹ-PBDM'FBUVSFʹண͢ΔҰํͰɺͪ͜Β(MPCBM'FBUVSFʹண͖͢λεΫͰ͋Δɻ • <$-4>Λ&ODͨ͠ಛྔ͔Β༧͍ͯ͠Δͷɺ"UUFOUJPOͰΠΠײ͡ʹશମ͔ΒಛྔΛू͍ͯ͠ΔͱظͰ͖Δ͔Βɻ ֤छϞʔμϧɺ-PDBM'FBUVSFɺ(MPCBM'FBUVSFΛ׆༻͢ΔΑ͏ ڭࢣͳ͠λεΫΛόϥϯεΑ͘ఏҊͯ͠ɺֶशΛΠΠײ͡ʹͨ͠ -
ఏҊΞʔΩςΫνϟͷৄࡉ w -BOHVBHFͱ7JTJPOͦΕͧΕʹઐ༻ͷMBZFSʢࢵͱʣΛઃ͚ͯɺಛྔΛมͯ͠࠷ޙʹՃࢉ͢Δ w -BOHVBHF'FBUVSFɺMBZFSΛॏͶΔ͝ͱʹɺҟͳΔϞʔμϧಛΛࠞͥͳ͕Β܁Γฦ͠5SBOTGPSN͞ΕΔ w ҰํͰɺ7JTJPO'FBUVSFͱ4QBUJBM'FBUVSFɺຖճॳظঢ়ଶͰ֤-BZFSʹԣ͔Βྲྀ͠ࠐ·ΕΔ w 4QBUJBM'FBUVSFΛ,FZ2VFSZʹ͢ΔMBZFSڞ௨Խʢʣ͍ͯ͠Δ w
ͨͩ͠ɺͦͦ4QBUJBM'FBUVSF-BOHVBHFઐ༻ͱ7JTJPOઐ༻Ͱผʑʹ४උͨ͠ʹҙ w 8SFM%SFMBUJWFQPTJUJPOFNCFEEJOHͱݺΕɺJ൪ͱK൪ͷؔੑΛ֫ಘ͢ΔͨΊͷֶशՄೳͳύϥϝλ w จͰɺྑ͍࣮ݧ݁Ռ͕ಘΒΕͨ͜ͱ͔Βɺʮ֤ϞʔμϧͷΞΫηεΛྑͨ͘͜͠ͱ͕ॏཁʂʯͱओு͍ͯ͠Δ https://openaccess.thecvf.com/content/ICCV2021/supplemental/Appalaraju_DocFormer_End-to-End_Transformer_ICCV_2021_supplemental.pdf ελϯμʔυͳܗ͜Εʂ -
࣮ݧ݁ՌɿैདྷݚڀΑΓߴੑೳ͔ʁ ͏·͍ͬͯ͘Δྫ .FOVJUFN 5PBM ੨ 4VC5PUBM ʹྨ )FBEFS
2VFTUJPO ੨ "OTXFS ʹྨ -
࣮ݧ݁Ռɿ7JTJPO'FBUVSFʹཱ͍ͬͯΔ͔ʁ ֤λεΫͰɺʮ5 4ʯΑΓ7JTJPOΛՃ͑ͨʮ5 4 7ʯͷํ͕'4DPSF͕େ͖͘ͳͬͨɻ ͨͩ͠ɺ্෯͕QPJOUఔͷ߹͋ΕɺQPJOUҎԼͷ߹͋ͬͨɻ %PD'PSNFSCBTFˠUSBOTGPSNFSMBZFST %PD'PSNFSMBSHFˠUSBOTGPSNFSMBZFST * 5
4ˠ5FYU *NBHF 4QBUJBM'FBUVSFΛར༻ 5 4ˠ5FYU 4QBUJBM'FBUVSFͷΈΛར༻ -
ຊͷ·ͱΊ %PDVNFOU6OEFSTUBOEJOHͷͨΊͷ5SBOTGPSNFSϞσϧΛఏҊ ೖྗΛΠΠײ͡ʹͨ͠ʂʢ5FYU 4QBUJBM 7JTJPOʣ ֶशΛΠΠײ͡ʹͨ͠ʂʢ3FDVOTUSVDUJPOλεΫΛ৽نఏҊʣ ΞʔΩςΫνϟΛΠΠײ͡ʹͨ͠ʂʢTLJQߏͰ֤ϞʔμϧʹΞΫηεͨ͘͢͠͠ʣ 4P5"ୡ͚ͩͰͳ͘ϞσϧܰྔԽͷ؍Ͱྑ͍࣮ݧ݁ՌΛࣔͨ͠ɻ ݸਓతʹؾʹͳͬͨ O൪ͷ7JTJPO'FBUVSF͕ɺO൪ͷ5FYU'FBUVSFͷCCPYྖҬͷը૾ಛͰͳ͍͕ؾʹͳΔɻ
4QBUJBM'FBUVSFΛࠞͥΔաఔͰରԠؔऔΕ͍ͯΔͱਪଌ͢Δ͕ɺ 7JTJPO'FBUVSFΛ5SBOTGPSNͤ͞Δํ๏ɺ·ͩ·ͩݚڀͷ༨͕͋Δͱײͨ͡ɻ