第６章　テキストデータのための素性

!LNPUPIBT ຊڷݴޠॲཧษڧձ ୈ6ষ ςΩετσʔλͷͨΊͷૉੑ

ࣗݾ঺հ !3 ຊڮ࿨و !LNPUPIBT ‣ ౦޻େجૅ෺ཧֶઐ߈1I%
‣ ιϑτόϯΫגࣜձࣾ ‣ %&&1$03&*OD ‣ εΧΠϚΠϯυגࣜձࣾ w ਂ૚ֶशϞσϧͷ։ൃɾӡ༻ϓϥοτϑΥʔϜ࡞ͬͯΔձࣾ +BWB4DBSBͰͰ͖ΔEFFQMFBSOJOHKͷ։ൃ ,FSBTOE UI DPOUSJCVUPST͕͍ͨΓ͢Δ w ๻͸ϓϦηʔϧεΤϯδχΞ

໨࣍ !4 ୈ̎ฤɹࣗવݴޠσʔλͷѻ͍ ‣ ୈ̒ষɹςΩετσʔλͷͨΊͷૉੑ w ࣗવݴޠॲཧʹ͓͚Δ෼ྨ໰୊ͷλΠϓ෼͚ w ࣗવݴޠॲཧ໰୊ͷͨΊͷૉੑ
௚઀؍࡯Ͱ͖Δૉੑ ਪଌ͞ΕΔݴޠతಛ௃ ֩ͱͳΔૉੑͱ૊Έ߹Θͤૉੑ OάϥϜૉੑ ෼෍࿦తૉੑ

ୈ̒ষͰѻ͏಺༰ !5 %//͸ૉੑΤϯδχΞϦϯάͷ͢΂ͯΛ୲ͬͯ͸͘Εͳ͍ λεΫ͝ͱʹ֩ͱͳΔૉੑͷू߹Λద੾ʹܾఆ͢Δ ςΩετσʔλ ࣮਺஋ϕΫτϧ ૉੑ ؔ਺
ɾૉੑநग़ GFBUVSFFYUSBDUJPO ɾૉੑදݱ GFBUVSFSFQSFTFOUBUJPO ͜ͷຊͰ͸lGFBUVSFzΛʮಛ௃ྔʯͰ͸ͳ͘ʮૉੑʯͱ༁͢໛༷

ࣗવݴޠॲཧʹ͓͚Δ෼ྨ໰୊ͷλΠϓ෼͚ !6 ୯ޠ ‣ ੜ͖෺ʁɺͲͷݴޠʁɺҰൠతʁɺࣅͨ୯ޠ͸ʁɺλΠϙʁ ςΩετʢจॻ෼ྨEPDVNFOUDMBTTJpDBUJPOʣ ‣ εύϜʁɺ੓࣏εϙʔπʁɺൽ೑ʁɺߠఆతʁɺ୭ͷஶ࡞ʁɺ޷·ΕΔʁ
ςΩετର ‣ ಉٛޠʁɺద੾ͳ຋༁ʁɺಉ͡ஶऀʁɺཁ໿ʁ จ຺ʹຒΊࠐ·Εͨ୯ޠ ‣ CPPL໊ࢺಈࢺܗ༰ࢺʁɺBQQMFاۀՌ෺ʁɺݻ༗දݱʁ ೋͭͷ୯ޠͷؔ܎ ‣ ୯ޠ"͸ಈࢺ#ͷओޠʁɺ໨తޠʁ

ίϥϜɿ୯ޠͱ͸Կ͔ !7 ୯ޠۭനͰ۠੾ΒΕͨจࣈͷܥྻʁ ‣ lEPH zlEPH zlEPHzlEPHz ୯ޠۭന͋Δ͍͸۟ಡ఺Ͱ۠੾ΒΕͨจࣈͷܥྻ
UPLFO ʁ ‣ l*#.zl.SzlEPO`UzϔϒϥΠޠɺΞϥϏΞޠɺதࠃޠɺ೔ຊޠɺʜ ‣ lDBUzl$BUzl/FX:PSLzlJDFDSFBNzlJDFDSFBNzlLJDLUIFCVDLFUz ୯ޠ XPSE ͱτʔΫϯ UPLFO ͸۠ผ͞ΕΔ ‣ τʔΫϯτʔΫϯԽॲཧͷग़ྗ ‣ ୯ޠҙຯΛӡͿ୯Ґ

ࣗવݴޠॲཧ໰୊ͷͨΊͷૉੑ !8 ୯ޠ΍จࣈ͸཭ࢄత‎ࢦඪܭ਺ΛΑ͘༻͍Δ w ࢦඪૉੑ JOEJDBUPSGFBUVSF w ͋Δ৚݅Λຬͨͨ͠Β̍ɺ͞΋ͳ͚Ε͹̌
ྫʣ୯ޠEPH͕จॻதʹҰ౓Ҏ্ݱΕͨΒ̍ɺͳ͚Ε͹̌ ‣ ܭ਺ DPVOU w ͋Δग़དྷࣄ͕ੜͨ͡ճ਺ ྫʣςΩετதʹ୯ޠEPH͕ݱΕͨճ਺ ௚઀؍࡯Ͱ͖Δૉੑ ਪଌ͞ΕΔݴޠతಛ௃ ֩ͱͳΔૉੑͱ૊Έ߹Θͤૉੑ OάϥϜૉੑ ෼෍࿦తૉੑ

௚઀؍࡯Ͱ͖Δૉੑ !9 Ұͭͷ୯ޠͷૉੑ ςΩετͷૉੑ จ຺தͷ୯ޠͷૉੑ ୯ޠؔ܎ͷૉੑ

Ұͭͷ୯ޠͷૉੑ !10 ϨϯϚԽ MFNNBUJ[JOH ‣ ϨϯϚ MFNNB ࣙॻݟग़͠ޠ
‣ CPPLJOH CPPLFE CPPLTCPPL εςϛϯά TUFNNJOH ‣ ϨϯϚԽΑΓ΋ૈ͍ॲཧ ‣ QJDUVSF QJDUVSFT QJDUVSFEQJDUVS ݪจ "TUFNNFSNBQTTFRVFODFTPGXPSETUPTIPSUFSTFRVFODFT 8PSE/FU-FNNBUJ[FS "TUFNNFSNBQTFRVFODFPGXPSEUPTIPSUFSTFRVFODF 1PSUFS "TUFNNFSNBQTFRVFODPGXPSEUPTIPSUFSTFRVFOD -BODBTUFS BTUFNNBQTFRVPGXPSEUPTIPSUTFRV 4UFNNJOHBOE-FNNBUJ[BUJPOXJUI1ZUIPO/-5,

Ұͭͷ୯ޠͷૉੑ !11 ޠኮࢿݯ MFYJDBMSFTPVSDFT ‣ 8PSE/FU<'FMMCBVN > w
୯ޠʹ͍ͭͯͷ֓೦తҙຯత஌ࣝΛ֫ಘ͢Δ͜ͱΛ໨త w ୯ޠΛෳ਺ͷTZOTFU ಉٛޠू߹ ʹଐ͢Δ w ਓखͰاըߏங IUUQXPSEOFUXFCQSJODFUPOFEVQFSMXFCXO *#.8BUTPO͸8PSE/FU %#1FEJB FUD ͳͲ͔Β஌ࣝମܥΛ֫ಘ ΫΠζ൪૊+FPQBSEZͰਓؒͷνϟϯϐϣϯΛഁͬͨ IUUQXXXNBUISZVLPLVBDKQdXXXRNBMBC*14+.(/QEG

Ұͭͷ୯ޠͷૉੑ !12 ޠኮࢿݯ MFYJDBMSFTPVSDFT ‣ 'SBNF/FU<'JMMNPSFFUBM > ‣
7FSC/FU<,JQQFSFUBM > w ಈࢺͷपΓʹয఺Λ౰ͯͯؔ࿈͢Δ߲Λྻڍ ֩ͱͳΔ߲ w ఏڙऀ %0/&3 w डྖऀ 3&$*1*&/5 w ର৅ 5)&.& ֩Ͱͳ͍߲ w ࣌ 5*.& w ໨త 1&3104& w ৔ॴ 1-"$& w ༷ଶ ."//&3 w ਓखͰاըߏங IUUQTGSBNFOFUJDTJCFSLFMFZFEVGO3FQPSUTEBUBGSBNF*OEFYYNM

Ұͭͷ୯ޠͷૉੑ !13 ޠኮࢿݯ MFYJDBMSFTPVSDFT ‣ 5IF1BSBQISBTF%BUBCBTF 11%#
<(BOJULFWJUDIFUBM 1BWMJDLFUBM > w ࣗಈߏங͞Εͨݴ͍׵͑ͷσʔληοτ ෼෍࿦తૉੑ EJTUSJCVUJPOBMGFBUVSF ‣ ޙड़ IUUQQBSBQISBTFPSH ޠኮࢿݯΛޮ཰తʹར༻͢Δํ๏͸λεΫʹґଘ ΤϯδχΞϦϯάతਚྗɾ૑ҙ޻෉͕ඞཁ //ϞσϧͰ͸සൟʹ׆༻͞Ε͍ͯͳ͍ ʢঢ়گ͸มԽ͢Δ͔΋͠Εͳ͍ʣ

ࢀߟʣ6OJ%JDCZࠃཱࠃޠݚڀॴ !14 IUUQTVOJEJDOJOKBMBDKQBCPVU@VOJEJD ‏ܗଶૉղੳث.F$BC༻ͷղੳ༻ࣙॻ

ςΩετͷૉੑ !16 ୯ޠόοά #BHPG8PSET#08 ‣ ςΩετதͷ୯ޠͷग़ݱ਺ͷώετάϥϜ 5'*%'ͷॏΈ෇͚<.BOOJOHFUBM
> w 5FSN'SFRVFODZ*OWFSTF%PDVNFOU'SFRVFODZ ‣ ಛఆͷจॻͰଟ਺ճग़ݱ͍ͯ͠Δ୯ޠΛॏཁࢹ w BGPSͳͲ͸ଟ਺ճݱΕΔ͕ಛ௃తͰ͸ͳ͍ IUUQTXXXQZUSZHDPNFOUSZ #d(w) P w02d #d(w0) <latexit sha1_base64="eq9A2AGwHGKX4XAIKzdU/Yx30FY=">AAACl3ichVHLLgRBFD3aa4zXYENsJiaEzaQaCbEhJGJpMEg0ne6eGir6le6emdCZH/ADJFYkIuIfbCz4AQufIJYkNhbu9HQiCG6nq06de8+tU1W6awo/YOyxQWpsam5pTbQl2zs6u7pTPb3rvlPyDJ43HNPxNnXN56aweT4Qgck3XY9rlm7yDX1/oZbfKHPPF469Fhy4fNvSdm1RFIYWEKWmxpWipxmhklELo5Wxaqj4JUsNKzuK6wmLK8JOF6r1ZEyNVdVUhmVZFOmfQI5BBnEsO6lLKCjAgYESLHDYCAib0ODTtwUZDC5x2wiJ8wiJKM9RRZK0JariVKERu0/jLq22Ytamda2nH6kN2sWk3yNlGsPsgV2xF3bPrtkTe/+1Vxj1qHk5oFmva7mrdh/1r779q7JoDrD3qfrTc4AipiOvgry7EVM7hVHXlw+PX1ZnVobDEXbOnsn/GXtkt3QCu/xqXOT4yimS9ADy9+v+CdbHs/JEVs5NZubm46dIYBBDGKX7nsIclrCMPO17ghvc4V4akGalRWmpXio1xJo+fAkp9wEvRp1/</latexit> |{d 2 D : w 2 d}| |D| <latexit sha1_base64="mbLSZYgS7PJ8YfNe2KQ734QEkeg=">AAAChHichVHLSsNAFD3Gd31V3QguLBbFhZSJD5QuRNSFS19VwUhJ0mkdTJOQpJWadunGH3DhSkFE3OoXuPEHXPgJ4lLBjQtv0oCoqDdk7pkz99w5M6PZhnA9xh4bpMam5pbWtvZYR2dXd0+8t2/TtUqOzjO6ZVjOtqa63BAmz3jCM/i27XC1qBl8S9tfDNa3ytxxhWVueBWb7xbVginyQlc9orLxISXvqLpfVfycIszEUvogSDmlVq351aVqLRtPshQLI/ETyBFIIooVK34JBTlY0FFCERwmPMIGVLj07UAGg03cLnziHEIiXOeoIUbaElVxqlCJ3aexQLOdiDVpHvR0Q7VOuxj0O6RMYIQ9sCv2wu7ZNXti77/28sMegZcKZa2u5Xa253hg/e1fVZGyh71P1Z+ePeQxG3oV5N0OmeAUel1fPjx5WU+vjfij7Jw9k/8z9sju6ARm+VW/WOVrp4jRA8jfr/sn2JxIyZMpeXUqOb8QPUUbBjGMMbrvGcxjGSvI0L5HuMYNbqUWaVyalKbrpVJDpOnHl5DmPgDhU5YQ</latexit> #d(w) P w02d #d(w0) ⇥ log |D| |{d 2 D : w 2 d}| <latexit sha1_base64="j4TbcpsK1GId4xrumkAPx+Vug3E=">AAACwHichVHLThRBFD20CDgojLIxYTNhgoHNpBpNNKwIsnDJwwGSKex099QMFfpFd81MoKd/gB9gwUoSQoxfYdzwAy74AmLcQCBx48Lbj8QAUW+l6p46955bt6qswJGRYux8QHsw+HBoeORRafTxk7Hx8tNn65HfCW1Rt33HDzctMxKO9ERdSeWIzSAUpms5YsPaeZvGN7oijKTvvVd7gdhyzbYnW9I2FVFGucFboWnHvGo0Z3qzScyjjmvEvQ88CKUruPQqzSQPFtRswhW5iDt+O9f2l/pJ3OdxM81emu9lIp70E6NcZTWWWeU+0AtQRWHLfvkUHE34sNGBCwEPirADExGNBnQwBMRtISYuJCSzuECCEmk7lCUowyR2h9Y27RoF69E+rRllaptOcWiGpKxgmn1jn9g1O2Of2Xf266+14qxG2sseeSvXisAYP3i+9vO/Kpe8wvYf1T97VmjhTdarpN6DjElvYef67v7h9dr86nT8gh2zH9T/R3bOvtINvO6NfbIiVo9Qog/Q7z73fbA+V9Nf1vSVV9WFxeIrRjCJKczQe7/GAt5hGXU69wsucIkrbVHb1nxtN0/VBgrNBG6Ztv8bMP+vZA==</latexit> ୯ޠස౓ 5' จॻස౓ %' 5'*%' จॻEதͷશ୯ޠग़ݱ਺Ͱਖ਼نԽͨ͠୯ޠXͷग़ݱ਺ ୯ޠX͕ग़ݱ͢ΔจॻEͷ਺ͷίʔύεαΠζc%cʹର͢Δׂ߹

จ຺தͷ୯ޠͷૉੑ !18 ૭ XJOEPX ‣ ͋Δ୯ޠͷ௚ۙͷจ຺ʹয఺ w lର৅ͷ୯ޠͷपΓʹ୯ޠ9͕ݱΕͨz͜ͱ͕ૉੑ
ର৅ͷۙ͘ʹ͋Δ୯ޠ͸ԕ͘཭Εͨ୯ޠΑΓ΋ର৅ͷ৘ใΛଟ͍࣋ͬͯ͘Δ͜ͱ͕ଟ͍ UIFCSPXOGPYKVNQFEPWFSUIFMB[ZCPZ XJOEPXTJ[FL యܕతʹ͸ \XPSECSPXO XPSEGPY XPSEPWFS XPSEUIF^ \XPSECSPXO XPSEGPY XPSE PWFS XPSE UIF^ Ґஔ෇͖૭ɿ૬ରతҐஔૉੑ SFMBUJWFQPTJUJPOBMGFBUVSFT^

‣ จ຺தͷೋͭͷ୯ޠͷؒͷڑ཭ EJTUBODF ‣ ؒʹݱΕΔ୯ޠͦͷ΋ͷ JEFOUJUZ

ਪଌ͞ΕΔݴޠֶతಛ௃ !20 ౷ޠ TZOUBY ‣ ࣗવݴޠʹ͓͚Δจͷߏ଄Λࢧ഑͢Δنଇ UIFCPZXJUIUIFCMBDLTIJSUPQFOFEUIFEPPSXJUIBLFZ
'&"563&4 '03 /-1 130#-&.4 JOH EFHSFFT PG BDDVSBDZ 4QFDJBMJ[FE TZTUFNT FYJTU GPS UIF QSFEJDUJPO PG QBSUT PG TQFFDI TZOUBDUJD USFFT TFNBOUJD SPMFT EJTDPVSTF SFMBUJPOT BOE PUIFS MJOHVJTUJD QSPQFSUJFT XJUI WBSJPVT EFHSFFT PG BDDVSBDZ © BOE UIFTF QSFEJDUJPOT PGUFO TFSWF BT HPPE GFBUVSFT GPS GVSUIFS DMBTTJmDBUJPO QSPCMFNT -JOHVJTUJD "OOPUBUJPO -FUT FYQMPSF TPNF GPSNT PG MJOHVJTUJD BOOPUBUJPOT $POTJEFS UIF TFOUFODF UIF CPZ XJUI UIF CMBDL TIJSU PQFOFE UIF EPPS XJUI B LFZ 0OF MFWFM PG BOOPUBUJPO BTTJHOT UP FBDI XPSE JUT QBSU PG TQFFDI UIF CPZ XJUI UIF CMBDL TIJSU PQFOFE UIF EPPS XJUI B LFZ %Ėĥ /ĠĦğ 1ģĖġ %Ėĥ "ĕě /ĠĦğ 7Ėģē %Ėĥ /ĠĦğ 1ģĖġ %Ėĥ /ĠĦğ (PJOH GVSUIFS VQ UIF DIBJO XF NBSL TZOUBDUJD DIVOL CPVOEBSJFT JOEJDBUJOH UIF UIF CPZ JT B OPVO QISBTF < NP UIF CPZ > < PP XJUI > < NP UIF CMBDL TIJSU > < VP PQFOFE > < NP UIF EPPS > < PP XJUI > < NP B LFZ > /PUF UIBU UIF XPSE PQFOFE JT NBSLFE BT B WFSCBMDIVOL 71 ɩJT NBZ OPU TFFN WFSZ VTFGVM CFDBVTF XF BMSFBEZ LOPX JUT B WFSC )PXFWFS 71 DIVOLT NBZ DPOUBJO NPSF FMFNFOUT DPWFSJOH BMTP DBTFT TVDI BT XJMM PQFOFE BOE EJE OPU PQFO ɩF DIVOLJOH JOGPSNBUJPO JT MPDBM " NPSF HMPCBM TZOUBDUJD TUSVDUVSF JT B DPOTUJUVFODZ USFF BMTP DBMMFE B QISBTFTUSVDUVSF USFF S NP DT NN NP PP PP VBD VP '&"563&4 '03 /-1 130#-&.4 JOH EFHSFFT PG BDDVSBDZ 4QFDJBMJ[FE TZTUFNT FYJTU GPS UIF QSFEJDUJPO PG QBSUT PG TQFFDI TZOUBDUJD USFFT TFNBOUJD SPMFT EJTDPVSTF SFMBUJPOT BOE PUIFS MJOHVJTUJD QSPQFSUJFT XJUI WBSJPVT EFHSFFT PG BDDVSBDZ © BOE UIFTF QSFEJDUJPOT PGUFO TFSWF BT HPPE GFBUVSFT GPS GVSUIFS DMBTTJmDBUJPO QSPCMFNT -JOHVJTUJD "OOPUBUJPO -FUT FYQMPSF TPNF GPSNT PG MJOHVJTUJD BOOPUBUJPOT $POTJEFS UIF TFOUFODF UIF CPZ XJUI UIF CMBDL TIJSU PQFOFE UIF EPPS XJUI B LFZ 0OF MFWFM PG BOOPUBUJPO BTTJHOT UP FBDI XPSE JUT QBSU PG TQFFDI UIF CPZ XJUI UIF CMBDL TIJSU PQFOFE UIF EPPS XJUI B LFZ %Ėĥ /ĠĦğ 1ģĖġ %Ėĥ "ĕě /ĠĦğ 7Ėģē %Ėĥ /ĠĦğ 1ģĖġ %Ėĥ /ĠĦğ (PJOH GVSUIFS VQ UIF DIBJO XF NBSL TZOUBDUJD DIVOL CPVOEBSJFT JOEJDBUJOH UIF UIF CPZ JT B OPVO QISBTF < NP UIF CPZ > < PP XJUI > < NP UIF CMBDL TIJSU > < VP PQFOFE > < NP UIF EPPS > < PP XJUI > < NP B LFZ > /PUF UIBU UIF XPSE PQFOFE JT NBSLFE BT B WFSCBMDIVOL 71 ɩJT NBZ OPU TFFN WFSZ VTFGVM CFDBVTF XF BMSFBEZ LOPX JUT B WFSC )PXFWFS 71 DIVOLT NBZ DPOUBJO NPSF FMFNFOUT DPWFSJOH BMTP DBTFT TVDI BT XJMM PQFOFE BOE EJE OPU PQFO ɩF DIVOLJOH JOGPSNBUJPO JT MPDBM " NPSF HMPCBM TZOUBDUJD TUSVDUVSF JT B DPOTUJUVFODZ USFF BMTP DBMMFE B QISBTFTUSVDUVSF USFF S NP VP '&"563&4 '03 /-1 130#-&.4 JOH EFHSFFT PG BDDVSBDZ 4QFDJBMJ[FE TZTUFNT FYJTU GPS UIF QSFEJDUJPO PG QBSUT PG TQFFDI TZOUBDUJD USFFT TFNBOUJD SPMFT EJTDPVSTF SFMBUJPOT BOE PUIFS MJOHVJTUJD QSPQFSUJFT XJUI WBSJPVT EFHSFFT PG BDDVSBDZ © BOE UIFTF QSFEJDUJPOT PGUFO TFSWF BT HPPE GFBUVSFT GPS GVSUIFS DMBTTJmDBUJPO QSPCMFNT -JOHVJTUJD "OOPUBUJPO -FUT FYQMPSF TPNF GPSNT PG MJOHVJTUJD BOOPUBUJPOT $POTJEFS UIF TFOUFODF UIF CPZ XJUI UIF CMBDL TIJSU PQFOFE UIF EPPS XJUI B LFZ 0OF MFWFM PG BOOPUBUJPO BTTJHOT UP FBDI XPSE JUT QBSU PG TQFFDI UIF CPZ XJUI UIF CMBDL TIJSU PQFOFE UIF EPPS XJUI B LFZ %Ėĥ /ĠĦğ 1ģĖġ %Ėĥ "ĕě /ĠĦğ 7Ėģē %Ėĥ /ĠĦğ 1ģĖġ %Ėĥ /ĠĦğ (PJOH GVSUIFS VQ UIF DIBJO XF NBSL TZOUBDUJD DIVOL CPVOEBSJFT JOEJDBUJOH UIF UIF CPZ JT B OPVO QISBTF < NP UIF CPZ > < PP XJUI > < NP UIF CMBDL TIJSU > < VP PQFOFE > < NP UIF EPPS > < PP XJUI > < NP B LFZ > /PUF UIBU UIF XPSE PQFOFE JT NBSLFE BT B WFSCBMDIVOL 71 ɩJT NBZ OPU TFFN WFSZ VTFGVM CFDBVTF XF BMSFBEZ LOPX JUT B WFSC )PXFWFS 71 DIVOLT NBZ DPOUBJO NPSF FMFNFOUT DPWFSJOH BMTP DBTFT TVDI BT XJMM PQFOFE BOE EJE OPU PQFO ɩF DIVOLJOH JOGPSNBUJPO JT MPDBM " NPSF HMPCBM TZOUBDUJD TUSVDUVSF JT B DPOTUJUVFODZ USFF BMTP DBMMFE B QISBTFTUSVDUVSF USFF S NP DT the boy opened IN IN NN NP NP DT DT with with the the black shirt a key door JJ NN NN DT NN NP PP PP VBD VP $POTUJUVFODZ USFFT BSF OFTUFE MBCFMFE CSBDLFUJOH PWFS UIF TFOUFODF JOEJDBUJOH UIF IJ FSBSDIZ PG TZOUBDUJD VOJUT UIF OPVO QISBTF UIF CPZ XJUI UIF CMBDL TIJSU JT NBEF PG UIF OPVO '&"563&4 '03 5&956"- %"5" QISBTF UIF CPZ BOE UIF QSFQPTJUJPO QISBTF 11 XJUI UIF CMBDL TIJSU ɩF MBUUFS JUTFMG DPOUB UIF OPVO QISBTF UIF CMBDL TIJSU )BWJOH XJUI B LFZ OFTUFE VOEFS UIF 71 BOE OPU VOEFS /1 UIF EPPS TJHOBMT UIBU XJUI B LFZ NPEJmFT UIF WFSC PQFOFE PQFOFE XJUI B LFZ SBUIFS U UIF /1 B EPPS XJUI B LFZ " EJĊFSFOU LJOE PG TZOUBDUJD BOOPUBUJPO JT B EFQFOEFODZ USFF 6OEFS EFQFOEF TZOUBY FBDI XPSE JO UIF TFOUFODF JT B NPEJmFS PG BOPUIFS XPSE XIJDI JT DBMMFE JUT IF &BDI XPSE JO UIF TFOUFODF JT IFBEFE CZ BOPUIFS TFOUFODF XPSE FYDFQU GPS UIF NBJO XP VTVBMMZ B WFSC XIJDI JT UIF SPPU PG UIF TFOUFODF BOE JT IFBEFE CZ B TQFDJBM iSPPUw OPEF the boy with the black shirt opened the door with a key det prep prep amod det pobj nsubj root dobj pobj det det 8IJMF DPOTUJUVFODZ USFFT NBLF FYQMJDJU UIF HSPVQJOH PG XPSET JOUP QISBTFT EFQFOEF USFFT NBLF FYQMJDJU UIF NPEJmDBUJPO SFMBUJPOT BOE DPOOFDUJPOT CFUXFFO XPSET 8PSET UIBU ඼ࢺ 1BSUPG4QFFDI104 ౷ޠతνϟϯΫ TZOUBDUJDDIVOL ߏ੒ཁૉ໦ DPOTUJUVFODZUSFF ۟ߏ଄໦ QISBTFTUSVDUVSFUSFF ґଘߏ଄໦ EFQFOEFODZUSFF

ਪଌ͞ΕΔݴޠֶతಛ௃ !21 '&"563&4 '03 5&956"- %"5"
QISBTF UIF CPZ BOE UIF QSFQPTJUJPO QISBTF 11 XJUI UIF CMBDL TIJSU ɩF MBUUFS JUTFMG DPOUBJOT UIF OPVO QISBTF UIF CMBDL TIJSU )BWJOH XJUI B LFZ OFTUFE VOEFS UIF 71 BOE OPU VOEFS UIF /1 UIF EPPS TJHOBMT UIBU XJUI B LFZ NPEJmFT UIF WFSC PQFOFE PQFOFE XJUI B LFZ SBUIFS UIBO UIF /1 B EPPS XJUI B LFZ " EJĊFSFOU LJOE PG TZOUBDUJD BOOPUBUJPO JT B EFQFOEFODZ USFF 6OEFS EFQFOEFODZ TZOUBY FBDI XPSE JO UIF TFOUFODF JT B NPEJmFS PG BOPUIFS XPSE XIJDI JT DBMMFE JUT IFBE &BDI XPSE JO UIF TFOUFODF JT IFBEFE CZ BOPUIFS TFOUFODF XPSE FYDFQU GPS UIF NBJO XPSE VTVBMMZ B WFSC XIJDI JT UIF SPPU PG UIF TFOUFODF BOE JT IFBEFE CZ B TQFDJBM iSPPUw OPEF the boy with the black shirt opened the door with a key det prep prep amod det pobj nsubj root dobj pobj det det 8IJMF DPOTUJUVFODZ USFFT NBLF FYQMJDJU UIF HSPVQJOH PG XPSET JOUP QISBTFT EFQFOEFODZ USFFT NBLF FYQMJDJU UIF NPEJmDBUJPO SFMBUJPOT BOE DPOOFDUJPOT CFUXFFO XPSET 8PSET UIBU BSF GBS BQBSU JO UIF TVSGBDF GPSN PG UIF TFOUFODF NBZ CF DMPTF JO JUT EFQFOEFODZ USFF 'PS FYBNQMF CPZ BOE PQFOFE IBWF GPVS XPSET CFUXFFO UIFN JO UIF TVSGBDF GPSN CVU IBWF B EJSFDU OTVCK FEHF DPOOFDUJOH UIFN JO UIF EFQFOEFODZ USFF ɩF EFQFOEFODZ SFMBUJPOT BSF TZOUBDUJD UIFZ BSF DPODFSOFE XJUI UIF TUSVDUVSF PG UIF TFOUFODF 0UIFS LJOET PG SFMBUJPOT BSF NPSF TFNBOUJD 'PS FYBNQMF DPOTJEFS UIF NPEJmFST PG UIF WFSC PQFO BMTP DBMMFE UIF BSHVNFOUT PG UIF WFSC ɩF TZOUBDUJD USFF DMFBSMZ NBSLT UIF CPZ XJUI UIF CMBDL TIJSU UIF EPPS BOE XJUI B LFZ BT BSHVNFOUT BOE BMTP UFMMT VT UIBU XJUI B LFZ JT BO BSHVNFOU PG PQFO SBUIFS UIBO B NPEJmFS PG EPPS *U EPFT OPU UFMM VT IPXFWFS XIBU BSF UIF TFNBOUJDSPMFT PG UIF BSHVNFOUT XJUI SFTQFDU UP UIF WFSC JF UIBU UIF CPZ JT UIF "ĘĖğĥ QFSGPSNJOH UIF BDUJPO BOE UIBU B LFZ JT BO *ğĤĥģĦĞĖğĥ DPNQBSF UIBU UP UIF CPZ PQFOFE UIF EPPS XJUI B TNJMF )FSF UIF TFOUFODF XJMM IBWF UIF TBNF TZOUBDUJD TUSVDUVSF CVU VOMFTT XF BSF JO B NBHJDBMXPSME B TNJMF JT B .ĒğğĖģ SBUIFS UIBO BO *ğĤĥģĦĞĖğĥ ɩF TFNBOUJD SPMF MBCFMJOH BOOPUBUJPOT SFWFBM UIFTF TUSVDUVSFT ґଘߏ଄໦ EFQFOEFODZUSFF '&"563&4 '03 /-1 130#-&.4 the boy with the black shirt opened the door with a smile the boy with the black shirt opened the door with a key Agent Instrument Patient Manner Patient Agent #FTJEFT UIF PCTFSWBCMF QSPQFSUJFT MFUUFST XPSET DPVOUT MFOHUIT MJOFBS EJTUBODFT GSFRVFO DJFT FUD XF DBO BMTP MPPL TVDI JOGFSSFE MJOHVJTUJD QSPQFSUJFT PG XPSET TFOUFODFT BOE EPDVNFOUT 'PS FYBNQMF XF DPVME MPPL BU UIF QBSUPGTQFFDI UBH 104 PG B XPSE XJUIJO B EPDVNFOU *T JU B OPVO B WFSC BEKFDUJWF PS B EFUFSNJOFS UIF TZOUBDUJD SPMF PG B XPSE %PFT JU TFSWF BT B TVCKFDU PS ҙຯ໾ׂϥϕϧ෇͚ TFNBOUJDSPMFMBCFMJOH

ਪଌ͞ΕΔݴޠֶతಛ௃ !22 ඼ࢺ 1BSUPG4QFFDIUBH104 ‣ ໊ࢺɺಈࢺɺܗ༰ࢺɺݶఆࢺ ౷ޠత໾ׂ
TZOUBDUJDSPMF ‣ ಈࢺͷओޠɺ໨తޠɺओಈࢺɺ෭ࢺతम০ޠ ҙຯ໾ׂ TFNBOUJDSPMF ‣ UIFLFZ ಓ۩ PQFOUIFEPPSUIFCPZ ಈ࡞ओ PQFOTUIFEPPS ౷ޠతґଘߏ଄໦ TZOUBDUJDEFQFOEFODZUSFF ‣ ଟ͘ͷ୯ޠͰִͯΒΕΔೋͭͷ୯ޠ͕౷ޠߏ଄ʹ͓͍ͯ͸ྡ઀ͯ͠Δ͜ͱ΋ ஊ࿩ؔ܎ EJTDPVSTFSFMBUJPOT ‣ จͱจͱͷؔ܎ɻৄࡉԽ NPSFPWFS ɺ൓࿦ IPXFWFS ɺҼՌ CFDBVTF ͳͲʢ઀ଓࢺͳ͍৔߹΋ʣ রԠ BOBQIPSB ‣ UIFCPZPQFOFEUIFEPPSXJUIBLFZ*UXBTO`UMPDLFEBOEIFFOUFSFEUIFSPPN)FTBXBNBO)FXBTTNJMJOH

ݴޠֶత֓೦͸ඞཁ͔ !23 ݴޠֶతಛ௃͸ඞཁͳ͘ɺ//͸தؒදݱΛࣗ෼Ͱֶश͢Δ ʢਂ૚ֶऀࢧ࣋ऀͷҰ෦ʣ े෼ͳྔͷσʔλͱਖ਼͍͠ํ޲෇͚͕͋Ε͹Մೳʢචऀʣ ‣ े෼ͳσʔλ͕ͳ͍৔߹͸໌ࣔతʹҰൠతͳ֓೦Λ//ʹఏڙ͢Δͷ͸ ༗ӹ
‣ ݴޠֶతಛ௃Λೖྗૉੑʹར༻͠ͳ͍৔߹Ͱ΋ɺϚϧνλεΫֶशʹ ͓͚Δ௥Ճతͳڭࢣͱͯ͠༻͍Δ͜ͱ͕Ͱ͖Δ %FFQ-FBSOJOH"$SJUJDBM"QQSBJTBM<(BSZ.BSDVT>

֩ͱͳΔૉੑͱ૊Έ߹Θͤૉੑ !24 ֩ͱͳΔૉੑ DPSFGFBUVSFT ૊Έ߹Θͤૉੑ DPNCJOBUJPOGFBUVSFT
‣ ৘ใྔ͕͋Γɺ͔ͭແବ͕ൺֱతগͳ͍Α͏ͳૉੑΛ࡞Δͷ͸೉͍͠ ɾ ઢܗϞσϧͰ͸ɺ૊Έ߹Θͤʢ9㱸:ʣ΋9 :ͦΕͧΕͷ߹ܭҎ্ͷείΞʹͳΒͳ͍ ‣ χϡʔϥϧωοτϫʔΫͷΑ͏ͳඇઢܗϞσϧͰ͜ͷ໰୊Λճආ ɾ ֩ૉੑͷू߹͚ͩఆٛ͠ɺॏཁͳ૊Έ߹Θͤૉੑ͸ωοτϫʔΫͷ܇࿅Ͱ֫ಘ \XJOEPXOPU XJOEPX9 XJOEPX : XJOEPX ;^ \XJOEPX9` XJOEPXMJLF XJOEPX :` XJOEPX ;`^ \XJOEPXOPU XJOEPXMJLF XJOEPX 9`` XJOEPX :``^ 㲉 1BSJT‎ཱྀߦΧςΰϦ )JMUPO‎ཱྀߦΧςΰϦ 1BSJT)JMUPO‎༗໊ਓPSΰγοϓΧςΰϦ

OάϥϜૉੑ !25 OάϥϜ OHSBN ‣ /FX:PSLOPUHPPE1BSJT)JMUPOͷΑ͏ͳߏ଄Λ֫ಘͰ͖Δ w PGUIFPOBUIFCPZͳͲͷόΠάϥϜ͸Ұൠత
Ϟσϧͷௐ੔Ͱখ͍͞ॏΈΛׂΓ౰ͯΔ ‣ .-1͸9㱸:ͱ͍͏૊Έ߹Θͤ͸ֶशͰ͖Δ͕9: CJHSBN ͸ೝࣝͰ͖ͳ͍ w ৞ΈࠐΈωοτϫʔΫ͸OHSBNGFBUVSFΛݟ͚ͭग़͢Α͏ʹઃܭ͞Ε͍ͯΔ w CJEJSFDUJPOBM3//͸Մม௕OHSBN΍ɺؒʹΪϟοϓΛؚΉOHSBNΛײ஌

෼෍࿦తૉੑ !26 ݴޠͷ෼෍Ծઆ EJTUSJCVUJPOBMIZQPUIFTJT <'JSUI )BSSJT> ‣ 'JSUIl:PVTIBMMLOPXBXPSECZUIFDPNQBOZJULFFQTz w
୯ޠͷҙຯ͸ͦΕ͕༻͍ΒΕΔจ຺͔Βਪ࿦Ͱ͖Δ CZ8JUUHFOTUFJO Firth, J.R. (1957). A synopsis of linguistic theory 1930-1955 ɾQJ[[B DIBJSɾ ɾCVSHFS ɾQJ[[B DIBJSɾ ɾCVSHFS ݪจΊͪΌͪ͘ΌԼ඼ͳྫͰস͏

จ຺͔Β୯ޠͷҰൠԽΛֶश !27 ΫϥελϦϯάʹجͮ͘ख๏ DMVTUFSJOHCBTFENFUIPET ‣ ྨࣅͨ͠୯ޠΛಉ͡୯ޠΛಉ͡ΫϥελʹׂΓ౰ͯΔ ‣ Ϋϥελʹॴଐ͢Δ౓߹͍ʹΑͬͯͦΕͧΕͷ୯ޠΛදݱ
ຒΊࠐΈʹجͮ͘ख๏ FNCFEEJOHCBTFENFUIPET ‣ ͦΕͧΕͷ୯ޠΛϕΫτϧͰදݱ ‣ ྨࣅͨ͠୯ޠ͸ྨࣅͨ͠ϕΫτϧͰදݱ͞ΕΔ ɾQJ[[B DIBJSɾ ɾCVSHFS ɾQJ[[B DIBJSɾ ɾCVSHFS ˎ-POEPO#FSMJO͸ৗʹྨࣅͨ͠୯ޠͱͯ͠ѻ͏΂͖ʁʢߤۭνέοτ༧໿ɺจॻ຋༁ͳͲͰ͸۠ผ͍ͨ͠ʣ

·ͱΊ !28 ςΩετσʔλͷૉੑͱͯ͠࢖ΘΕ͍ͯΔ΋ͷΛྻڍ ‣ ෼ྨର৅͸୯ޠɺςΩετɺςΩετରɺจষதͷ୯ޠɺ୯ޠͷؔ܎ ‣ ௚઀؍࡯Ͱ͖Δૉੑ w ϨϯϚɺεςϛϯάɺޠኮࢿݯɺ5'*%'ɺ8JOEPX
‣ ਪଌ͞ΕΔݴޠతಛ௃ w จ๏ ‣ ֩ͱͳΔૉੑͱ૊Έ߹Θͤૉੑ w //Λ༻͍ͯ૊Έ߹ΘͤૉੑΛ֫ಘ ‣ OάϥϜૉੑ w l/FX:PSLzͳͲෳ਺ͷ୯ޠ͔Βߏ੒͞ΕΔݴ༿Λଊ͑Δ ‣ ෼෍࿦తૉੑ w l:PVTIBMMLOPXBXPSECZUIFDPNQBOZJULFFQTz

第６章　テキストデータのための素性

第６章　テキストデータのための素性

kmotohas

More Decks by kmotohas

Other Decks in Technology

Featured

Transcript

!LNPUPIBT ຊڷݴޠॲཧษڧձ ୈ6ষ ςΩετσʔλͷͨΊͷૉੑ

ࣗݾ঺հ !3 ຊڮ࿨و !LNPUPIBT ‣ ౦޻େجૅ෺ཧֶઐ߈1I%

໨࣍ !4 ୈ̎ฤɹࣗવݴޠσʔλͷѻ͍ ‣ ୈ̒ষɹςΩετσʔλͷͨΊͷૉੑ w ࣗવݴޠॲཧʹ͓͚Δ෼ྨ໰୊ͷλΠϓ෼͚ w ࣗવݴޠॲཧ໰୊ͷͨΊͷૉੑ

ୈ̒ষͰѻ͏಺༰ !5 %//͸ૉੑΤϯδχΞϦϯάͷ͢΂ͯΛ୲ͬͯ͸͘Εͳ͍ λεΫ͝ͱʹ֩ͱͳΔૉੑͷू߹Λద੾ʹܾఆ͢Δ ςΩετσʔλ ࣮਺஋ϕΫτϧ ૉੑ ؔ਺

ࣗવݴޠॲཧʹ͓͚Δ෼ྨ໰୊ͷλΠϓ෼͚ !6 ୯ޠ ‣ ੜ͖෺ʁɺͲͷݴޠʁɺҰൠతʁɺࣅͨ୯ޠ͸ʁɺλΠϙʁ ςΩετʢจॻ෼ྨEPDVNFOUDMBTTJpDBUJPOʣ ‣ εύϜʁɺ੓࣏εϙʔπʁɺൽ೑ʁɺߠఆతʁɺ୭ͷஶ࡞ʁɺ޷·ΕΔʁ

ίϥϜɿ୯ޠͱ͸Կ͔ !7 ୯ޠۭനͰ۠੾ΒΕͨจࣈͷܥྻʁ ‣ lEPH zlEPH zlEPHzlEPHz ୯ޠۭന͋Δ͍͸۟ಡ఺Ͱ۠੾ΒΕͨจࣈͷܥྻ

ࣗવݴޠॲཧ໰୊ͷͨΊͷૉੑ !8 ୯ޠ΍จࣈ͸཭ࢄత‎ࢦඪܭ਺ΛΑ͘༻͍Δ w ࢦඪૉੑ JOEJDBUPSGFBUVSF w ͋Δ৚݅Λຬͨͨ͠Β̍ɺ͞΋ͳ͚Ε͹̌

௚઀؍࡯Ͱ͖Δૉੑ !9 Ұͭͷ୯ޠͷૉੑ ςΩετͷૉੑ จ຺தͷ୯ޠͷૉੑ ୯ޠؔ܎ͷૉੑ

Ұͭͷ୯ޠͷૉੑ !10 ϨϯϚԽ MFNNBUJ[JOH ‣ ϨϯϚ MFNNB ࣙॻݟग़͠ޠ

Ұͭͷ୯ޠͷૉੑ !11 ޠኮࢿݯ MFYJDBMSFTPVSDFT ‣ 8PSE/FU<'FMMCBVN > w

Ұͭͷ୯ޠͷૉੑ !12 ޠኮࢿݯ MFYJDBMSFTPVSDFT ‣ 'SBNF/FU<'JMMNPSFFUBM > ‣

Ұͭͷ୯ޠͷૉੑ !13 ޠኮࢿݯ MFYJDBMSFTPVSDFT ‣ 5IF1BSBQISBTF%BUBCBTF 11%#

ࢀߟʣ6OJ%JDCZࠃཱࠃޠݚڀॴ !14 IUUQTVOJEJDOJOKBMBDKQBCPVU@VOJEJD ‏ܗଶૉղੳث.F$BC༻ͷղੳ༻ࣙॻ

௚઀؍࡯Ͱ͖Δૉੑ !15 Ұͭͷ୯ޠͷૉੑ ςΩετͷૉੑ จ຺தͷ୯ޠͷૉੑ ୯ޠؔ܎ͷૉੑ

ςΩετͷૉੑ !16 ୯ޠόοά #BHPG8PSET#08 ‣ ςΩετதͷ୯ޠͷग़ݱ਺ͷώετάϥϜ 5'*%'ͷॏΈ෇͚<.BOOJOHFUBM

௚઀؍࡯Ͱ͖Δૉੑ !17 Ұͭͷ୯ޠͷૉੑ ςΩετͷૉੑ จ຺தͷ୯ޠͷૉੑ ୯ޠؔ܎ͷૉੑ

จ຺தͷ୯ޠͷૉੑ !18 ૭ XJOEPX ‣ ͋Δ୯ޠͷ௚ۙͷจ຺ʹয఺ w lର৅ͷ୯ޠͷपΓʹ୯ޠ9͕ݱΕͨz͜ͱ͕ૉੑ

௚઀؍࡯Ͱ͖Δૉੑ !19 Ұͭͷ୯ޠͷૉੑ ςΩετͷૉੑ จ຺தͷ୯ޠͷૉੑ ୯ޠؔ܎ͷૉੑ

ਪଌ͞ΕΔݴޠֶతಛ௃ !20 ౷ޠ TZOUBY ‣ ࣗવݴޠʹ͓͚Δจͷߏ଄Λࢧ഑͢Δنଇ UIFCPZXJUIUIFCMBDLTIJSUPQFOFEUIFEPPSXJUIBLFZ

ਪଌ͞ΕΔݴޠֶతಛ௃ !21 '&"563&4 '03 5&956"- %"5"

ਪଌ͞ΕΔݴޠֶతಛ௃ !22 ඼ࢺ 1BSUPG4QFFDIUBH104 ‣ ໊ࢺɺಈࢺɺܗ༰ࢺɺݶఆࢺ ౷ޠత໾ׂ

ݴޠֶత֓೦͸ඞཁ͔ !23 ݴޠֶతಛ௃͸ඞཁͳ͘ɺ//͸தؒදݱΛࣗ෼Ͱֶश͢Δ ʢਂ૚ֶऀࢧ࣋ऀͷҰ෦ʣ े෼ͳྔͷσʔλͱਖ਼͍͠ํ޲෇͚͕͋Ε͹Մೳʢචऀʣ ‣ े෼ͳσʔλ͕ͳ͍৔߹͸໌ࣔతʹҰൠతͳ֓೦Λ//ʹఏڙ͢Δͷ͸ ༗ӹ

֩ͱͳΔૉੑͱ૊Έ߹Θͤૉੑ !24 ֩ͱͳΔૉੑ DPSFGFBUVSFT ૊Έ߹Θͤૉੑ DPNCJOBUJPOGFBUVSFT

OάϥϜૉੑ !25 OάϥϜ OHSBN ‣ /FX:PSLOPUHPPE1BSJT)JMUPOͷΑ͏ͳߏ଄Λ֫ಘͰ͖Δ w PGUIFPOBUIFCPZͳͲͷόΠάϥϜ͸Ұൠత

෼෍࿦తૉੑ !26 ݴޠͷ෼෍Ծઆ EJTUSJCVUJPOBMIZQPUIFTJT <'JSUI )BSSJT> ‣ 'JSUIl:PVTIBMMLOPXBXPSECZUIFDPNQBOZJULFFQTz w

จ຺͔Β୯ޠͷҰൠԽΛֶश !27 ΫϥελϦϯάʹجͮ͘ख๏ DMVTUFSJOHCBTFENFUIPET ‣ ྨࣅͨ͠୯ޠΛಉ͡୯ޠΛಉ͡ΫϥελʹׂΓ౰ͯΔ ‣ Ϋϥελʹॴଐ͢Δ౓߹͍ʹΑͬͯͦΕͧΕͷ୯ޠΛදݱ

·ͱΊ !28 ςΩετσʔλͷૉੑͱͯ͠࢖ΘΕ͍ͯΔ΋ͷΛྻڍ ‣ ෼ྨର৅͸୯ޠɺςΩετɺςΩετରɺจষதͷ୯ޠɺ୯ޠͷؔ܎ ‣ ௚઀؍࡯Ͱ͖Δૉੑ w ϨϯϚɺεςϛϯάɺޠኮࢿݯɺ5'*%'ɺ8JOEPX

第６章 テキストデータのための素性

第６章 テキストデータのための素性

More Decks by kmotohas

Other Decks in Technology

Featured

Transcript

第６章　テキストデータのための素性

第６章　テキストデータのための素性