ػցֶशͱֶशऀίʔύεʹجͮ͘ ࣗಈ࠾఺γεςϜͷվྑ খྛ༤Ұ࿠   ʢ౦༸େֶʣ   2016೥3݄19೔ɺૣҴాେֶ 1

໨࣍ •  ͸͡Ίʹ   •  ࣗಈධՁγεςϜখ࢙   •  ਓؒͱػց   •  ֶशऀίʔύεݚڀ   •  ػցֶशͷ׆༻   •  Case  study:  εϐʔΩϯάͷࣗಈධՁ   •  ͓ΘΓʹ   2

͸͡Ίʹ •  ݴޠςετ   –  ҰൠతʹɺਓؒͷධՁऀֶ͕शऀͷύϑΥʔϚϯεΛ ૯߹తʹධՁ   ɹɹɹɹɹˣ   –  ༷ʑͳ໰୊   •  halo  effects   •  sequence  effects   •  central  tendency  effects            etc.   ɹɹɹɹɹˣ   –  ਓؒͷධՁ͸ɺͱ͖ͱͯ͠όΠΞεͷӨڹΛڧ͘ड͚ Δ(e.g.,  Bejar,  Williamson,  and  Mislevy,  2006)   3

•  ͞ΒͳΔ໰୊   –  ख़ୡͨ͠ධՁऀΛҭ੒͢ΔͨΊʹ͸ɺലେͳ͓ۚͱ ͕͔͔࣌ؒΔ   –  ख़ୡͨ͠ධՁऀͰ͋ͬͯ΋ɺධՁऀؒ৴པੑ͕ߴ͍ͱ ͸ݶΒͳ͍(e.g.,  Page,  2003)   –  ӳޠڭҭͷधཁ͕ߴ·ΔʹͭΕͯɺֶशऀ͕૿͑Δ   ɹetc.  etc.   ɹɹɹɹɹˣ   •  ࣗಈධՁͷՄೳੑ   –  ॲཧ଎౓ͷඈ༂తͳ޲্   –  Ұఆͷ৴པੑΛ୲อ   4

•  େֶೖࢼͱࣗಈධՁ   –  ؖࠃͰ͸ɺKICE  (Korea  InsItute  for  Curriculum  and   EvaluaIon)  ʹΑΔେن໛ͳࣗಈ࠾఺γεςϜͷݚڀ (e.g.,  Shin,  Min,  Park,  Jung,  Joo,  and  Kim,  2013)   –  ೔ຊͰ͸ɺMEXT  (Ministry  of  EducaIon,  Culture,  Sports,   Science  and  Technology)  ͕2020೥Ҏ߱ͷେֶೖࢼʹࣗ ಈ࠾఺ͷಋೖΛݕ౼            etc.     5

ࣗಈධՁγεςϜখ࢙ •  1960೥୅Ҏ߱ɺϥΠςΟϯάͷධՁΛ໨తͱ͢ ΔγεςϜͷ։ൃ(e.g.,  Shermis  and  Burstein,   2003;  ੴԬ,  2009)   –  PEG  (Page,  1994)   –  e-­‐rater  (Burstein,  Kukich,  Wolff,  Lu,  Chodorow,  Braden-­‐ Harder,  and  Harris,  1998)   –  IEA  (Landauer,  Laham,  and  Foltz,  2000)   –  Betsy  (Rudener  and  Liang,  2002)   –  IntelliMetric  (Elliot,  2003)   –  JESS    (ੴԬ,  2003)   6

•  ۙ೥ɺίϛϡχέʔγϣϯೳྗҭ੒ͷॏཁੑʹ ΑΓɺεϐʔΩϯάͷධՁγεςϜʹ΋௅ઓ   –  Versant  (Downey,  Farhady,  Present-­‐Thomas,  Suzuki,  and   Van  Moere,  2008)   –  SpeechRater  (Zechner,  Higgins,  and  Williamson,  2009)   ɹɹɹɹɹˣ   –  Ի੠ೝࣝͷਫ਼౓͕ϘτϧωοΫ   •  λεΫʹ΋ΑΔ͕ɺֶशऀͷࣗવൃ࿩ͷ৔߹͸50%ఔ౓ (Zechner,  Higgins,  and  Williamson,  2009)   •  ࠷৽ͷݚڀͰ͸ɺ70%ఔ౓·Ͱ޲্(Bhat  and  Yoon,  2015)   7

ਓؒͱػց •  ݴޠςετͷ෼໺Ͱ͸ɺਓؒͷධՁͱػցͷධ Ձ͸ରরతͳ΋ͷͰ͋Δͱ৴͡ΒΕ͖ͯͨ (Williamson,  2013)   ɹɹɹɹɹˣ   •  ਓؒ   –  ߴ͍ଥ౰ੑ   –  ௿͍৴པੑ   •  ػց   –  ߴ͍৴པੑ   –  ௿͍ଥ౰ੑ   8 ৴པੑͱଥ౰ੑͷؒʹ͸ɺ   ৗʹ͋Δఔ౓ͷߝҾ͖ͷ   ؔ܎͕͋Δ(Hughes,  2003)  

•  ࣗಈධՁγεςϜʢओʹɺϥΠςΟϯάʣʹର ͢Δ൷൑   –  Can  systems  be  gamed?   –  Does  the  use  of  AEE  (automated  essay  evalua8on)  foster   limited  signaling  effects  by  encouraging  a>en8on  only  to   formal  aspects  of  wri8ng,  excluding  richer  aspects  of  the   wri8ng  construct?   –  Does  AEE  subvert  the  wri8ng  act  fundamentally  by   depriving  the  student  of  a  true  “audience”?            etc.  (quoted  by  Shermis,  et  al.,  2013)   –  ࣗಈ࠾఺͸ɺbrute-­‐empirical  approachͳͷ͔ʁ   9

•  ࣗಈධՁγεςϜͷ໰୊   –  ਓؒͷߏ੒֓೦ͷશͯΛΧόʔͰ͖͍ͯͳ͍   ɹɹɹɹɹˣ   •  ਓؒͷධՁऀͷ໰୊   –  ࣗ෼ͷߏ੒֓೦ͷશͯΛݴޠԽ͢Δ͜ͱ͕Ͱ͖ͳ͍   •  ධՁ߲໨ͷ໢ཏతͳϦετʢcan-­‐doϦετͰ͸ෆे෼ʣ   •  ݸʑͷධՁ߲໨͝ͱͷॏΈ͚ͮʢશͯͷධՁ߲໨͕ಉఔ౓ͷ ॏཁੑΛ͍࣋ͬͯΔ༁Ͱ͸ͳ͍ʣ   ɹɹɹɹɹˣ   •  ࣮ࡍͷσʔλ͔Βߏ੒֓೦Λਪఆ   –  େྔͷֶशऀσʔλʢֶशऀίʔύεʣ   –  ਪఆͷͨΊͷ౷ܭख๏ʢػցֶशʣ   10

ֶशऀίʔύεݚڀ •  ൃୡࢦඪͷಛఆ   –  ࣮ࡍͷύϑΥʔϚϯεɾσʔλʹج͖ͮɺֶशऀͷश ख़౓൑ఆʹ༗ޮͳʢʹҟͳΔशख़ஈ֊ͷֶशऀΛಛ ௃͚ͮΔʣݴޠ߲໨Λ୳ࡧ   –  Meunier  and  Liere  (2013)   •  Longitudinal  Database  of  Learner  English  (Longdale)  Λ༻͍ͯɺӳ ޠֶशऀͷςϯεɾΞεϖΫτͷशಘաఔΛهड़   –  Thewissen  (2013)   •  InternaIonal  Corpus  of  Learner  English  (ICLE)  Λ༻͍ͯɺӳޠֶ शऀͷޡΓύλʔϯΛهड़    etc.  etc.   11

•  CEFRͱج४ಛੑ   –  Hawkins and Filipović (2012)   •  Cambridge  Learner  Corpus  (CLC)  Λ༻͍ͯɺݴޠධՁʹԠ༻Ͱ ͖ͦ͏ͳֶशऀͷਖ਼༻(posiIve  features)  ͱޡ༻(negaIve   features)  Λ໢ཏతʹهड़   ɹɹɹɹɹˣ   •  ֶशऀίʔύεݚڀͷ໰୊఺   –  ͦΕͧΕͷशख़ஈ֊ʹ͋ΔֶशऀσʔλΛ1ͭͷ ʮέʔεʯͱ͍ͯ͠ΔͨΊʹɺݸਓࠩͷӨڹ͕ߟྀ͞ Ε͍ͯͳ͍   –  ࣮ࡍͷݴޠςετʢ͋Δ͍͸ɺܭࢉػʹΑΔशख़౓ਪ ఆ࣮ݧʣʹ͓͚Δ༗༻ੑͷݕূ͕ෆे෼   12

ػցֶशͷ׆༻ •  ಛ௃நग़   –  ֶशऀͷύϑΥʔϚϯεɾσʔλʹજΉύλʔϯΛൃ ݟʢൃୡࢦඪͷಛఆʣ   –  ݸʑͷൃୡࢦඪͷॏཁ౓Λਪఆʢ͋·ΓॏཁͰͳ͍ ͱࢥΘΕΔࢦඪ͸࢖Θͳ͍ʣ   •  ࣗಈ࠾఺   –  σʔλ͔Βൃݟͨ͠ύλʔϯʹج͖ͮɺֶशऀͷशख़ ౓Λ൑ఆ   •  ࣗಈϑΟʔυόοΫ   –  ݸʑͷֶशऀʹ޲͚ͯ࠷దԽ͞ΕͨϑΟʔυόοΫΛ ఏڙ   13

•  ֶशऀίʔύεͱػցֶशΛ༻͍ͯɺߏ੒֓೦ Λਪఆ   –  शख़౓৘ใ͕෇༩͞ΕֶͨशऀίʔύεΛ࢖༻   –  शख़౓൑ఆʹ໾ཱͭʢͱࢥΘΕΔʣݴޠతಛ௃Λ໢ ཏతʹ࢖༻   –  (a)  ධՁऀͷ൑ఆ݁Ռͱ(b)  ࣮ࡍͷֶशऀͷύϑΥʔϚ ϯεɺͷؔ܎Λ౷ܭతʹ෼ੳ   –  शख़౓൑ఆʹ༗ޮͳݴޠతಛ௃Λಛఆ   14

•  ࢒͞Εͨ໰୊   –  ͲͷΑ͏ͳධՁ߲໨Λػցֶशʹجͮ͘ਪఆ࣮ݧʹ༻ ͍Δ͔ʁ   ɹɹɹɹɹˣ   –  ػցֶशʹΑΔม਺બ୒   •  ෼ੳʹ࢖༻͢ΔશͯͷධՁ߲໨ʢม਺ʣͷॏཁ౓Λਪఆ͠ɺ ॏཁ౓ͷ௿͍߲໨Λ෼ੳ͔Βআ֎   •  ͔͠͠ɺ෼ੳऀ͕ʮ෼ੳʹ࢖༻͢ΔશͯͷධՁ߲໨ʯΛఆٛ ͢Δඞཁ͕͋Δ   ɹɹɹɹɹˣ   –  ʢલड़ͷͱ͓Γʣਓؒ͸ߏ੒֓೦ͷʮશͯʯΛݴޠ Խ͢Δ͜ͱ͕Ͱ͖ͳ͍ 15

•  ࣗಈධՁʹ༻͍Δ߲໨ͷબఆ   –  ֶशऀίʔύεʹجͮ͘ݴޠशಘݚڀͷ஌ݟΛ׆༻͢ ΂͖(Higgins,  Ramineni,  and  Zechner,  2015)   –  ୈೋݴޠशಘݚڀͷཧ࿦త࿮૊ΈΛ׆༻   ʢ͔͠͠ɺݱঢ়Ͱ͸ɺ·ͩ·ͩ஌ݟ͕଍Γͳ͍ʣ   ɹɹɹɹɹˣ   –  Aeali  (2013)  ͷఏҊ͢Δݴޠ߲໨   •  Biber  (1988)  ͷଟ࣍ݩ෼ੳʹ༻͍ΒΕ͍ͯΔݴޠ߲໨Ϧετͷ ׆༻   •  ͦΕʹՃ͑ͯɺྲྀெ͞ɺޠኮతɾ౷ޠతෳࡶ͞ɺਖ਼֬͞ͳͲ ʹؔ͢Δࢦඪͷ׆༻   16

$BTFTUVEZ 17   Kobayashi,  Y.,  &  Abe,  M.  (2014).     Automated  scoring  of  L2  spoken  English   with  random  forests     Second  Asia  Pacific  Corpus  LinguisIcs  Conference   (at  Hong  Kong  Polytechnic  University)

෼ੳσʔλ •  NICT  JLE  Corpus  (࿨ઘɾ಺ݩɾҪࠤݪ,  2004)   –  ೔ຊਓӳޠֶशऀ1,281ਓͷൃ࿩σʔλʢॻ͖ى͜ ͠ʣ –  Speaking  Standard  Test  (SST)  ʹجͮ͘शख़౓৘ใ –  ख࡞ۀͰ෇༩͞ΕͨޡΓ৘ใ –  hep://alaginrc.nict.go.jp/nict_jle/   –  Standard  Speaking  Test   •  ACTFL  OPIʹ४ڌ͢Δ9ஈ֊ͷशख़౓Λ൑ఆ •  15෼ؒͰߦ͏ෳ਺ͷλεΫ •  ΢ΥʔϜΞοϓɺΠϥετඳࣸɺϩʔϧϓϨΠɺετʔϦʔ ࡞ΓɺϑΥϩʔΞοϓ •  ຊݚڀͰ͸ɺλεΫʹΑΔݴޠ࢖༻ͷࠩ͸ߟྀ͠ͳ͍   18

•  SSTͷධՁํ๏ –  ࢼݧ׭͕ΠϯλϏϡʔͨ͠΋ͷΛςʔϓʹ࿥Ի –  ͦͷςʔϓʹج͖ͮɺ࠷௿2໊ͷධՁ׭͕ಠཱͯ͠ධՁ –  2ਓͷධՁ͕෼͔Εͨ৔߹ɺୈ3ͷධՁ׭͕൑அ •  SSTͷධՁج४ –  ૯߹తλεΫɾػೳʢݴ༿Λ࢖ͬͯԿ͕Ͱ͖Δ͔ʣ –  ςΩετͷܕʢͲΜͳঢ়گͰԿʹ͍ͭͯ࿩͢͜ͱ͕Ͱ͖Δ ͔ʣ –  ࿩୊ɾঢ়گʢͲΜͳߏจ΍ߏ੒Ͱ࿩͢͜ͱ͕Ͱ͖Δ͔ʣ –  ਖ਼֬͞ʢฉ͖खʹͲΕ͚͖ͩͪΜͱ఻͑Δ͜ͱ͕Ͱ͖Δ ͔ʣ –  ৄ͘͠͸ɺ࿨ઘɾ಺ݩɾҪࠤݪ(2004)  Λࢀর 19

•  SSTϨϕϧผͷֶशऀσʔλ 20 Level Par(cipants Tokens 1 3  (0.23%) 428  (0.04%) 2 35  (2.73%) 7,701  (0.81%) 3 222  (17.33%) 95,169  (9.98%) 4 482  (37.63%) 308,177  (32.31%) 5 236  (18.42%) 203,759  (21.36%) 6 130  (10.15%) 130,492  (13.68%) 7 77  (6.01%) 85,309  (8.94%) 8 56  (4.37%) 68,470  (7.18%) 9 40  (3.12%) 54,341  (5.70%) Total 1,281  (100.00%) 953,846  (100.00%)

෼ੳର৅ͱ͢Δݴޠ߲໨ •  Biber  (1988)  ͷݴޠ߲໨ –  ޠኮɺ඼ࢺɺߏจɺஊ࿩ʹؔ͢Δ58߲໨   –  type-­‐token  raIo,  mean  length  of  wordsͷ2߲໨ ʢݩʑ͸ɺ฼ޠ࿩ऀͷॻ͖ݴ༿ͱ࿩͠ݴ༿ͷ෼ੳʹ ࢖༻ʣ   •  ྲྀெ͞ʹؔ͢Δݴޠ߲໨ –  tokens,  type,  mean  length  of  sentenceͷ3߲໨ ɹɹɹɹɹˣ –  ߹ܭͰ63छྨͷݴޠ߲໨ –  ৄࡉ͸ɺখྛɾѨ෦(2013)  Λࢀর 21

෼ੳख๏ •  ϥϯμϜϑΥϨετ(Breiman,  2001)   –  ػցֶशͷख๏ͷ1ͭ   –  ٕज़తʹ͸ɺܾఆ໦ʹجͮ͘Ξϯαϯϒϧֶशͱఆٛ   –  ଟ͘ͷධՁ߲໨Λޮ཰తʹѻ͏͜ͱ͕Մೳ   –  ଞͷςΩετ෼ྨख๏ʹൺ΂ͯɺߴ͍෼ྨਫ਼౓(ۚɾ ଜ্,  2007;  খྛɾాதɾ෌Ӝ,  2011)   –  ݸʑͷධՁ߲໨͕ධՁʹد༩͢Δ౓߹͍ʢม਺ॏཁ ౓ʣΛਪఆ 22

•  ܾఆ໦   –  σʔλΛ෼ྨ͢ΔͨΊͷύλʔϯΛथ໦ߏ଄ͱͯ͠ࢹ ֮Խ   23

•  Ξϯαϯϒϧֶश   –  ෳ਺ͷ෼ྨϞσϧΛ૊Έ߹ΘͤΔ͜ͱͰɺߴ͍෼ྨਫ਼ ౓Λ࣮ݱ   24 Bootstrap   sampling

൑ఆ݁Ռ •  ༧ଌਫ਼౓͸ɺ60.11%  (baseline:  37.63%)   •  SSTͷධՁ׭ʹΑΔධՁͱͷ૬ؔ܎਺͸ɺ0.85   25 L1 L2 L3 L4 L5 L6 L7 L8 L9 accuracy L1 0 3 0 0 0 0 0 0 0 0.00% L2 0 21 14 0 0 0 0 0 0 60.00% L3 0 2 145 74 1 0 0 0 0 65.32% L4 0 0 33 407 40 2 0 0 0 84.44% L5 0 0 0 102 119 14 1 0 0 50.42% L6 0 0 0 19 61 39 5 5 1 30.00% L7 0 0 0 1 21 24 21 9 1 27.27% L8 0 0 0 0 12 14 22 4 4 7.14% L9 0 0 0 0 1 5 14 6 14 35.00%

•  SSTධՁ׭ͷධՁͱີ઀ͳؔ܎ʹ͋Δݴޠ߲໨   26 DPOUSBDUJPOT DBVTBUJWFBEWFSCJBMTVCPSEJOBUPST BOBMZUJDOFHBUJPOT QPTTJCJMJUZNPEBMT JOGJOJUJWFT FNQIBUJDT QBTUUFOTF PUIFSUPUBMOPVOT UZQFT UPLFOT WBSJBCMFJNQPSUBODF

–  ਓؒʹΑΔධՁʹ΋ɺ૯ޠ਺΍ҟޠ਺ͱ͍ͬͨྔతͳ ߲໨͕େ͖͘د༩͍ͯ͠Δʢ൴Β͕“brutal-­‐empirical   approach”ͱඇ೉͢Δࣗಈ࠾఺γεςϜͱಉ༷ʹʣ   –  ͜Ε͸௚઀ͷ૬ؔʁͦΕͱ΋ٙࣅ૬ؔʁ   27 Tokens Types Proficiency Tokens Types Proficiency ? ? ? ???

•  ධՁ߲໨ͷgold  standard͸ଘࡏ͠ͳ͍   –  ධՁର৅ͱ͢Δσʔλͷੑ࣭ʢֶशऀͷशख़౓ɺλ εΫͳͲʣʹΑͬͯɺ༗ޮͳධՁ߲໨͸ҟͳΔ   –  ͨͱ͑͹ɺॳڃऀͷϥΠςΟϯάΛ෼ੳ͢Δ৔߹͸ɺ จ๏߲໨ͷ࢖༻ස౓΍ޡΓͷස౓͕ʢ૬ରతʹʣ༗ ޮ   –  Ұํɺ্ڃऀͷϥΠςΟϯάΛ෼ੳ͢Δ৔߹͸ɺςΩ ετͷஊ࿩ߏ଄΍ҙຯΛߟྀʹೖΕΔඞཁ(খྛɾా த,  2014)   28

͓ΘΓʹ •  ΑΓΑ͍ධՁ߲໨ͷݕ౼   –  ೔ຊਓֶशऀʹಛԽͨ͠ݴޠ߲໨(e.g.,  ףࢺ)   –  ൑ఆਫ਼౓ͷඈ༂త޲্ʹͭͳ͕Δͱ͸ݶΒͳ͍͕ɺ ڭҭతʹϑΟʔυόοΫ͕ٻΊΒΕΔݴޠ߲໨  (e.g.,   จ๏తޡΓɺֶߍจ๏ͷ࢖༻ස౓)   •  ਓؒͱػցͷ໾ׂ෼୲   –  ݱঢ়ͰػցʹՄೳͳ಺༰໘ͷ൑ఆ͸͔ͳΓݶΒΕΔ (e.g.,  ϕΫτϧۭؒϞσϧɺτϐοΫɾϞσϦϯά)   –  ਓ͕ؒओʹ಺༰໘ɺػց͕ओʹܗࣜ໘Λ൑ఆ͢Δͷ ͕ݱ࣮త(e.g.,  TOEFL  wriIng)   29

•  ػցֶशͷ͞ΒͳΔ׆༻   –  ࣗಈධՁͷޡ൑ఆ཰ΛධՁͷଌఆޡࠩͱΈͳ͢͜ͱ Ͱɺशख़౓ई౓ͷଥ౰ੑΛݕূ(খྛɾాதɾѨ෦,   2014)   –  طଘͷࣗಈධՁγεςϜʹΑΔධՁͷଥ౰ੑͷݕূʹ ΋Ԡ༻Մೳ(খྛɾؙۚ,  2012a;  খྛɾؙۚ,  2012b)   30 8100000 8200000 8300000 8400000 8500000 8600000 8700000 0.6 0.7 0.8 0.9 1.0 AIC Accuracy 1-189 190-224 225-255 Accuracy Proportion 0.0 0.2 0.4 0.6 0.8 1.0 -0.60 0.60+ 0.65+ 0.70+ 0.75+ 0.80+ 0.85+ 0.90+ 0.95+ 1|2 2|3 3|4 4|5 5|6 6|7 7|8 8|9

ओͳࢀߟจݙ •  Kobayashi,  Y.,  &  Abe,  M.  (2014,  March).  Automated  scoring  of  L2  spoken   English  with  random  forests.  Paper  given  at  the  Second  Asia  Pacific  Corpus   LinguisIcs  Conference.   •  খྛ༤Ұ࿠ɾؙۚහ޾(2012a).  ʮύλʔϯೝࣝΛ༻͍ͨ՝୊ӳ࡞จͷ ࣗಈධՁͷࢼΈʯʰిࢠ৘ใ௨৴ֶձٕज़ݚڀใࠂʱ  112(103),   37-­‐42.   •  খྛ༤Ұ࿠ɾؙۚහ޾(2012b).  ʮCoh-­‐MetrixͱύλʔϯೝࣝΛ༻͍ͨ ՝୊ӳ࡞จͷࣗಈධՁʯʰਓจՊֶͱίϯϐϡʔλγϯϙδ΢Ϝ࿦ จू―ͭͳ͕ΔσδλϧΞʔΧΠϒʱ(pp.  259-­‐266).  ౦ژ:  ৘ใॲཧֶ ձ.   •  খྛ༤Ұ࿠ɾాதল࡞(2014).  ʮϝλஊ࿩ඪࣝΛૉੑͱ͢ΔϥϯμϜ ϑΥϨετʹΑΔӳޠՊֶ࿦จͷ࣭൑ఆʯ؛ߐ৴հɾాാஐ࢘(ฤ) ʰςΩετϚΠχϯάʹΑΔݴޠݚڀʱ(pp.  137-­‐151).  ౦ژ:  ͻͭ͡ॻ ๪.   •  খྛ༤Ұ࿠ɾాதল࡞ɾѨ෦ਅཧࢠ(2014).  ʮ৘ใྔج४ʹجͮ͘श ख़౓ई౓ͷ࠶ݕ౼ʯʰՊֶٕज़ίʔύεͷಛ௃දݱ෼ੳͱLRTʹجͮ ͍ͨCATͷ࡞੒ʱ(pp.  29-­‐43).  ౦ژ:  ౷ܭ਺ཧݚڀॴ.   31