Meta-Augmented Human: Augmenting the Capability of Augmenting the Capability / 2022-12-07

.FUB"VHNFOUFE)VNBO   "VHNFOUJOHUIF$BQBCJMJUZPG "VHNFOUJOHUIF$BQBCJMJUZ +VO1SPG%S4IPZB*TIJNBSV 56,BJTFSTMBVUFSO%',* %BHTUVIM4FNJOBS$PHOJUJWF"VHNFOUBUJPO

.FUB"VHNFOUFE)VNBO Meta-skill augmentation Skill augmentation • .PTUBVHNFOUJOHIVNBOUFDIOPMPHJFTIFMQBVTFSXIJMFUIFZBSFXPSO • *GXFBVHNFOUUIFTLJMMPGTLJMMBDRVJTJUJPO JF
NFUBTLJMM UIFBDRVJSFETLJMMIFMQTUIF VTFSJOBMPOHUFSN 4LJMMBDRVJTJUJPOMFBSOJOH TQPSUTUSBJOJOH NVTJDQSBDUJDF FUD • 5IJTUBMLJOUSPEVDFTTPNFTUVEJFTBQQMJDBUJPOTUIBUSFDPHOJ[FJOUFSOBMTUBUFTXIJMF MFBSOJOHCZNBDIJOFMFBOSJOHBOEHJWFBQQSPQSJBUFGFFECBDLCBTFEPOUIFDPOUFYU

Physical Activity walking, standing, cycling, sleeping, ... Iceberg Model of
Activity Recognition

Physical Activity walking, standing, cycling, sleeping, ... Cognitive Activity reading,
writing, memorizing, talking, ... Iceberg Model of Activity Recognition

Physical Activity walking, standing, cycling, sleeping, ... Cognitive Activity reading,
writing, memorizing, talking, ... Iceberg Model of Activity Recognition Cognitive and   A ff ective State interest, workload, con fi dence, fatigue, ...

"VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF -FDUVSFBOEUFYUCPPL 2VFTUJPOT UFTU %BJMZIBCJUTBOEDVSJPTJUZ 8IJDIRVFTUJPOTIPVMECFSFWJFXFE   DBSFGVMMZ DPOTJEFSJOHQSJPSJUZMFBLBHF
$BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT   BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ

"VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF -FDUVSFBOEUFYUCPPL 2VFTUJPOT UFTU 8IJDIRVFTUJPOTIPVMECFSFWJFXFE   DBSFGVMMZ DPOTJEFSJOHQSJPSJUZMFBLBHF $BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT
  BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ %BJMZIBCJUTBOEDVSJPTJUZ

Sleeping Walking Attending a lecture Reading a book Watching a
movie Eating 6OEFSTUBOEJOH%BJMZ"DUJWJUJFT GSPN1IZTJDBMUP$PHOJUJWF 8FBSBCMFEFWJDF<#VMMJOH FUBM> 4NBSUFZFXFBS $PNNFSDJBMMZBWBJMBCMFBDUJWJUZUSBDLFST 4UVEFOUTEBJMZBDUJWJUJFT A. Bulling, et al. Eye movement analysis for activity recognition using electrooculography. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33.4 (2010): 741-753. J. Steil & A. Bulling. Discovery of everyday human activities from long-term visual behaviour using topic models. Proc. UbiComp 2015, pp. 75–85, 2015. S. Ishimaru, et al. Smart Eyewear for Interaction and Activity Recognition. CHI 2015 Extended Abstracts, pp. 307–310, 2015. ɾɾɾ

+*/4.&.&

# 3 - &MFDUSPEFT &MFDUSPPDVMPHSBQIZ ɾ7FSUJDBM# - 3 <N7> ɾ)PSJ[POUBM-3<N7>

+*/4.&.&&MFDUSPPDVMPHSBQIZ &0( (MBTTFT Electrooculography # 3 - Electrodes Horizontal axis:
L - R [mV] Vertical axis: B - (L + R)/2 [mV]

8PSEPNFUFS

8PSEPNFUFS&TUJNBUJOHUIFOVNCFSPGSFBEXPSET 2VBOUJGZJOHSFBEJOHTBNFBTXBMLJOH *OUFSWFOUJPO read! Nice condition. Read one paragraph more!
There are interesting articles for you. S. Ishimaru, et al. The Wordometer 2.0: Estimating the Number of Words You Read in Real Life using Commercial EOG Glasses. Proc. UbiComp 2016 Adjunct, pp. 293–296, 2016. S. Ishimaru, et al. Reading Interventions: Tracking Reading State and Designing Interventions. Proc. UbiComp 2016 Adjunct, pp. 1759–1764, 2016.

&0( )FBENPUJPOCBTFE"DUJWJUZ$MBTTJ fi DBUJPO 69% 23% 3% 10% 18%
66% 1% 0% 0% 0% 77% 14% 12% 9% 18% 74% Predicted class reading typing eating talking Actual class talking eating typing reading 78% 36% 0% 12% 8% 19% 6% 37% 0% 1% 93% 7% 12% 42% 0% 41% Predicted class reading typing eating talking Actual class talking eating typing reading EOG (Acc.: 54%)  majority vote -> 72 % EOG and head (Acc.: 72%)  majority vote -> 100 % • 5IFSFXFSFGSFRVFOUDIBSBDUFSJTUJD QBUUFSOTPO&0(XIJMFSFBEJOHPS FBUJOH&0(CBTFEBQQSPBDIDPVME DMBTTJGZUIFN 4UVEZ&0(#BTFE"DUJWJUZ$MBTTJ fi DBUJPO S. Ishimaru, et al. “Smarter Eyewear: Using Commercial EOG Glasses for Activity Recognition”. Proc. UbiComp '14 Adjunct, pp. 239–242, 2014.

&0( )FBENPUJPOCBTFE"DUJWJUZ$MBTTJ fi DBUJPO 69% 23% 3% 10% 18%
66% 1% 0% 0% 0% 77% 14% 12% 9% 18% 74% Predicted class reading typing eating talking Actual class talking eating typing reading 78% 36% 0% 12% 8% 19% 6% 37% 0% 1% 93% 7% 12% 42% 0% 41% Predicted class reading typing eating talking Actual class talking eating typing reading EOG (Acc.: 54%)  majority vote -> 72 % EOG and head (Acc.: 72%)  majority vote -> 100 % • 5IFSFXFSFGSFRVFOUDIBSBDUFSJTUJD QBUUFSOTPO&0(XIJMFSFBEJOHPS FBUJOH&0(CBTFEBQQSPBDIDPVME DMBTTJGZUIFN • 5ZQJOHBOEUBMLJOHXFSFPGUFONJT DMBTTJ fi FEJOUIF&0(CBTFEBQQSPBDI TJNJMBSFZFNPWFQBUUFSOT #VUUIF DPNCJOBUJPOXJUIIFBENPUJPOTPMWFE UIJTQSPCMFN 4UVEZ&0(#BTFE"DUJWJUZ$MBTTJ fi DBUJPO S. Ishimaru, et al. “Smarter Eyewear: Using Commercial EOG Glasses for Activity Recognition”. Proc. UbiComp '14 Adjunct, pp. 239–242, 2014.

• 'PVSGFBUVSFT UIFOVNCFSTPGGPSXBSECBDLXBSETBDDBEFT NFBO&0(XIJMFUIFTBDDBEFT DPVMEFTUJNBUFUIFOVNCFSPGSFBEXPSET • 8PSEDPVOUFTUJNBUJPOFSSPSJOVTFSEFQFOEFOU JOVTFSJOEFQFOEFOUBQQSPBDI FWBMVBUFEJOBOFYQFSJNFOUXJUIQBSUJDJQBOUTSFBEEPDVNFOUT
8PSEPNFUFS4ZTUFN0WFSWJFX Recording EOG signals Detecting forward/backward-saccades Support Vector Regression 4UVEZ&0(#BTFE8PSE$PVOU&TUJNBUJPO S. Ishimaru, et al. "The wordometer 2.0: estimating the number of words you read in real life using commercial EOG glasses." UbiComp '16 Adjunct, pp. 293–296, 2016.

"VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF 2VFTUJPOT UFTU %BJMZIBCJUTBOEDVSJPTJUZ 8IJDIRVFTUJPOTIPVMECFSFWJFXFE   DBSFGVMMZ DPOTJEFSJOHQSJPSJUZMFBLBHF $BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT
  BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ -FDUVSFBOEUFYUCPPL

4FOTPS#BTFE$PHOJUJWF4UBUF3FDPHOJUJPO *OUFSFTU $PNQSFIFOTJPO $PHOJUJWFMPBE &ZFUSBDLFS 5IFSNBMDBNFSB 4BDDBEFT 'BDFUFNQFSBUVSF 'JYBUJPOT S.
Ishimaru, et al. Cognitive State Measurement on Learning Materials by Utilizing Eye Tracker and Thermal Camera. Proc. ICDAR HDI 2017, pp. 32–36, 2017. S. Ishimaru, et al. Augmented Learning on Anticipating Textbooks with Eye Tracking. Positive Learning in the Age of Information (PLATO), pp. 387–398, 2018.

%FNPBU")T

)ZQFS.JOE

)ZQFS.JOE4UVEJP

&YQFSJNFOUBM%FTJHO Introduction De nitions Applications Ap (a) text 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT

"UUFOUJPO"OBMZTJT intro. def. appl. intro. def. appl. intro. def.
appl. intro. def. appl. intro. def. appl. intro. def. appl. Expert Intermediate Novice reading solving reading solving reading solving B SFBEJOH C TPMWJOH F SFBEJOH G TPMWJOH /PWJDF D SFBEJOH E TPMWJOH *OUFSNFEJBUF &YQFSU Introduction De nitions Applications (a) text $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT

$PNQSFIFOTJPO3FDPHOJUJPO 'FBUVSFSFQSFTFOUBUJPOPGBMMQBSUJDJQBOUTEBUB   JOTVCTFRVFODFT NJO CBTFEBQQSPBDI • BDDVSBDZGPSDMBTTJGZJOH  
FYQFSUJTFTJOUPUISFFDMBTTFT   VTFSJOEFQFOEFOUUSBJOJOH $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT S. Ishimaru, et al. "Augmented learning on anticipating textbooks with eye tracking." Positive Learning in the Age of Information. Springer VS, Wiesbaden, 2018. 387-398.

FYQFSUJTFTJOUPUISFFDMBTTFT   VTFSJOEFQFOEFOUUSBJOJOH • /PWJDFTUVEFOUTSFBEB   UFYUCPPLTMPXMZXJUITNBMMTUFQT $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT S. Ishimaru, et al. "Augmented learning on anticipating textbooks with eye tracking." Positive Learning in the Age of Information. Springer VS, Wiesbaden, 2018. 387-398.

FYQFSUJTFTJOUPUISFFDMBTTFT   VTFSJOEFQFOEFOUUSBJOJOH • /PWJDFTUVEFOUTSFBEB   UFYUCPPLTMPXMZXJUITNBMMTUFQT • &YQFSUTUVEFOUTTPNFUJNFTTLJQ   UIFJSFZFTPOUIFUFYUBOEGPDVTPO   UIFDPOUFOUUIFZBSFJOUFSFTUFEJO $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT S. Ishimaru, et al. "Augmented learning on anticipating textbooks with eye tracking." Positive Learning in the Age of Information. Springer VS, Wiesbaden, 2018. 387-398.

&YQFSJNFOUBM%FTJHO • VOJWFSTJUZTUVEFOUTSFBEOFXTQBQFSBSUJDMFT ʶXPSET • %FWJDF4.*3&%OTDJFOUJ fi D)[FZFUSBDLFS
• $BMJCSBUFUIFEFWJDFˠ3FTU TFD ˠ3FBEBEPDVNFOUˠ"OTXFSTVSWFZT 4UVEZ(B[F#BTFE*OUFSFTU%FUFDUJPO S. Jacob, et al. "Gaze-based interest detection on newspaper articles." Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction. 2018.

*OUFSFTU3FDPHOJUJPO0WFSWJFXBOE3FTVMU 'PSXBSETBDDBEF 3FHSFTTJPO 'FBUVSFDBMDVMBUJPO 47. 4UVEZ(B[F#BTFE*OUFSFTU%FUFDUJPO + ongoing work
S. Jacob, et al. "Gaze-based interest detection on newspaper articles." Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction. 2018. NFBO NFBO 62% 16% 2% 1% 25% 66% 7% 0% 6% 15% 83% 6% 7% 3% 8% 93% Predicted class 1 2 3 4 Actual class 4 3 2 1 *OVTFSEFQFOEFOUFWBMVBUJPO MFBWFPOFEPDVNFOUPVUDSPTTWBMJEBUJPOJOBQBSUJDJQBOU   47.DMBTTJ fi FEJOUFSFTUTJOUPMFWFMTXJUIBDDVSBDZ $//-45.DMBTTJ fi FEXJUIBDD $//-45.

"VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF -FDUVSFBOEUFYUCPPL %BJMZIBCJUTBOEDVSJPTJUZ 8IJDIRVFTUJPOTIPVMECFSFWJFXFE   DBSFGVMMZ DPOTJEFSJOHQSJPSJUZMFBLBHF $BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT  
BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ 2VFTUJPOT UFTU

$P"-"$PO fi EFODF"XBSF-FBSOJOH"TTJTUBOU $PSSFDU *ODPSSFDU $PO fi EFOU 6ODPO fi
EFOU .VMUJQMF$IPJDF2VFTUJPO

(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO %FDJTJPOXJUIDPO fi EFODF %FDJTJPOXJUIPVUDPO fi EFODF S.
Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts, p. 8, 2020. S. Ishimaru, et al. Confidence-Aware Learning Assistant. In arXiv preprint arXiv:2102.07312, 2021

4ZTUFN0WFSWJFX 1SFQSPDFTTJOH   fi YBUJPOTBDDBEFEFUFDUJPO 'FBUVSFDBMDVMBUJPO $MBTTJ fi DBUJPO
8FVUJMJ[FUIFGPMMPXJOH fi WFF ff FDUJWFGFBUVSFTTFMFDUFECZ'PSXBSETUFQXJTFGFBUVSFTFMFDUJPO   GTVNPG fi YBUJPOEVSBUJPOTPO$IPJDFT GWBSJBODFPGYDPPSEJOBUFPG fi YBUJPOT   GUIFOVNCFSPGTBDDBEFTCFUXFFODIPJDFBSFBT GTVNPGTBDDBEFEVSBUJPOT GSFBEJOHUJNF • #JOBSZDMBTTJ fi DBUJPO • 4VQQPSU7FDUPS .BDIJOF 47. • 3#'LFSOFM • $ Ѝ 4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO S. Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts (Asian CHI Symposium), p. 8, 2020.

3FTFBSDI)ZQPUIFTFT 4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO • 3)$PSSFDUBOTXFSTXJUIPVUDPO fi EFODF WBHVFLOPXMFEHF
UFOEUP CFJODPSSFDUMZBOTXFSFEXIFOBTLFEBHBJODPNQBSFEUPDPSSFDU BOTXFSXJUIDPO fi EFODF • 3)*ODPSSFDUBOTXFSTXJUIDPO fi EFODF NJTVOEFSTUBOEJOH UFOEUP CFNJTUBLFOBHBJODPNQBSFEUPJODPSSFDUBOTXFSTXJUIPVUDPO fi EFODF • 3)&TUJNBUJOHTFMGDPO fi EFODFGSPNMFBSOJOHCFIBWJPSTBOEHJWJOH GFFECBDL FH BEEJOHRVFTUJPOTUPBSFWJFXMJTU IJHIMJHIUJOHUIFN XIJMFSFWJFXJOH BWPJETUIFTDFOBSJPTPVUMJOFEJO3)BOE3) S. Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts (Asian CHI Symposium), p. 8, 2020.

&YQFSJNFOUBM%FTJHO 4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO • BOE B )PXPGUFOBOTXFSTXJUIPVUDPO
fi EFODFBSFGPSHPUUFO • B BOE C )PX$P"-"JTF ff FDUJWFJOBTVDIDPOEJUJPO • BOE B )PXPGUFOBOTXFSTXJUIDPO fi EFODFBSFJODPSSFDUMZBOTXFSFEBHBJO • B BOE C )PX$P"-"JTF ff FDUJWFJOBTVDIDPOEJUJPO $PNQBSJTPOPG

&YQFSJNFOUBM3FTVMUT 4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO • 3)$PSSFDUBOTXFSTXJUIPVUDPO fi EFODFUFOEUPCFJODPSSFDUMZ BOTXFSFEXIFOBTLFEBHBJODPNQBSFEUPBOTXFSXJUIDPO
fi EFODF • 4VQQPSUFE B ESPQQFEDPNQBSFEUP Q • 3)*ODPSSFDUBOTXFSTXJUIDPO fi EFODFUFOEUPCFNJTUBLFOBHBJO DPNQBSFEUPJODPSSFDUBOTXFSTXJUIPVUDPO fi EFODF • /PUTVQQPSUFE/PTJHOJ fi DBOUEJ ff FSFODFCFUXFFO BOE B • 3)&TUJNBUJOHTFMGDPO fi EFODFGSPNMFBSOJOHCFIBWJPSTBOEHJWJOH GFFECBDL FH BEEJOHRVFTUJPOTUPBSFWJFXMJTU IJHIMJHIUJOHUIFN XIJMFSFWJFXJOH BWPJETTVDITDFOBSJPT • 4VQQPSUFE$PNQBSJTPOTCFUXFFO B WT C B WT C B C DPSSXDPOG B DPSSXPDPOG C DPSSXPDPOGGFFECBDL B C JODPSSXPDPOG B JODPSXDPOG C JODPSSXDPOGGFFECBDL S. Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts (Asian CHI Symposium), p. 8, 2020.

"VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF 8PSEPNFUFS )ZQFS.JOE $P"-" -FDUVSFBOEUFYUCPPL 2VFTUJPOT UFTU %BJMZIBCJUTBOEDVSJPTJUZ 8IJDIRVFTUJPOTIPVMECFSFWJFXFE  
DBSFGVMMZ DPOTJEFSJOHQSJPSJUZMFBLBHF $BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT   BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ

$POUBDUJTIJNBSV!DTVOJLMEF   IUUQTTIPZBJP !TIPZB .FUB"VHNFOUFE)VNBO   "VHNFOUJOHUIF$BQBCJMJUZPG "VHNFOUJOHUIF$BQBCJMJUZ +VO1SPG%S4IPZB*TIJNBSV 56,BJTFSTMBVUFSO%',*
%BHTUVIM4FNJOBS$PHOJUJWF"VHNFOUBUJPO

Meta-Augmented Human: Augmenting the Capability of Augmenting the Capability / 2022-12-07

Meta-Augmented Human: Augmenting the Capability of Augmenting the Capability / 2022-12-07

More Decks by Shoya Ishimaru

Featured

Transcript