Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Meta-Augmented Human: Augmenting the Capability...

Shoya Ishimaru
December 07, 2022
15

Meta-Augmented Human: Augmenting the Capability of Augmenting the Capability / 2022-12-07

My talk at the Dagstuhl Seminar Cognitive Augmentation on 07.12.2022.

Shoya Ishimaru

December 07, 2022
Tweet

More Decks by Shoya Ishimaru

Transcript

  1. .FUB"VHNFOUFE)VNBO Meta-skill augmentation Skill augmentation • .PTUBVHNFOUJOHIVNBOUFDIOPMPHJFTIFMQBVTFSXIJMFUIFZBSFXPSO • *GXFBVHNFOUUIFTLJMMPGTLJMMBDRVJTJUJPO JF

    NFUBTLJMM UIFBDRVJSFETLJMMIFMQTUIF VTFSJOBMPOHUFSN 4LJMMBDRVJTJUJPOMFBSOJOH TQPSUTUSBJOJOH NVTJDQSBDUJDF FUD  • 5IJTUBMLJOUSPEVDFTTPNFTUVEJFTBQQMJDBUJPOTUIBUSFDPHOJ[FJOUFSOBMTUBUFTXIJMF MFBSOJOHCZNBDIJOFMFBOSJOHBOEHJWFBQQSPQSJBUFGFFECBDLCBTFEPOUIFDPOUFYU
  2. Physical Activity walking, standing, cycling, sleeping, ... Cognitive Activity reading,

    writing, memorizing, talking, ... Iceberg Model of Activity Recognition
  3. Physical Activity walking, standing, cycling, sleeping, ... Cognitive Activity reading,

    writing, memorizing, talking, ... Iceberg Model of Activity Recognition Cognitive and 
 A ff ective State interest, workload, con fi dence, fatigue, ...
  4. Sleeping Walking Attending a lecture Reading a book Watching a

    movie Eating 6OEFSTUBOEJOH%BJMZ"DUJWJUJFT GSPN1IZTJDBMUP$PHOJUJWF 8FBSBCMFEFWJDF<#VMMJOH FUBM> 4NBSUFZFXFBS $PNNFSDJBMMZBWBJMBCMFBDUJWJUZUSBDLFST 4UVEFOUTEBJMZBDUJWJUJFT A. Bulling, et al. Eye movement analysis for activity recognition using electrooculography. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33.4 (2010): 741-753. J. Steil & A. Bulling. Discovery of everyday human activities from long-term visual behaviour using topic models. Proc. UbiComp 2015, pp. 75–85, 2015. S. Ishimaru, et al. Smart Eyewear for Interaction and Activity Recognition. CHI 2015 Extended Abstracts, pp. 307–310, 2015. ɾɾɾ
  5. 8PSEPNFUFS&TUJNBUJOHUIFOVNCFSPGSFBEXPSET 2VBOUJGZJOHSFBEJOHTBNFBTXBMLJOH *OUFSWFOUJPO read! Nice condition. Read one paragraph more!

    There are interesting articles for you. S. Ishimaru, et al. The Wordometer 2.0: Estimating the Number of Words You Read in Real Life using Commercial EOG Glasses. Proc. UbiComp 2016 Adjunct, pp. 293–296, 2016. S. Ishimaru, et al. Reading Interventions: Tracking Reading State and Designing Interventions. Proc. UbiComp 2016 Adjunct, pp. 1759–1764, 2016.
  6. &0( )FBENPUJPOCBTFE"DUJWJUZ$MBTTJ fi DBUJPO  69% 23% 3% 10% 18%

    66% 1% 0% 0% 0% 77% 14% 12% 9% 18% 74% Predicted class reading typing eating talking Actual class talking eating typing reading 78% 36% 0% 12% 8% 19% 6% 37% 0% 1% 93% 7% 12% 42% 0% 41% Predicted class reading typing eating talking Actual class talking eating typing reading EOG (Acc.: 54%)
 majority vote -> 72 % EOG and head (Acc.: 72%)
 majority vote -> 100 % • 5IFSFXFSFGSFRVFOUDIBSBDUFSJTUJD QBUUFSOTPO&0(XIJMFSFBEJOHPS FBUJOH&0(CBTFEBQQSPBDIDPVME DMBTTJGZUIFN 4UVEZ&0(#BTFE"DUJWJUZ$MBTTJ fi DBUJPO S. Ishimaru, et al. “Smarter Eyewear: Using Commercial EOG Glasses for Activity Recognition”. Proc. UbiComp '14 Adjunct, pp. 239–242, 2014.
  7. &0( )FBENPUJPOCBTFE"DUJWJUZ$MBTTJ fi DBUJPO  69% 23% 3% 10% 18%

    66% 1% 0% 0% 0% 77% 14% 12% 9% 18% 74% Predicted class reading typing eating talking Actual class talking eating typing reading 78% 36% 0% 12% 8% 19% 6% 37% 0% 1% 93% 7% 12% 42% 0% 41% Predicted class reading typing eating talking Actual class talking eating typing reading EOG (Acc.: 54%)
 majority vote -> 72 % EOG and head (Acc.: 72%)
 majority vote -> 100 % • 5IFSFXFSFGSFRVFOUDIBSBDUFSJTUJD QBUUFSOTPO&0(XIJMFSFBEJOHPS FBUJOH&0(CBTFEBQQSPBDIDPVME DMBTTJGZUIFN • 5ZQJOHBOEUBMLJOHXFSFPGUFONJT DMBTTJ fi FEJOUIF&0(CBTFEBQQSPBDI TJNJMBSFZFNPWFQBUUFSOT #VUUIF DPNCJOBUJPOXJUIIFBENPUJPOTPMWFE UIJTQSPCMFN 4UVEZ&0(#BTFE"DUJWJUZ$MBTTJ fi DBUJPO S. Ishimaru, et al. “Smarter Eyewear: Using Commercial EOG Glasses for Activity Recognition”. Proc. UbiComp '14 Adjunct, pp. 239–242, 2014.
  8. • 'PVSGFBUVSFT UIFOVNCFSTPGGPSXBSECBDLXBSETBDDBEFT NFBO&0(XIJMFUIFTBDDBEFT  DPVMEFTUJNBUFUIFOVNCFSPGSFBEXPSET • 8PSEDPVOUFTUJNBUJPOFSSPSJOVTFSEFQFOEFOU JOVTFSJOEFQFOEFOUBQQSPBDI FWBMVBUFEJOBOFYQFSJNFOUXJUIQBSUJDJQBOUTSFBEEPDVNFOUT

     8PSEPNFUFS4ZTUFN0WFSWJFX  Recording EOG signals Detecting forward/backward-saccades Support Vector Regression 4UVEZ&0(#BTFE8PSE$PVOU&TUJNBUJPO S. Ishimaru, et al. "The wordometer 2.0: estimating the number of words you read in real life using commercial EOG glasses." UbiComp '16 Adjunct, pp. 293–296, 2016.
  9. 4FOTPS#BTFE$PHOJUJWF4UBUF3FDPHOJUJPO *OUFSFTU $PNQSFIFOTJPO $PHOJUJWFMPBE &ZFUSBDLFS 5IFSNBMDBNFSB 4BDDBEFT 'BDFUFNQFSBUVSF 'JYBUJPOT S.

    Ishimaru, et al. Cognitive State Measurement on Learning Materials by Utilizing Eye Tracker and Thermal Camera. Proc. ICDAR HDI 2017, pp. 32–36, 2017. S. Ishimaru, et al. Augmented Learning on Anticipating Textbooks with Eye Tracking. Positive Learning in the Age of Information (PLATO), pp. 387–398, 2018.
  10. "UUFOUJPO"OBMZTJT  intro. def. appl. intro. def. appl. intro. def.

    appl. intro. def. appl. intro. def. appl. intro. def. appl. Expert Intermediate Novice reading solving reading solving reading solving B SFBEJOH C TPMWJOH F SFBEJOH G TPMWJOH /PWJDF D SFBEJOH E TPMWJOH *OUFSNFEJBUF &YQFSU Introduction De nitions Applications (a) text $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT
  11. $PNQSFIFOTJPO3FDPHOJUJPO  'FBUVSFSFQSFTFOUBUJPOPGBMMQBSUJDJQBOUTEBUB 
 JOTVCTFRVFODFT NJO CBTFEBQQSPBDI • BDDVSBDZGPSDMBTTJGZJOH 


    FYQFSUJTFTJOUPUISFFDMBTTFT 
 VTFSJOEFQFOEFOUUSBJOJOH $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT S. Ishimaru, et al. "Augmented learning on anticipating textbooks with eye tracking." Positive Learning in the Age of Information. Springer VS, Wiesbaden, 2018. 387-398.
  12. $PNQSFIFOTJPO3FDPHOJUJPO  'FBUVSFSFQSFTFOUBUJPOPGBMMQBSUJDJQBOUTEBUB 
 JOTVCTFRVFODFT NJO CBTFEBQQSPBDI • BDDVSBDZGPSDMBTTJGZJOH 


    FYQFSUJTFTJOUPUISFFDMBTTFT 
 VTFSJOEFQFOEFOUUSBJOJOH  • /PWJDFTUVEFOUTSFBEB 
 UFYUCPPLTMPXMZXJUITNBMMTUFQT $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT S. Ishimaru, et al. "Augmented learning on anticipating textbooks with eye tracking." Positive Learning in the Age of Information. Springer VS, Wiesbaden, 2018. 387-398.
  13. $PNQSFIFOTJPO3FDPHOJUJPO  'FBUVSFSFQSFTFOUBUJPOPGBMMQBSUJDJQBOUTEBUB 
 JOTVCTFRVFODFT NJO CBTFEBQQSPBDI • BDDVSBDZGPSDMBTTJGZJOH 


    FYQFSUJTFTJOUPUISFFDMBTTFT 
 VTFSJOEFQFOEFOUUSBJOJOH  • /PWJDFTUVEFOUTSFBEB 
 UFYUCPPLTMPXMZXJUITNBMMTUFQT • &YQFSUTUVEFOUTTPNFUJNFTTLJQ 
 UIFJSFZFTPOUIFUFYUBOEGPDVTPO 
 UIFDPOUFOUUIFZBSFJOUFSFTUFEJO $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT S. Ishimaru, et al. "Augmented learning on anticipating textbooks with eye tracking." Positive Learning in the Age of Information. Springer VS, Wiesbaden, 2018. 387-398.
  14. &YQFSJNFOUBM%FTJHO  • VOJWFSTJUZTUVEFOUTSFBEOFXTQBQFSBSUJDMFT ʶXPSET  • %FWJDF4.*3&%OTDJFOUJ fi D)[FZFUSBDLFS

    • $BMJCSBUFUIFEFWJDFˠ3FTU TFD ˠ3FBEBEPDVNFOUˠ"OTXFSTVSWFZT 4UVEZ(B[F#BTFE*OUFSFTU%FUFDUJPO S. Jacob, et al. "Gaze-based interest detection on newspaper articles." Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction. 2018.
  15. *OUFSFTU3FDPHOJUJPO0WFSWJFXBOE3FTVMU  'PSXBSETBDDBEF 3FHSFTTJPO 'FBUVSFDBMDVMBUJPO 47. 4UVEZ(B[F#BTFE*OUFSFTU%FUFDUJPO + ongoing work

    S. Jacob, et al. "Gaze-based interest detection on newspaper articles." Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction. 2018. NFBO NFBO 62% 16% 2% 1% 25% 66% 7% 0% 6% 15% 83% 6% 7% 3% 8% 93% Predicted class 1 2 3 4 Actual class 4 3 2 1 *OVTFSEFQFOEFOUFWBMVBUJPO MFBWFPOFEPDVNFOUPVUDSPTTWBMJEBUJPOJOBQBSUJDJQBOU 
 47.DMBTTJ fi FEJOUFSFTUTJOUPMFWFMTXJUIBDDVSBDZ $//-45.DMBTTJ fi FEXJUIBDD $//-45.
  16. (B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO %FDJTJPOXJUIDPO fi EFODF %FDJTJPOXJUIPVUDPO fi EFODF S.

    Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts, p. 8, 2020. S. Ishimaru, et al. Confidence-Aware Learning Assistant. In arXiv preprint arXiv:2102.07312, 2021
  17. 4ZTUFN0WFSWJFX  1SFQSPDFTTJOH 
 fi YBUJPOTBDDBEFEFUFDUJPO 'FBUVSFDBMDVMBUJPO $MBTTJ fi DBUJPO

    8FVUJMJ[FUIFGPMMPXJOH fi WFF ff FDUJWFGFBUVSFTTFMFDUFECZ'PSXBSETUFQXJTFGFBUVSFTFMFDUJPO 
 GTVNPG fi YBUJPOEVSBUJPOTPO$IPJDFT GWBSJBODFPGYDPPSEJOBUFPG fi YBUJPOT 
 GUIFOVNCFSPGTBDDBEFTCFUXFFODIPJDFBSFBT GTVNPGTBDDBEFEVSBUJPOT GSFBEJOHUJNF • #JOBSZDMBTTJ fi DBUJPO • 4VQQPSU7FDUPS .BDIJOF 47.  • 3#'LFSOFM  • $ Ѝ 4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO S. Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts (Asian CHI Symposium), p. 8, 2020.
  18. 3FTFBSDI)ZQPUIFTFT  4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO • 3)$PSSFDUBOTXFSTXJUIPVUDPO fi EFODF WBHVFLOPXMFEHF

    UFOEUP CFJODPSSFDUMZBOTXFSFEXIFOBTLFEBHBJODPNQBSFEUPDPSSFDU BOTXFSXJUIDPO fi EFODF • 3)*ODPSSFDUBOTXFSTXJUIDPO fi EFODF NJTVOEFSTUBOEJOH UFOEUP CFNJTUBLFOBHBJODPNQBSFEUPJODPSSFDUBOTXFSTXJUIPVUDPO fi EFODF • 3)&TUJNBUJOHTFMGDPO fi EFODFGSPNMFBSOJOHCFIBWJPSTBOEHJWJOH GFFECBDL FH BEEJOHRVFTUJPOTUPBSFWJFXMJTU IJHIMJHIUJOHUIFN XIJMFSFWJFXJOH BWPJETUIFTDFOBSJPTPVUMJOFEJO3)BOE3) S. Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts (Asian CHI Symposium), p. 8, 2020.
  19. &YQFSJNFOUBM%FTJHO  4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO •  BOE B )PXPGUFOBOTXFSTXJUIPVUDPO

    fi EFODFBSFGPSHPUUFO • B BOE C )PX$P"-"JTF ff FDUJWFJOBTVDIDPOEJUJPO •  BOE B )PXPGUFOBOTXFSTXJUIDPO fi EFODFBSFJODPSSFDUMZBOTXFSFEBHBJO • B BOE C )PX$P"-"JTF ff FDUJWFJOBTVDIDPOEJUJPO $PNQBSJTPOPG
  20. &YQFSJNFOUBM3FTVMUT  4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO • 3)$PSSFDUBOTXFSTXJUIPVUDPO fi EFODFUFOEUPCFJODPSSFDUMZ BOTXFSFEXIFOBTLFEBHBJODPNQBSFEUPBOTXFSXJUIDPO

    fi EFODF • 4VQQPSUFE B ESPQQFEDPNQBSFEUP   Q  • 3)*ODPSSFDUBOTXFSTXJUIDPO fi EFODFUFOEUPCFNJTUBLFOBHBJO DPNQBSFEUPJODPSSFDUBOTXFSTXJUIPVUDPO fi EFODF • /PUTVQQPSUFE/PTJHOJ fi DBOUEJ ff FSFODFCFUXFFO  BOE B  • 3)&TUJNBUJOHTFMGDPO fi EFODFGSPNMFBSOJOHCFIBWJPSTBOEHJWJOH GFFECBDL FH BEEJOHRVFTUJPOTUPBSFWJFXMJTU IJHIMJHIUJOHUIFN XIJMFSFWJFXJOH BWPJETTVDITDFOBSJPT • 4VQQPSUFE$PNQBSJTPOTCFUXFFO B WT C  B WT C    B C           DPSSXDPOG B DPSSXPDPOG C DPSSXPDPOGGFFECBDL   B C           JODPSSXPDPOG B JODPSXDPOG C JODPSSXDPOGGFFECBDL S. Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts (Asian CHI Symposium), p. 8, 2020.
  21. "VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF 8PSEPNFUFS )ZQFS.JOE $P"-" -FDUVSFBOEUFYUCPPL 2VFTUJPOT UFTU %BJMZIBCJUTBOEDVSJPTJUZ 8IJDIRVFTUJPOTIPVMECFSFWJFXFE 


    DBSFGVMMZ  DPOTJEFSJOHQSJPSJUZMFBLBHF $BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT 
 BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ