Janet's phd defense @ 2018.06.04

Slide 1

Slide 1 text

Designing for Complex Creative Task Solving Yi-Ching (Janet) Huang 戔懯薹究蕦褾ጱ獺蝨௔犨率 2018.06.04 PhD Oral Defense 讙௑覌 Advisor: Jane Yung-jen Hsu, PhD

Slide 2

Slide 2 text

https://www.facebook.com/womaninthestriped/ Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !2

Slide 3

Slide 3 text

A Rejection Letter !3

Slide 4

Slide 4 text

Complex Creative Tasks Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !4

Slide 5

Slide 5 text

human-centered design from IDEO A Creative Task as An Iterative Process Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !5

Slide 6

Slide 6 text

Score: 2 Score: 2.5 Score: 2.75 1st version 2nd version 3rd version Writing as an iterative process !6

Slide 7

Slide 7 text

http://push.m-iti.org User Interface Design as An Iterative Process Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !7

Slide 8

Slide 8 text

Complex Creative Process Uncertainty A Concrete Solution Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !8

Slide 9

Slide 9 text

Properties of Creative Tasks 1. Open-ended and ill-defined 3. Quality is usually evaluated by multiple criteria 4. Quality can be improved by iterative refinement 2. Answer is not true or false, but how good the answer is Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !9

Slide 10

Slide 10 text

Teevan, Iqbal, and von Veh. Supporting Collaborative Writing with Microtasks. CHI 2016. Sadauskas, Byrne, and Atkinson. Mining Memories: Designing a Platform to Support Social Media Based Writing. CHI 2015 Bernstein, Little, Miller, Hartmann, Ackerman, Karger, Crowell, and Panovich. Soylent: A Word Processor with a Crowd Inside. UIST 2010. Kim, Cheng, and Bernstein. Ensemble: Exploring Complementary Strengths of Leaders and Crowds in Creative Collaboration. CSCW 2014 Hahn, Chang, Kim, and Kittur. The Knowledge Accelerator: Big Picture Thinking in Small Pieces. CHI 2016. Nebeling, To, Guo, de Freitas, Teevan, Dow, and Bigham. WearWrite: Crowd-Assisted Writing from Smartwatches. CHI 2016. Kittur, Smus, Khamkar, and Kraut. CrowdForge: Crowdsourcing Complex Work. UIST 2011. Agapie, Teevan, and Monroy- Hernández. Crowdsourcing in the Field: A Case Study Using Local Crowds for Event Reporting. HCOMP 2015. Luther, Hahn, Dow, and Kittur. Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines. HCOMP 2015. 10

Slide 11

Slide 11 text

(Boisson et al., 2013) Criterion (Burstein et al., 2004) (Chen et al., 2016) IEA(1997) (Yen et al., 2016) !11

Slide 12

Slide 12 text

Prior Work Ideation Outlining Creation Revision Publishing CrowdLines   (Luther et al.,2015) MicroWriter   (Teevan et al.,2016) CrowdForge   (Kittur et al.,2011) Sparkfolio  (Sadauskas et al.,2015) Ensemble  (Kim et al.,2014) Soylent  (Bernstei et al.,2010) Crowdsourcing in the Field   (Agapie et al.,2015) Knowledge Accelerator  (Hahn et al.,2016) WearWrite (Nebeling et al.,2016) IntroAssist (Hui et al.,2018) MechanicalNovel  (Kim et al.,2017) reﬂect and revise Writing Process WriteAhead (Chang and Chang, 2015) Rephraser 2.0 Linggle Knows (Chen et al., 2016) !12

Slide 13

Slide 13 text

“The best writing is rewriting.” — E.B. White — !13

Slide 14

Slide 14 text

Shortn: Text Shortening Crowdproof: Crowdsourced Proofreading Soylent: A Word Processor with a Crowd Inside (Bernstein et al., UIST’ 10) Applications M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, UIST '10, pages 313-322, New York, NY, USA, 2010. ACM. Microsoft Word Mechanical Turk Fix Verify Find Soylent select texts Find-Fix-Verify Workﬂow !14

Slide 15

Slide 15 text

Does rewriting improve quality ? !15

Slide 16

Slide 16 text

Iteration Quality The Beneﬁts of Iteration Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !16

Slide 17

Slide 17 text

Iteration Quality The Beneﬁts of Iteration Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !16

Slide 18

Slide 18 text

Iteration Quality The Beneﬁts of Iteration o x Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !16

Slide 19

Slide 19 text

Feedback Facilitates High Quality Results Evaluate the writing Improve the writing Feedback Work Iterative process Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !17 J. Hattie and H. Timperley. The power of feedback. Review of Educational Research, 77(1):81–112, March 2007.

Slide 20

Slide 20 text

Author Learning Supporting Collaboration Feedback Provider Creative Task Solving Framework Feedback Creative Work Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !18

Slide 21

Slide 21 text

Feedback Creative Work Author Learning Supporting Feedback Utilization Collaboration Feedback Generation Feedback Provider Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !19 Creative Task Solving Framework

Slide 22

Slide 22 text

Outline - Introduction - Part I: Feedback Generation - Part II: Feedback Utilization - Part III: Learning through Reﬂection - Conclusion Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !20

Slide 23

Slide 23 text

Part I: Feedback Generation Writing Feedback How do we generate eﬀective feedback for supporting authors to improve the quality of writing? Supporting Feedback Provider Author Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !21

Slide 24

Slide 24 text

Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion 22

Slide 25

Slide 25 text

Machine can help for correcting surface errors Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion 23

Slide 26

Slide 26 text

- Holistic Scoring: - provide diagnostic feedback on grammar, usage, and mechanics; style and diction; and organization and development - Templated-based feedback Criterion Feedback for the highest score “6” Feedback for the highest score “1” Template-based feedback Holistic Scoring Jill Burstein, Martin Chodorow, and Claudia Leacock. Automated essay evaluation: The criterion online writing service. AI Magazine, 25(3):27–36, 2004. (Burstein et al., 2004) Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion !24

Slide 27

Slide 27 text

Disadvantage of Existing Feedback Systems -Require large amounts of labeled data -Support limited topics -Static feedback template !25 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 28

Slide 28 text

Experts Peers Crowds Where can we get feedback? !26 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 29

Slide 29 text

Current Rewriting Support Tools Rewriting Feedback global + local Local issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 30

Slide 30 text

Current Rewriting Support Tools Rewriting Feedback global + local Spelling checker Grammar checker sentence word Local issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 31

Slide 31 text

Current Rewriting Support Tools Rewriting Feedback global + local Free Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Writer Reader Structure helps deliver message to a reader Idea !28 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 34

Slide 34 text

English Oriental (Kaplan, 1966) Rhetorical Patterns of Diﬀerent Languages Robert B. Kaplan. Cultural thought patterns in inter-cultural education. Language Learning, 1966. !29 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 35

Slide 35 text

Author Feedback Writing Crowd Machine Supporting Writing Revision by Crowdsourced Structural Feedback Revision Crowdsourcing Workﬂow Data Annotations StructFeed Yi-Ching Huang, Jiunn-Chia Huang, and Jane Yung-jen Hsu. Supporting ESL writing by prompting crowdsourced structural feedback. In Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017) Yi-Ching Huang, Hao-Chuan Wang, and Jane Yung-jen Hsu. Bridging learning gap in writing education with a crowd-powered system. CHI 2017 Workshop on Designing for Curiosity, Denver, Colorado, USA, 2017. !30 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 36

Slide 36 text

Topic Sentence Supporting Sentence Concluding Sentence Introduction Body Conclusion Essay Structure Paragraph Structure paragraph paragraph paragraph Key point Supporting Sentence Supporting Sentence !31 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 37

Slide 37 text

A paragraph is a group of sentences organized around a central topic. Four Key Elements of Great Writing Element #1: Unity Element #2: Order Element #3: Coherence Element #4: Completeness All sentences in a paragraph should speak about one single idea or one main subject. Order refers to the way you organize your supporting sentences. Sentences within a paragraph need to connect to each other and work together as a whole. Completeness means a paragraph is well-developed. !32 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion Holly L. Jacobs, Stephen A. Zinkgraf, Deanna R. Wormuth, V. Faye Hartﬁel, and Jane B. Hughey. Testing ESL Composition: A Practical Approach. Newbury House, 1981.

Slide 38

Slide 38 text

1. All sub-points centering on one central idea 2. Using no irrelevant sentences Key points to achieve unity: Topic Sentence Supporting Sentence #1 Concluding Sentence related to the topic sentence Supporting Sentence #2 Supporting Sentence #3 The First Key Element of a Great Writing - All sentences in a paragraph should speak about one single idea. Unity !33 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 39

Slide 39 text

Crowdsourcing Workflow Structural Feedback Unity Identification Writing Criteria 1. multiple topic issue 2. missing topic issue 3. irrelevance issue Topic sentence prediction Irrelevant sentence prediction Crowd Annotations System Overview of StructFeed Topic sentence annotation Relevant keyword annotation !34 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 40

Slide 40 text

Crowdsourcing Workflow for Unity Identification Topic Identify topic sentence topic + ideas Crowdsourcing Workflow Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation !35 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 41

Slide 41 text

Topic Task - identify topic sentence Quality Control - native speakers as workers - brief explanation of concept - worked example - annotate sentence by click !36 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 42

Slide 42 text

Topic Task - identify topic sentence Quality Control - native speakers as workers - brief explanation of concept - worked example - annotate sentence by click Explanation Worked example Working area annotate sentence by click !36 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 43

Slide 43 text

Relevance Task annotate word by click !37 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 44

Slide 44 text

Relevance Task Worked example annotate word by click !37 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 45

Slide 45 text

Topic Identify topic sentence topic + ideas Crowdsourcing Workflow Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation Structural Feedback Writing Criteria 1. multiple topic issue 2. missing topic issue 3. irrelevance issue Crowd Annotations Unity Identification Topic sentence prediction Irrelevant sentence prediction !38 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 46

Slide 46 text

Eﬀective Feedback 1. Obtain a concept of the standard or goal 2. Compare the actual level of performance with the standard 3. Engage in action which leads to closure of the gap (Sadler, D. R. 1989) D. R. Sadler. Formative assessment and the design of instructional systems. Instructional Science, 18(2):119{144, 1989 !39 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 47

Slide 47 text

Structural Feedback !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 48

Slide 48 text

Structural Feedback Feedback Summary - type of issue - suggested action !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 49

Slide 49 text

Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 50

Slide 50 text

Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action - topic sentence !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 51

Slide 51 text

Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action - topic sentence - irrelevant sentence !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 52

Slide 52 text

Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action - topic sentence - irrelevant sentence - relevant keywords !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 53

Slide 53 text

!41

Slide 54

Slide 54 text

!41

Slide 55

Slide 55 text

Field Experiment on ESL Writers - 18 self-motivated ESL learners (8 females, 10 males) - 19~34 years old - A between subjects study Conditions - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed Writing original version R Rewriting revised version R’ Feedback Measure - time, quantity, cost - quality improvement (R’-R) - perceived helpfulness !42 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 56

Slide 56 text

Expert Feedback Crowd Feedback StructFeed -Diff-rating: 0.29 (.43) -Time: 1~2 days -Cost: $16 -Quantity: 55.44 suggestions - # of equal rating: 1 - # of decreased rating: 1 - Diff-rating: 0.38 (.44) - Time: 10~30 mins - Cost: $2 - Quantity: 8.11 suggestions - # of equal rating: 1 - # of decreased rating: 1 -Diff-rating: 0.54 (.25) -Time: 1~5 hrs -Costs: $1~1.7 -15-25 workers -All participants improve the quality of writing ! Field Experiment Results !43 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 57

Slide 57 text

“I don’t understand what he means. His comments contain diﬃcult terminology and it’s hard for me to capture the key points.” (P15) Observation I Knowledge gaps between an expert and a novice writer !44 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 58

Slide 58 text

Structural feedback promotes self-reﬂection “I’m so surprised that no one annotates it as relevant keywords. I originally think that is a common example for other people. But, I am wrong. I will carefully choose a more common and understandable example to describe my idea next time.” (P7) Observation 2 !45 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 59

Slide 59 text

Crowd Machine Crowdsourcing Workﬂow Data Annotations Expert Crowd Iterative Revision Process Author Feedback Writing Feedback Feedback !46 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 60

Slide 60 text

Writing Rewriting Feedback 1 Rewriting Feedback 2 Rewriting Feedback 3 v1 v2 v3 v4 Writing Iteration Experiment - 18 self-motivate ESL learners (8 females, 10 males) - 19~34 years old - A within-subjects counter-balanced design Conditions - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed !47 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 61

Slide 61 text

Experiment Results Red: Grade decreased Green: Grade increase White: No improvement Expert feedback (C1) Crowd feedback (C2) StructFeed (C3) Avg diff rating 0.15 0.21 0.43 Avg standard deviation 0.32 0.32 0.44 # of decreased diff rating 3 2 0 # of equal rating 7 7 5 !48 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 62

Slide 62 text

Red: Grade decreased Green: Grade increase White: No improvement Expert feedback (C1) Crowd feedback (C2) StructFeed (C3) Ensemble Avg diff rating 0.15 0.21 0.43 0.79 Avg standard deviation 0.32 0.32 0.44 0.45 # of decreased diff rating 3 2 0 0 # of equal rating 7 7 5 1 Experiment Results !49 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 63

Slide 63 text

Different types of feedback support writers in different perspectives “All types of feedback are useful to me. I’d like to use them for different purpose or at different stage. For example, I will use StructFeed at the beginning, then crowd feedback. If I need to write SOP, I’ll use expert feedback for final check.” (P5) Insight !50 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Expert Crowd Crowd Machine StructFeed ҄ ҄ Ensemble feedback supports writers in different perspectives

Slide 64

Slide 64 text

(1) We designed a crowd-powered system that enables structural feedback for supporting ESL writing (2) We leveraged domain rubrics in designing crowdsourcing workflow (3) StructFeed outperformed free-form feedback from both experts and crowd (4) Ensemble feedback may support writers in different perspectives Summary !51 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 65

Slide 65 text

Learning Part II: Feedback Utilization Writing Feedback Feedback Provider Author How do we support authors to integrate feedback into revisions and facilitate high-quality outcome? Revision !52 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 66

Slide 66 text

Evaluate the writing Improve the writing Feedback Work Iterative process Good feedback NOT always facilitates good results! !53 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 67

Slide 67 text

Too many suggestions may cause problems 1. Information overload 2. Cost of task switching 3. People focus on easier problems !54 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 68

Slide 68 text

Formative Study 6 participants (1 female) with the age 18-23 - Novices deal with feedback in an “unstructured” way - Varying revision strategies (1) browse all comments (2) group similar comments (3) deal with comments in a sequence (1) beginning-to-end editing (2) specificity-first editing (3) high-to-low editing (4) low-to-high editing novice writers !55 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 69

Slide 69 text

Expert Revision Practice Novice writers Expert writers ESL writers revise in a linear process revise in a recursive way revise in a disorganized way Expert writers think high-level goals, and break it into low-level steps !56 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion A Lack of Reﬂection

Slide 70

Slide 70 text

Challenges of ESL Revision ESL students lack abilities in a foreign language - metarhetorical awareness (knowledge of themselves as writers) - metastrategic awareness (knowledge of their own personality type and its inﬂuence on their writing behaviors, including revision) - metalinguistic awareness (terminology to discuss language issues) (Alice S. Horning, 2006) !57 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion A Lack of Awareness

Slide 71

Slide 71 text

Goals 1. Support awareness and reﬂection 2. Guide writers to think and revise “structurally” !58 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 72

Slide 72 text

Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 73

Slide 73 text

Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision 1. get feedback Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 74

Slide 74 text

Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision 1. get feedback 2. classify feedback Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 75

Slide 75 text

Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision 1. get feedback 2. classify feedback 3. revise an article in a revision workflow Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 76

Slide 76 text

1) Rhetorical structure 2) Meta-feedback 3) Flexible revision workflow Three Design Considerations Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback w/ meta-feedback Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback content organization grammar mechanics language !60 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 77

Slide 77 text

Feedback Type Definition Examples High Content Content refers to the substance of writing. It includes topic sentence expressing main argument and supporting ideas. e.g. unity of argument, supporting idea, relevant example, addresses the question, etc. Organization Organization refers to the logical organization of the content. e.g. coherence of the content, relation between sentences, logical sequencing, etc. Medium Vocabulary Vocabulary refers to the selection or words those are suitable with the content. e.g. word choice, etc. Language Use Language Use refers to the use of the correct grammatical forms and syntactical pattern. e.g. fixing grammatical errors, or paraphrasing, shortening, etc. Low Mechanics Mechanic refers to all the arbitrary technical stuff in writing like spelling, capitalization, punctuation, etc. e.g. spelling errors, punctuation, capitalization, format, etc. Feedback Classification (Jacobs et al.,1981) !61 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 78

Slide 78 text

Automated Classification content organization vocabulary grammar mechanics Feedback Mechanics Vocabulary Grammar Rule-based classifier Comment Edit Feedback punctuation, spelling, capitalization stemming Non-Mechanics High-level Medium-level Low-level !62 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 79

Slide 79 text

Flexible Revision Workflow High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics mechanics vocabulary gram m ar organization content organization content vocabulary gram m ar mechanics Sequential Workflow Concurrent Workflow high-to-low (HML) low-to-high (LMH) vocabulary gram m ar mechanics content organization all (ALL) or !63 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 80

Slide 80 text

Rhetorical Structure content organization vocabulary grammar mechanics Feedback Rhetorical Categories Feedback Classify High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Collect Structure Crowdsourcing Workﬂows !64 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 81

Slide 81 text

ReviseO ReviseO !65 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 82

Slide 82 text

High-to-low editing Low-to-high editing Mechanics Feedback Language Feedback Content & Organization Feedback High-level Middle-level Low-level !66

Slide 83

Slide 83 text

Experiment Design High-to-low (HML) Low-to-high (LMH) High - a within-subjects, counterbalanced experiment design - 12 self-motivated non-native writers - each participants performed 3 rewriting tasks with diﬀerent topics - 3 experimental conditions - (1) show feedback together (ALL) - (2) show feedback sequentially from high to low (HML) - (3) show feedback sequentially from low to high (LMH) Together (ALL) Medium Low High Medium Low ALL !67 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 84

Slide 84 text

Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion RevisO System Categorization Writing Quality Revision Effort Ease of use  M(SE) Helpfulness  M(SE) Helpfulness  M(SE) Diff of Rating  M(SE) Time spent  M(SE) Edit distance  M(SE) LMH 5.58(.43) 6.22(.21) 6.58(.23) 6.79(.88) 1731.58(147.81) 353.33(.68) HML 4.92(.47) 6.22(.25) 6.50(.26) 7.21(.53) 1727.75(204.05) 370.00(60.85) ALL 5.25(.33) 6.25(.18) 5.67(.48) 7.25(.44) 1612.92(124.57) 444.33(72.35) Experiment Results - All participants improved the quality of writing - The ReviseO system got high perceived helpfulness and usefulness - No significant difference among three conditions post-questionnaire: 0~7 score: 0~100

Slide 85

Slide 85 text

Insight 1 Structured feedback helps filter information, identify weaknesses, and support reflection “This categorized feedback helps me identify my common mistakes easily! When I see the same type of writing issues appearing frequently, I understand that I need to pay more attention to this type of problem in my next writing. (P2)” 69 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 86

Slide 86 text

Flexible revision supports varying strategies and helps develop new revision strategies Insight 2 70 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion 25% 17% 33% 25% (3) (3) (4) (2) HML LMH ALL HLM

Slide 87

Slide 87 text

Separating feedback in a sequence may cause editing conflicts Insight 3 1) Mis-classified feedback leads to misunderstanding 2) Too many low-level fixes leads to decreased motivation for high-level improvement 71 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 88

Slide 88 text

Summary (1) Feedback Orchestration uses a rhetorical structure, meta- feedback, and flexible workflows to guide effective revision (2) Structured feedback helps identify weaknesses and support reflection (3) Flexible workflows help develop personalized revision strategies. (4) ReviseO support writers to think and revise structurally 72 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 89

Slide 89 text

Creative Work Feedback Provider Author Structural Feedback Feedback-Driven Revision Feedback Crowd Machine Crowdsourcing Workflow Data Annotations Revision Workflow Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback High Medium Low Feedback Collaboration 1 2 3 Structure Matters We use “structure” to guide feedback providers to contribute high-quality results. 1 2 3 We use “structure” to present feedback from different levels. We use “structure” to guide users to integrate feedback into revision !73 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 90

Slide 90 text

Iteration Quality Revision Effort individual’s ability !74 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 91

Slide 91 text

Iteration Quality Revision Effort individual’s ability !74 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 92

Slide 92 text

Iteration Quality Revision Effort individual’s ability !74 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 93

Slide 93 text

Iteration Quality Revision Effort Knowledge Gap individual’s ability !74 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 94

Slide 94 text

“We do not learn from experience… we learn from reﬂecting on experience” — John Dewey — !75 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 95

Slide 95 text

Creative Work User as Author Practice Review Collaboration Feedback Provider Feedback Feedback-Driven Revision Structural Feedback !76 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 96

Slide 96 text

Creative Work User as Author Practice Review Collaboration Feedback Provider Feedback Feedback-Driven Revision Structural Feedback Never-Ending Creative Learning !76 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion User as Learner Reflection Reflection Collaboration Creative Work Learning through Reflection

Slide 97

Slide 97 text

!77 Learning Professional Skills for Drawing Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 98

Slide 98 text

!78 The “knowledge” is in the details… Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 99

Slide 99 text

Leaner Creative Work Author Before/After-Practice Reflection Workflow Generate Learning Points learning points Identify Self-Explain Reflection Workflows Practice Extracting Learning Points for Drawing Support Identify Self-Explain Practice Identify Self-Explain Learn & Reflect Create Yi-Ching Huang, Jerry Yu-Heng Chan, and Jane Yung-jen Hsu. Reflection before/after practice: Learnersourcing for drawing support. In CHI ’18 Extended Abstracts on Human Factors in Computing Systems, 2018. !79 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 100

Slide 100 text

!80 drafting outlining details What is a learning point? 1.where is it? a clip of video/process 2.what is it? a description 3.why do you think it is important? a reason coloring with large area Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 101

Slide 101 text

- identify start and end point - describe what it is - explain why Reflection Workflow for Extracting Learning Points Identify Identify one learning point Self-Explain Explain why you choose it Why? start point end point !81 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 102

Slide 102 text

Reﬂection Practice !82 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion

Slide 103

Slide 103 text

Reflection Reflection is defined as a purposeful thinking toward a goal. (Dewey, 1933) 1. Reflection-in-action 2. Reflection-on-action - refers to the monitoring and modification of actions during the learning process - refers to reasonable evaluation and strategic planning for improvement after the learning process. Reflection Practice (Schön, 1983) !83 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 104

Slide 104 text

Before/After-Practice Reflection Workflow Practice Learning by doing Identify Identify learning points Self-Explain Explain it is important helpful difficult interest Why? !84 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Before-annotation Reflection Practice Learning by doing Identify Identify learning points Self-Explain Explain it is important helpful difficult interest Why? After-annotation Reflection

Slide 105

Slide 105 text

Practice Learning by doing Identify Identify learning points Self-Explain Explain it is important helpful difficult interest Why? Identify Identify learning points Self-Explain Explain it is important helpful difficult interest Why? Pilot study: Before/After-Practice Reflection Workflow - 8 participants (5 male, 3 female), 20-23 years old - 80 annotations (48 learning points, 16 difficult points, 16 interest points) !85 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion - Learners discover new learning points or revise their previous findings after practice Lessons Learned - After-annotation augments before-annotation

Slide 106

Slide 106 text

!86 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion ䷱䋊螂ܹ璾牧眤憽氅ᜋӞ䍅Ӟ䍅吩Ӥ݄ጱ向ဩ盄蠐 (P5, before) Ӥᜋጱොဩ牧᩻ڊᇔ誢क़ᶎฎݢ犥ጱ牧磧盅ٚ硄ധ疰অ牧蝡䰬Ӥᜋ眤憽穉斃অӞ讨 (P3, after) ضय़膌向ڊ斪ୄ牧ٚୌ缏奞℄ (P1,before) አ܈ਁ斔ۗ娄向Ո腷 (P4, after) 硄ധ毣誧螲翣ጱ茐ᜋ㬵蕣蝨ط୽ጱ硳ຎ (P4,after) Reflective annotations help learners obtain new knowledge in different perspectives

Slide 107

Slide 107 text

!87 Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion Creative learning is never ending …

Slide 108

Slide 108 text

Feedback Provider Crowd Machine Crowdsourcing Workflow Data Annotations Expert Crowd Part I: Feedback Generation Feedback Author Revision Workflow Part II: Feedback Utilization Rhetorical Structure High-Level Medium-Level Low-Level Feedback High Medium Low Feedback Collaboration Learner Creative Work Extract Reflection Annotations Learn & Reflect Learning Part III: Learning through Reflection !88

Slide 109

Slide 109 text

Creative Work Author Practice Review Feedback-Driven Revision Collaboration Structural Feedback Feedback Provider Learner Reflection Feedback Learning through Reflection Reflection Collaboration Creative Work !89 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 110

Slide 110 text

Slide 111

Slide 111 text

Accomplishments - Two Intelligent Systems for Supporting Writing Revision - StructFeed for generating structural feedback - ReviseO for facilitating feedback-driven revision - Studies on understandings between feedback, revision, and writing quality - Workflows for collecting feedback, facilitate revision, and promote reflection - Unity & Coherence workflow - Flexible Revision workflows - Reflection-based workflows !90 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion

Slide 112

Slide 112 text

!91 Contributions Intro | Feedback Generation | Feedback Utilization | Learning thro Reﬂection | Conclusion - Never-Ending Creative Learning for support creative task solving and learning - Leverage structure to facilitate feedback generation and feedback-driven revision - Techniques for enabling collaborations among users, crowds, and machines

Slide 113

Slide 113 text

Future Directions - Effective collaborations between crowd and machine - Adaptive Feedback for supporting users with different levels - Never-Ending Creative Learning for Other Domains !92 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Crowd Machine

Slide 114

Slide 114 text

Thank You Q&A !93