Janet's phd defense @ 2018.06.04

I passed my phd oral defense!!!! :D

Transcript

1. Designing for Complex Creative Task Solving Yi-Ching (Janet) Huang 戔懯薹究蕦褾ጱ獺蝨௔犨率 2018.06.04 PhD Oral Defense 讙௑覌 Advisor: Jane Yung-jen Hsu, PhD
2. https://www.facebook.com/womaninthestriped/ Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !2
3. A Rejection Letter !3
4. Complex Creative Tasks Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !4
5. human-centered design from IDEO A Creative Task as An Iterative Process Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !5
6. Score: 2 Score: 2.5 Score: 2.75 1st version 2nd version 3rd version Writing as an iterative process !6
7. http://push.m-iti.org User Interface Design as An Iterative Process Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !7
8. Complex Creative Process Uncertainty A Concrete Solution Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !8
9. Properties of Creative Tasks 1. Open-ended and ill-defined 3. Quality is usually evaluated by multiple criteria 4. Quality can be improved by iterative refinement 2. Answer is not true or false, but how good the answer is Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !9
10. Teevan, Iqbal, and von Veh. Supporting Collaborative Writing with Microtasks. CHI 2016. Sadauskas, Byrne, and Atkinson. Mining Memories: Designing a Platform to Support Social Media Based Writing. CHI 2015 Bernstein, Little, Miller, Hartmann, Ackerman, Karger, Crowell, and Panovich. Soylent: A Word Processor with a Crowd Inside. UIST 2010. Kim, Cheng, and Bernstein. Ensemble: Exploring Complementary Strengths of Leaders and Crowds in Creative Collaboration. CSCW 2014 Hahn, Chang, Kim, and Kittur. The Knowledge Accelerator: Big Picture Thinking in Small Pieces. CHI 2016. Nebeling, To, Guo, de Freitas, Teevan, Dow, and Bigham. WearWrite: Crowd-Assisted Writing from Smartwatches. CHI 2016. Kittur, Smus, Khamkar, and Kraut. CrowdForge: Crowdsourcing Complex Work. UIST 2011. Agapie, Teevan, and Monroy- Hernández. Crowdsourcing in the Field: A Case Study Using Local Crowds for Event Reporting. HCOMP 2015. Luther, Hahn, Dow, and Kittur. Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines. HCOMP 2015. 10
11. (Boisson et al., 2013) Criterion (Burstein et al., 2004) (Chen et al., 2016) IEA(1997) (Yen et al., 2016) !11
12. Prior Work Ideation Outlining Creation Revision Publishing CrowdLines 
 (Luther et al.,2015) MicroWriter 
 (Teevan et al.,2016) CrowdForge 
 (Kittur et al.,2011) Sparkfolio
 (Sadauskas et al.,2015) Ensemble
 (Kim et al.,2014) Soylent
 (Bernstei et al.,2010) Crowdsourcing in the Field 
 (Agapie et al.,2015) Knowledge Accelerator
 (Hahn et al.,2016) WearWrite (Nebeling et al.,2016) IntroAssist (Hui et al.,2018) MechanicalNovel
 (Kim et al.,2017) reflect and revise Writing Process WriteAhead (Chang and Chang, 2015) Rephraser 2.0 Linggle Knows (Chen et al., 2016) !12
13. “The best writing is rewriting.” — E.B. White — !13
14. Shortn: Text Shortening Crowdproof: Crowdsourced Proofreading Soylent: A Word Processor with a Crowd Inside (Bernstein et al., UIST’ 10) Applications M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, UIST '10, pages 313-322, New York, NY, USA, 2010. ACM. Microsoft Word Mechanical Turk Fix Verify Find Soylent select texts Find-Fix-Verify Workflow !14
15. Does rewriting improve quality ? !15
16. Iteration Quality The Benefits of Iteration Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !16
17. Iteration Quality The Benefits of Iteration Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !16
18. Iteration Quality The Benefits of Iteration o x Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !16
19. Feedback Facilitates High Quality Results Evaluate the writing Improve the writing Feedback Work Iterative process Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !17 J. Hattie and H. Timperley. The power of feedback. Review of Educational Research, 77(1):81–112, March 2007.
20. Author Learning Supporting Collaboration Feedback Provider Creative Task Solving Framework Feedback Creative Work Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !18
21. Feedback Creative Work Author Learning Supporting Feedback Utilization Collaboration Feedback Generation Feedback Provider Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !19 Creative Task Solving Framework
22. Outline - Introduction - Part I: Feedback Generation - Part II: Feedback Utilization - Part III: Learning through Reflection - Conclusion Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !20
23. Part I: Feedback Generation Writing Feedback How do we generate effective feedback for supporting authors to improve the quality of writing? Supporting Feedback Provider Author Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !21
24. Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion 22
25. Machine can help for correcting surface errors Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion 23
26. - Holistic Scoring: - provide diagnostic feedback on grammar, usage, and mechanics; style and diction; and organization and development - Templated-based feedback Criterion Feedback for the highest score “6” Feedback for the highest score “1” Template-based feedback Holistic Scoring Jill Burstein, Martin Chodorow, and Claudia Leacock. Automated essay evaluation: The criterion online writing service. AI Magazine, 25(3):27–36, 2004. (Burstein et al., 2004) Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !24
27. Disadvantage of Existing Feedback Systems -Require large amounts of labeled data -Support limited topics -Static feedback template !25 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
28. Experts Peers Crowds Where can we get feedback? !26 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
29. Current Rewriting Support Tools Rewriting Feedback global + local Local issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
30. Current Rewriting Support Tools Rewriting Feedback global + local Spelling checker Grammar checker sentence word Local issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
31. Current Rewriting Support Tools Rewriting Feedback global + local Free Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
32. Current Rewriting Support Tools Rewriting Feedback global + local Free Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
33. Writer Reader Structure helps deliver message to a reader Idea !28 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
34. English Oriental (Kaplan, 1966) Rhetorical Patterns of Different Languages Robert B. Kaplan. Cultural thought patterns in inter-cultural education. Language Learning, 1966. !29 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
35. Author Feedback Writing Crowd Machine Supporting Writing Revision by Crowdsourced Structural Feedback Revision Crowdsourcing Workflow Data Annotations StructFeed Yi-Ching Huang, Jiunn-Chia Huang, and Jane Yung-jen Hsu. Supporting ESL writing by prompting crowdsourced structural feedback. In Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017) Yi-Ching Huang, Hao-Chuan Wang, and Jane Yung-jen Hsu. Bridging learning gap in writing education with a crowd-powered system. CHI 2017 Workshop on Designing for Curiosity, Denver, Colorado, USA, 2017. !30 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
36. Topic Sentence Supporting Sentence Concluding Sentence Introduction Body Conclusion Essay Structure Paragraph Structure paragraph paragraph paragraph Key point Supporting Sentence Supporting Sentence !31 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
37. A paragraph is a group of sentences organized around a central topic. Four Key Elements of Great Writing Element #1: Unity Element #2: Order Element #3: Coherence Element #4: Completeness All sentences in a paragraph should speak about one single idea or one main subject. Order refers to the way you organize your supporting sentences. Sentences within a paragraph need to connect to each other and work together as a whole. Completeness means a paragraph is well-developed. !32 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Holly L. Jacobs, Stephen A. Zinkgraf, Deanna R. Wormuth, V. Faye Hartfiel, and Jane B. Hughey. Testing ESL Composition: A Practical Approach. Newbury House, 1981.
38. 1. All sub-points centering on one central idea 2. Using no irrelevant sentences Key points to achieve unity: Topic Sentence Supporting Sentence #1 Concluding Sentence related to the topic sentence Supporting Sentence #2 Supporting Sentence #3 The First Key Element of a Great Writing - All sentences in a paragraph should speak about one single idea. Unity !33 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
39. Crowdsourcing Workflow Structural Feedback Unity Identification Writing Criteria 1. multiple topic issue 2. missing topic issue 3. irrelevance issue Topic sentence prediction Irrelevant sentence prediction Crowd Annotations System Overview of StructFeed Topic sentence annotation Relevant keyword annotation !34 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
40. Crowdsourcing Workflow for Unity Identification Topic Identify topic sentence topic + ideas Crowdsourcing Workflow Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation !35 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
41. Topic Task - identify topic sentence Quality Control - native speakers as workers - brief explanation of concept - worked example - annotate sentence by click !36 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
42. Topic Task - identify topic sentence Quality Control - native speakers as workers - brief explanation of concept - worked example - annotate sentence by click Explanation Worked example Working area annotate sentence by click !36 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
43. Relevance Task annotate word by click !37 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
44. Relevance Task Worked example annotate word by click !37 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
45. Topic Identify topic sentence topic + ideas Crowdsourcing Workflow Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation Structural Feedback Writing Criteria 1. multiple topic issue 2. missing topic issue 3. irrelevance issue Crowd Annotations Unity Identification Topic sentence prediction Irrelevant sentence prediction !38 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
46. Effective Feedback 1. Obtain a concept of the standard or goal 2. Compare the actual level of performance with the standard 3. Engage in action which leads to closure of the gap (Sadler, D. R. 1989) D. R. Sadler. Formative assessment and the design of instructional systems. Instructional Science, 18(2):119{144, 1989 !39 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
47. Structural Feedback !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
48. Structural Feedback Feedback Summary - type of issue - suggested action !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
49. Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
50. Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action - topic sentence !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
51. Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action - topic sentence - irrelevant sentence !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
52. Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action - topic sentence - irrelevant sentence - relevant keywords !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
53. !41
54. !41
55. Field Experiment on ESL Writers - 18 self-motivated ESL learners (8 females, 10 males) - 19~34 years old - A between subjects study Conditions - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed Writing original version R Rewriting revised version R’ Feedback Measure - time, quantity, cost - quality improvement (R’-R) - perceived helpfulness !42 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
56. Expert Feedback Crowd Feedback StructFeed -Diff-rating: 0.29 (.43) -Time: 1~2 days -Cost: $16 -Quantity: 55.44 suggestions - # of equal rating: 1 - # of decreased rating: 1 - Diff-rating: 0.38 (.44) - Time: 10~30 mins - Cost: $2 - Quantity: 8.11 suggestions - # of equal rating: 1 - # of decreased rating: 1 -Diff-rating: 0.54 (.25) -Time: 1~5 hrs -Costs: $1~1.7 -15-25 workers -All participants improve the quality of writing ! Field Experiment Results !43 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
57. “I don’t understand what he means. His comments contain difficult terminology and it’s hard for me to capture the key points.” (P15) Observation I Knowledge gaps between an expert and a novice writer !44 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
58. Structural feedback promotes self-reflection “I’m so surprised that no one annotates it as relevant keywords. I originally think that is a common example for other people. But, I am wrong. I will carefully choose a more common and understandable example to describe my idea next time.” (P7) Observation 2 !45 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
59. Crowd Machine Crowdsourcing Workflow Data Annotations Expert Crowd Iterative Revision Process Author Feedback Writing Feedback Feedback !46 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
60. Writing Rewriting Feedback 1 Rewriting Feedback 2 Rewriting Feedback 3 v1 v2 v3 v4 Writing Iteration Experiment - 18 self-motivate ESL learners (8 females, 10 males) - 19~34 years old - A within-subjects counter-balanced design Conditions - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed !47 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
61. Experiment Results Red: Grade decreased Green: Grade increase White: No improvement Expert feedback (C1) Crowd feedback (C2) StructFeed (C3) Avg diff rating 0.15 0.21 0.43 Avg standard deviation 0.32 0.32 0.44 # of decreased diff rating 3 2 0 # of equal rating 7 7 5 !48 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
62. Red: Grade decreased Green: Grade increase White: No improvement Expert feedback (C1) Crowd feedback (C2) StructFeed (C3) Ensemble Avg diff rating 0.15 0.21 0.43 0.79 Avg standard deviation 0.32 0.32 0.44 0.45 # of decreased diff rating 3 2 0 0 # of equal rating 7 7 5 1 Experiment Results !49 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
63. Different types of feedback support writers in different perspectives “All types of feedback are useful to me. I’d like to use them for different purpose or at different stage. For example, I will use StructFeed at the beginning, then crowd feedback. If I need to write SOP, I’ll use expert feedback for final check.” (P5) Insight !50 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Expert Crowd Crowd Machine StructFeed ҄ ҄ Ensemble feedback supports writers in different perspectives
64. (1) We designed a crowd-powered system that enables structural feedback for supporting ESL writing (2) We leveraged domain rubrics in designing crowdsourcing workflow (3) StructFeed outperformed free-form feedback from both experts and crowd (4) Ensemble feedback may support writers in different perspectives Summary !51 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
65. Learning Part II: Feedback Utilization Writing Feedback Feedback Provider Author How do we support authors to integrate feedback into revisions and facilitate high-quality outcome? Revision !52 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
66. Evaluate the writing Improve the writing Feedback Work Iterative process Good feedback NOT always facilitates good results! !53 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
67. Too many suggestions may cause problems 1. Information overload 2. Cost of task switching 3. People focus on easier problems !54 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
68. Formative Study 6 participants (1 female) with the age 18-23 - Novices deal with feedback in an “unstructured” way - Varying revision strategies (1) browse all comments (2) group similar comments (3) deal with comments in a sequence (1) beginning-to-end editing (2) specificity-first editing (3) high-to-low editing (4) low-to-high editing novice writers !55 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
69. Expert Revision Practice Novice writers Expert writers ESL writers revise in a linear process revise in a recursive way revise in a disorganized way Expert writers think high-level goals, and break it into low-level steps !56 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion A Lack of Reflection
70. Challenges of ESL Revision ESL students lack abilities in a foreign language - metarhetorical awareness (knowledge of themselves as writers) - metastrategic awareness (knowledge of their own personality type and its influence on their writing behaviors, including revision) - metalinguistic awareness (terminology to discuss language issues) (Alice S. Horning, 2006) !57 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion A Lack of Awareness
71. Goals 1. Support awareness and reflection 2. Guide writers to think and revise “structurally” !58 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
72. Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
73. Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision 1. get feedback Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
74. Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision 1. get feedback 2. classify feedback Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
75. Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision 1. get feedback 2. classify feedback 3. revise an article in a revision workflow Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
76. 1) Rhetorical structure 2) Meta-feedback 3) Flexible revision workflow Three Design Considerations Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback w/ meta-feedback Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback content organization grammar mechanics language !60 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
77. Feedback Type Definition Examples High Content Content refers to the substance of writing. It includes topic sentence expressing main argument and supporting ideas. e.g. unity of argument, supporting idea, relevant example, addresses the question, etc. Organization Organization refers to the logical organization of the content. e.g. coherence of the content, relation between sentences, logical sequencing, etc. Medium Vocabulary Vocabulary refers to the selection or words those are suitable with the content. e.g. word choice, etc. Language Use Language Use refers to the use of the correct grammatical forms and syntactical pattern. e.g. fixing grammatical errors, or paraphrasing, shortening, etc. Low Mechanics Mechanic refers to all the arbitrary technical stuff in writing like spelling, capitalization, punctuation, etc. e.g. spelling errors, punctuation, capitalization, format, etc. Feedback Classification (Jacobs et al.,1981) !61 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
78. Automated Classification content organization vocabulary grammar mechanics Feedback Mechanics Vocabulary Grammar Rule-based classifier Comment Edit Feedback punctuation, spelling, capitalization stemming Non-Mechanics High-level Medium-level Low-level !62 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
79. Flexible Revision Workflow High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics mechanics vocabulary gram m ar organization content organization content vocabulary gram m ar mechanics Sequential Workflow Concurrent Workflow high-to-low (HML) low-to-high (LMH) vocabulary gram m ar mechanics content organization all (ALL) or !63 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
80. Rhetorical Structure content organization vocabulary grammar mechanics Feedback Rhetorical Categories Feedback Classify High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Collect Structure Crowdsourcing Workflows !64 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
81. ReviseO ReviseO !65 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
82. High-to-low editing Low-to-high editing Mechanics Feedback Language Feedback Content & Organization Feedback High-level Middle-level Low-level !66
83. Experiment Design High-to-low (HML) Low-to-high (LMH) High - a within-subjects, counterbalanced experiment design - 12 self-motivated non-native writers - each participants performed 3 rewriting tasks with different topics - 3 experimental conditions - (1) show feedback together (ALL) - (2) show feedback sequentially from high to low (HML) - (3) show feedback sequentially from low to high (LMH) Together (ALL) Medium Low High Medium Low ALL !67 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
84. Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion RevisO System Categorization Writing Quality Revision Effort Ease of use
 M(SE) Helpfulness
 M(SE) Helpfulness
 M(SE) Diff of Rating
 M(SE) Time spent
 M(SE) Edit distance
 M(SE) LMH 5.58(.43) 6.22(.21) 6.58(.23) 6.79(.88) 1731.58(147.81) 353.33(.68) HML 4.92(.47) 6.22(.25) 6.50(.26) 7.21(.53) 1727.75(204.05) 370.00(60.85) ALL 5.25(.33) 6.25(.18) 5.67(.48) 7.25(.44) 1612.92(124.57) 444.33(72.35) Experiment Results - All participants improved the quality of writing - The ReviseO system got high perceived helpfulness and usefulness - No significant difference among three conditions post-questionnaire: 0~7 score: 0~100
85. Insight 1 Structured feedback helps filter information, identify weaknesses, and support reflection “This categorized feedback helps me identify my common mistakes easily! When I see the same type of writing issues appearing frequently, I understand that I need to pay more attention to this type of problem in my next writing. (P2)” 69 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
86. Flexible revision supports varying strategies and helps develop new revision strategies Insight 2 70 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion 25% 17% 33% 25% (3) (3) (4) (2) HML LMH ALL HLM
87. Separating feedback in a sequence may cause editing conflicts Insight 3 1) Mis-classified feedback leads to misunderstanding 2) Too many low-level fixes leads to decreased motivation for high-level improvement 71 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
88. Summary (1) Feedback Orchestration uses a rhetorical structure, meta- feedback, and flexible workflows to guide effective revision (2) Structured feedback helps identify weaknesses and support reflection (3) Flexible workflows help develop personalized revision strategies. (4) ReviseO support writers to think and revise structurally 72 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
89. Creative Work Feedback Provider Author Structural Feedback Feedback-Driven Revision Feedback Crowd Machine Crowdsourcing Workflow Data Annotations Revision Workflow Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback High Medium Low Feedback Collaboration 1 2 3 Structure Matters We use “structure” to guide feedback providers to contribute high-quality results. 1 2 3 We use “structure” to present feedback from different levels. We use “structure” to guide users to integrate feedback into revision !73 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
90. Iteration Quality Revision Effort individual’s ability !74 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
91. Iteration Quality Revision Effort individual’s ability !74 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
92. Iteration Quality Revision Effort individual’s ability !74 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
93. Iteration Quality Revision Effort Knowledge Gap individual’s ability !74 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
94. “We do not learn from experience… we learn from reflecting on experience” — John Dewey — !75 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
95. Creative Work User as Author Practice Review Collaboration Feedback Provider Feedback Feedback-Driven Revision Structural Feedback !76 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
96. Creative Work User as Author Practice Review Collaboration Feedback Provider Feedback Feedback-Driven Revision Structural Feedback Never-Ending Creative Learning !76 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion User as Learner Reflection Reflection Collaboration Creative Work Learning through Reflection
97. !77 Learning Professional Skills for Drawing Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
98. !78 The “knowledge” is in the details… Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
99. Leaner Creative Work Author Before/After-Practice Reflection Workflow Generate Learning Points learning points Identify Self-Explain Reflection Workflows Practice Extracting Learning Points for Drawing Support Identify Self-Explain Practice Identify Self-Explain Learn & Reflect Create Yi-Ching Huang, Jerry Yu-Heng Chan, and Jane Yung-jen Hsu. Reflection before/after practice: Learnersourcing for drawing support. In CHI ’18 Extended Abstracts on Human Factors in Computing Systems, 2018. !79 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
100. !80 drafting outlining details What is a learning point? 1.where is it? a clip of video/process 2.what is it? a description 3.why do you think it is important? a reason coloring with large area Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
101. - identify start and end point - describe what it is - explain why Reflection Workflow for Extracting Learning Points Identify Identify one learning point Self-Explain Explain why you choose it Why? start point end point !81 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
102. Reflection Practice !82 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
103. Reflection Reflection is defined as a purposeful thinking toward a goal. (Dewey, 1933) 1. Reflection-in-action 2. Reflection-on-action - refers to the monitoring and modification of actions during the learning process - refers to reasonable evaluation and strategic planning for improvement after the learning process. Reflection Practice (Schön, 1983) !83 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
104. Before/After-Practice Reflection Workflow Practice Learning by doing Identify Identify learning points Self-Explain Explain it is important helpful difficult interest Why? !84 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Before-annotation Reflection Practice Learning by doing Identify Identify learning points Self-Explain Explain it is important helpful difficult interest Why? After-annotation Reflection
105. Practice Learning by doing Identify Identify learning points Self-Explain Explain it is important helpful difficult interest Why? Identify Identify learning points Self-Explain Explain it is important helpful difficult interest Why? Pilot study: Before/After-Practice Reflection Workflow - 8 participants (5 male, 3 female), 20-23 years old - 80 annotations (48 learning points, 16 difficult points, 16 interest points) !85 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion - Learners discover new learning points or revise their previous findings after practice Lessons Learned - After-annotation augments before-annotation
106. !86 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion ䷱䋊螂ܹ璾牧眤憽氅ᜋӞ䍅Ӟ䍅吩Ӥ݄ጱ向ဩ盄蠐 (P5, before) Ӥᜋጱොဩ牧᩻ڊᇔ誢क़ᶎฎݢ犥ጱ牧磧盅ٚ硄ധ疰অ牧蝡䰬Ӥᜋ眤憽穉斃অӞ讨 (P3, after) ضय़膌向ڊ斪ୄ牧ٚୌ缏奞℄ (P1,before) አ܈ਁ斔ۗ娄向Ո腷 (P4, after) 硄ധ毣誧螲翣ጱ茐ᜋ㬵蕣蝨ط୽ጱ硳ຎ (P4,after) Reflective annotations help learners obtain new knowledge in different perspectives
107. !87 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Creative learning is never ending …
108. Feedback Provider Crowd Machine Crowdsourcing Workflow Data Annotations Expert Crowd Part I: Feedback Generation Feedback Author Revision Workflow Part II: Feedback Utilization Rhetorical Structure High-Level Medium-Level Low-Level Feedback High Medium Low Feedback Collaboration Learner Creative Work Extract Reflection Annotations Learn & Reflect Learning Part III: Learning through Reflection !88
109. Creative Work Author Practice Review Feedback-Driven Revision Collaboration Structural Feedback Feedback Provider Learner Reflection Feedback Learning through Reflection Reflection Collaboration Creative Work !89 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
110. Creative Work Author Practice Review Feedback-Driven Revision Collaboration Structural Feedback Feedback Provider Learner Reflection Feedback Learning through Reflection Reflection Collaboration Creative Work !89 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Never-Ending Creative Learning
111. Accomplishments - Two Intelligent Systems for Supporting Writing Revision - StructFeed for generating structural feedback - ReviseO for facilitating feedback-driven revision - Studies on understandings between feedback, revision, and writing quality - Workflows for collecting feedback, facilitate revision, and promote reflection - Unity & Coherence workflow - Flexible Revision workflows - Reflection-based workflows !90 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
112. !91 Contributions Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion - Never-Ending Creative Learning for support creative task solving and learning - Leverage structure to facilitate feedback generation and feedback-driven revision - Techniques for enabling collaborations among users, crowds, and machines
113. Future Directions - Effective collaborations between crowd and machine - Adaptive Feedback for supporting users with different levels - Never-Ending Creative Learning for Other Domains !92 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Crowd Machine
114. Thank You Q&A !93