Janet's phd defense @ 2018.06.04

Fc1b497e61d3adca89516740944ad1e5?s=47 janetyc
June 04, 2018

Janet's phd defense @ 2018.06.04

I passed my phd oral defense!!!! :D

Fc1b497e61d3adca89516740944ad1e5?s=128

janetyc

June 04, 2018
Tweet

Transcript

  1. Designing for Complex Creative Task Solving Yi-Ching (Janet) Huang 戔懯薹究蕦褾ጱ獺蝨௔犨率

    2018.06.04 PhD Oral Defense 讙௑覌 Advisor: Jane Yung-jen Hsu, PhD
  2. https://www.facebook.com/womaninthestriped/ Intro | Feedback Generation | Feedback Utilization | Learning

    thro Reflection | Conclusion !2
  3. A Rejection Letter !3

  4. Complex Creative Tasks Intro | Feedback Generation | Feedback Utilization

    | Learning thro Reflection | Conclusion !4
  5. human-centered design from IDEO A Creative Task as An Iterative

    Process Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !5
  6. Score: 2 Score: 2.5 Score: 2.75 1st version 2nd version

    3rd version Writing as an iterative process !6
  7. http://push.m-iti.org User Interface Design as An Iterative Process Intro |

    Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !7
  8. Complex Creative Process Uncertainty A Concrete Solution Intro | Feedback

    Generation | Feedback Utilization | Learning thro Reflection | Conclusion !8
  9. Properties of Creative Tasks 1. Open-ended and ill-defined 3. Quality

    is usually evaluated by multiple criteria 4. Quality can be improved by iterative refinement 2. Answer is not true or false, but how good the answer is Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !9
  10. Teevan, Iqbal, and von Veh. Supporting Collaborative Writing with Microtasks.

    CHI 2016. Sadauskas, Byrne, and Atkinson. Mining Memories: Designing a Platform to Support Social Media Based Writing. CHI 2015 Bernstein, Little, Miller, Hartmann, Ackerman, Karger, Crowell, and Panovich. Soylent: A Word Processor with a Crowd Inside. UIST 2010. Kim, Cheng, and Bernstein. Ensemble: Exploring Complementary Strengths of Leaders and Crowds in Creative Collaboration. CSCW 2014 Hahn, Chang, Kim, and Kittur. The Knowledge Accelerator: Big Picture Thinking in Small Pieces. CHI 2016. Nebeling, To, Guo, de Freitas, Teevan, Dow, and Bigham. WearWrite: Crowd-Assisted Writing from Smartwatches. CHI 2016. Kittur, Smus, Khamkar, and Kraut. CrowdForge: Crowdsourcing Complex Work. UIST 2011. Agapie, Teevan, and Monroy- Hernández. Crowdsourcing in the Field: A Case Study Using Local Crowds for Event Reporting. HCOMP 2015. Luther, Hahn, Dow, and Kittur. Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines. HCOMP 2015. 10
  11. (Boisson et al., 2013) Criterion (Burstein et al., 2004) (Chen

    et al., 2016) IEA(1997) (Yen et al., 2016) !11
  12. Prior Work Ideation Outlining Creation Revision Publishing CrowdLines 
 (Luther

    et al.,2015) MicroWriter 
 (Teevan et al.,2016) CrowdForge 
 (Kittur et al.,2011) Sparkfolio
 (Sadauskas et al.,2015) Ensemble
 (Kim et al.,2014) Soylent
 (Bernstei et al.,2010) Crowdsourcing in the Field 
 (Agapie et al.,2015) Knowledge Accelerator
 (Hahn et al.,2016) WearWrite (Nebeling et al.,2016) IntroAssist (Hui et al.,2018) MechanicalNovel
 (Kim et al.,2017) reflect and revise Writing Process WriteAhead (Chang and Chang, 2015) Rephraser 2.0 Linggle Knows (Chen et al., 2016) !12
  13. “The best writing is rewriting.” — E.B. White — !13

  14. Shortn: Text Shortening Crowdproof: Crowdsourced Proofreading Soylent: A Word Processor

    with a Crowd Inside (Bernstein et al., UIST’ 10) Applications M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, UIST '10, pages 313-322, New York, NY, USA, 2010. ACM. Microsoft Word Mechanical Turk Fix Verify Find Soylent select texts Find-Fix-Verify Workflow !14
  15. Does rewriting improve quality ? !15

  16. Iteration Quality The Benefits of Iteration Intro | Feedback Generation

    | Feedback Utilization | Learning thro Reflection | Conclusion !16
  17. Iteration Quality The Benefits of Iteration Intro | Feedback Generation

    | Feedback Utilization | Learning thro Reflection | Conclusion !16
  18. Iteration Quality The Benefits of Iteration o x Intro |

    Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !16
  19. Feedback Facilitates High Quality Results Evaluate the writing Improve the

    writing Feedback Work Iterative process Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !17 J. Hattie and H. Timperley. The power of feedback. Review of Educational Research, 77(1):81–112, March 2007.
  20. Author Learning Supporting Collaboration Feedback Provider Creative Task Solving Framework

    Feedback Creative Work Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !18
  21. Feedback Creative Work Author Learning Supporting Feedback Utilization Collaboration Feedback

    Generation Feedback Provider Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !19 Creative Task Solving Framework
  22. Outline - Introduction - Part I: Feedback Generation - Part

    II: Feedback Utilization - Part III: Learning through Reflection - Conclusion Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !20
  23. Part I: Feedback Generation Writing Feedback How do we generate

    effective feedback for supporting authors to improve the quality of writing? Supporting Feedback Provider Author Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !21
  24. Intro | Feedback Generation | Feedback Utilization | Learning thro

    Reflection | Conclusion 22
  25. Machine can help for correcting surface errors Intro | Feedback

    Generation | Feedback Utilization | Learning thro Reflection | Conclusion 23
  26. - Holistic Scoring: - provide diagnostic feedback on grammar, usage,

    and mechanics; style and diction; and organization and development - Templated-based feedback Criterion Feedback for the highest score “6” Feedback for the highest score “1” Template-based feedback Holistic Scoring Jill Burstein, Martin Chodorow, and Claudia Leacock. Automated essay evaluation: The criterion online writing service. AI Magazine, 25(3):27–36, 2004. (Burstein et al., 2004) Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion !24
  27. Disadvantage of Existing Feedback Systems -Require large amounts of labeled

    data -Support limited topics -Static feedback template !25 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  28. Experts Peers Crowds Where can we get feedback? !26 Intro

    | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  29. Current Rewriting Support Tools Rewriting Feedback global + local Local

    issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  30. Current Rewriting Support Tools Rewriting Feedback global + local Spelling

    checker Grammar checker sentence word Local issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  31. Current Rewriting Support Tools Rewriting Feedback global + local Free

    Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  32. Current Rewriting Support Tools Rewriting Feedback global + local Free

    Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue !27 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  33. Writer Reader Structure helps deliver message to a reader Idea

    !28 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  34. English Oriental (Kaplan, 1966) Rhetorical Patterns of Different Languages Robert

    B. Kaplan. Cultural thought patterns in inter-cultural education. Language Learning, 1966. !29 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  35. Author Feedback Writing Crowd Machine Supporting Writing Revision by Crowdsourced

    Structural Feedback Revision Crowdsourcing Workflow Data Annotations StructFeed Yi-Ching Huang, Jiunn-Chia Huang, and Jane Yung-jen Hsu. Supporting ESL writing by prompting crowdsourced structural feedback. In Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017) Yi-Ching Huang, Hao-Chuan Wang, and Jane Yung-jen Hsu. Bridging learning gap in writing education with a crowd-powered system. CHI 2017 Workshop on Designing for Curiosity, Denver, Colorado, USA, 2017. !30 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  36. Topic Sentence Supporting Sentence Concluding Sentence Introduction Body Conclusion Essay

    Structure Paragraph Structure paragraph paragraph paragraph Key point Supporting Sentence Supporting Sentence !31 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  37. A paragraph is a group of sentences organized around a

    central topic. Four Key Elements of Great Writing Element #1: Unity Element #2: Order Element #3: Coherence Element #4: Completeness All sentences in a paragraph should speak about one single idea or one main subject. Order refers to the way you organize your supporting sentences. Sentences within a paragraph need to connect to each other and work together as a whole. Completeness means a paragraph is well-developed. !32 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Holly L. Jacobs, Stephen A. Zinkgraf, Deanna R. Wormuth, V. Faye Hartfiel, and Jane B. Hughey. Testing ESL Composition: A Practical Approach. Newbury House, 1981.
  38. 1. All sub-points centering on one central idea 2. Using

    no irrelevant sentences Key points to achieve unity: Topic Sentence Supporting Sentence #1 Concluding Sentence related to the topic sentence Supporting Sentence #2 Supporting Sentence #3 The First Key Element of a Great Writing - All sentences in a paragraph should speak about one single idea. Unity !33 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  39. Crowdsourcing Workflow Structural Feedback Unity Identification Writing Criteria 1. multiple

    topic issue 2. missing topic issue 3. irrelevance issue Topic sentence prediction Irrelevant sentence prediction Crowd Annotations System Overview of StructFeed Topic sentence annotation Relevant keyword annotation !34 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  40. Crowdsourcing Workflow for Unity Identification Topic Identify topic sentence topic

    + ideas Crowdsourcing Workflow Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation !35 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  41. Topic Task - identify topic sentence Quality Control - native

    speakers as workers - brief explanation of concept - worked example - annotate sentence by click !36 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  42. Topic Task - identify topic sentence Quality Control - native

    speakers as workers - brief explanation of concept - worked example - annotate sentence by click Explanation Worked example Working area annotate sentence by click !36 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  43. Relevance Task annotate word by click !37 Intro | Feedback

    Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  44. Relevance Task Worked example annotate word by click !37 Intro

    | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  45. Topic Identify topic sentence topic + ideas Crowdsourcing Workflow Relevance

    Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation Structural Feedback Writing Criteria 1. multiple topic issue 2. missing topic issue 3. irrelevance issue Crowd Annotations Unity Identification Topic sentence prediction Irrelevant sentence prediction !38 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  46. Effective Feedback 1. Obtain a concept of the standard or

    goal 2. Compare the actual level of performance with the standard 3. Engage in action which leads to closure of the gap (Sadler, D. R. 1989) D. R. Sadler. Formative assessment and the design of instructional systems. Instructional Science, 18(2):119{144, 1989 !39 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  47. Structural Feedback !40 Intro | Feedback Generation | Feedback Utilization

    | Learning thro Reflection | Conclusion
  48. Structural Feedback Feedback Summary - type of issue - suggested

    action !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  49. Rhetorical Visualization Structural Feedback Feedback Summary - type of issue

    - suggested action !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  50. Rhetorical Visualization Structural Feedback Feedback Summary - type of issue

    - suggested action - topic sentence !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  51. Rhetorical Visualization Structural Feedback Feedback Summary - type of issue

    - suggested action - topic sentence - irrelevant sentence !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  52. Rhetorical Visualization Structural Feedback Feedback Summary - type of issue

    - suggested action - topic sentence - irrelevant sentence - relevant keywords !40 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  53. !41

  54. !41

  55. Field Experiment on ESL Writers - 18 self-motivated ESL learners

    (8 females, 10 males) - 19~34 years old - A between subjects study Conditions - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed Writing original version R Rewriting revised version R’ Feedback Measure - time, quantity, cost - quality improvement (R’-R) - perceived helpfulness !42 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  56. Expert Feedback Crowd Feedback StructFeed -Diff-rating: 0.29 (.43) -Time: 1~2

    days -Cost: $16 -Quantity: 55.44 suggestions - # of equal rating: 1 - # of decreased rating: 1 - Diff-rating: 0.38 (.44) - Time: 10~30 mins - Cost: $2 - Quantity: 8.11 suggestions - # of equal rating: 1 - # of decreased rating: 1 -Diff-rating: 0.54 (.25) -Time: 1~5 hrs -Costs: $1~1.7 -15-25 workers -All participants improve the quality of writing ! Field Experiment Results !43 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  57. “I don’t understand what he means. His comments contain difficult

    terminology and it’s hard for me to capture the key points.” (P15) Observation I Knowledge gaps between an expert and a novice writer !44 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  58. Structural feedback promotes self-reflection “I’m so surprised that no one

    annotates it as relevant keywords. I originally think that is a common example for other people. But, I am wrong. I will carefully choose a more common and understandable example to describe my idea next time.” (P7) Observation 2 !45 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  59. Crowd Machine Crowdsourcing Workflow Data Annotations Expert Crowd Iterative Revision

    Process Author Feedback Writing Feedback Feedback !46 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  60. Writing Rewriting Feedback 1 Rewriting Feedback 2 Rewriting Feedback 3

    v1 v2 v3 v4 Writing Iteration Experiment - 18 self-motivate ESL learners (8 females, 10 males) - 19~34 years old - A within-subjects counter-balanced design Conditions - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed !47 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  61. Experiment Results Red: Grade decreased Green: Grade increase White: No

    improvement Expert feedback (C1) Crowd feedback (C2) StructFeed (C3) Avg diff rating 0.15 0.21 0.43 Avg standard deviation 0.32 0.32 0.44 # of decreased diff rating 3 2 0 # of equal rating 7 7 5 !48 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  62. Red: Grade decreased Green: Grade increase White: No improvement Expert

    feedback (C1) Crowd feedback (C2) StructFeed (C3) Ensemble Avg diff rating 0.15 0.21 0.43 0.79 Avg standard deviation 0.32 0.32 0.44 0.45 # of decreased diff rating 3 2 0 0 # of equal rating 7 7 5 1 Experiment Results !49 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  63. Different types of feedback support writers in different perspectives “All

    types of feedback are useful to me. I’d like to use them for different purpose or at different stage. For example, I will use StructFeed at the beginning, then crowd feedback. If I need to write SOP, I’ll use expert feedback for final check.” (P5) Insight !50 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Expert Crowd Crowd Machine StructFeed ҄ ҄ Ensemble feedback supports writers in different perspectives
  64. (1) We designed a crowd-powered system that enables structural feedback

    for supporting ESL writing (2) We leveraged domain rubrics in designing crowdsourcing workflow (3) StructFeed outperformed free-form feedback from both experts and crowd (4) Ensemble feedback may support writers in different perspectives Summary !51 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  65. Learning Part II: Feedback Utilization Writing Feedback Feedback Provider Author

    How do we support authors to integrate feedback into revisions and facilitate high-quality outcome? Revision !52 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  66. Evaluate the writing Improve the writing Feedback Work Iterative process

    Good feedback NOT always facilitates good results! !53 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  67. Too many suggestions may cause problems 1. Information overload 2.

    Cost of task switching 3. People focus on easier problems !54 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  68. Formative Study 6 participants (1 female) with the age 18-23

    - Novices deal with feedback in an “unstructured” way - Varying revision strategies (1) browse all comments (2) group similar comments (3) deal with comments in a sequence (1) beginning-to-end editing (2) specificity-first editing (3) high-to-low editing (4) low-to-high editing novice writers !55 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  69. Expert Revision Practice Novice writers Expert writers ESL writers revise

    in a linear process revise in a recursive way revise in a disorganized way Expert writers think high-level goals, and break it into low-level steps !56 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion A Lack of Reflection
  70. Challenges of ESL Revision ESL students lack abilities in a

    foreign language - metarhetorical awareness (knowledge of themselves as writers) - metastrategic awareness (knowledge of their own personality type and its influence on their writing behaviors, including revision) - metalinguistic awareness (terminology to discuss language issues) (Alice S. Horning, 2006) !57 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion A Lack of Awareness
  71. Goals 1. Support awareness and reflection 2. Guide writers to

    think and revise “structurally” !58 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  72. Author Feedback Provider Feedback Revision Workflow Writing High Medium Low

    Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  73. Author Feedback Provider Feedback Revision Workflow Writing High Medium Low

    Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision 1. get feedback Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  74. Author Feedback Provider Feedback Revision Workflow Writing High Medium Low

    Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision 1. get feedback 2. classify feedback Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  75. Author Feedback Provider Feedback Revision Workflow Writing High Medium Low

    Feedback Feedback Orchestration Structuring Feedback for Promoting Reflection and Awareness in Revision 1. get feedback 2. classify feedback 3. revise an article in a revision workflow Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback !59 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  76. 1) Rhetorical structure 2) Meta-feedback 3) Flexible revision workflow Three

    Design Considerations Author Feedback Provider Feedback Revision Workflow Writing High Medium Low Feedback w/ meta-feedback Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback content organization grammar mechanics language !60 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  77. Feedback Type Definition Examples High Content Content refers to the

    substance of writing. It includes topic sentence expressing main argument and supporting ideas. e.g. unity of argument, supporting idea, relevant example, addresses the question, etc. Organization Organization refers to the logical organization of the content. e.g. coherence of the content, relation between sentences, logical sequencing, etc. Medium Vocabulary Vocabulary refers to the selection or words those are suitable with the content. e.g. word choice, etc. Language Use Language Use refers to the use of the correct grammatical forms and syntactical pattern. e.g. fixing grammatical errors, or paraphrasing, shortening, etc. Low Mechanics Mechanic refers to all the arbitrary technical stuff in writing like spelling, capitalization, punctuation, etc. e.g. spelling errors, punctuation, capitalization, format, etc. Feedback Classification (Jacobs et al.,1981) !61 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  78. Automated Classification content organization vocabulary grammar mechanics Feedback Mechanics Vocabulary

    Grammar Rule-based classifier Comment Edit Feedback punctuation, spelling, capitalization stemming Non-Mechanics High-level Medium-level Low-level !62 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  79. Flexible Revision Workflow High-Level Medium-Level Low-Level content organization vocabulary grammar

    mechanics mechanics vocabulary gram m ar organization content organization content vocabulary gram m ar mechanics Sequential Workflow Concurrent Workflow high-to-low (HML) low-to-high (LMH) vocabulary gram m ar mechanics content organization all (ALL) or !63 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  80. Rhetorical Structure content organization vocabulary grammar mechanics Feedback Rhetorical Categories

    Feedback Classify High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Collect Structure Crowdsourcing Workflows !64 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  81. ReviseO ReviseO !65 Intro | Feedback Generation | Feedback Utilization

    | Learning thro Reflection | Conclusion
  82. High-to-low editing Low-to-high editing Mechanics Feedback Language Feedback Content &

    Organization Feedback High-level Middle-level Low-level !66
  83. Experiment Design High-to-low (HML) Low-to-high (LMH) High - a within-subjects,

    counterbalanced experiment design - 12 self-motivated non-native writers - each participants performed 3 rewriting tasks with different topics - 3 experimental conditions - (1) show feedback together (ALL) - (2) show feedback sequentially from high to low (HML) - (3) show feedback sequentially from low to high (LMH) Together (ALL) Medium Low High Medium Low ALL !67 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  84. Intro | Feedback Generation | Feedback Utilization | Learning thro

    Reflection | Conclusion RevisO System Categorization Writing Quality Revision Effort Ease of use
 M(SE) Helpfulness
 M(SE) Helpfulness
 M(SE) Diff of Rating
 M(SE) Time spent
 M(SE) Edit distance
 M(SE) LMH 5.58(.43) 6.22(.21) 6.58(.23) 6.79(.88) 1731.58(147.81) 353.33(.68) HML 4.92(.47) 6.22(.25) 6.50(.26) 7.21(.53) 1727.75(204.05) 370.00(60.85) ALL 5.25(.33) 6.25(.18) 5.67(.48) 7.25(.44) 1612.92(124.57) 444.33(72.35) Experiment Results - All participants improved the quality of writing - The ReviseO system got high perceived helpfulness and usefulness - No significant difference among three conditions post-questionnaire: 0~7 score: 0~100
  85. Insight 1 Structured feedback helps filter information, identify weaknesses, and

    support reflection “This categorized feedback helps me identify my common mistakes easily! When I see the same type of writing issues appearing frequently, I understand that I need to pay more attention to this type of problem in my next writing. (P2)” 69 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  86. Flexible revision supports varying strategies and helps develop new revision

    strategies Insight 2 70 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion 25% 17% 33% 25% (3) (3) (4) (2) HML LMH ALL HLM
  87. Separating feedback in a sequence may cause editing conflicts Insight

    3 1) Mis-classified feedback leads to misunderstanding 2) Too many low-level fixes leads to decreased motivation for high-level improvement 71 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  88. Summary (1) Feedback Orchestration uses a rhetorical structure, meta- feedback,

    and flexible workflows to guide effective revision (2) Structured feedback helps identify weaknesses and support reflection (3) Flexible workflows help develop personalized revision strategies. (4) ReviseO support writers to think and revise structurally 72 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  89. Creative Work Feedback Provider Author Structural Feedback Feedback-Driven Revision Feedback

    Crowd Machine Crowdsourcing Workflow Data Annotations Revision Workflow Rhetorical Structure High-Level Medium-Level Low-Level content organization vocabulary grammar mechanics Feedback High Medium Low Feedback Collaboration 1 2 3 Structure Matters We use “structure” to guide feedback providers to contribute high-quality results. 1 2 3 We use “structure” to present feedback from different levels. We use “structure” to guide users to integrate feedback into revision !73 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  90. Iteration Quality Revision Effort individual’s ability !74 Intro | Feedback

    Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  91. Iteration Quality Revision Effort individual’s ability !74 Intro | Feedback

    Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  92. Iteration Quality Revision Effort individual’s ability !74 Intro | Feedback

    Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  93. Iteration Quality Revision Effort Knowledge Gap individual’s ability !74 Intro

    | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  94. “We do not learn from experience… we learn from reflecting

    on experience” — John Dewey — !75 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  95. Creative Work User as Author Practice Review Collaboration Feedback Provider

    Feedback Feedback-Driven Revision Structural Feedback !76 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  96. Creative Work User as Author Practice Review Collaboration Feedback Provider

    Feedback Feedback-Driven Revision Structural Feedback Never-Ending Creative Learning !76 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion User as Learner Reflection Reflection Collaboration Creative Work Learning through Reflection
  97. !77 Learning Professional Skills for Drawing Intro | Feedback Generation

    | Feedback Utilization | Learning thro Reflection | Conclusion
  98. !78 The “knowledge” is in the details… Intro | Feedback

    Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  99. Leaner Creative Work Author Before/After-Practice Reflection Workflow Generate Learning Points

    learning points Identify Self-Explain Reflection Workflows Practice Extracting Learning Points for Drawing Support Identify Self-Explain Practice Identify Self-Explain Learn & Reflect Create Yi-Ching Huang, Jerry Yu-Heng Chan, and Jane Yung-jen Hsu. Reflection before/after practice: Learnersourcing for drawing support. In CHI ’18 Extended Abstracts on Human Factors in Computing Systems, 2018. !79 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  100. !80 drafting outlining details What is a learning point? 1.where

    is it? a clip of video/process 2.what is it? a description 3.why do you think it is important? a reason coloring with large area Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  101. - identify start and end point - describe what it

    is - explain why Reflection Workflow for Extracting Learning Points Identify Identify one learning point Self-Explain Explain why you choose it Why? start point end point !81 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  102. Reflection Practice !82 Intro | Feedback Generation | Feedback Utilization

    | Learning thro Reflection | Conclusion
  103. Reflection Reflection is defined as a purposeful thinking toward a

    goal. (Dewey, 1933) 1. Reflection-in-action 2. Reflection-on-action - refers to the monitoring and modification of actions during the learning process - refers to reasonable evaluation and strategic planning for improvement after the learning process. Reflection Practice (Schön, 1983) !83 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  104. Before/After-Practice Reflection Workflow Practice Learning by doing Identify Identify learning

    points Self-Explain Explain it is important helpful difficult interest Why? !84 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Before-annotation Reflection Practice Learning by doing Identify Identify learning points Self-Explain Explain it is important helpful difficult interest Why? After-annotation Reflection
  105. Practice Learning by doing Identify Identify learning points Self-Explain Explain

    it is important helpful difficult interest Why? Identify Identify learning points Self-Explain Explain it is important helpful difficult interest Why? Pilot study: Before/After-Practice Reflection Workflow - 8 participants (5 male, 3 female), 20-23 years old - 80 annotations (48 learning points, 16 difficult points, 16 interest points) !85 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion - Learners discover new learning points or revise their previous findings after practice Lessons Learned - After-annotation augments before-annotation
  106. !86 Intro | Feedback Generation | Feedback Utilization | Learning

    thro Reflection | Conclusion ䷱䋊螂ܹ璾牧眤憽氅ᜋӞ䍅Ӟ䍅吩Ӥ݄ጱ向ဩ盄蠐 (P5, before) Ӥᜋጱොဩ牧᩻ڊᇔ誢क़ᶎฎݢ犥ጱ牧磧盅ٚ硄ധ疰অ牧蝡䰬Ӥᜋ眤憽穉斃অӞ讨 (P3, after) ضय़膌向ڊ斪ୄ牧ٚୌ缏奞℄ (P1,before) አ܈ਁ斔ۗ娄向Ո腷 (P4, after) 硄ധ毣誧螲翣ጱ茐ᜋ㬵蕣蝨ط୽ጱ硳ຎ (P4,after) Reflective annotations help learners obtain new knowledge in different perspectives
  107. !87 Intro | Feedback Generation | Feedback Utilization | Learning

    thro Reflection | Conclusion Creative learning is never ending …
  108. Feedback Provider Crowd Machine Crowdsourcing Workflow Data Annotations Expert Crowd

    Part I: Feedback Generation Feedback Author Revision Workflow Part II: Feedback Utilization Rhetorical Structure High-Level Medium-Level Low-Level Feedback High Medium Low Feedback Collaboration Learner Creative Work Extract Reflection Annotations Learn & Reflect Learning Part III: Learning through Reflection !88
  109. Creative Work Author Practice Review Feedback-Driven Revision Collaboration Structural Feedback

    Feedback Provider Learner Reflection Feedback Learning through Reflection Reflection Collaboration Creative Work !89 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  110. Creative Work Author Practice Review Feedback-Driven Revision Collaboration Structural Feedback

    Feedback Provider Learner Reflection Feedback Learning through Reflection Reflection Collaboration Creative Work !89 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Never-Ending Creative Learning
  111. Accomplishments - Two Intelligent Systems for Supporting Writing Revision -

    StructFeed for generating structural feedback - ReviseO for facilitating feedback-driven revision - Studies on understandings between feedback, revision, and writing quality - Workflows for collecting feedback, facilitate revision, and promote reflection - Unity & Coherence workflow - Flexible Revision workflows - Reflection-based workflows !90 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
  112. !91 Contributions Intro | Feedback Generation | Feedback Utilization |

    Learning thro Reflection | Conclusion - Never-Ending Creative Learning for support creative task solving and learning - Leverage structure to facilitate feedback generation and feedback-driven revision - Techniques for enabling collaborations among users, crowds, and machines
  113. Future Directions - Effective collaborations between crowd and machine -

    Adaptive Feedback for supporting users with different levels - Never-Ending Creative Learning for Other Domains !92 Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion Crowd Machine
  114. Thank You Q&A !93