Janet's phd defense @ 2018.06.04

I passed my phd oral defense!!!! :D

Transcript

1. Designing for Complex
   Creative Task Solving
   Yi-Ching (Janet) Huang
   戔懯薹究蕦褾ጱ獺蝨௔犨率
   2018.06.04
   PhD Oral Defense
   讙௑覌
   Advisor: Jane Yung-jen Hsu, PhD
   

   View Slide

2. https://www.facebook.com/womaninthestriped/
   Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
   !2
   

   View Slide

3. A Rejection Letter
   !3
   

   View Slide

4. Complex Creative Tasks
   Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
   !4
   

   View Slide

5. human-centered design from IDEO
   A Creative Task as An Iterative Process
   Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
   !5
   

   View Slide

6. Score: 2
   Score: 2.5
   Score: 2.75
   1st version
   2nd version
   3rd version
   Writing as an iterative process
   !6
   

   View Slide

7. http://push.m-iti.org
   User Interface Design as An Iterative Process
   Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
   !7
   

   View Slide

8. Complex Creative Process
   Uncertainty A Concrete Solution
   Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
   !8
   

   View Slide

9. Properties of Creative Tasks
   1. Open-ended and ill-defined
   3. Quality is usually evaluated by multiple criteria
   4. Quality can be improved by iterative refinement
   2. Answer is not true or false, but how good the
   answer is
   Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
   !9
   

   View Slide

10. Teevan, Iqbal, and von Veh.
    Supporting Collaborative Writing with
    Microtasks. CHI 2016.
    Sadauskas, Byrne, and Atkinson.
    Mining Memories: Designing a
    Platform to Support Social Media
    Based Writing. CHI 2015
    Bernstein, Little, Miller,
    Hartmann, Ackerman,
    Karger, Crowell, and
    Panovich. Soylent: A
    Word Processor with a
    Crowd Inside. UIST 2010.
    Kim, Cheng, and Bernstein.
    Ensemble: Exploring
    Complementary Strengths of
    Leaders and Crowds in Creative
    Collaboration. CSCW 2014
    Hahn, Chang, Kim, and Kittur. The
    Knowledge Accelerator: Big Picture
    Thinking in Small Pieces. CHI 2016.
    Nebeling, To, Guo, de Freitas, Teevan,
    Dow, and Bigham. WearWrite:
    Crowd-Assisted Writing from
    Smartwatches. CHI 2016.
    Kittur, Smus, Khamkar,
    and Kraut. CrowdForge:
    Crowdsourcing Complex
    Work. UIST 2011.
    Agapie, Teevan, and Monroy-
    Hernández. Crowdsourcing in
    the Field: A Case Study Using
    Local Crowds for Event
    Reporting. HCOMP 2015.
    Luther, Hahn, Dow, and Kittur.
    Crowdlines: Supporting
    Synthesis of Diverse Information
    Sources through Crowdsourced
    Outlines. HCOMP 2015.
    10
    

    View Slide

11. (Boisson et al., 2013)
    Criterion
    (Burstein et al., 2004)
    (Chen et al., 2016)
    IEA(1997)
    (Yen et al., 2016)
    !11
    

    View Slide

12. Prior Work
    Ideation Outlining Creation Revision Publishing
    CrowdLines 

    (Luther et al.,2015)
    MicroWriter 

    (Teevan et al.,2016)
    CrowdForge 

    (Kittur et al.,2011)
    Sparkfolio

    (Sadauskas et al.,2015)
    Ensemble

    (Kim et al.,2014)
    Soylent

    (Bernstei et al.,2010)
    Crowdsourcing in the Field 

    (Agapie et al.,2015)
    Knowledge Accelerator

    (Hahn et al.,2016)
    WearWrite
    (Nebeling et al.,2016)
    IntroAssist
    (Hui et al.,2018)
    MechanicalNovel

    (Kim et al.,2017)
    reflect and revise
    Writing Process
    WriteAhead
    (Chang and Chang, 2015)
    Rephraser 2.0
    Linggle Knows
    (Chen et al., 2016)
    !12
    

    View Slide

13. “The best writing is rewriting.”
    
    — E.B. White —
    !13
    

    View Slide

14. Shortn: Text Shortening
    Crowdproof: Crowdsourced Proofreading
    Soylent: A Word Processor with a Crowd Inside
    (Bernstein et al., UIST’ 10)
    Applications
    M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside.
    In Proceedings of the 23nd annual ACM symposium on User interface software and technology, UIST '10, pages 313-322, New York, NY, USA, 2010. ACM.
    Microsoft Word Mechanical Turk
    Fix
    Verify
    Find
    Soylent
    select texts
    Find-Fix-Verify Workflow
    !14
    

    View Slide

15. Does rewriting improve quality ?
    !15
    

    View Slide

16. Iteration
    Quality
    The Benefits of Iteration
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    !16
    

    View Slide

17. Iteration
    Quality
    The Benefits of Iteration
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    !16
    

    View Slide

18. Iteration
    Quality
    The Benefits of Iteration
    o
    x
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    !16
    

    View Slide

19. Feedback Facilitates High Quality Results
    Evaluate the writing
    Improve the writing
    Feedback
    Work
    Iterative process
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    !17
    J. Hattie and H. Timperley. The power of feedback. Review of Educational Research, 77(1):81–112, March 2007.
    

    View Slide

20. Author
    Learning
    Supporting
    Collaboration
    Feedback
    Provider
    Creative Task Solving Framework
    Feedback
    Creative Work
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    !18
    

    View Slide

21. Feedback
    Creative Work
    Author
    Learning
    Supporting
    Feedback Utilization
    Collaboration
    Feedback Generation
    Feedback
    Provider
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    !19
    Creative Task Solving Framework
    

    View Slide

22. Outline
    - Introduction
    
    - Part I: Feedback Generation
    
    - Part II: Feedback Utilization
    
    - Part III: Learning through Reflection
    
    - Conclusion
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    !20
    

    View Slide

23. Part I: Feedback Generation
    Writing
    Feedback
    How do we generate effective feedback for supporting
    authors to improve the quality of writing?
    Supporting
    Feedback
    Provider
    Author
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    !21
    

    View Slide

24. Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    22
    

    View Slide

25. Machine can help for correcting surface errors
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    23
    

    View Slide

26. - Holistic Scoring:
    
    - provide diagnostic feedback on grammar, usage, and mechanics;
    style and diction; and organization and development
    
    - Templated-based feedback
    Criterion
    Feedback for the highest score “6”
    Feedback for the highest score “1”
    Template-based feedback
    Holistic Scoring
    Jill Burstein, Martin Chodorow, and Claudia Leacock. Automated essay evaluation: The criterion online writing service. AI Magazine, 25(3):27–36, 2004.
    (Burstein et al., 2004)
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    !24
    

    View Slide

27. Disadvantage of Existing Feedback Systems
    -Require large amounts of labeled data
    
    -Support limited topics
    
    -Static feedback template
    !25
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

28. Experts Peers Crowds
    Where can we get feedback?
    !26
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

29. Current Rewriting Support Tools
    Rewriting
    Feedback
    global + local
    Local issue
    Global issue
    !27
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

30. Current Rewriting Support Tools
    Rewriting
    Feedback
    global + local
    Spelling checker
    Grammar checker sentence
    word
    Local issue
    Global issue
    !27
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

31. Current Rewriting Support Tools
    Rewriting
    Feedback
    global + local
    Free Comment idea
    Organization checker structure
    Spelling checker
    Grammar checker sentence
    word
    Local issue
    Global issue
    !27
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

32. Current Rewriting Support Tools
    Rewriting
    Feedback
    global + local
    Free Comment idea
    Organization checker structure
    Spelling checker
    Grammar checker sentence
    word
    Local issue
    Global issue
    !27
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

33. Writer
    Reader
    Structure helps deliver message to a reader
    Idea
    !28
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

34. English Oriental
    (Kaplan, 1966)
    Rhetorical Patterns of Different Languages
    Robert B. Kaplan. Cultural thought patterns in inter-cultural education. Language Learning, 1966. !29
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

35. Author
    Feedback
    Writing
    Crowd Machine
    Supporting Writing Revision by Crowdsourced Structural Feedback
    Revision
    Crowdsourcing Workflow
    Data Annotations
    StructFeed
    Yi-Ching Huang, Jiunn-Chia Huang, and Jane Yung-jen Hsu. Supporting ESL writing by prompting crowdsourced structural feedback. In Proceedings
    of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017)
    Yi-Ching Huang, Hao-Chuan Wang, and Jane Yung-jen Hsu. Bridging learning gap in writing education with a crowd-powered system. CHI 2017
    Workshop on Designing for Curiosity, Denver, Colorado, USA, 2017.
    !30
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

36. Topic Sentence
    Supporting Sentence
    Concluding Sentence
    Introduction
    Body
    Conclusion
    Essay Structure Paragraph Structure
    paragraph
    paragraph
    paragraph
    Key point
    Supporting Sentence
    Supporting Sentence
    !31
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

37. A paragraph is a group of sentences organized around a
    central topic.
    Four Key Elements of Great Writing
    Element #1: Unity
    Element #2: Order
    Element #3: Coherence
    Element #4: Completeness
    All sentences in a paragraph should speak about one single idea or one main
    subject.
    Order refers to the way you organize your supporting sentences.
    Sentences within a paragraph need to connect to each other and work
    together as a whole.
    Completeness means a paragraph is well-developed.
    !32
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    Holly L. Jacobs, Stephen A. Zinkgraf, Deanna R. Wormuth, V. Faye Hartfiel, and Jane B. Hughey. Testing ESL
    Composition: A Practical Approach. Newbury House, 1981.
    

    View Slide

38. 1. All sub-points centering on one central idea
    2. Using no irrelevant sentences
    Key points to achieve unity:
    Topic Sentence
    Supporting Sentence #1
    Concluding Sentence
    related to the topic sentence
    Supporting Sentence #2
    Supporting Sentence #3
    The First Key Element of a Great Writing
    - All sentences in a paragraph should speak about one single idea.
    Unity
    !33
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

39. Crowdsourcing Workflow
    Structural Feedback
    Unity Identification
    Writing Criteria
    1. multiple topic issue
    2. missing topic issue
    3. irrelevance issue
    Topic sentence prediction Irrelevant sentence prediction
    Crowd Annotations
    System Overview of StructFeed
    Topic sentence annotation Relevant keyword annotation
    !34
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

40. Crowdsourcing Workflow for
    Unity Identification
    Topic
    Identify topic sentence
    topic + ideas
    Crowdsourcing Workflow
    Relevance
    Highlight the relevant
    words between two
    sentences
    relevance
    topic
    Filter
    Filter paragraphs
    with no topic
    sentence
    (weight>=2)
    Topic sentence annotation Relevant keyword annotation
    !35
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

41. Topic Task
    - identify topic sentence
    Quality Control
    - native speakers as workers
    
    - brief explanation of concept
    
    - worked example
    
    - annotate sentence by click
    !36
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

42. Topic Task
    - identify topic sentence
    Quality Control
    - native speakers as workers
    
    - brief explanation of concept
    
    - worked example
    
    - annotate sentence by click
    Explanation
    Worked example
    Working area
    annotate sentence by click
    !36
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

43. Relevance Task
    annotate word by click
    !37
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

44. Relevance Task
    Worked example
    annotate word by click
    !37
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

45. Topic
    Identify topic sentence
    topic + ideas
    Crowdsourcing Workflow
    Relevance
    Highlight the relevant
    words between two
    sentences
    relevance
    topic
    Filter
    Filter paragraphs
    with no topic
    sentence
    (weight>=2)
    Topic sentence annotation Relevant keyword annotation
    Structural Feedback
    Writing Criteria
    1. multiple topic issue
    2. missing topic issue
    3. irrelevance issue
    Crowd Annotations
    Unity Identification
    Topic sentence prediction Irrelevant sentence prediction
    !38
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

46. Effective Feedback
    1. Obtain a concept of the standard or goal
    
    2. Compare the actual level of performance with the
    standard
    
    3. Engage in action which leads to closure of the gap
    (Sadler, D. R. 1989)
    D. R. Sadler. Formative assessment and the design of instructional systems. Instructional Science, 18(2):119{144, 1989 !39
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

47. Structural Feedback
    !40
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

48. Structural Feedback
    Feedback Summary
    - type of issue
    
    - suggested action
    !40
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

49. Rhetorical Visualization
    Structural Feedback
    Feedback Summary
    - type of issue
    
    - suggested action
    !40
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

50. Rhetorical Visualization
    Structural Feedback
    Feedback Summary
    - type of issue
    
    - suggested action
    - topic sentence
    !40
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

51. Rhetorical Visualization
    Structural Feedback
    Feedback Summary
    - type of issue
    
    - suggested action
    - topic sentence
    - irrelevant sentence
    !40
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

52. Rhetorical Visualization
    Structural Feedback
    Feedback Summary
    - type of issue
    
    - suggested action
    - topic sentence
    - irrelevant sentence
    - relevant keywords
    !40
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

53. !41
    

    View Slide

54. !41
    

    View Slide

55. Field Experiment on ESL Writers
    - 18 self-motivated ESL learners (8 females, 10 males)
    
    - 19~34 years old
    
    - A between subjects study
    Conditions
    - C1 (expert feedback): free-form feedback from an expert
    
    - C2 (crowd feedback): free-form feedback from a crowd worker
    
    - C3 (structural feedback): structural feedback from StructFeed
    Writing
    original version R
    Rewriting
    revised version R’
    Feedback
    Measure
    - time, quantity, cost
    
    - quality improvement (R’-R)
    
    - perceived helpfulness
    !42
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

56. Expert Feedback
    Crowd Feedback
    StructFeed
    -Diff-rating: 0.29 (.43)
    -Time: 1~2 days
    -Cost: $16
    -Quantity: 55.44 suggestions
    - # of equal rating: 1
    - # of decreased rating: 1
    - Diff-rating: 0.38 (.44)
    - Time: 10~30 mins
    - Cost: $2
    - Quantity: 8.11 suggestions
    - # of equal rating: 1
    - # of decreased rating: 1
    -Diff-rating: 0.54 (.25)
    -Time: 1~5 hrs
    -Costs: $1~1.7
    -15-25 workers
    -All participants
    improve the quality of
    writing
    !
    Field Experiment Results
    !43
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

57. “I don’t understand what he means. His comments contain difficult
    terminology and it’s hard for me to capture the key points.” (P15)
    Observation I
    Knowledge gaps between an expert and a novice writer
    !44
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

58. Structural feedback promotes self-reflection
    “I’m so surprised that no one annotates it as relevant keywords.
    I originally think that is a common example for other people.
    But, I am wrong. I will carefully choose a more common and
    understandable example to describe my idea next time.” (P7)
    
    Observation 2
    !45
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

59. Crowd Machine
    Crowdsourcing Workflow
    Data Annotations
    Expert
    Crowd
    Iterative Revision Process
    Author
    Feedback
    Writing
    Feedback
    Feedback
    !46
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

60. Writing Rewriting
    Feedback 1 Rewriting
    Feedback 2 Rewriting
    Feedback 3
    v1 v2 v3 v4
    Writing Iteration Experiment
    - 18 self-motivate ESL learners (8 females, 10 males)
    
    - 19~34 years old
    
    - A within-subjects counter-balanced design
    Conditions
    - C1 (expert feedback): free-form feedback from an expert
    
    - C2 (crowd feedback): free-form feedback from a crowd worker
    
    - C3 (structural feedback): structural feedback from StructFeed
    !47
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

61. Experiment Results
    Red: Grade decreased
    Green: Grade increase
    White: No improvement
    Expert feedback
    (C1)
    Crowd feedback
    
    (C2)
    StructFeed
    
    (C3)
    Avg diff rating 0.15 0.21 0.43
    Avg standard deviation 0.32 0.32 0.44
    # of decreased diff rating 3 2 0
    # of equal rating 7 7 5
    !48
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

62. Red: Grade decreased
    Green: Grade increase
    White: No improvement
    Expert feedback
    (C1)
    Crowd feedback
    
    (C2)
    StructFeed
    
    (C3)
    Ensemble
    Avg diff rating 0.15 0.21 0.43 0.79
    Avg standard deviation 0.32 0.32 0.44 0.45
    # of decreased diff rating 3 2 0 0
    # of equal rating 7 7 5 1
    Experiment Results
    !49
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

63. Different types of feedback support writers in
    different perspectives
    “All types of feedback are useful to me. I’d like to use them for
    different purpose or at different stage. For example, I will use
    StructFeed at the beginning, then crowd feedback. If I need to
    write SOP, I’ll use expert feedback for final check.” (P5)
    
    Insight
    !50
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    Expert Crowd
    Crowd Machine
    StructFeed
    ҄ ҄
    Ensemble feedback supports writers in different perspectives
    

    View Slide

64. (1) We designed a crowd-powered system that enables
    structural feedback for supporting ESL writing
    
    (2) We leveraged domain rubrics in designing crowdsourcing
    workflow
    
    (3) StructFeed outperformed free-form feedback from both
    experts and crowd
    
    (4) Ensemble feedback may support writers in different
    perspectives
    Summary
    !51
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

65. Learning
    Part II: Feedback Utilization
    Writing
    Feedback
    Feedback
    Provider
    Author
    How do we support authors to integrate feedback into
    revisions and facilitate high-quality outcome?
    Revision
    !52
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

66. Evaluate the writing
    Improve the writing
    Feedback
    Work
    Iterative process
    Good feedback NOT always facilitates good results!
    !53
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

67. Too many suggestions may cause problems
    1. Information overload
    2. Cost of task switching
    3. People focus on easier problems
    !54
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

68. Formative Study
    6 participants (1 female) with the age 18-23
    - Novices deal with feedback in an “unstructured” way
    - Varying revision strategies
    (1) browse all comments
    (2) group similar comments
    (3) deal with comments in a sequence
    (1) beginning-to-end editing
    (2) specificity-first editing
    (3) high-to-low editing
    (4) low-to-high editing
    novice writers
    !55
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

69. Expert Revision Practice
    Novice writers
    Expert writers
    ESL writers
    revise in a linear process
    revise in a recursive way
    revise in a disorganized way
    Expert writers think high-level goals, and break it into
    low-level steps
    !56
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    A Lack of Reflection
    

    View Slide

70. Challenges of ESL Revision
    ESL students lack abilities in a foreign language
    
    - metarhetorical awareness (knowledge of themselves as writers)
    
    - metastrategic awareness (knowledge of their own personality type and its
    influence on their writing behaviors, including revision)
    
    - metalinguistic awareness (terminology to discuss language issues)
    (Alice S. Horning, 2006)
    !57
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    A Lack of Awareness
    

    View Slide

71. Goals
    1. Support awareness and reflection
    
    2. Guide writers to think and revise “structurally”
    !58
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

72. Author
    Feedback
    Provider
    Feedback
    Revision Workflow
    Writing
    High Medium Low
    Feedback
    Feedback Orchestration
    Structuring Feedback for Promoting Reflection and Awareness in Revision
    Rhetorical Structure
    High-Level
    Medium-Level
    Low-Level
    content
    organization
    vocabulary
    grammar
    mechanics
    Feedback
    !59
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

73. Author
    Feedback
    Provider
    Feedback
    Revision Workflow
    Writing
    High Medium Low
    Feedback
    Feedback Orchestration
    Structuring Feedback for Promoting Reflection and Awareness in Revision
    1. get feedback
    Rhetorical Structure
    High-Level
    Medium-Level
    Low-Level
    content
    organization
    vocabulary
    grammar
    mechanics
    Feedback
    !59
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

74. Author
    Feedback
    Provider
    Feedback
    Revision Workflow
    Writing
    High Medium Low
    Feedback
    Feedback Orchestration
    Structuring Feedback for Promoting Reflection and Awareness in Revision
    1. get feedback
    2. classify feedback
    Rhetorical Structure
    High-Level
    Medium-Level
    Low-Level
    content
    organization
    vocabulary
    grammar
    mechanics
    Feedback
    !59
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

75. Author
    Feedback
    Provider
    Feedback
    Revision Workflow
    Writing
    High Medium Low
    Feedback
    Feedback Orchestration
    Structuring Feedback for Promoting Reflection and Awareness in Revision
    1. get feedback
    2. classify feedback
    3. revise an article in a revision workflow
    Rhetorical Structure
    High-Level
    Medium-Level
    Low-Level
    content
    organization
    vocabulary
    grammar
    mechanics
    Feedback
    !59
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

76. 1) Rhetorical structure
    
    2) Meta-feedback
    
    3) Flexible revision workflow
    Three Design Considerations
    Author
    Feedback
    Provider
    Feedback
    Revision Workflow
    Writing
    High Medium Low
    Feedback w/ meta-feedback
    Rhetorical Structure
    High-Level
    Medium-Level
    Low-Level
    content
    organization
    vocabulary
    grammar
    mechanics
    Feedback
    content organization grammar mechanics
    language
    !60
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

77. Feedback Type Definition Examples
    High
    Content
    Content refers to the substance
    of writing. It includes topic
    sentence expressing main
    argument and supporting ideas.
    e.g. unity of argument,
    supporting idea, relevant
    example, addresses the
    question, etc.
    Organization
    Organization refers to the
    logical organization of the
    content.
    e.g. coherence of the
    content, relation between
    sentences, logical
    sequencing, etc.
    Medium
    Vocabulary
    Vocabulary refers to the
    selection or words those are
    suitable with the content.
    e.g. word choice, etc.
    Language Use
    Language Use refers to the use
    of the correct grammatical
    forms and syntactical pattern.
    e.g. fixing grammatical
    errors, or paraphrasing,
    shortening, etc.
    Low Mechanics
    Mechanic refers to all the
    arbitrary technical stuff in writing
    like spelling, capitalization,
    punctuation, etc.
    e.g. spelling errors,
    punctuation, capitalization,
    format, etc.
    Feedback Classification (Jacobs et al.,1981)
    !61
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

78. Automated Classification
    content
    organization
    vocabulary
    grammar
    mechanics
    Feedback
    Mechanics
    Vocabulary Grammar
    Rule-based classifier
    Comment Edit
    Feedback
    punctuation,
    spelling,
    capitalization
    stemming
    Non-Mechanics
    High-level
    Medium-level
    Low-level
    !62
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

79. Flexible Revision Workflow
    High-Level
    Medium-Level
    Low-Level
    content
    organization
    vocabulary
    grammar
    mechanics
    mechanics
    vocabulary
    gram
    m
    ar
    organization
    content
    organization
    content
    vocabulary
    gram
    m
    ar
    mechanics
    Sequential Workflow Concurrent Workflow
    high-to-low (HML)
    low-to-high (LMH)
    vocabulary
    gram
    m
    ar
    mechanics
    content
    organization
    all (ALL)
    or
    !63
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

80. Rhetorical Structure
    content
    organization
    vocabulary
    grammar
    mechanics
    Feedback
    Rhetorical Categories
    Feedback
    Classify
    High-Level
    Medium-Level
    Low-Level
    content
    organization
    vocabulary
    grammar
    mechanics
    Collect Structure
    Crowdsourcing
    Workflows
    !64
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

81. ReviseO
    ReviseO
    !65
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

82. High-to-low editing
    Low-to-high editing
    Mechanics Feedback
    Language Feedback
    Content & Organization Feedback High-level
    Middle-level
    Low-level
    !66
    

    View Slide

83. Experiment Design
    High-to-low (HML)
    Low-to-high (LMH)
    High
    - a within-subjects, counterbalanced experiment design
    
    - 12 self-motivated non-native writers
    
    - each participants performed 3 rewriting tasks with different topics
    
    - 3 experimental conditions
    
    - (1) show feedback together (ALL)
    
    - (2) show feedback sequentially from high to low (HML)
    
    - (3) show feedback sequentially from low to high (LMH)
    Together (ALL)
    Medium Low
    High
    Medium
    Low
    ALL
    !67
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

84. Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    RevisO System Categorization Writing Quality Revision Effort
    Ease of use

    M(SE)
    Helpfulness

    M(SE)
    Helpfulness

    M(SE)
    Diff of Rating

    M(SE)
    Time spent

    M(SE)
    Edit distance

    M(SE)
    LMH 5.58(.43) 6.22(.21) 6.58(.23) 6.79(.88) 1731.58(147.81) 353.33(.68)
    HML 4.92(.47) 6.22(.25) 6.50(.26) 7.21(.53) 1727.75(204.05) 370.00(60.85)
    ALL 5.25(.33) 6.25(.18) 5.67(.48) 7.25(.44) 1612.92(124.57) 444.33(72.35)
    Experiment Results
    - All participants improved the quality of writing
    
    - The ReviseO system got high perceived helpfulness and usefulness
    
    - No significant difference among three conditions
    post-questionnaire: 0~7
    score: 0~100
    

    View Slide

85. Insight 1
    Structured feedback helps filter information, identify
    weaknesses, and support reflection
    “This categorized feedback helps me identify my common
    mistakes easily! When I see the same type of writing issues
    appearing frequently, I understand that I need to pay more
    attention to this type of problem in my next writing. (P2)”
    69
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

86. Flexible revision supports varying strategies and
    helps develop new revision strategies
    Insight 2
    70
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    25%
    17%
    33%
    25%
    (3) (3)
    (4)
    (2)
    HML
    LMH
    ALL
    HLM
    

    View Slide

87. Separating feedback in a sequence may cause
    editing conflicts
    Insight 3
    1) Mis-classified feedback leads to misunderstanding
    
    2) Too many low-level fixes leads to decreased motivation
    for high-level improvement
    71
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

88. Summary
    (1) Feedback Orchestration uses a rhetorical structure, meta-
    feedback, and flexible workflows to guide effective revision
    
    (2) Structured feedback helps identify weaknesses and support
    reflection
    
    (3) Flexible workflows help develop personalized revision
    strategies.
    
    (4) ReviseO support writers to think and revise structurally
    72
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

89. Creative Work
    Feedback
    Provider
    Author
    Structural Feedback Feedback-Driven Revision
    Feedback
    Crowd Machine
    Crowdsourcing Workflow
    Data Annotations
    Revision Workflow
    Rhetorical Structure
    High-Level
    Medium-Level
    Low-Level
    content
    organization
    vocabulary
    grammar
    mechanics
    Feedback
    High Medium Low
    Feedback
    Collaboration
    1
    2
    3
    Structure Matters
    We use “structure” to guide feedback providers to contribute high-quality results.
    1
    2
    3
    We use “structure” to present feedback from different levels.
    We use “structure” to guide users to integrate feedback into revision
    !73
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

90. Iteration
    Quality
    Revision Effort
    individual’s ability
    !74
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

91. Iteration
    Quality
    Revision Effort
    individual’s ability
    !74
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

92. Iteration
    Quality
    Revision Effort
    individual’s ability
    !74
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

93. Iteration
    Quality
    Revision Effort
    Knowledge Gap
    individual’s ability
    !74
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

94. “We do not learn from experience…
    
    we learn from reflecting on experience”
    — John Dewey —
    !75
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

95. Creative Work
    User as
    Author
    Practice
    Review
    Collaboration
    Feedback
    Provider
    Feedback
    Feedback-Driven
    Revision
    Structural Feedback
    !76
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

96. Creative Work
    User as
    Author
    Practice
    Review
    Collaboration
    Feedback
    Provider
    Feedback
    Feedback-Driven
    Revision
    Structural Feedback
    Never-Ending Creative Learning
    !76
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    User as
    Learner
    Reflection
    Reflection
    Collaboration
    Creative Work
    Learning through
    Reflection
    

    View Slide

97. !77
    Learning Professional Skills for Drawing
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

98. !78
    The “knowledge” is in the details…
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

99. Leaner
    Creative Work
    Author
    Before/After-Practice Reflection Workflow
    Generate Learning Points
    learning points
    Identify Self-Explain
    Reflection Workflows
    Practice
    Extracting Learning Points for Drawing Support
    Identify Self-Explain
    Practice
    Identify Self-Explain
    Learn & Reflect
    Create
    Yi-Ching Huang, Jerry Yu-Heng Chan, and Jane Yung-jen Hsu. Reflection before/after practice: Learnersourcing for drawing support.
    In CHI ’18 Extended Abstracts on Human Factors in Computing Systems, 2018.
    !79
    Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
    

    View Slide

100. !80
     drafting outlining details
     What is a learning point?
     1.where is it? a clip of video/process
     
     2.what is it? a description
     
     3.why do you think it is important? a reason
     coloring with large area
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     

     View Slide

101. - identify start and end point
     
     - describe what it is
     - explain why
     Reflection Workflow for Extracting Learning Points
     Identify
     Identify one learning point
     Self-Explain
     Explain why you choose it
     Why?
     start point
     end point
     !81
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     

     View Slide

102. Reflection
     Practice
     !82
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     

     View Slide

103. Reflection
     Reflection is defined as a purposeful thinking toward a goal. (Dewey, 1933)
     1. Reflection-in-action
     2. Reflection-on-action
     - refers to the monitoring and modification of actions during the
     learning process
     - refers to reasonable evaluation and strategic planning for improvement
     after the learning process.
     Reflection Practice (Schön, 1983)
     !83
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     

     View Slide

104. Before/After-Practice Reflection Workflow
     Practice
     Learning by doing
     Identify
     Identify learning points
     Self-Explain
     Explain it is important
     helpful
     difficult
     interest
     Why?
     !84
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     Before-annotation
     Reflection
     Practice
     Learning by doing
     Identify
     Identify learning points
     Self-Explain
     Explain it is important
     helpful
     difficult
     interest
     Why?
     After-annotation
     Reflection
     

     View Slide

105. Practice
     Learning by doing
     Identify
     Identify learning points
     Self-Explain
     Explain it is important
     helpful
     difficult
     interest
     Why?
     Identify
     Identify learning points
     Self-Explain
     Explain it is important
     helpful
     difficult
     interest
     Why?
     Pilot study: Before/After-Practice Reflection Workflow
     - 8 participants (5 male, 3 female), 20-23 years old
     
     - 80 annotations (48 learning points, 16 difficult points, 16 interest points)
     !85
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     - Learners discover new learning points or revise their previous
     findings after practice
     Lessons Learned
     - After-annotation augments before-annotation
     

     View Slide

106. !86
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     ䷱䋊螂ܹ璾牧眤憽氅ᜋӞ䍅Ӟ䍅吩Ӥ݄ጱ向ဩ盄蠐 (P5, before)
     Ӥᜋጱොဩ牧᩻ڊᇔ誢क़ᶎฎݢ犥ጱ牧磧盅ٚ硄ധ疰অ牧蝡䰬Ӥᜋ眤憽穉斃অӞ讨 (P3, after)
     ضय़膌向ڊ斪ୄ牧ٚୌ缏奞℄ (P1,before)
     አ܈ਁ斔ۗ娄向Ո腷 (P4, after)
     硄ധ毣誧螲翣ጱ茐ᜋ㬵蕣蝨ط୽ጱ硳ຎ (P4,after)
     Reflective annotations help learners obtain new knowledge in
     different perspectives
     

     View Slide

107. !87
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     Creative learning is never ending …
     

     View Slide

108. Feedback
     Provider
     Crowd Machine
     Crowdsourcing Workflow
     Data Annotations
     Expert
     Crowd
     Part I: Feedback Generation
     Feedback
     Author
     Revision Workflow
     Part II: Feedback Utilization
     Rhetorical Structure
     High-Level
     Medium-Level
     Low-Level
     Feedback
     High Medium Low
     Feedback
     Collaboration
     Learner
     Creative Work
     Extract Reflection
     Annotations
     Learn & Reflect
     Learning
     Part III: Learning through Reflection
     !88
     

     View Slide

109. Creative Work
     Author
     Practice
     Review
     Feedback-Driven
     Revision
     Collaboration
     Structural Feedback
     Feedback
     Provider
     Learner
     Reflection
     Feedback
     Learning through
     Reflection
     Reflection
     Collaboration
     Creative Work
     !89
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     

     View Slide

110. Creative Work
     Author
     Practice
     Review
     Feedback-Driven
     Revision
     Collaboration
     Structural Feedback
     Feedback
     Provider
     Learner
     Reflection
     Feedback
     Learning through
     Reflection
     Reflection
     Collaboration
     Creative Work
     !89
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     Never-Ending Creative Learning
     

     View Slide

111. Accomplishments
     - Two Intelligent Systems for Supporting Writing Revision
     - StructFeed for generating structural feedback
     
     - ReviseO for facilitating feedback-driven revision
     
     - Studies on understandings between feedback, revision, and
     writing quality
     - Workflows for collecting feedback, facilitate revision, and
     promote reflection
     - Unity & Coherence workflow
     
     - Flexible Revision workflows
     
     - Reflection-based workflows
     !90
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     

     View Slide

112. !91
     Contributions
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     - Never-Ending Creative Learning for support creative task
     solving and learning
     
     - Leverage structure to facilitate feedback generation and
     feedback-driven revision
     
     - Techniques for enabling collaborations among users,
     crowds, and machines
     

     View Slide

113. Future Directions
     - Effective collaborations between crowd and machine
     - Adaptive Feedback for supporting users with different levels
     - Never-Ending Creative Learning for Other Domains
     !92
     Intro | Feedback Generation | Feedback Utilization | Learning thro Reflection | Conclusion
     Crowd Machine
     

     View Slide

114. Thank You
     Q&A
     !93
     

     View Slide