Supporting ESL Writing by Prompting Crowdsourced Structural Feedback

Fc1b497e61d3adca89516740944ad1e5?s=47 janetyc
October 25, 2017

Supporting ESL Writing by Prompting Crowdsourced Structural Feedback

Presented on HCOMP 2017
http://www.humancomputation.com/2017/

Fc1b497e61d3adca89516740944ad1e5?s=128

janetyc

October 25, 2017
Tweet

Transcript

  1. Supporting ESL Writing by Prompting Crowdsourced Structural Feedback Yi-Ching (Janet)

    Huang National Taiwan University janetyc@gmail.com Hao-Chuan Wang National Tsing Hua University haochuan@cs.nthu.edu.tw Jane Yung-jen Hsu National Taiwan University yjhsu@csie.ntu.edu.tw 2017.10.25 Yi-Ching (Janet) Huang
  2. 2 https://www.facebook.com/womaninthestriped/

  3. 2 First, open a new word file https://www.facebook.com/womaninthestriped/

  4. 2 First, open a new word file Only one word-“index”

    https://www.facebook.com/womaninthestriped/
  5. 2 First, open a new word file My brain is

    all blank, just like everything I do in life. Only one word-“index” https://www.facebook.com/womaninthestriped/
  6. 2 First, open a new word file My brain is

    all blank, just like everything I do in life. I feel useless Only one word-“index” … And nobody loves me https://www.facebook.com/womaninthestriped/
  7. 3 A Rejection Letter in HCOMP 2016

  8. “The best writing is rewriting.” ― E.B. White 4

  9. 5

  10. 5 Local Issues

  11. 5 Local Issues Global issues

  12. 6 Current Rewriting Support Tools Rewriting Feedback global local Local

    issue Global issue
  13. 6 Current Rewriting Support Tools Rewriting Feedback global local Spelling

    checker Grammar checker sentence word Local issue Global issue
  14. 6 Current Rewriting Support Tools Rewriting Feedback global local Free

    Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue
  15. 6 Current Rewriting Support Tools Rewriting Feedback global local Free

    Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue
  16. 7 Writer Reader Structure helps deliver message to a reader

    Idea
  17. 8 English Oriental (Kaplan, 1966) Rhetorical Patterns of Different Languages

    Robert B. Kaplan. Cultural thought patterns in inter-cultural education. Language Learning, 1966.
  18. 9 Writer Structural Feedback Writing Crowd Machine Revision Crowdsourcing Workflow

    Data Annotations StructFeed
  19. 10 Topic Sentence Supporting Sentence Concluding Sentence Introduction Body Conclusion

    Essay Structure Paragraph Structure paragraph paragraph paragraph Key point Supporting Sentence Supporting Sentence
  20. A paragraph is a group of sentences organized around a

    central topic. Four Key Elements of a Great Writing Element #1: Unity Element #2: Order Element #3: Coherence Element #4: Completeness All sentences in a paragraph should speak about one single idea or one main subject. Order refers to the way you organize your supporting sentences. Sentences within a paragraph need to connect to each other and work together as a whole. Completeness means a paragraph is well-developed.
  21. 12 1. All sub-points centering on one central idea 2.

    Using no irrelevant sentences Key points to achieve unity: Topic Sentence Supporting Sentence #1 Concluding Sentence related to the topic sentence Supporting Sentence #2 Supporting Sentence #3 The First Key Element of a Great Writing - All sentences in a paragraph should speak about one single idea. Unity
  22. 13 Crowdsourcing Workflow Structural Feedback Writing Criteria 1. multiple topic

    issue 2. missing topic issue 3. irrelevance issue Crowd Annotations StructFeed for Improving Unity Topic sentence annotation Relevant keyword annotation Unity Identification Topic sentence prediction Irrelevant sentence prediction
  23. 14 Crowdsourcing Workflow for Unity Identification Topic Identify topic sentence

    topic + ideas Crowdsourcing Workflow Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation
  24. 15 Topic Task - identify topic sentence Quality Control -

    native speakers as workers - brief explanation of concept - worked example - annotate sentence by click
  25. 15 Topic Task - identify topic sentence Quality Control -

    native speakers as workers - brief explanation of concept - worked example - annotate sentence by click Explanation Worked example Working area annotate sentence by click
  26. 16 Relevance Task annotate word by click

  27. 16 Relevance Task Worked example annotate word by click

  28. 17 Topic Identify topic sentence topic + ideas Crowdsourcing Workflow

    Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation Structural Feedback Writing Criteria 1. multiple topic issue 2. missing topic issue 3. irrelevance issue Crowd Annotations Unity Identification Topic sentence prediction Irrelevant sentence prediction StructFeed
  29. 18 Structural Feedback

  30. 18 Structural Feedback Feedback Summary - type of issue -

    suggested action
  31. 18 Rhetorical Visualization Structural Feedback Feedback Summary - type of

    issue - suggested action
  32. 18 Rhetorical Visualization Structural Feedback Feedback Summary - type of

    issue - suggested action - topic sentence
  33. 18 Rhetorical Visualization Structural Feedback Feedback Summary - type of

    issue - suggested action - topic sentence - irrelevant sentence
  34. 18 Rhetorical Visualization Structural Feedback Feedback Summary - type of

    issue - suggested action - topic sentence - irrelevant sentence - relevant keywords
  35. 19

  36. 19

  37. 20 A Field Experiment on ESL Writers - 18 self-motivate

    ESL writers (8 females, 10 males) - 19~34 years old Conditions - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed Writing Original version R Rewriting Revised version R’ Feedback Measure - a writing task: write an essay in 30 mins (300-400 words) - a rewriting task: improve the original writing using feedback in 30 mins Tasks - time, quantity, cost - quality improvement (R’-R) - perceived helpfulness
  38. Experiment Results Expert Feedback Crowd Feedback StructFeed -Diff-rating: 0.29 (.43)

    -Time: 1~2 days -Cost: $16 -Quantity: 55.44 suggestions - # of equal rating: 1 - # of decreased rating: 1 -Diff-rating: 0.38 (.44) -Time: 10~30 mins -Cost: $2 -Quantity: 8.11 suggestions - # of equal rating: 1 - # of decreased rating: 1 -Diff-rating: 0.54 (.25) -Time: 1~5 hrs -Costs: $1~1.7 - All participants improve the quality of writing 21 !
  39. 22 “I don’t understand what he means. His comments contain

    difficult terminology and it’s hard for me to capture the key points.” (P15) Observation I Knowledge gap between an expert and a novice writer
  40. 23 Observation 2 Structural feedback promotes self-reflection “I’m so surprised

    that no one annotates it as relevant keywords. I originally think that is a common example for other people. But, I am wrong. I will carefully choose a more common and understandable example to describe my idea next time.” (P7)
  41. - 18 self-motivate ESL learners (8 females, 10 males) -

    19~34 years old - A within-subjects counter-balanced design Writing Rewriting - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed Feedback 1 Rewriting Feedback 2 Rewriting Feedback 3 v1 v2 v3 v4 An Extended Field Experiment Tasks
  42. Extended Experiment Results Red: Grade decreased Green: Grade increase White:

    No improvement Expert feedback (C1) Crowd feedback (C2) StructFeed (C3) Avg diff rating 0.15 0.21 0.43 Avg standard deviation 0.32 0.32 0.44 # of decreased diff rating 3 2 0 # of equal rating 7 7 5
  43. 26 Conclusion - We designed a crowd-powered system that enables

    structural feedback for supporting ESL writing - We leveraged domain rubrics in designing workflow - StructFeed outperformed free-form feedback from both experts and crowd
  44. 27 Thank You Q&A