Supporting ESL Writing by Prompting Crowdsourced Structural Feedback

Presented on HCOMP 2017
http://www.humancomputation.com/2017/

Transcript

1. Supporting ESL Writing by Prompting Crowdsourced Structural Feedback Yi-Ching (Janet) Huang National Taiwan University janetyc@gmail.com Hao-Chuan Wang National Tsing Hua University haochuan@cs.nthu.edu.tw Jane Yung-jen Hsu National Taiwan University yjhsu@csie.ntu.edu.tw 2017.10.25 Yi-Ching (Janet) Huang
2. 2 https://www.facebook.com/womaninthestriped/
3. 2 First, open a new word file https://www.facebook.com/womaninthestriped/
4. 2 First, open a new word file Only one word-“index” https://www.facebook.com/womaninthestriped/
5. 2 First, open a new word file My brain is all blank, just like everything I do in life. Only one word-“index” https://www.facebook.com/womaninthestriped/
6. 2 First, open a new word file My brain is all blank, just like everything I do in life. I feel useless Only one word-“index” … And nobody loves me https://www.facebook.com/womaninthestriped/
7. 3 A Rejection Letter in HCOMP 2016
8. “The best writing is rewriting.” ― E.B. White 4
10. 5 Local Issues
11. 5 Local Issues Global issues
12. 6 Current Rewriting Support Tools Rewriting Feedback global local Local issue Global issue
13. 6 Current Rewriting Support Tools Rewriting Feedback global local Spelling checker Grammar checker sentence word Local issue Global issue
14. 6 Current Rewriting Support Tools Rewriting Feedback global local Free Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue
15. 6 Current Rewriting Support Tools Rewriting Feedback global local Free Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue
16. 7 Writer Reader Structure helps deliver message to a reader Idea
17. 8 English Oriental (Kaplan, 1966) Rhetorical Patterns of Different Languages Robert B. Kaplan. Cultural thought patterns in inter-cultural education. Language Learning, 1966.
18. 9 Writer Structural Feedback Writing Crowd Machine Revision Crowdsourcing Workflow Data Annotations StructFeed
19. 10 Topic Sentence Supporting Sentence Concluding Sentence Introduction Body Conclusion Essay Structure Paragraph Structure paragraph paragraph paragraph Key point Supporting Sentence Supporting Sentence
20. A paragraph is a group of sentences organized around a central topic. Four Key Elements of a Great Writing Element #1: Unity Element #2: Order Element #3: Coherence Element #4: Completeness All sentences in a paragraph should speak about one single idea or one main subject. Order refers to the way you organize your supporting sentences. Sentences within a paragraph need to connect to each other and work together as a whole. Completeness means a paragraph is well-developed.
21. 12 1. All sub-points centering on one central idea 2. Using no irrelevant sentences Key points to achieve unity: Topic Sentence Supporting Sentence #1 Concluding Sentence related to the topic sentence Supporting Sentence #2 Supporting Sentence #3 The First Key Element of a Great Writing - All sentences in a paragraph should speak about one single idea. Unity
22. 13 Crowdsourcing Workflow Structural Feedback Writing Criteria 1. multiple topic issue 2. missing topic issue 3. irrelevance issue Crowd Annotations StructFeed for Improving Unity Topic sentence annotation Relevant keyword annotation Unity Identification Topic sentence prediction Irrelevant sentence prediction
23. 14 Crowdsourcing Workflow for Unity Identification Topic Identify topic sentence topic + ideas Crowdsourcing Workflow Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation
24. 15 Topic Task - identify topic sentence Quality Control - native speakers as workers - brief explanation of concept - worked example - annotate sentence by click
25. 15 Topic Task - identify topic sentence Quality Control - native speakers as workers - brief explanation of concept - worked example - annotate sentence by click Explanation Worked example Working area annotate sentence by click
26. 16 Relevance Task annotate word by click
27. 16 Relevance Task Worked example annotate word by click
28. 17 Topic Identify topic sentence topic + ideas Crowdsourcing Workflow Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation Structural Feedback Writing Criteria 1. multiple topic issue 2. missing topic issue 3. irrelevance issue Crowd Annotations Unity Identification Topic sentence prediction Irrelevant sentence prediction StructFeed
29. 18 Structural Feedback
30. 18 Structural Feedback Feedback Summary - type of issue - suggested action
31. 18 Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action
32. 18 Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action - topic sentence
33. 18 Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action - topic sentence - irrelevant sentence
34. 18 Rhetorical Visualization Structural Feedback Feedback Summary - type of issue - suggested action - topic sentence - irrelevant sentence - relevant keywords
37. 20 A Field Experiment on ESL Writers - 18 self-motivate ESL writers (8 females, 10 males) - 19~34 years old Conditions - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed Writing Original version R Rewriting Revised version R’ Feedback Measure - a writing task: write an essay in 30 mins (300-400 words) - a rewriting task: improve the original writing using feedback in 30 mins Tasks - time, quantity, cost - quality improvement (R’-R) - perceived helpfulness
38. Experiment Results Expert Feedback Crowd Feedback StructFeed -Diff-rating: 0.29 (.43) -Time: 1~2 days -Cost: $16 -Quantity: 55.44 suggestions - # of equal rating: 1 - # of decreased rating: 1 -Diff-rating: 0.38 (.44) -Time: 10~30 mins -Cost: $2 -Quantity: 8.11 suggestions - # of equal rating: 1 - # of decreased rating: 1 -Diff-rating: 0.54 (.25) -Time: 1~5 hrs -Costs: $1~1.7 - All participants improve the quality of writing 21 !
39. 22 “I don’t understand what he means. His comments contain difficult terminology and it’s hard for me to capture the key points.” (P15) Observation I Knowledge gap between an expert and a novice writer
40. 23 Observation 2 Structural feedback promotes self-reflection “I’m so surprised that no one annotates it as relevant keywords. I originally think that is a common example for other people. But, I am wrong. I will carefully choose a more common and understandable example to describe my idea next time.” (P7)
41. - 18 self-motivate ESL learners (8 females, 10 males) - 19~34 years old - A within-subjects counter-balanced design Writing Rewriting - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed Feedback 1 Rewriting Feedback 2 Rewriting Feedback 3 v1 v2 v3 v4 An Extended Field Experiment Tasks
42. Extended Experiment Results Red: Grade decreased Green: Grade increase White: No improvement Expert feedback (C1) Crowd feedback (C2) StructFeed (C3) Avg diff rating 0.15 0.21 0.43 Avg standard deviation 0.32 0.32 0.44 # of decreased diff rating 3 2 0 # of equal rating 7 7 5
43. 26 Conclusion - We designed a crowd-powered system that enables structural feedback for supporting ESL writing - We leveraged domain rubrics in designing workflow - StructFeed outperformed free-form feedback from both experts and crowd
44. 27 Thank You Q&A