Share
Embed
Supporting ESL Writing by Prompting Crowdsourced Structural Feedback
Presented on HCOMP 2017
http://www.humancomputation.com/2017/
janetyc
October 25, 2017
Transcript
44 slides
-
Supporting ESL Writing by Prompting Crowdsourced Structural Feedback Yi-Ching (Janet)
Huang National Taiwan University [email protected] Hao-Chuan Wang National Tsing Hua University [email protected] Jane Yung-jen Hsu National Taiwan University [email protected] 2017.10.25 Yi-Ching (Janet) Huang -
2 First, open a new word file Only one word-“index”
https://www.facebook.com/womaninthestriped/ -
2 First, open a new word file My brain is
all blank, just like everything I do in life. Only one word-“index” https://www.facebook.com/womaninthestriped/ -
2 First, open a new word file My brain is
all blank, just like everything I do in life. I feel useless Only one word-“index” … And nobody loves me https://www.facebook.com/womaninthestriped/ -
6 Current Rewriting Support Tools Rewriting Feedback global local Local
issue Global issue -
6 Current Rewriting Support Tools Rewriting Feedback global local Spelling
checker Grammar checker sentence word Local issue Global issue -
6 Current Rewriting Support Tools Rewriting Feedback global local Free
Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue -
6 Current Rewriting Support Tools Rewriting Feedback global local Free
Comment idea Organization checker structure Spelling checker Grammar checker sentence word Local issue Global issue -
8 English Oriental (Kaplan, 1966) Rhetorical Patterns of Different Languages
Robert B. Kaplan. Cultural thought patterns in inter-cultural education. Language Learning, 1966. -
9 Writer Structural Feedback Writing Crowd Machine Revision Crowdsourcing Workflow
Data Annotations StructFeed -
10 Topic Sentence Supporting Sentence Concluding Sentence Introduction Body Conclusion
Essay Structure Paragraph Structure paragraph paragraph paragraph Key point Supporting Sentence Supporting Sentence -
A paragraph is a group of sentences organized around a
central topic. Four Key Elements of a Great Writing Element #1: Unity Element #2: Order Element #3: Coherence Element #4: Completeness All sentences in a paragraph should speak about one single idea or one main subject. Order refers to the way you organize your supporting sentences. Sentences within a paragraph need to connect to each other and work together as a whole. Completeness means a paragraph is well-developed. -
12 1. All sub-points centering on one central idea 2.
Using no irrelevant sentences Key points to achieve unity: Topic Sentence Supporting Sentence #1 Concluding Sentence related to the topic sentence Supporting Sentence #2 Supporting Sentence #3 The First Key Element of a Great Writing - All sentences in a paragraph should speak about one single idea. Unity -
13 Crowdsourcing Workflow Structural Feedback Writing Criteria 1. multiple topic
issue 2. missing topic issue 3. irrelevance issue Crowd Annotations StructFeed for Improving Unity Topic sentence annotation Relevant keyword annotation Unity Identification Topic sentence prediction Irrelevant sentence prediction -
14 Crowdsourcing Workflow for Unity Identification Topic Identify topic sentence
topic + ideas Crowdsourcing Workflow Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation -
15 Topic Task - identify topic sentence Quality Control -
native speakers as workers - brief explanation of concept - worked example - annotate sentence by click -
15 Topic Task - identify topic sentence Quality Control -
native speakers as workers - brief explanation of concept - worked example - annotate sentence by click Explanation Worked example Working area annotate sentence by click -
17 Topic Identify topic sentence topic + ideas Crowdsourcing Workflow
Relevance Highlight the relevant words between two sentences relevance topic Filter Filter paragraphs with no topic sentence (weight>=2) Topic sentence annotation Relevant keyword annotation Structural Feedback Writing Criteria 1. multiple topic issue 2. missing topic issue 3. irrelevance issue Crowd Annotations Unity Identification Topic sentence prediction Irrelevant sentence prediction StructFeed -
18 Structural Feedback Feedback Summary - type of issue -
suggested action -
18 Rhetorical Visualization Structural Feedback Feedback Summary - type of
issue - suggested action -
18 Rhetorical Visualization Structural Feedback Feedback Summary - type of
issue - suggested action - topic sentence -
18 Rhetorical Visualization Structural Feedback Feedback Summary - type of
issue - suggested action - topic sentence - irrelevant sentence -
18 Rhetorical Visualization Structural Feedback Feedback Summary - type of
issue - suggested action - topic sentence - irrelevant sentence - relevant keywords -
20 A Field Experiment on ESL Writers - 18 self-motivate
ESL writers (8 females, 10 males) - 19~34 years old Conditions - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed Writing Original version R Rewriting Revised version R’ Feedback Measure - a writing task: write an essay in 30 mins (300-400 words) - a rewriting task: improve the original writing using feedback in 30 mins Tasks - time, quantity, cost - quality improvement (R’-R) - perceived helpfulness -
Experiment Results Expert Feedback Crowd Feedback StructFeed -Diff-rating: 0.29 (.43)
-Time: 1~2 days -Cost: $16 -Quantity: 55.44 suggestions - # of equal rating: 1 - # of decreased rating: 1 -Diff-rating: 0.38 (.44) -Time: 10~30 mins -Cost: $2 -Quantity: 8.11 suggestions - # of equal rating: 1 - # of decreased rating: 1 -Diff-rating: 0.54 (.25) -Time: 1~5 hrs -Costs: $1~1.7 - All participants improve the quality of writing 21 ! -
22 “I don’t understand what he means. His comments contain
difficult terminology and it’s hard for me to capture the key points.” (P15) Observation I Knowledge gap between an expert and a novice writer -
23 Observation 2 Structural feedback promotes self-reflection “I’m so surprised
that no one annotates it as relevant keywords. I originally think that is a common example for other people. But, I am wrong. I will carefully choose a more common and understandable example to describe my idea next time.” (P7) -
- 18 self-motivate ESL learners (8 females, 10 males) -
19~34 years old - A within-subjects counter-balanced design Writing Rewriting - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed Feedback 1 Rewriting Feedback 2 Rewriting Feedback 3 v1 v2 v3 v4 An Extended Field Experiment Tasks -
Extended Experiment Results Red: Grade decreased Green: Grade increase White:
No improvement Expert feedback (C1) Crowd feedback (C2) StructFeed (C3) Avg diff rating 0.15 0.21 0.43 Avg standard deviation 0.32 0.32 0.44 # of decreased diff rating 3 2 0 # of equal rating 7 7 5 -
26 Conclusion - We designed a crowd-powered system that enables
structural feedback for supporting ESL writing - We leveraged domain rubrics in designing workflow - StructFeed outperformed free-form feedback from both experts and crowd