Quality is usually evaluated by multiple criteria 4. Quality can be improved by iterative refinement 2. Answer is not true or false, but how good the answer is Intro | Feedback Generation | Feedback Integration | Iteration Process
Case Study Using Local Crowds for Event Reporting. HCOMP 2015. Sadauskas, Byrne, and Atkinson. Mining Memories: Designing a Platform to Support Social Media Based Writing. CHI 2015 Bernstein, Little, Miller, Hartmann, Ackerman, Karger, Crowell, and Panovich. Soylent: A Word Processor with a Crowd Inside. UIST 2010. Kim, Cheng, and Bernstein. Ensemble: Exploring Complementary Strengths of Leaders and Crowds in Creative Collaboration. CSCW 2014 Hahn, Chang, Kim, and Kittur. The Knowledge Accelerator: Big Picture Thinking in Small Pieces. CHI 2016. Nebeling, To, Guo, de Freitas, Teevan, Dow, and Bigham. WearWrite: Crowd-Assisted Writing from Smartwatches. CHI 2016. Kittur, Smus, Khamkar, and Kraut. CrowdForge: Crowdsourcing Complex Work. UIST 2011. Agapie, Teevan, and Monroy- Hernández. Crowdsourcing in the Field: A Case Study Using Local Crowds for Event Reporting. HCOMP 2015. Luther, Hahn, Dow, and Kittur. Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines. HCOMP 2015.
Case Study Using Local Crowds for Event Reporting. HCOMP 2015. Sadauskas, Byrne, and Atkinson. Mining Memories: Designing a Platform to Support Social Media Based Writing. CHI 2015 Bernstein, Little, Miller, Hartmann, Ackerman, Karger, Crowell, and Panovich. Soylent: A Word Processor with a Crowd Inside. UIST 2010. Kim, Cheng, and Bernstein. Ensemble: Exploring Complementary Strengths of Leaders and Crowds in Creative Collaboration. CSCW 2014 Hahn, Chang, Kim, and Kittur. The Knowledge Accelerator: Big Picture Thinking in Small Pieces. CHI 2016. Nebeling, To, Guo, de Freitas, Teevan, Dow, and Bigham. WearWrite: Crowd-Assisted Writing from Smartwatches. CHI 2016. Kittur, Smus, Khamkar, and Kraut. CrowdForge: Crowdsourcing Complex Work. UIST 2011. Agapie, Teevan, and Monroy- Hernández. Crowdsourcing in the Field: A Case Study Using Local Crowds for Event Reporting. HCOMP 2015. Luther, Hahn, Dow, and Kittur. Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines. HCOMP 2015.
Natural Language Crowd Scripting Soylent: A Word Processor with a Crowd Inside (Bernstein et al., UIST’ 10) Find-Fix-Verify workflow M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, UIST '10, pages 313-322, New York, NY, USA, 2010. ACM. Intro | Feedback Generation | Feedback Integration | Iteration Process
(Luther et al.,2015) MicroWriter (Teevan et al.,2016) CrowdForge (Kittur et al.,2011) Sparkfolio (Sadauskas et al.,2015) Ensemble (Kim et al.,2014) Soylent (Bernstei et al.,2011) Crowdsourcing in the Field (Agapie et al.,2015) Knowledge Accelerator (Hahn et al.,2016) WearWrite (Nebeling et al.,2016) Intro | Feedback Generation | Feedback Integration | Iteration Process
usage, and mechanics; style and diction; and organization and development - Templated-based feedback Criterion Feedback for the highest score “6” Feedback for the highest score “1” Template-based feedback Intro | Feedback Generation | Feedback Integration | Iteration Process Holistic Scoring Jill Burstein, Martin Chodorow, and Claudia Leacock. Automated essay evaluation: The criterion online writing service. AI Magazine, 25(3):27–36, 2004. (Burstein et al., 2004)
checker Free Comment idea Organization checker structure sentence word Local Global Intro | Feedback Generation | Feedback Integration | Iteration Process Human Machine
checker Free Comment idea Organization checker structure sentence word Local Global ??? Intro | Feedback Generation | Feedback Integration | Iteration Process Human Machine
Crowdsourced Structural Feedback Revision Crowdsourcing Workflow Data Annotations StructFeed Yi-Ching Huang, Jiunn-Chia Huang, and Jane Yung-jen Hsu. Supporting ESL writing by prompting crowdsourced structural feedback. In Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2017) Yi-Ching Huang, Hao-Chuan Wang, and Jane Yung-jen Hsu. Bridging learning gap in writing education with a crowd-powered system. CHI 2017 Workshop on Designing for Curiosity, Denver, Colorado, USA, 2017. Intro | Feedback Generation | Feedback Integration | Iteration Process
Intro | Feedback Generation | Feedback Integration | Iteration Process Robert B. Kaplan. Cultural thought patterns in inter-cultural education. Language Learning, 1966.
native speakers as workers - brief explanation of concept - worked example - annotate sentence by click Intro | Feedback Generation | Feedback Integration | Iteration Process
native speakers as workers - brief explanation of concept - worked example - annotate sentence by click Explanation Worked example Working area annotate sentence by click Intro | Feedback Generation | Feedback Integration | Iteration Process
Rule Intro | Feedback Generation | Feedback Integration | Iteration Process TFIDF-based - Topic sentence is a sentence that consist of the most amount of key ideas Similarity-based (ASS) - Topic sentence is a sentence that is the most similar sentence to other sentences in the paragraph. Rule-based - choose the first sentence as topic sentence
or goal 2. Compare the actual level of performance with the standard 3. Engage in action which leads to closure of the gap (Sadler, D. R. 1989) D. R. Sadler. Formative assessment and the design of instructional systems. Instructional Science, 18(2):119{144, 1989 Intro | Feedback Generation | Feedback Integration | Iteration Process
sentence - relevant keywords Feedback summary - type of issue - suggested action Writing Tip - provide a quick reminder about writing basic knowledge https://writingfeedback.herokuapp.com/topic_vis?workflow_id=78 StructFeed
learners (8 females, 10 males) - 19~34 years old - A between subjects study Conditions - C1 (expert feedback): free-form feedback from an expert - C2 (crowd feedback): free-form feedback from a crowd worker - C3 (structural feedback): structural feedback from StructFeed Writing Original version R Rewriting Revised version R’ Feedback Measure - time, quantity, cost - quality improvement (R’-R) - perceived helpfulness Intro | Feedback Generation | Feedback Integration | Iteration Process
machine - data-driven template is “4-paragraph essay template” - StructFeed not only identifies writing issues but also promotes reflection - Expert feedback perform worse than crowd feedback? - may exist knowledge gap or misunderstanding between experts and novice writers - writers tend to fix local issues like grammatical error rather than global issues like improve unity or coherence Intro | Feedback Generation | Feedback Integration | Iteration Process
crowd workers to annotate topic sentence and relevant keywords for identifying paragraph unity in an article (2) We demonstrate that our crowd-based method outperformed naive machine- based methods on ill-structure ESL writing with no labeled data (3) We designed and implement a crowd-powered system, StructFeed, that generates structural feedback consisting of writing hints and crowd annotations. (4) A field experiment showed that people who received structural feedback from StructFeed obtained better performance than people who received free-form feedback generated from a single expert and a single crowd worker. Intro | Feedback Generation | Feedback Integration | Iteration Process
Writer How do we support writers to integrate feedback into revisions and facilitate high-quality outcome? Revision Intro | Feedback Generation | Feedback Integration | Iteration Process
Writing High Medium Low Feedback Low Medium High H M M L L L L Feedback Classification Feedback Orchestration Facilitating Better Revision Behavior and Task Performance Intro | Feedback Generation | Feedback Integration | Iteration Process
Writing High Medium Low Feedback Low Medium High H M M L L L L Feedback Classification Feedback Orchestration Facilitating Better Revision Behavior and Task Performance 1. get feedback Intro | Feedback Generation | Feedback Integration | Iteration Process
Writing High Medium Low Feedback Low Medium High H M M L L L L Feedback Classification Feedback Orchestration Facilitating Better Revision Behavior and Task Performance 1. get feedback 2. classify feedback Intro | Feedback Generation | Feedback Integration | Iteration Process
Writing High Medium Low Feedback Low Medium High H M M L L L L Feedback Classification Feedback Orchestration Facilitating Better Revision Behavior and Task Performance 1. get feedback 2. classify feedback 3. revise an article in a revision workflow Intro | Feedback Generation | Feedback Integration | Iteration Process
the substance of writing. It includes topic sentence expressing main argument and supporting ideas. e.g. unity of argument, supporting idea, relevant example, addresses the question, etc. Organization Organization refers to the logical organization of the content. e.g. coherence of the content, relation between sentences, logical sequencing, etc. Medium Vocabulary Vocabulary refers to the selection or words those are suitable with the content. e.g. word choice, etc. Language Use Language Use refers to the use of the correct grammatical forms and syntactical pattern. e.g. fixing grammatical errors, or paraphrasing, shortening, etc. Low Mechanics Mechanic refers to all the arbitrary technical stuff in writing like spelling, capitalization, punctuation, etc. e.g. spelling errors, punctuation, capitalization, format, etc. Feedback Classification (Jacobs et al.,1981) Intro | Feedback Generation | Feedback Integration | Iteration Process
High Middle Low All - a within-subjects, counterbalanced experiment design - 12 self-motivated non-native writers - each participants performed 3 rewriting tasks with different topics - 3 experimental conditions - (1) show feedback together (ALL) - (2) show feedback sequentially from high to low (HML) - (3) show feedback sequentially from low to high (LMH) Together (ALL) Intro | Feedback Generation | Feedback Integration | Iteration Process
2) Increase awareness and reduce learning anxiety - it reduces learning anxiety about uncertain or unknown matters 3) Promote learning behaviors - It provides new knowledge about how to evaluate the writing - It helps memorize and organize writing issues and and promote focused learning The Benefits of Feedback Classification “This categorized feedback helps me identify my common mistakes easily! When I see the same type of writing issues appearing frequently, I understand that I need to pay more attention to this type of problem in my next writing. (P2)” “I learn new knowledge about writing from categorized feedback, and it helps me learn how to evaluate the quality of the writing by those good or bad examples. I will use the same way to review others’ writings in the next time!! (P7)” Intro | Feedback Generation | Feedback Integration | Iteration Process
- sequential process: clear and edit smoothly - concurrent process: same time and edit efficiently - Preference of editing process is related to the past experience - people who have writing experience preferred high-to-low process, for it is consistent to their knowledge obtained from the writing courses. - others preferred low-to-high process - Concurrent process reduce the influence of feedback classification - people tend to follow the original editing behaviors - people tend to take easy way out and ignore high-level issues Sequential Process vs. Concurrent Process Intro | Feedback Generation | Feedback Integration | Iteration Process
to consider big picture first and dive into the details later - Bottom-up thinking - the low-to-high process guides people to develop meaning and momentum to solve high-level issues like content or organization problems. A Sequential Process Guides Logical Thinking “Either low-to-high or high-to-low process helps me develop a thinking order. It’s convenient for me to edit my writing. (P10)” “I start with fixing language-level issues over the whole article and re-read my writing sentence by sentence. Then, I am getting more and more enjoyable because I become familiar with the content and easily recall the situation in which I was writing and understand why I generated those words. It helps me solve the structure issues more easily. (P4)” Intro | Feedback Generation | Feedback Integration | Iteration Process
- separating related comments into different stages leads to misunderstanding - appearing inexistent low-level issues that are solved at the high-level stage also leads to confusion - Mental conflicts - uncertainty impedes editing at the high-level stage - too many micro contributions lead to loss aversion “In the second stage, I rejected a comment suggesting an incorrect edit. However, I realized that the suggestion is correct at the third stage when I recall the previous comment I obtained. (P11)” “In the last stage, the feedback suggests that adding one connecting sentence in a specific paragraph increase the unity of the paragraph. I feel that all my previous efforts would be in vain if I fixed this issue. (P12)” Intro | Feedback Generation | Feedback Integration | Iteration Process
learning behaviors - Both writers and feedback providers need to be guided by a category structure to support feedback generation and integration process. - Low-to-high sequence facilitates difficult problem solving - Applying this strategy should carefully consider the number of low-level feedback. - Many micro-contributions may decrease work performance - Two possible reasons - 1) a large number of repetitive and monotonous tasks cause boredom. - 2) many micro-contributions lead to a feeling of loss aversion - Incorporate automated methods to reduce the duplicated tasks and distributing low-level feedback around the content containing high-level writing issues Intro | Feedback Generation | Feedback Integration | Iteration Process
related comments and presenting them at the same stage. - Increase transparency of the whole process by providing an overview - Flexible writing support - Offer varying writing support based on their writing abilities, current writing stage (drafting, early stage of revision, or editing), or preference. Discussion & Design Implications (II) Intro | Feedback Generation | Feedback Integration | Iteration Process
strategies applied to revisions from ESL writers. (2) The concept of feedback orchestration that uses a category structure to guide effective revision process by orchestrating feedback of different type. (3) The system ReviseO, which supports three feedback presentation strategies for helping writers resolve issues in a concurrent or sequential process by expert feedback structured by a standard rubric. (4) Our system helps individuals to identify weaknesses, facilitate self-assessment, and promote reflective practice. The results also demonstrate that sequential process guides users toward logical thinking and low-to-high sequence helps complex problem solving. Intro | Feedback Generation | Feedback Integration | Iteration Process
13.17 Cost (USD) $16 $2 Feedback Type more less Feedback Form (Direct/Indirect) 2.7 3.2 Results of Content Analysis Intro | Feedback Generation | Feedback Integration | Iteration Process
feedback than a single crowd worker; multiple crowd workers generated diverse types of feedback as expert created. - Both of experts and crowds generated less high-level feedback (i.e., content & organization) - Both of them used “direct editing” to correct the errors. Intro | Feedback Generation | Feedback Integration | Iteration Process
Data Annotations Expert Crowd Writer Revision Workflow High Medium Low Feedback Low Medium High H M M L L L L Feedback Classification Feedback Rate Feedback Part I: Feedback Generation Part II: Feedback Integration Part III: Iterative Revision Process Roadmap of Research
high-level thinking in revision process and enabling reflective practice - Feedback orchestration - Provide a categorical structure to scaffolding learning and revision behaviors - Enabling concurrent and sequential workflow for supporting feedback integration (CHI Workshop 2017; HCOMP 2017) (CHI 2018 in-submission) What we have done What we plan do - A series of studies to explore significant factors of feedback that impact revision quality - crowd feedback vs expert feedback - iterative writing experiment - Collaborative feedback by crowd collaboration - Improve global quality of feedback (HCOMP 2015 WIP) Intro | Feedback Generation | Feedback Integration | Iteration Process
janetyc
sansan_randd
hponka
masatoto
zacky1972
holyshun
chemical_tree
hpprc
kampersanda
onely7
beomi
maguro27
productmarketing
sachag
kneath
shlominoach
maltzj
holman
quramy
tammyeverts
zakiyama
colly
reverentgeek
