Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Pilot study: Longitudinal analysis of peer feedback on writing-intensive course

Pilot study: Longitudinal analysis of peer feedback on writing-intensive course

A presentation for the Festival of Learning, Burnaby BC, Canada, June 2016

3cd8f8111c34336b77da90efab71822e?s=128

Christina Hendricks

June 09, 2016
Tweet

Transcript

  1. Longitudinal Analysis of Peer Feedback in a Writing-Intensive Course: A

    Pilot Study PI: Christina Hendricks Co-PI: Jeremy Biesanz University of British Columbia-Vancouver Funded by the UBC Institute for the Scholarship of Teaching and Learning SoTL Seed Fund Festival of Learning, June 2016 Slides licensed CC-BY 4.0
  2. Literature on peer feedback Receiving peer feedback improves writing (Paulus,

    1999; Cho & Schunn, 2007; Cho & MacArthur, 2010; Crossman & Kite, 2012) Giving peer feedback improves writing (Cho & Cho, 2011; Li, Liu & Steckelberg, 2010)
  3. GAPS: Most studies look at revisions to a single essay,

    not changes across different essays Dra$ 1 Dra$ 2 Dra$ 3 Essay 1 Essay 2 Essay 3 Essay 4 Essay …n PFB PFB PFB PFB PFB PFB Few studies look at “dose-response curve”
  4. Pilot study research questions 1.  How do students use peer

    comments given and received for improving different essays rather than drafts of the same essay? 2.  Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal? 3.  Does the quality of peer comments improve over time?
  5. •  Interdisciplinary, full year course for first-years •  18 credits

    (English, History, Philosophy) •  Students write 10-12 essays (1500-2000 words) •  Peer feedback tutorials every week (4 students) http://artsone.arts.ubc.ca Toni Morrison, Wikimedia Commons, licensed CC BY-SA 2.0 Osamu Tezuka, public domain on Wikimedia Commons Jane Austen, public domain on Wikimedia Commons Friedrich Nietzsche, public domain, Wikimedia Commons
  6. Data for pilot study 2013-2014 •  10 essays by 12

    participants (n=120) •  Comments by 3 peers on essays (n=1218) •  Comments by instructor (n=3291) •  All coded with same rubric
  7. Coding Rubric Categories (plus subcategories, for 11 options) •  Strength

    of argument •  Organization •  Insight •  Style & Mechanics Numerical value 1: Significant problem 2: Moderate problem 3: Positive comment/praise E.g., STREV 2: could use more textual evidence to support your claims Change for future
  8. Inter-coder reliability Fleiss’ Kappa Intra-class correla2on Student comments (n=141) All

    categories: 0.61 (moderate) Most used categories: 0.8 (excellent) 0.96 (excellent) Essays (n=120) 0.71 (adequate) 3 coders: •  Daniel Munro & Kosta Prodanovic (undergrads, former Arts One) •  Jessica Stewart (author, editor) Change for future
  9. LOOKING AT TRENDS IN COMMENTS OVER TIME

  10. 0 2 4 6 8 10 12 0 2 4

    6 8 10 12 Essay Number Instructor Number of Comments Argument Strength Style Insight Organization INSTRUCTOR Comments -.28** Strength Style Organiz. Insight -.04* Number of 2 comments over time
  11. 0 2 4 6 8 10 12 0 1 2

    3 4 Essay Number Student Number of Comments Argument Strength Style Insight Organization STUDENT comments Strength Style Organiz. Insight -.16** Number of 2 comments over time
  12. 0 2 4 6 8 10 12 0 1 2

    3 4 5 Essay Number Instructor Number of Comments Argument Strength Style Insight Organization INSTRUCTOR Comments .31*** Strength Style Organiz. Insight .08** .19** .11** Number of 3 comments
  13. 0 2 4 6 8 10 12 0.0 0.5 1.0

    1.5 2.0 2.5 3.0 Essay Number Student Number of Comments Argument Strength Style Insight Organization STUDENT Comments Strength Style Organiz. Insight Number of 3 comments over time
  14. HOW DOES ESSAY QUALITY CHANGE OVER TIME?

  15. Essay quality improves linearly b = .038 t(107) = 2.1

    p = .037 Essays rated on a 7-point scale
  16. MORE COMPLEX ANALYSES

  17. Cross-lagged panel design with auto-regressive structure Essay Quality Time 1

    Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N
  18. Path A: Instructor Comments Essay Quality Time 1 Essay Quality

    Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant relationships •  Ratings of 1 in Strength (-.12*) & Org. (-.23**) •  Ratings of 2 in Strength (-.06*) & Style (-.08*) •  Ratings of 3 in Str, (.11*), Insight (.35*), Style (.15*) *p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
  19. Path A: Student comments Essay Quality Time 1 Essay Quality

    Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant relationships •  Ratings of 2 in Insight (-.53*) •  Ratings of 3 in Organization (.13*) *p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
  20. Path C: instructor comments Essay Quality Time 1 Essay Quality

    Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant effects don’t show up if split out by category •  Comments ratings of 1 (.29**) •  Comments ratings of 2 (.23*) •  Comments ratings of 3 (.21, p=.057) *p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
  21. Path C: student comments Essay Quality Time 1 Essay Quality

    Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant relationships •  Comments rated 2 in Strength (.22*) & Style (.33**) •  Comments rated 3 in Style (.31*) *p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
  22. Path D: Student & Instructor comments Essay Quality Time 1

    Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N Significant relationship ONLY if combine student & instructor comments, & only for comments rated 1 (all categories combined): (.05, p=.06)
  23. Research question 1 How do students use peer comments given

    and received for improving different essays rather than drafts of same essay? o  Very little significant evidence of relationships in Path D o  No difference between comments given & received
  24. Research question 2 Are students more likely to use peer

    comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal? o  No evidence that there is any change over time in path D o  No difference between comments given or received
  25. Research question 3 Does the quality of peer comments improve

    over time? o  No evidence of change over time in path A Essay Quality Time 1 Essay Quality Time 2 Comments Time 1 Comments Time 2 B A C D E … N … N
  26. Research Question 3, cont’d Student/instructor agreement on average numerical ratings

    on each essay •  tends to go down over time (-.04**) •  student ratings increase at only half the rate (.16*) that instructor’s ratings increase (.33*****) *p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
  27. Research Question 3, cont’d Correlations on number of comments, students

    & instructor •  No change in these relationships over time *p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001 Comment value 1 Comment value 2 Comment value 3 Strength 0.23* Organiza\on 0.21* 0.17* Insight 0.17* Style
  28. Some conclusions Pilot study: feasible for larger sample? Yes, if:

    o  instructors code essay quality rather than coders o  “chunk” essays for cross-lagged analyses o  have easy collection of comments
  29. References •  Cho, K., & MacArthur, C. (2010). Student revision

    with peer and expert reviewing, Learning and Instruction. 20, 328-338. •  Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving comments. Instructional Science, 39, 629-643. •  Cho, K. & Schunn, C. D. (2007). Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system. Computers & Education, 48, 409–426 •  Crossman, J. M., & Kite, S. L. (2012). Facilitating improved writing among students through directed peer review, Active Learning in Higher Education, 13, 219-229. •  Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525–536. •  Paulus, T. M. (1999). The effect of peer and teacher feedback on student writing. Journal of Second Language Writing, 8, 265-289.
  30. Thank you! Christina Hendricks University of British Columbia-Vancouver Website: http://blogs.ubc.ca/christinahendricks

    Blog: http://blogs.ubc.ca/chendricks Twitter: @clhendricksbc Slides available: https://is.gd/PeerFeedbackPilot_FOL16 Slides licensed CC-BY 4.0 Capitals needed underscore