Beyond Surveys: Analyzing Software Development Artifacts to Assess Teaching Efforts

Slides of the talk on the paper "Beyond Surveys: Analyzing Software Development Artifacts to Assess Teaching Efforts" by Christoph Matthies, Ralf Teusner and Guenter Hesse given at the Frontiers in Education 2018 conference in San Jose, CA, USA in October 2018.

Preprints of the paper are available on arXiv (https://arxiv.org/abs/1807.02400)

Transcript

1. Hasso Plattner Institute University of Potsdam, Germany christoph.matthies@hpi.de @chrisma0 Beyond Surveys: Analyzing Software Development Artifacts to Assess Teaching Efforts Christoph Matthies, Ralf Teusner, Guenter Hesse ’18, San Jose, CA, October 2018
2. “ Background Course Focus You will learn how to manage a long-running software project with a large number of developers. [1] 2 [1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html ” An undergraduate software engineering capstone course
3. “ Background Course Focus You will learn how to manage a long-running software project with a large number of developers. [1] 3 [1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html ▪ All participants (in teams) jointly develop a software ▪ Self-organizing teams ▪ Collaboration > technical skills ” An undergraduate software engineering capstone course
4. “ Background Course Focus You will learn how to manage a long-running software project with a large number of developers. [1] 4 [1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html ▪ All participants (in teams) jointly develop a software ▪ Self-organizing teams ▪ Collaboration > technical skills ▪ Project work & intro exercises & lectures & tutors ▪ Learn and apply agile methods ▪ Real-world scenario, real development tools ” An undergraduate software engineering capstone course
5. Challenges & Ideas ▪ Course employed Scrum ▪ Structured, prescriptive ▪ Good for starting [2] ▪ Kanban gaining popularity in industry 5 Evaluate and adapt the course over time [2] V. Mahnic, “From Scrum to Kanban: Introducing Lean Principles to a Software Engineering Capstone Course”
6. Challenges & Ideas ▪ Course employed Scrum ▪ Structured, prescriptive ▪ Good for starting [2] ▪ Kanban gaining popularity in industry ▪ Idea: Update course and project! ▪ Employ Scrum Sprints first, then switch to Kanban ▪ Kanban for “finishing touches” 6 Evaluate and adapt the employed development methodology [2] V. Mahnic, “From Scrum to Kanban: Introducing Lean Principles to a Software Engineering Capstone Course”
7. Challenges & Ideas 7 Scrum: the usual agile development process Scrum key ideas ▪ Sprints: timeboxed iterations ▪ Planning and estimation ▪ Review and retrospectives ▪ Prescriptive process
8. Challenges & Ideas 8 Kanban: the new kid on the block Kanban key ideas ▪ Visualize work items on Kanban board ▪ “Pull” them through the process ▪ Limit work-in-progress ▪ Handle bottlenecks
9. Research Question 9 Goals of the research and background How can we gauge (the effect of curriculum) changes in student behavior during project work?
10. Research Question ▪ Usual approach: end-of-term surveys 10 Goals of the research and background How can we gauge (the effect of curriculum) changes in student behavior during project work?
11. ▪ Performed regularly after end of course (before grades) ▪ Allows student feedback on the course ▪ Standardized questions, overwhelmingly positive responses → Hard to gauge changes in curriculum design End-of-term Surveys 11 General satisfaction indicators of the last 5 years
12. Research Question ▪ Usual approach: end-of-term surveys 12 Goals of the research and background How can we gauge (the effect of curriculum) changes in student behavior during project work?
13. Research Question ▪ Usual approach: end-of-term surveys ▪ Programming project provides unique opportunity ▪ Developers regularly produce software artifacts ▪ GitHub: Version control system, issue tracker 13 Goals of the research and background How can we gauge (the effect of curriculum) changes in student behavior during project work?
14. ▪ Collected data from the last 5 course instalments ▪ Last 3 course iterations introduced Kanban ▪ The 2 before these used only Scrum Development Artifact Collection 14 Comparing development artifacts over course instalments
15. ▪ Collected data from the last 5 course instalments ▪ Last 3 course iterations introduced Kanban ▪ The 2 before these used only Scrum Crawled GitHub APIs and extracted artifacts Development Artifact Collection 15 Comparing development artifacts over course instalments commits tickets / user stories
16. Development Artifact Analysis Kanban usage had some noticeable effects 16 Gaining insights into team behaviors
17. Development Artifact Analysis Kanban usage had some noticeable effects ▪ Higher mean amount of non-comment events ▪ Assigning labels (status, priority) & developers (responsibility) → More interaction with issues 17 Gaining insights into team behaviors
18. Development Artifact Analysis Kanban usage had some noticeable effects ▪ Higher mean amount of non-comment events ▪ Assigning labels (status, priority) & developers (responsibility) → More interaction with issues ▪ Commits towards end of Sprint higher in Kanban ▪ Scrum (planning) vs Kanban (dynamic) → Better ability to adapt to changes 18 Gaining insights into team behaviors
19. Development Artifact Analysis Key development artifacts measures did not change significantly ▪ Mean amount of commits & touched files ▪ Mean line changes per commit ▪ Mean amount of unique issues referenced ▪ Mean issues closed, mean comments 19 Gaining insights into team behaviors
20. Development Artifact Analysis Hypotheses regarding changes in artifacts were violated 20 Gaining insights into team behaviors
21. Development Artifact Analysis Hypotheses regarding changes in artifacts were violated ▪ Similar percentage of issues opened and closed by same person ▪ No dedicated Product Owner role → Expected higher engagement of entire team 21 Gaining insights into team behaviors
22. Development Artifact Analysis Hypotheses regarding changes in artifacts were violated ▪ Similar percentage of issues opened and closed by same person ▪ No dedicated Product Owner role → Expected higher engagement of entire team ▪ No change in user story length ▪ No need to estimate, focus on throughput → Expected smaller user stories 22 Gaining insights into team behaviors
23. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) 23 Asking the questions that actually matter
24. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) ▪ “Was the Kanban sprint more useful and productive than another Scrum sprint?” ▪ Yes!, mean 4.08 24 Asking the questions that actually matter
25. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) ▪ “Was the Kanban sprint more useful and productive than another Scrum sprint?” ▪ Yes!, mean 4.08 ▪ “Did you adapt your workflow?” ▪ Yes., 3.83 25 Asking the questions that actually matter
26. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) ▪ “Was the Kanban sprint more useful and productive than another Scrum sprint?” ▪ Yes!, mean 4.08 ▪ “Did you adapt your workflow?” ▪ Yes., 3.83 ▪ “Biggest (dis)advantages of Kanban?” (free text) ▪ Advantages: Efficiency & Autonomy ▪ Drawbacks: Only work on small stories, uneven task distribution 26 Asking the questions that actually matter
27. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) ▪ “How did user stories change from using Scrum to Kanban?” ▪ More bug-oriented (11 mentions) ▪ Shorter (11 mentions) ▪ With more detailed requirements (8 mentions) 27 Asking the questions that actually matter
28. Kanban Survey Survey in 2017/18 course instalment (N=18, 5 point Likert scale) ▪ “How did user stories change from using Scrum to Kanban?” ▪ More bug-oriented (11 mentions) ▪ Shorter (11 mentions) ▪ With more detailed requirements (8 mentions) ▪ “Would you recommend using Kanban to next year’s participants?” ▪ YES!, mean 4.33 28 Asking the questions that actually matter
29. Summary & Conclusion ▪ Kanban introduction was liked by students, but w/ mixed success 29 Take-away messages christoph.matthies@hpi.de @chrisma0
30. Summary & Conclusion ▪ Kanban introduction was liked by students, but w/ mixed success ▪ Development artifacts represent another dimension of analysis ▪ Beyond the perceptions of students ▪ Based on data naturally produced, high “response rate” 30 Take-away messages christoph.matthies@hpi.de @chrisma0
31. Summary & Conclusion ▪ Kanban introduction was liked by students, but w/ mixed success ▪ Development artifacts represent another dimension of analysis ▪ Beyond the perceptions of students ▪ Based on data naturally produced, high “response rate” ▪ Analysis allowed finding those areas where expectations are… ▪ Confirmed ▪ Violated! (even more interesting) 31 Take-away messages christoph.matthies@hpi.de @chrisma0
32. Summary & Conclusion ▪ Kanban introduction was liked by students, but w/ mixed success ▪ Development artifacts represent another dimension of analysis ▪ Beyond the perceptions of students ▪ Based on data naturally produced, high “response rate” ▪ Analysis allowed finding those areas where expectations are… ▪ Confirmed ▪ Violated! (even more interesting) → Opportunity for conversation and improvement 32 Take-away messages christoph.matthies@hpi.de @chrisma0
33. Image Credits 33 In order of appearance ▪ Archaeologist by Gan Khoon Lay from the Noun Project (CC BY 3.0 US) ▪ Mortar Board by Mike Chum from the Noun Project (CC BY 3.0 US) ▪ Target by Arthur Shlain from the Noun Project (CC BY 3.0 US) ▪ Process by Laymik from the Noun Project (CC BY 3.0 US) ▪ Questions by Gregor Cresnar from the Noun Project (CC BY 3.0 US) ▪ Data collection by H Alberto Gongora from the Noun Project (CC BY 3.0 US) ▪ Search Code by icon 54 from the Noun Project (CC BY 3.0 US) ▪ Clipboard by David from the Noun Project (CC BY 3.0 US) ▪ Idea by Gilbert Bages from the Noun Project (CC BY 3.0 US)