Using GitHub in the classroom predicts student learning outcomes and classroom experiences Vanessa Gennarelli GitHub Education

We wanted to find out: • Student experiences • Using GitHub for classroom vs. not • Which / what kind of implementations work best?

GitHub in Education • GitHub is recognized as an industry standard and developers view knowing how to use GitHub as an essential part of securing a job • >31M developers building on GitHub • >20k teachers have used GitHub in their courses • >1.3M students have learned to code with GitHub

Past research: what’s the impact of using GitHub? • Teachers are motivated to adopt GitHub is so students get familiar with an industry tool, which makes them more competitive on the job market (Feliciano) • Boosts student engagement in the classroom (Gunnarsson et al. 2017) • Students develop soft skills that prepare them for working in industry (Feliciano, Story, and Zagalsky 2016)

Methodology

Survey design overview Participants • Both users and non-users of GitHub participated • We emailed students with currently active student coupons • We emailed teachers who applied for the teacher discount either for personal repositories or for free organization Survey eligibility • Must have taken or taught a class with a programming component this past semester Two versions of the survey • Student version (n = 7530) • Teacher version (n = 300)

Respondent Characteristics: Year in school Student Survey Teacher Survey 7.8% 12.5% 15.4% 33.7% 1.2% 1.3% 2.3% 5.1% 3.3% 17.4% College Freshman College Sophomore College Junior College Senior HS Freshman HS Sophomore HS Junior HS Senior Non-traditional student Other Year in school What year in school was the average student in your course? 8.2% 20.4% 17.0% 17.0% 2.0% 1.4% 6.1% 5.8% 2.0% 20.1%

Diary study overview Study structure: • 4-month long structured diary study • Data collection through: • Online surveys (open-ended and scale responses) • In-depth interviews at the beginning and at the end of the semester Participants • N = 8 (3 female; 5 male) • New to GitHub or slightly familiar with GitHub • Undergraduate students (1 freshman, 1 sophomore, 5 juniors, 1 senior) • Instructor implemented GitHub in the classroom this semester Schedule: Initial: Week 1 - 2 Survey 4: Week 5 - 6 Survey 1: Week 2 - 3 Survey 5: Week 7 - 8 Survey 2: Week 3 - 4 Survey 6: Week 9 - 10 Survey 3: Week 4 - 5 Survey 7: Week 14 - 15

Findings

Survey results: Prepared for the future: GitHub versus non GitHub classrooms Student Survey Teacher Survey Rate how much you have learned about each of the following through [course] this past semester: Rate how much your students have learned about each of the following through [course] this past semester: 1 2 3 4 5 Teamwork and collaboration Popular industry tool(s) Project management Very much Very little 1 2 3 4 5 Teamwork and collaboration Popular industry tool(s) Project management • Student version (n = 7530) • Teacher version (n = 300)

The specific GitHub implementation matters Student Survey Teacher Survey 1 2 3 4 5 Fewer features (-1SD) More features (+1SD) Very much Very little Teamwork and collaborations Project management Preparation for future internship/career Preparation for being part of the developer community 1 2 3 4 5 Fewer features (-1SD) More features (+1SD) Very much Very little Teamwork and collaborations Project management Preparation for future internship/career Preparation for being part of the developer community Rate how much [course] has prepared you for each of the following Rate how much [course] has prepared your students for each of the following • Student version (n = 7530) • Teacher version (n = 300)

Diary study: GitHub features Which of the following GitHub features have you used in your class? (n = 8)

• Students in the study felt that, about halfway through the semester, they stopped learning new things about GitHub and felt comfortable using GitHub for class. • However, students who only interacted with a limited set of GitHub features wished they could have learned more about GitHub features in their course • “The only time we really used GitHub was pulling the starter code for our repositories. Unfortunately that was about it, which is a little disappointing because I thought we would be using it more…I wanted to get a better understanding of everything it could do…” • “I wish we would’ve gone more in depth…I didn’t really want to explore features on my own without someone else to show me what it is…definitely think there should have been more instruction…forks come to mind. I’m not sure what that is.” • “I know I want to use my GitHub as a portfolio, but I’ve never had a teacher show me how to do that. I don’t want to mess up.” Diary study: Exposure to GitHub features N = 8

Survey results: Instructor Feedback Student Survey 1 2 3 4 5 6 7 I understand the instructor's feedback I pay attention to the instructor's feedback I use the instructor's feedback effectively I find the instructor's feedback helpful The instructor understands my needs as a student Strongly agree Strongly disagree Used GitHub to provide feedback Did not use GitHub to provide feedback ns ns Use your general impression of the instructor’s feedback to rate the items below: • Student version (n = 7530)

Student Survey 1 2 3 4 5 6 7 I understand feedback from peers I pay attention to feedback from peers I use feedback from peers effectively I find feedback from peers helpful Strongly agree Strongly disagree Used GitHub to provide feedback Did not use GitHub to provide feedback ns ns Survey results: Peer Feedback • Student version (n = 7530)

• Students’ sense of belonging can positive predict variables tied to academic success, such as intrinsic motivation and academic self-efficacy (Freeman, Anderman, and Jensen 2007) Both Belonging in class and Belonging in field variables were composite scores of multiple items: Use your general impression of [course] to rate the items below 1 (Strongly disagree) to 7 (Strongly agree) •Belonging in class: •My contributions in [course] are valued •I feel comfortable in [course] •People in [course] accept me •Belonging in field: •I see myself as part of the developer community •I feel that I am a member of the developer community •I feel a sense of belonging to the developer community Survey results: Feelings of belonging

Survey results: Sense of belonging: GitHub versus non GitHub classrooms 1 2 3 4 5 6 7 Belonging in class Belonging in field High Low Student Survey • Student version (n = 7530)

• Learning to use GitHub was one variable that directly contributed to students’ feelings of belonging: • “[Using GitHub] increased my feelings of belonging because learning tools like that always does…it is a very common and necessary tool. I feel like it increased my…desire to be a programmer.” • “…I actually switched out of CS at one point because I had a lot of problems with being a girl in CS…sometimes it’s difficult with the amount of boys and it being an old boys’ club…it’s better now…” (Before Survey 1) • “…over the course of the semester, I feel a lot more that I can identify with the label that I am a developer because I am more familiar with how GitHub works…I don’t feel like an imposter…” • “…now that I know how to use GitHub… I think there’s a lot more that I can do to kind of get involved like with other developers…[GitHub] provides a place to do all of that. If I want to do this as a career path…it’s important that I’m not just working by myself because that’s not how I’m going to work after college…” Diary study: Feelings of belonging

Survey results: Learning GitHub Student Survey Teacher Survey 49% 36% 13% 3% 1% 2% 14% 54% 26% 4% 0% 10% 20% 30% 40% 50% 60% Not at all familiar Slightly familiar Moderately familiar Very familiar Extremely familiar How familiar were your students with GitHub at the beginning of the semester? How familiar are your students with GitHub now? 8% 21% 26% 29% 15% 0% 0% 22% 49% 26% 0% 10% 20% 30% 40% 50% 60% Not at all familiar Slightly familiar Moderately familiar Very familiar Extremely familiar How familiar were you with GitHub at the beginning of the semester? How familiar are you with GitHub now? • Student version (n = 7530) • Teacher version (n = 300)

• On average, at the beginning of the semester, participants expected needing 2 to 2.5 months before feeling familiar with GitHub • Participants who were not at all to slightly familiar with GitHub at the beginning of the semester were moderately familiar with GitHub by the end of the semester. “Creating a repo…using the branches and pull requests…I feel like in the beginning of the semester I would have been scared to do all that but like now…it’s just like easy, comfortable.” How familiar are you with GitHub right now? 1 2 3 4 5 Initial Survey 1 Survey 2 Survey 3 Survey 4 Survey 5 Survey 6 Survey 7 Extremely familiar Moderately familiar Not at all familiar Diary study: Learning GitHub (n = 8)

Survey results: Learning GitHub Student Survey Teacher Survey 30% 27% 28% 14% 1% 0% 10% 20% 30% 40% 1 2 3 4 5 Easy Difficult How easy or difficult was it to learn how to use GitHub? 8% 24% 29% 34% 6% 0% 10% 20% 30% 40% 1 2 3 4 5 Easy Difficult How easy or difficult was it for the average student in [course] to learn how to use GitHub? • Student version (n = 7530)

Survey: Support while learning GitHub Student Survey Teacher Survey 10% 15% 55% 13% 8% 12% 20% 47% 15% 6% 0% 10% 20% 30% 40% 50% 60% Too little support Just the right amount of support Too much support Describe the amount of support you received from each of the following as you were learning GitHub [course] teachers [course] peers 1% 11% 59% 19% 6% 5% 9% 19% 45% 13% 1% 13% 0% 10% 20% 30% 40% 50% 60% Too little support Just the right amount of support Too much support I don't know/not sure Describe the amount of support students received on average from each of the following as they were learning GitHub [course] teachers [course] peers • Student version (n = 7530)

• Students who received too little support from instructors thought instructors were making assumptions about their familiarity with GitHub and typically relied on online resources. “The instructors assume we know it more…” “It’s mostly googling what I need to figure out.” “I would like my instructor to show us how to use GitHub in depth, especially in an intro class like the one I am in.” Support while learning GitHub

• Access to resources made learning to use GitHub easy • “Good resources to learn it from my class.” • New terminology made learning to use GitHub difficult • “There are many commands that GitHub does not make clear to learn…” 1 2 3 4 5 Survey 1 Survey 2 Survey 3 Survey 4 Survey 5 Survey 6 Survey 7 Difficult Neither easy nor difficult Easy How easy or difficult is it learning to use GitHub right now? Diary study: Ease of learning GitHub (n = 8)

• Nearly all students relied on peers at the beginning of the semester. • However, when there was a perceived gap in familiarity between the participant and their peers, they may have been more hesitant to reach out to peers for help. “I feel like that [asking peers] will be a very last resort because it’s also like embarrassing to ask them” • As students started collaborating with peers on larger group projects towards the end of the semester, reliance on peers for support increased. Diary study: Support while learning GitHub (n = 8)

Diary study: Exploration of GitHub Percentage of participants who explored GitHub beyond what was necessary for their course 0% 20% 40% 60% 80% 100% Survey 1 Survey 2 Survey 3 Survey 4 Survey 5 Survey 6 Survey 7 (n = 8)

• Some participants felt their course did not motivate them to explore GitHub beyond what was necessary for the course: • “I think they could have added in part of the project…like, you have to make a branch or like make a pull request…to like understand what it is. Just like add little elements into the project so you can explore additional functionalities. You can work that into the project…just a little extra part…then I’d definitely walk away with a bigger understanding” • “At a certain point, we were learning less in the classroom…Like, everything we were doing for the class stayed at one level…the responsibility to learn more shifted to me personally…” Exploration of GitHub

Conclusion

Conclusion 1.Using GitHub for coursework (versus not using GitHub) predicts outcomes around: • Teamwork and collaboration, popular industry tool(s), and project management • Sense of belonging in the field 2.More features used, more outcomes experienced 3.Feedback is more effective , helpful, and students felt understood by their teachers 4.Students feel moderately familiar with GitHub 2-2.5 months into the semester 5. Students may require more support when learning to use GitHub

GitHub Education education.github.com

Survey results: GitHub in the classroom predicts greater student preparedness for the future • Students in GitHub (versus non GitHub) classrooms learned more about: • Teamwork and collaboration • Popular industry tool(s) • Project management • Students in GitHub (versus non GitHub) were more prepared for: • Being a part of the developer community • Developing a portfolio of their work