The Incredible Machine: Developer Productivity and the Impact of AI

Keynote at FSE 2024, Porto de Galinhas, Brazil.

Developer productivity is about more than an individual’s activity levels or the efficiency of the engineering systems, and it cannot be measured by a single metric or dimension. In this talk, I will discuss a decade of my productivity research. I will show how to use the SPACE framework to measure developer productivity across multiple dimensions to better understand productivity in practice. I will also discuss common myths around developer productivity and propose a collection of sample metrics to navigate around those pitfalls. Measuring developer productivity at Microsoft has allowed us to build new insights about the challenges remote work has introduced for software engineers, and how to overcome many of those challenges moving forward into a new future of work. Finally, I will talk about how I expect that the AI revolution will change developers and their productivity.

https://2024.esec-fse.org/

Transcript

1. The Incredible Machine 3: Developer Productivity and the Impact of AI on Productivity Thomas Zimmermann Microsoft Research FSE 2024, Porto de Galinhas, Brazil
5. Developer Productivity is like The Incredible Machine Many components/aspects Unintended side effects More complex than you think
6. Why does developer productivity matter?
7. 11 «Software is eating the world.» Marc Andreesen Image: Pixels Movie
8. Developer productivity is also important for software teams Create higher- quality software more efficiently Improve well-being and satisfaction Improve team collaboration
9. Software development is a team sport Developer Team Organization Consumer
10. What is productivity?
11. What is productivity? Productivity = Output Input
12. When are software developers productive?
13. Inspiration from fitness tracking
15. I have a productive workday when I get through my list of tasks for the day. I have at most one meeting. I have no code reviews to complete. I have a productive workday when I have autonomy on what I do and don't have to wait for response from other side of earth. I am focused on one work item and have all the inputs I need to close on that work item. I write more than 10 lines of code I wrote at least 1 line of code I complete the goals I set for myself that day. I am in a good mood and slept well :) When the weather is dry, and I can work at a silent and well-ventilated place. I get enough rest and can take nap when I feel tired. I am not randomized too much by meetings and interruptions from others André N. Meyer, Thomas Fritz, Gail C. Murphy, Thomas Zimmermann: Software developers' perceptions of productivity. SIGSOFT FSE 2014: 19-29
16. Observing developers
17. Observing developers 148 activity switches, 27 task switches
18. Observing developers Participants felt productive, yet switched frequently between tasks (13.5 times/hour) and between activities (47 times/hour)
19. Personal Analytics collects data about • Input activity (keyboard, mouse clicks) • Application usage • Meeting information • Email statistics • Self-reported productivity Personal Analytics André N. Meyer, Laura E. Barton, Gail C. Murphy, Thomas Zimmermann, Thomas Fritz: The Work Life of Developers: Activities, Switches and Perceived Productivity. IEEE Trans. Software Eng. 43(12): 1178-1193 (2017)
20. Retrospection with Personal Analytics André N. Meyer, Gail C. Murphy, Thomas Zimmermann, Thomas Fritz: Retrospecting on Work and Productivity: A Study on Self-Monitoring Software Developers' Work. Proc. ACM Hum. Comput. Interact. 1(CSCW): 79:1-79:24 (2017)
21. Productivity patterns Lower productivity during lunch Higher productivity in the afternoon Equal productivity throughout day André N. Meyer, Laura E. Barton, Gail C. Murphy, Thomas Zimmermann, Thomas Fritz: The Work Life of Developers: Activities, Switches and Perceived Productivity. IEEE Trans. Software Eng. 43(12): 1178-1193 (2017)
22. Modelling productivity Coding Time (0.18) + Review Time (0.18) + Other Time (0.09) + Time spent + Special events User + uid ([-0.93;1.45]) + I was travelling (-0.93) + I was a Designated Response Individual (-0.46) + I had more interruptions than usual (-0.49) + I had fewer interruptions than usual (0.58) + I slept well (0.31) + I did not sleep well (-0.38) + I worked longer than usual (0.24) + I worked shorter than usual (-0.35) + I had more or longer meetings than usual (-0.26) + I had fewer or shorter meetings than usual (0.23) + None of the above applied (-0.32) Email at the end of every workday. 81 participants. 5 weeks. 91% liked the emails. 74% completion rate. Moritz Beller, Vince R. Orgovan, Spencer Buja, Thomas Zimmermann: Mind the Gap: On the Relationship Between Automatically Measured and Self-Reported Productivity. IEEE Softw. 38(5): 24-31 (2021)
23. What factors influence developer satisfaction and productivity?
24. Managers Eirini Kalliamvakou, Christian Bird, Thomas Zimmermann, Andrew Begel, Robert DeLine, Daniel M. Germán: What Makes a Great Manager of Software Engineers? IEEE Trans. Software Eng. 45(1): 87-106 (2019)
25. Workdays feeling (372) stress overtime good or not good workday ability to work focused progress learn help others value creation (2725) efficient use of time (2164) meets expectations constructive discussions meaningful work productivity André N. Meyer, Earl T. Barr, Christian Bird, Thomas Zimmermann: Today Was a Good Day: The Daily Life of Software Developers. IEEE Trans. Software Eng. 47(5): 863-880 (2021)
26. Office space Brittany Johnson, Thomas Zimmermann, Christian Bird: The Effect of Work Environments on Productivity and Satisfaction of Software Engineers. IEEE Trans. Software Eng. 47(4): 736-757 (2021)
27. Towards a theory of satisfaction and productivity Margaret-Anne D. Storey, Thomas Zimmermann, Christian Bird, Jacek Czerwonka, Brendan Murphy, Eirini Kalliamvakou: Towards a Theory of Software Developer Job Satisfaction and Perceived Productivity. IEEE Trans. Software Eng. 47(10): 2125-2142 (2021)
28. We identified 45 factors important to developers from literature reviews and surveys at Microsoft (1000+ developers). Category Social and technical factor(s) important to developers Engineering systems and processes Engineering processes Hardware Engineering tools Collaboration tools Management Manager Work feedback Appreciation shown for work Well defined goals Clear priorities Autonomy Organization Organization culture Vision Personal productivity Perceived productivity Technical competency Ability to achieve goals Important member on one's team Rewards, benefits, career Rewards Salary Benefits Promotions Job security Lateral move opportunities Team Team culture Skilled co-workers Collaborative team Training/documentation Learning skills useful for future Availability of documentation Training for engineering technologies Training for soft skills Training for engineering tools Work type/impact Skills are well used Impactful work Type of work Working environment Physical working environment Working privately (interruption free) Proximity to team A private office / Number of people in workspace Work complexity Can complete tasks Job requires a lot of skill Time it takes to learn job Work complexity
29. We identified 24 challenges from literature reviews and surveys at Microsoft. Category Challenge(s) that may impede developer satisfaction and productivity Management Poorly defined goals Manager Organization Lack of vision Team Poorly qualified co-workers Team culture Personal productivity Technical competency Work type/impact Unclear requirements Legacy code Poor software architecture Too many external dependencies Development stack/architecture Interacting with people Engineering system and process Poor software development processes Poor engineering tools Poor hardware resources Poor collaboration tools Too many emails Too many meetings Too many communication channels Training/ documentation Insufficient training Finding relevant information Working environment Too many interruptions Lack of quiet space for work Work life Time to do required work
30. Challenges Factors A link between a challenge and factor indicates that when the challenge was present, the factor had statistically significant lower satisfaction.
32. SPACE: A framework for understanding developer productivity https://aka.ms/SpaceFramework
33. Satisfaction and well-being Performance Activity Communication and collaboration Efficiency and flow Nicole Forsgren, Margaret-Anne D. Storey, Chandra Shekhar Maddila, Thomas Zimmermann, Brian Houck, Jenna L. Butler: The SPACE of Developer Productivity: There's more to it than you think. ACM Queue 19(1): 20-48 (2021)
34. Myths and misconceptions about developer productivity • One productivity metric can tell you everything • Productivity is all about developer activity • Productivity is only about individual performance • Productivity measures are only useful for managers • Productivity comes down to engineering systems and developer tools
35. SPACE framework in action Individual One person Team or group People that work together Level System End-to-end work through a system (like a development pipeline) Satisfaction & Well-being How fulfilled, happy, and healthy one it Developer satisfaction Retention* Satisfaction with code reviews assigned Perception of code reviews Developer satisfaction Retention* Satisfaction with engineering system (e.g., CI/CD pipeline) Performance An outcome of a process Code review velocity Code review velocity Story points shipped* Code review velocity Code review (acceptance rate) Customer satisfaction Reliability (uptime) Activity The count of actions or outputs Number of code reviews completed Coding time # commits Lines of code* # story points completed* Frequency of deployments Communication & collaboration How people talk and work together Code review score (quality or thoughtfulness) PR merge times Quality of meetings* Knowledge sharing, discoverability (quality of documentation) Efficiency & flow Doing work with minimal delays or interruptions Code review timing Productivity perception Lack of interruptions PR merge times Quality of meetings* Knowledge sharing, discoverability (quality of documentation) Code review timing Handoffs Knowledge sharing, discoverability (quality of documentation) Code review timing Velocity/flow through the system * Use these metrics with (even more) caution – they can proxy more things.
36. Work from home and productivity
37. Multi-survey study How has productivity changed? Why? What is good about working from home? What is bad about working from home? What challenges have you encountered working from home? How has communication with your team members changed since working from home? How do you stay socially engaged with your colleagues (in place of hallway conversation)? Denae Ford, Margaret-Anne D. Storey, Thomas Zimmermann, Christian Bird, Sonia Jaffe, Chandra Shekhar Maddila, Jenna L. Butler, Brian Houck, Nachiappan Nagappan: A Tale of Two Cities: Software Developers Working from Home during the COVID-19 Pandemic. ACM Trans. Softw. Eng. Methodol. 31(2): 27:1-27:37 (2022)
38. Compared to working in office, how has your productivity changed? (N=2,254, April/May) 11.18% 25.60% 31.72% 25.51% 5.99% Significantly more productive More productive About the same Less productive Significantly less productive Answer Percent Significantly less productive 6.06% Less productive 31.79% About the same 31.79% More productive 22.06% Significantly more productive 8.30% Previous survey: March 16-23, N=1,357
39. How frequent (Prevalence) and how important (Importance) are these benefits? How has productivity changed (Delta) when a benefit is present? Statistically significant differences indicated with (*) What work-related benefits have you experienced working from home?
40. How frequent (Prevalence) and how impactful (Impact) are these challenges? How has productivity changed (Delta) when a challenge is present? Statistically significant differences indicated with (*) What challenges have you experienced working from home?
41. Recommendations for effective remote teams Build and maintain team culture. Include social activities as part of “work.” Be mindful of other people’s time. Actively work to be inclusive. Managers should stay aware of contributions. Managers should hold weekly 1:1 meetings. Hold regular team meetings. Create communication standards. Courtney Miller, Paige Rodeghero, Margaret-Anne D. Storey, Denae Ford, Thomas Zimmermann: "How Was Your Weekend?" Software Development Teams Working From Home During COVID-19. ICSE 2021: 624-636
42. Recommendations for remote onboarding Promote communication & asking for help. Encourage teams to turn cameras on. Schedule frequent 1:1 meetings. Provide information about the organization. Assign an onboarding buddy. Assign an onboarding technical mentor. Support multiple onboarding speeds. Assign a simple first task. Provide up-to-date documentation. Paige Rodeghero, Thomas Zimmermann, Brian Houck, Denae Ford: Please Turn Your Cameras on: Remote Onboarding of Software Developers During a Pandemic. ICSE (SEIP) 2021: 41-50
43. The Impact of AI on Productivity
44. Disruption: AI-powered code editing with GitHub Copilot
45. Disruption: Speed of innovation has increased
46. Disruption: The rise of AI-powered apps and Copilots
47. Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. ACM Queue, Volume 20, Issue 6, November/December 2022, pp 35–57 Developers reported spending less time on Stack Overflow due to Copilot’s code suggestions. Developers' roles shifted from primarily writing code to reviewing and understanding code suggested by AI. Copilot opened new learning opportunities like mastering new programming languages. Developers' trust plays a crucial role in adoption, as any unexpected behavior can significantly impact its usage.
48. #1 AI for the entire software lifecycle GitHub Copilot was focused on code editing within the IDE. Software creation is more than writing code. Huge opportunity to apply AI to the entire software lifecycle. The ultimate “shift left”?
49. #2 Help people build AI-powered software Future software will be AI-powered (“Aiware”). How can we build, test, and deploy these systems in a scalable and in a disciplined way? Important to think about now to avoid “AI debt”.
50. AIware is democratizing software creation
51. New AI-focused conferences in SE “Software for all and by all” is the future of humanity.
52. #3 Provide great human-AI interaction One of the reasons for the success of GitHub Copilot: it built on a familiar experience. Important to figure out the experience that humans interact with AI system. Design an experience that makes the interaction seamless. HCI must be considered from the beginning.
53. Mixed Reality Environments
54. #4 Leverage AI for software science Huge potential for AI to be used in research design, data analysis. Great brainstorming partner. But keep in mind: AI isn't perfect, so people need to vet suggestions. Role of research is changing given the rapid speed of innovation.
55. Can GPT-4 Replicate Empirical Software Engineering Research? Jenny T. Liang, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, Thomas Zimmermann Talk: Thursday at 11:00 in Room Mandacaru Session Human Aspects 2 Poster: Thursday at 15:30-16:00 in the Lounge AI-generated images may be incorrect. None of the authors wore a lab coat during this research. :-)
56. #5 Apply AI in a responsible way How to design and build software systems using AI in a responsible, ethical way that users can trust and do not negatively affect society? What mechanism and regulations do we put into place to oversee AI systems?
57. Will AI replace software engineers?
58. Despite these advances, the role of humans in software development will remain essential. Here's why: 1. Creative Problem Solving: While AI can automate routine tasks, human developers are needed for their creativity, strategic thinking, and ability to solve complex, unstructured problems. They will focus more on designing solutions, making high-level decisions, and defining the behaviors and capabilities of software systems. 2. Understanding User Needs: Humans excel at empathizing with other humans, understanding their needs, and designing intuitive interfaces. This deep understanding of the user is key to developing software that meets their needs and provides a satisfying user experience. 3. Ethics and Oversight: As AI becomes more prevalent, the importance of ethical considerations and oversight increases. Human developers will play a critical role in ensuring that AI systems behave fairly, transparently, and responsibly. So, while the future of software development will be shaped significantly by AI, it will continue to be a human-centric field. Humans and AI will work together in a symbiotic relationship, each contributing their unique strengths to create more robust, efficient, and user-friendly software.
59. Thanks to my amazing collaborators!