Characterization of the Xen Project Code Review Process

Many software development projects have introduced mandatory code review for every change to the code. This means that the project needs to devote a significant effort to review all proposed changes, and that their merging into the code base may get considerably delayed. Therefore, all those projects need to understand how code review is working, and the delays it is causing in time to merge.

This is the case in the Xen project, which performs peer review using mailing lists. During the first half of 2015, some people in the project observed a large and sustained increase in the number of messages related to code review, which had started some years before. This observation led to concerns on whether the code review process was having some trouble, and too large an impact on the overall development process.

Those concerns were addressed with a quantitative study, which is presented in this paper. Based on the information in code review messages, some metrics were defined to infer delays imposed by code review.

The study produced quantitative data suitable for informed discussion, which the project is using to understand its code review process, and to take decisions to improve it.

Presented in the MSR 2016.

Transcript

1. Characterization of the Xen Project Code Review Process An Exprience Report Daniel Izquierdo-Cortazar, Lars Kurth, Jesus M. Gonzalez-Barahona, Santiago Due˜ nas, Nelson Sekitoleko jgb@bitergia.com @jgbarah Bitergia / LibreSoft (URJC) Intl. Working Conference on Mining Software Repositories (MSR 2016) Austin (TX, USA), May 15th 2016 Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 1 / 15
2. The Xen code review process (patch) Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 2 / 15
3. The Xen code review process (patchset) Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 3 / 15
4. The problem (messages) Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 4 / 15
5. The problem (messages/commit) Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 5 / 15
6. The analysis: GQM Goal: Is the review process getting slower? Questions: Is it taking longer until reviews are decided? Are “patch series” having a large impact in code review performance? Metrics: Time to merge over time Time to merge per number of patches in patch series Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 6 / 15
7. The procedure Retrieve data from mailing lists into a database tool: MLStats, store in MySQL Retrieve data from git repository into a database tool: CVSAnalY, store in MySQL Produce a combined database linking threads to commits Produce analytics reports Python / Pandas / Jupyter Notebooks Discuss with the Xen community Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 7 / 15
8. The devil is in the details Low-hanging fruits are not enough Time to merge: from first message to commit Review threads difficult to detect: specific algorithm Matching of subjects to commits difficult: several patterns Heuristics for abandoned reviews Testing hypotesis... with the community Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 8 / 15
9. Time to merge by semester Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 9 / 15
10. Time to merge by size of patchset Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 10 / 15
11. Time to merge by semester (5+ patchsets) Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 11 / 15
12. Further work: continuous monitoring http://tinyurl.com/xenproject-dashboard Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 12 / 15
13. Takeaways Helping practitioners (from research to practice): Discussions are difficult when you only have opinions But having “some numbers” is not good enough Signifficant effort in understanding the processes... ...and the needs of the project Some good news: Developers appreciate useful numbers Measuring allows tracking Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 13 / 15
14. Characterization of the Xen Project Code Review Process Daniel Izquierdo-Cortazar, Lars Kurth, Jesus M. Gonzalez-Barahona, Santiago Due˜ nas, Nelson Sekitoleko The problem The metrics: Time to merge over time Time to merge per number of patches in patch series The results Takeaways: Developers appreciate useful numbers Measuring allows tracking Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 14 / 15
15. License c 2016 Bitergia Some rights reserved. This presentation is distributed under the “Attribution-ShareAlike 3.0” license, by Creative Commons, available at http://creativecommons.org/licenses/by-sa/3.0/ Izquierdo-Cortazar et al. (Bitergia/URJC) Xen Code Review MSR 2016 15 / 15