CPython loves your Pull Requests

In this talk, I will present you the reason of the move to GitHub and the status after this move. Please read it and you can be ready for your future contribution, because we are ready for your Pull Requests.

Transcript

1. CPython loves your Pull Requests by Stéphane Wirtel 1 / 89
2. I live in Belgium @matrixise PythonFOSDEM CPython contributor #fellow member of @ThePSF member of EuroPython Society I am Stéphane 2 / 89
3. Once upon a time... 3 / 89
4. for a bunch of happy developers, 4 / 89
5. the main tool for the management, for the issues, and the contributions was the issue tracker! 5 / 89
6. The Issue Tracker bugs.python.org aka b.p.o 6 / 89
7. The Issue Tracker could only handle the contributions as patches 7 / 89
8. Yes, patches ! 8 / 89
9. Example 9 / 89
10. Example We can have patches for the same issue 10 / 89
11. Reviews with Rietveld 11 / 89
12. Rietveld 12 / 89
13. Rietveld 13 / 89
14. Rietveld Custom fork of the official Rietveld Maintained by one person Not really sexy No hightligted diff Not perfectly integrated with the Bug Tracker 14 / 89
15. Rietveld Custom fork of the official Rietveld Maintained by one person Not really sexy No hightligted diff Not perfectly integrated with the Bug Tracker 15 / 89
16. Typical Work ow 16 / 89
17. Typical Work ow As a contributor Open an issue Checkout the CPython source code from hg.python.org Make the fix Upload a patch Have a review with Rietveld 17 / 89
18. Typical Work ow As a core dev Download the patch Make sure it still applies cleanly Fix the conflicts or ask a new patch to the contributor Run test suite manually (10-15 min) Commit the change manually Write the commit message Commit in the maintenance branches first and then merged in the default branch Run the test suite manually again Commit the merge Push the changes 18 / 89
19. Typical Work ow About the patch Need to fix the conflicts Try to find the original author No description about the behaviour of the patch Problem, sometimes, there is no reference about the revision Difficult to rebase diff --git a/Misc/python.man b/Misc/python.man --- a/Misc/python.man +++ b/Misc/python.man 19 / 89
20. Typical Work ow About the continuous integration Check the buildbots (between 1h & 1d... if we are lucky) Frequent bugs on Windows the developers are on Linux or OSX or just too lazy If green, port to several branches can introduce new bugs in the other branches risk of conflicts tests executed by Buildbot 20 / 89
21. Typical Work ow About the continuous integration Check the buildbots (between 1h & 1d... if we are lucky) Frequent bugs on Windows the developers are on Linux or OSX or just too lazy If green, port to several branches can introduce new bugs in the other branches risk of conflicts tests executed by Buildbot CI: Only on the stable branches (2.7, 3.4+) 21 / 89
22. Typical Work ow Current limitation Not enought reviewers for the issues/patches Difficult to update the MISC/News file 22 / 89
23. Typical Work ow Status Download the sources from https://hg.python.org Use Patches Store them on bugs.python.org Use unmaintained version of Rietveld Complex Workflow 23 / 89
24. Issue with the work ow! 24 / 89
25. Di culty for the contribution 25 / 89
26. We have to change the Work ow 26 / 89
27. How? 27 / 89
28. PEP 507 PEP 512 PEP 507: Migrating to GitLab PEP 512: Migration from hg.python.org to Git/GitHub 28 / 89
29. Why Git? Based on the statistics of The Open Hub and PyPI itself. 29 / 89
30. The Open Hub 48% for SVN 37% for Git 2% for Mercurial PyPI 62% for Git 22% for Mercurial 13% for the rest Why Git? Git is 3 times more popupar as Mercurial for the top 100 projects on PyPI. 30 / 89
31. Why GitHub? 31 / 89
32. Why GitHub? GitHub is popular Primary place if you want to contribute to a project Avoid a custom infrastructure for the volunteers Example, Rietveld was a custom fork and not maintained a lot of developers are familiar with GitHub GitHub Pull Requests has a major advantage over the older "submit a patch to a bug tracker" model. GitHub Pull Requests are easier to review have nice syntax hightligted diffs can be commented 32 / 89
33. GitHub - Web interface 33 / 89
34. GitHub - Web interface 34 / 89
35. GitHub - Web interface 35 / 89
36. GitHub - Web interface 36 / 89
37. GitHub - Dashboards 37 / 89
38. GitHub - Reviews Integrated review Comment the code and ask for changes 38 / 89
39. GitHub - Reviews - Code Owners Auto attribution of the reviewers for a PR .github/CODEOWNERS */*asyncio* @1st1 */*import* @python/import-team */*ssl* @python/crypto-team */*mail* @python/email-team */*subprocess* @gpshead /PC/ @python/windows-team */*itertools* @rhettinger */*functools* @ncoghlan @rhettinger */*idlelib* @terryjreedy https://help.github.com/articles/about-codeowners/ 39 / 89
40. GitHub - Reviews - Ready to merge 40 / 89
41. GitHub - API REST API GraphQL API Really interesting for statistics 41 / 89
42. Migrations to GitHub 42 / 89
43. 15th January 2016 PEPs https://github.com/python/peps 43 / 89
44. 22nd July 2016 DevGuide https://github.com/python/devguide 44 / 89
45. 11st February 2017 CPython https://github.com/python/cpython 45 / 89
46. Anish Shah Barry Warsaw Benjamin Peterson Berker Peksag Carol Willing Donald Stufft Ernest W Durbin Ezio Melotti Maciej Szulik Mariatta Wijaya Nick Coghlan Oleg Broytman Senthil Kumaran and me Seriously, you can congratulate them Orchestrated by Brett Cannon and these volunteers https://mail.python.org/pipermail/python-committers/2017-February/004220.html 46 / 89
47. Automation 47 / 89
48. Run the test suite Run the code coverage Run the performance tests Compile the documentation Execute make patchcheck Check if files have been generated Execute for master and the stable branches (2.7, 3.6) Execute on the branches from the contributors Automation 48 / 89
49. The BOTS... 49 / 89
50. Job checks the title of the PR with bedevere/issue-number has a Misc/NEWS.d/next entry with bedevere/news play with the labels awaiting review, awaiting merge, awaiting core review Bedevere! https://github.com/python/bedevere 50 / 89
51. Job checks if the the commiter has signed the Contributor Agreement aka CLA The knights who say ni! https://github.com/python/the-knights-who-say-ni CLA: https://www.python.org/psf/contrib/contrib-form/ 51 / 89
52. The knights who say ni! if CLA is signed 52 / 89
53. The knights who say ni! if CLA is not signed 53 / 89
54. Job backports a Pull Request to an other branch based on needs backport to X.Y Miss Islington! https://github.com/python/miss-islington 54 / 89
55. BOTS + Automations = 55 / 89
56. BOTS + Automations = 56 / 89
57. BOTS + Automations = 57 / 89
58. BOTS + Automations = 58 / 89
59. as you can see BOTS are really useful for the core-devs 59 / 89
60. the MISC/News le 60 / 89
61. Problem the Misc/NEWS file is a very big file containing all the descriptions for each change. really difficult to merge when you receive a PR or a patch. => it's a real pain for the core-developers and the contributors discussion is here: https://github.com/python/core-workflow/issues/6 61 / 89
62. 62 / 89
63. Solution: blurb by Larry Hastings (Thank you) It's a CLI tool which manage the Misc/NEWS.d entries an individual file per news entry! > cat Misc/NEWS.d/next/IDLE/2018-03-05-01-29-05.bpo-32984.NGjgT4.rst And it's a ReStructured Text file format ;-) https://github.com/python/core-workflow 63 / 89
64. Status about the Work ow 64 / 89
65. Before Contributor Open an issue Clone Upload a patch Core Dev Wait a review of a Core Dev Download the patch Fix the conflicts Run the tests manually Run the code coverage manually Commit manually Write the commit message Merge into the branches Run the test suite manually Commit the merge Push the changes etc... process is long & manual & error prone 65 / 89
66. Before Contributor Open an issue Clone Upload a patch Core Dev Wait a review of a Core Dev Download the patch Fix the conflicts Run the tests manually Run the code coverage manually Commit manually Write the commit message Merge into the branches Run the test suite manually Commit the merge Push the changes etc... process is long & manual & error prone After Contributor Open an issue Clone Create a Pull Request Core Dev Review from contributors or core-dev Wait the feedback of the BOTS + Automation Wait the signal Mark the PR as needs backport to X.Y process is short & automatic 66 / 89
67. Before GitHub 67 / 89
68. After GitHub 68 / 89
69. Statistics from Feb 2017 to Apr 2018 April is not yet nished 69 / 89
70. 6542 70 / 89
71. 6542 The number of Pull Requests! 71 / 89
72. Pull Requests: 6542 72 / 89
73. 848 73 / 89
74. 848 The number of Contributors of CPython 74 / 89
75. 848 The number of Contributors of CPython 75 / 89
76. 848 The number of Contributors of CPython and 34 Core Devs 76 / 89
77. Number of Core-Devs & Contributors 77 / 89
78. Pull Requests from Core-Devs & Contributors 78 / 89
79. Pull Requests from Miss-Islington 79 / 89
80. Merge Time 80 / 89
81. Merge Time 81 / 89
82. and the winner of 368 days is 82 / 89
83. Top Merge Time 83 / 89
84. Merged Pull Requests: 4966 84 / 89
85. Yeah! CPython loves your Pull Requests 85 / 89
86. Yeah! CPython loves your Pull Requests and 86 / 89
87. We are ready for the future! 87 / 89
88. Last thing ;-) Now, you can translate the official documentation in your mother language. with PEP 545 88 / 89
89. stephane.wirtel@mgx.io @mgxio 89 / 89