CPython loves your Pull Requests

by Stéphane Wirtel

Slide 1

Slide 1 text

CPython loves your Pull Requests by Stéphane Wirtel 1 / 89

Slide 2

Slide 2 text

I live in Belgium @matrixise PythonFOSDEM CPython contributor #fellow member of @ThePSF member of EuroPython Society I am Stéphane 2 / 89

Slide 3

Slide 3 text

Once upon a time... 3 / 89

Slide 4

Slide 4 text

for a bunch of happy developers, 4 / 89

Slide 5

Slide 5 text

the main tool for the management, for the issues, and the contributions was the issue tracker! 5 / 89

Slide 6

Slide 6 text

The Issue Tracker bugs.python.org aka b.p.o 6 / 89

Slide 7

Slide 7 text

The Issue Tracker could only handle the contributions as patches 7 / 89

Slide 8

Slide 8 text

Yes, patches ! 8 / 89

Slide 9

Slide 9 text

Example 9 / 89

Slide 10

Slide 10 text

Example We can have patches for the same issue 10 / 89

Slide 11

Slide 11 text

Reviews with Rietveld 11 / 89

Slide 12

Slide 12 text

Rietveld 12 / 89

Slide 13

Slide 13 text

Rietveld 13 / 89

Slide 14

Slide 14 text

Rietveld Custom fork of the official Rietveld Maintained by one person Not really sexy No hightligted diff Not perfectly integrated with the Bug Tracker 14 / 89

Slide 15

Slide 15 text

Rietveld Custom fork of the official Rietveld Maintained by one person Not really sexy No hightligted diff Not perfectly integrated with the Bug Tracker 15 / 89

Slide 16

Slide 16 text

Typical Work ow 16 / 89

Slide 17

Slide 17 text

Typical Work ow As a contributor Open an issue Checkout the CPython source code from hg.python.org Make the fix Upload a patch Have a review with Rietveld 17 / 89

Slide 18

Slide 18 text

Typical Work ow As a core dev Download the patch Make sure it still applies cleanly Fix the conflicts or ask a new patch to the contributor Run test suite manually (10-15 min) Commit the change manually Write the commit message Commit in the maintenance branches first and then merged in the default branch Run the test suite manually again Commit the merge Push the changes 18 / 89

Slide 19

Slide 19 text

Typical Work ow About the patch Need to fix the conflicts Try to find the original author No description about the behaviour of the patch Problem, sometimes, there is no reference about the revision Difficult to rebase diff --git a/Misc/python.man b/Misc/python.man --- a/Misc/python.man +++ b/Misc/python.man 19 / 89

Slide 20

Slide 20 text

Typical Work ow About the continuous integration Check the buildbots (between 1h & 1d... if we are lucky) Frequent bugs on Windows the developers are on Linux or OSX or just too lazy If green, port to several branches can introduce new bugs in the other branches risk of conflicts tests executed by Buildbot 20 / 89

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Typical Work ow Current limitation Not enought reviewers for the issues/patches Difficult to update the MISC/News file 22 / 89

Slide 23

Slide 23 text

Typical Work ow Status Download the sources from https://hg.python.org Use Patches Store them on bugs.python.org Use unmaintained version of Rietveld Complex Workflow 23 / 89

Slide 24

Slide 24 text

Issue with the work ow! 24 / 89

Slide 25

Slide 25 text

Di culty for the contribution 25 / 89

Slide 26

Slide 26 text

We have to change the Work ow 26 / 89

Slide 27

Slide 27 text

How? 27 / 89

Slide 28

Slide 28 text

PEP 507 PEP 512 PEP 507: Migrating to GitLab PEP 512: Migration from hg.python.org to Git/GitHub 28 / 89

Slide 29

Slide 29 text

Why Git? Based on the statistics of The Open Hub and PyPI itself. 29 / 89

Slide 30

Slide 30 text

The Open Hub 48% for SVN 37% for Git 2% for Mercurial PyPI 62% for Git 22% for Mercurial 13% for the rest Why Git? Git is 3 times more popupar as Mercurial for the top 100 projects on PyPI. 30 / 89

Slide 31

Slide 31 text

Why GitHub? 31 / 89

Slide 32

Slide 32 text

Why GitHub? GitHub is popular Primary place if you want to contribute to a project Avoid a custom infrastructure for the volunteers Example, Rietveld was a custom fork and not maintained a lot of developers are familiar with GitHub GitHub Pull Requests has a major advantage over the older "submit a patch to a bug tracker" model. GitHub Pull Requests are easier to review have nice syntax hightligted diffs can be commented 32 / 89

Slide 33

Slide 33 text

GitHub - Web interface 33 / 89

Slide 34

Slide 34 text

GitHub - Web interface 34 / 89

Slide 35

Slide 35 text

GitHub - Web interface 35 / 89

Slide 36

Slide 36 text

GitHub - Web interface 36 / 89

Slide 37

Slide 37 text

GitHub - Dashboards 37 / 89

Slide 38

Slide 38 text

GitHub - Reviews Integrated review Comment the code and ask for changes 38 / 89

Slide 39

Slide 39 text

GitHub - Reviews - Code Owners Auto attribution of the reviewers for a PR .github/CODEOWNERS */*asyncio* @1st1 */*import* @python/import-team */*ssl* @python/crypto-team */*mail* @python/email-team */*subprocess* @gpshead /PC/ @python/windows-team */*itertools* @rhettinger */*functools* @ncoghlan @rhettinger */*idlelib* @terryjreedy https://help.github.com/articles/about-codeowners/ 39 / 89

Slide 40

Slide 40 text

GitHub - Reviews - Ready to merge 40 / 89

Slide 41

Slide 41 text

GitHub - API REST API GraphQL API Really interesting for statistics 41 / 89

Slide 42

Slide 42 text

Migrations to GitHub 42 / 89

Slide 43

Slide 43 text

15th January 2016 PEPs https://github.com/python/peps 43 / 89

Slide 44

Slide 44 text

22nd July 2016 DevGuide https://github.com/python/devguide 44 / 89

Slide 45

Slide 45 text

11st February 2017 CPython https://github.com/python/cpython 45 / 89

Slide 46

Slide 46 text

Anish Shah Barry Warsaw Benjamin Peterson Berker Peksag Carol Willing Donald Stufft Ernest W Durbin Ezio Melotti Maciej Szulik Mariatta Wijaya Nick Coghlan Oleg Broytman Senthil Kumaran and me Seriously, you can congratulate them Orchestrated by Brett Cannon and these volunteers https://mail.python.org/pipermail/python-committers/2017-February/004220.html 46 / 89

Slide 47

Slide 47 text

Automation 47 / 89

Slide 48

Slide 48 text

Run the test suite Run the code coverage Run the performance tests Compile the documentation Execute make patchcheck Check if files have been generated Execute for master and the stable branches (2.7, 3.6) Execute on the branches from the contributors Automation 48 / 89

Slide 49

Slide 49 text

The BOTS... 49 / 89

Slide 50

Slide 50 text

Job checks the title of the PR with bedevere/issue-number has a Misc/NEWS.d/next entry with bedevere/news play with the labels awaiting review, awaiting merge, awaiting core review Bedevere! https://github.com/python/bedevere 50 / 89

Slide 51

Slide 51 text

Job checks if the the commiter has signed the Contributor Agreement aka CLA The knights who say ni! https://github.com/python/the-knights-who-say-ni CLA: https://www.python.org/psf/contrib/contrib-form/ 51 / 89

Slide 52

Slide 52 text

The knights who say ni! if CLA is signed 52 / 89

Slide 53

Slide 53 text

The knights who say ni! if CLA is not signed 53 / 89

Slide 54

Slide 54 text

Job backports a Pull Request to an other branch based on needs backport to X.Y Miss Islington! https://github.com/python/miss-islington 54 / 89

Slide 55

Slide 55 text

BOTS + Automations = 55 / 89

Slide 56

Slide 56 text

BOTS + Automations = 56 / 89

Slide 57

Slide 57 text

BOTS + Automations = 57 / 89

Slide 58

Slide 58 text

BOTS + Automations = 58 / 89

Slide 59

Slide 59 text

as you can see BOTS are really useful for the core-devs 59 / 89

Slide 60

Slide 60 text

the MISC/News le 60 / 89

Slide 61

Slide 61 text

Problem the Misc/NEWS file is a very big file containing all the descriptions for each change. really difficult to merge when you receive a PR or a patch. => it's a real pain for the core-developers and the contributors discussion is here: https://github.com/python/core-workflow/issues/6 61 / 89

Slide 62

Slide 62 text

62 / 89

Slide 63

Slide 63 text

Solution: blurb by Larry Hastings (Thank you) It's a CLI tool which manage the Misc/NEWS.d entries an individual file per news entry! > cat Misc/NEWS.d/next/IDLE/2018-03-05-01-29-05.bpo-32984.NGjgT4.rst And it's a ReStructured Text file format ;-) https://github.com/python/core-workflow 63 / 89

Slide 64

Slide 64 text

Status about the Work ow 64 / 89

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Before Contributor Open an issue Clone Upload a patch Core Dev Wait a review of a Core Dev Download the patch Fix the conflicts Run the tests manually Run the code coverage manually Commit manually Write the commit message Merge into the branches Run the test suite manually Commit the merge Push the changes etc... process is long & manual & error prone After Contributor Open an issue Clone Create a Pull Request Core Dev Review from contributors or core-dev Wait the feedback of the BOTS + Automation Wait the signal Mark the PR as needs backport to X.Y process is short & automatic 66 / 89

Slide 67

Slide 67 text

Before GitHub 67 / 89

Slide 68

Slide 68 text

After GitHub 68 / 89

Slide 69

Slide 69 text

Statistics from Feb 2017 to Apr 2018 April is not yet nished 69 / 89

Slide 70

Slide 70 text

6542 70 / 89

Slide 71

Slide 71 text

6542 The number of Pull Requests! 71 / 89

Slide 72

Slide 72 text

Pull Requests: 6542 72 / 89

Slide 73

Slide 73 text

848 73 / 89

Slide 74

Slide 74 text

848 The number of Contributors of CPython 74 / 89

Slide 75

Slide 75 text

848 The number of Contributors of CPython 75 / 89

Slide 76

Slide 76 text

848 The number of Contributors of CPython and 34 Core Devs 76 / 89

Slide 77

Slide 77 text

Number of Core-Devs & Contributors 77 / 89

Slide 78

Slide 78 text

Pull Requests from Core-Devs & Contributors 78 / 89

Slide 79

Slide 79 text

Pull Requests from Miss-Islington 79 / 89

Slide 80

Slide 80 text

Merge Time 80 / 89

Slide 81

Slide 81 text

Merge Time 81 / 89

Slide 82

Slide 82 text

and the winner of 368 days is 82 / 89

Slide 83

Slide 83 text

Top Merge Time 83 / 89

Slide 84

Slide 84 text

Merged Pull Requests: 4966 84 / 89

Slide 85

Slide 85 text

Yeah! CPython loves your Pull Requests 85 / 89

Slide 86

Slide 86 text

Yeah! CPython loves your Pull Requests and 86 / 89

Slide 87

Slide 87 text

We are ready for the future! 87 / 89

Slide 88

Slide 88 text

Last thing ;-) Now, you can translate the official documentation in your mother language. with PEP 545 88 / 89

Slide 89

Slide 89 text

[email protected] @mgxio 89 / 89