Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CPython loves your Pull Requests

CPython loves your Pull Requests

In this talk, I will present you the reason of the move to GitHub and the status after this move. Please read it and you can be ready for your future contribution, because we are ready for your Pull Requests.

Stéphane Wirtel

November 19, 2017
Tweet

More Decks by Stéphane Wirtel

Other Decks in Programming

Transcript

  1. CPython loves your Pull Requests
    by Stéphane Wirtel
    1 / 89

    View Slide

  2. I live in
    Belgium
    @matrixise
    PythonFOSDEM
    CPython contributor
    #fellow member of @ThePSF
    member of EuroPython Society
    I am Stéphane
    2 / 89

    View Slide

  3. Once upon a time...
    3 / 89

    View Slide

  4. for a bunch of happy developers,
    4 / 89

    View Slide

  5. the main tool for the management,
    for the issues, and the contributions
    was
    the issue tracker!
    5 / 89

    View Slide

  6. The Issue Tracker
    bugs.python.org aka b.p.o
    6 / 89

    View Slide

  7. The Issue Tracker
    could only handle the contributions
    as patches
    7 / 89

    View Slide

  8. Yes, patches !
    8 / 89

    View Slide

  9. Example
    9 / 89

    View Slide

  10. Example
    We can have patches for the same issue
    10 / 89

    View Slide

  11. Reviews with Rietveld
    11 / 89

    View Slide

  12. Rietveld
    12 / 89

    View Slide

  13. Rietveld
    13 / 89

    View Slide

  14. Rietveld
    Custom fork of the official Rietveld
    Maintained by one person
    Not really sexy
    No hightligted diff
    Not perfectly integrated with the Bug Tracker
    14 / 89

    View Slide

  15. Rietveld
    Custom fork of the official Rietveld
    Maintained by one person
    Not really sexy
    No hightligted diff
    Not perfectly integrated with the Bug Tracker
    15 / 89

    View Slide

  16. Typical Work ow
    16 / 89

    View Slide

  17. Typical Work ow
    As a contributor
    Open an issue
    Checkout the CPython source code from hg.python.org
    Make the fix
    Upload a patch
    Have a review with Rietveld
    17 / 89

    View Slide

  18. Typical Work ow
    As a core dev
    Download the patch
    Make sure it still applies cleanly
    Fix the conflicts or ask a new patch to the contributor
    Run test suite manually (10-15 min)
    Commit the change manually
    Write the commit message
    Commit in the maintenance branches first and then merged in the default
    branch
    Run the test suite manually again
    Commit the merge
    Push the changes
    18 / 89

    View Slide

  19. Typical Work ow
    About the patch
    Need to fix the conflicts
    Try to find the original author
    No description about the behaviour of the patch
    Problem, sometimes, there is no reference about the revision
    Difficult to rebase
    diff --git a/Misc/python.man b/Misc/python.man
    --- a/Misc/python.man
    +++ b/Misc/python.man
    19 / 89

    View Slide

  20. Typical Work ow
    About the continuous integration
    Check the buildbots (between 1h & 1d... if we are lucky)
    Frequent bugs on Windows
    the developers are on Linux or OSX
    or just too lazy
    If green, port to several branches
    can introduce new bugs in the other branches
    risk of conflicts
    tests executed by Buildbot
    20 / 89

    View Slide

  21. Typical Work ow
    About the continuous integration
    Check the buildbots (between 1h & 1d... if we are lucky)
    Frequent bugs on Windows
    the developers are on Linux or OSX
    or just too lazy
    If green, port to several branches
    can introduce new bugs in the other branches
    risk of conflicts
    tests executed by Buildbot
    CI: Only on the stable branches (2.7, 3.4+)
    21 / 89

    View Slide

  22. Typical Work ow
    Current limitation
    Not enought reviewers for the issues/patches
    Difficult to update the MISC/News file
    22 / 89

    View Slide

  23. Typical Work ow
    Status
    Download the sources from https://hg.python.org
    Use Patches
    Store them on bugs.python.org
    Use unmaintained version of Rietveld
    Complex Workflow
    23 / 89

    View Slide

  24. Issue with
    the work ow!
    24 / 89

    View Slide

  25. Di culty for the contribution
    25 / 89

    View Slide

  26. We have to change the Work ow
    26 / 89

    View Slide

  27. How?
    27 / 89

    View Slide

  28. PEP 507
    PEP 512
    PEP 507: Migrating to GitLab
    PEP 512: Migration from hg.python.org to Git/GitHub
    28 / 89

    View Slide

  29. Why Git?
    Based on the statistics of The Open Hub and PyPI itself.
    29 / 89

    View Slide

  30. The Open Hub
    48% for SVN
    37% for Git
    2% for Mercurial
    PyPI
    62% for Git
    22% for Mercurial
    13% for the rest
    Why Git?
    Git is 3 times more popupar as
    Mercurial for the top 100 projects on
    PyPI.
    30 / 89

    View Slide

  31. Why GitHub?
    31 / 89

    View Slide

  32. Why GitHub?
    GitHub is popular
    Primary place if you want to contribute to a project
    Avoid a custom infrastructure for the volunteers
    Example, Rietveld was a custom fork and not maintained
    a lot of developers are familiar with GitHub
    GitHub Pull Requests has a major advantage over the older "submit a
    patch to a bug tracker" model.
    GitHub Pull Requests
    are easier to review
    have nice syntax hightligted diffs
    can be commented
    32 / 89

    View Slide

  33. GitHub - Web interface
    33 / 89

    View Slide

  34. GitHub - Web interface
    34 / 89

    View Slide

  35. GitHub - Web interface
    35 / 89

    View Slide

  36. GitHub - Web interface
    36 / 89

    View Slide

  37. GitHub - Dashboards
    37 / 89

    View Slide

  38. GitHub - Reviews
    Integrated review
    Comment the code and ask for changes
    38 / 89

    View Slide

  39. GitHub - Reviews - Code Owners
    Auto attribution of the reviewers for a PR
    .github/CODEOWNERS
    */*asyncio* @1st1
    */*import* @python/import-team
    */*ssl* @python/crypto-team
    */*mail* @python/email-team
    */*subprocess* @gpshead
    /PC/ @python/windows-team
    */*itertools* @rhettinger
    */*functools* @ncoghlan @rhettinger
    */*idlelib* @terryjreedy
    https://help.github.com/articles/about-codeowners/
    39 / 89

    View Slide

  40. GitHub - Reviews - Ready to merge
    40 / 89

    View Slide

  41. GitHub - API
    REST API
    GraphQL API
    Really interesting for statistics
    41 / 89

    View Slide

  42. Migrations to GitHub
    42 / 89

    View Slide

  43. 15th January 2016
    PEPs
    https://github.com/python/peps
    43 / 89

    View Slide

  44. 22nd July 2016
    DevGuide
    https://github.com/python/devguide
    44 / 89

    View Slide

  45. 11st February 2017
    CPython
    https://github.com/python/cpython
    45 / 89

    View Slide

  46. Anish Shah
    Barry Warsaw
    Benjamin Peterson
    Berker Peksag
    Carol Willing
    Donald Stufft
    Ernest W Durbin
    Ezio Melotti
    Maciej Szulik
    Mariatta Wijaya
    Nick Coghlan
    Oleg Broytman
    Senthil Kumaran
    and me
    Seriously, you can congratulate them
    Orchestrated by Brett Cannon and these volunteers
    https://mail.python.org/pipermail/python-committers/2017-February/004220.html
    46 / 89

    View Slide

  47. Automation
    47 / 89

    View Slide

  48. Run the test suite
    Run the code coverage
    Run the performance tests
    Compile the documentation
    Execute make patchcheck
    Check if files have been generated
    Execute for master and the stable branches (2.7, 3.6)
    Execute on the branches from the contributors
    Automation
    48 / 89

    View Slide

  49. The BOTS...
    49 / 89

    View Slide

  50. Job
    checks the title of the PR with bedevere/issue-number
    has a Misc/NEWS.d/next entry with bedevere/news
    play with the labels awaiting review, awaiting merge,
    awaiting core review
    Bedevere!
    https://github.com/python/bedevere
    50 / 89

    View Slide

  51. Job
    checks if the the commiter has signed the Contributor
    Agreement aka CLA
    The knights who say ni!
    https://github.com/python/the-knights-who-say-ni
    CLA: https://www.python.org/psf/contrib/contrib-form/
    51 / 89

    View Slide

  52. The knights who say ni!
    if CLA is signed
    52 / 89

    View Slide

  53. The knights who say ni!
    if CLA is not signed
    53 / 89

    View Slide

  54. Job
    backports a Pull Request to an other branch
    based on needs backport to X.Y
    Miss Islington!
    https://github.com/python/miss-islington
    54 / 89

    View Slide

  55. BOTS + Automations =
    55 / 89

    View Slide

  56. BOTS + Automations =
    56 / 89

    View Slide

  57. BOTS + Automations =
    57 / 89

    View Slide

  58. BOTS + Automations =
    58 / 89

    View Slide

  59. as you can see
    BOTS are really useful for the core-devs
    59 / 89

    View Slide

  60. the MISC/News le
    60 / 89

    View Slide

  61. Problem
    the Misc/NEWS file is a very big file containing all the descriptions for each
    change.
    really difficult to merge when you receive a PR or a patch.
    => it's a real pain for the core-developers and the contributors
    discussion is here: https://github.com/python/core-workflow/issues/6
    61 / 89

    View Slide

  62. 62 / 89

    View Slide

  63. Solution: blurb
    by Larry Hastings (Thank you)
    It's a CLI tool which manage the Misc/NEWS.d entries
    an individual file per news entry!
    > cat Misc/NEWS.d/next/IDLE/2018-03-05-01-29-05.bpo-32984.NGjgT4.rst
    And it's a ReStructured Text file format ;-)
    https://github.com/python/core-workflow
    63 / 89

    View Slide

  64. Status about the Work ow
    64 / 89

    View Slide

  65. Before
    Contributor
    Open an issue
    Clone
    Upload a patch
    Core Dev
    Wait a review of a Core Dev
    Download the patch
    Fix the conflicts
    Run the tests manually
    Run the code coverage
    manually
    Commit manually
    Write the commit message
    Merge into the branches
    Run the test suite manually
    Commit the merge
    Push the changes
    etc...
    process is long & manual & error
    prone 65 / 89

    View Slide

  66. Before
    Contributor
    Open an issue
    Clone
    Upload a patch
    Core Dev
    Wait a review of a Core Dev
    Download the patch
    Fix the conflicts
    Run the tests manually
    Run the code coverage
    manually
    Commit manually
    Write the commit message
    Merge into the branches
    Run the test suite manually
    Commit the merge
    Push the changes
    etc...
    process is long & manual & error
    prone
    After
    Contributor
    Open an issue
    Clone
    Create a Pull Request
    Core Dev
    Review from contributors or
    core-dev
    Wait the feedback of the BOTS
    + Automation
    Wait the signal
    Mark the PR as needs
    backport to X.Y
    process is short & automatic
    66 / 89

    View Slide

  67. Before GitHub
    67 / 89

    View Slide

  68. After GitHub
    68 / 89

    View Slide

  69. Statistics
    from Feb 2017 to Apr 2018
    April is not yet nished
    69 / 89

    View Slide

  70. 6542
    70 / 89

    View Slide

  71. 6542
    The number of Pull Requests!
    71 / 89

    View Slide

  72. Pull Requests: 6542
    72 / 89

    View Slide

  73. 848
    73 / 89

    View Slide

  74. 848
    The number of Contributors of CPython
    74 / 89

    View Slide

  75. 848
    The number of Contributors of CPython
    75 / 89

    View Slide

  76. 848
    The number of Contributors of CPython
    and 34 Core Devs
    76 / 89

    View Slide

  77. Number of
    Core-Devs & Contributors
    77 / 89

    View Slide

  78. Pull Requests from
    Core-Devs & Contributors
    78 / 89

    View Slide

  79. Pull Requests from Miss-Islington
    79 / 89

    View Slide

  80. Merge Time
    80 / 89

    View Slide

  81. Merge Time
    81 / 89

    View Slide

  82. and the winner of 368 days is
    82 / 89

    View Slide

  83. Top Merge Time
    83 / 89

    View Slide

  84. Merged Pull Requests: 4966
    84 / 89

    View Slide

  85. Yeah!
    CPython loves your Pull Requests
    85 / 89

    View Slide

  86. Yeah!
    CPython loves your Pull Requests
    and
    86 / 89

    View Slide

  87. We are ready for the future!
    87 / 89

    View Slide

  88. Last thing ;-)
    Now, you can translate the official documentation in your mother language.
    with PEP 545
    88 / 89

    View Slide

  89. [email protected]
    @mgxio
    89 / 89

    View Slide