Don't Be a Robot; Build the Bot

DjangoCon US 2018 Keynote

Transcript

1. Don’t Be a Robot;
   Build the Bot
   @mariatta
   

   View Slide

2. Python Core Developer
   Platform Engineer
   @mariatta
   mariatta.ca
   #IceCreamSelfie
   WHO AM I?
   Mariatta Wijaya
   !
   DjangoCon US 2018
 @mariatta
   https://zapier.com/jobs/
   

   View Slide

3. Problems?
   CORE PYTHON DEVELOPMENT
   DjangoCon US 2018
 @mariatta
   

   View Slide

4. Problem: We’re outnumbered
   Source: Stephane Wirtel, CPython
   loves your pull request
   https://speakerdeck.com/matrixise/
   python-loves-your-contributions
   DjangoCon US 2018
 @mariatta
   

   View Slide

5. Problem: Complicated Workflow
   Bug or feature?
   CLA signed?
   Tests passed?
   Needs an issue?
   Needs news entry?
   News entry added?
   Needs backport? How? Who? When?
   First time contributor?
   Commit message?
   DjangoCon US 2018
 @mariatta
   

   View Slide

6. Problem: Complicated Workflow
   
   Bug or feature?
   CLA signed?
   Tests passed?
   Needs an issue?
   Needs news entry?
   News entry added?
   Needs backport? How? Who? When?
   First time contributor?
   Commit message?
   DjangoCon US 2018
 @mariatta
   

   View Slide

7. Problem: Complicated Workflow
   Bug or feature?
   CLA signed?
   Tests passed?
   Needs an issue?
   Needs news entry?
   News entry added?
   Needs backport? How? Who? When?
   First time contributor?
   Commit message?
   DjangoCon US 2018
 @mariatta
   

   View Slide

8. Backport?
   DjangoCon US 2018
 @mariatta
   

   View Slide

9. Backport?
   Applying changes from newer version
   to older version
   DjangoCon US 2018
 @mariatta
   

   View Slide

10. • master (3.8.0a1, due January 2019)
    • 3.7 (3.7.1, due October 2018)
    • 3.6 (3.6.7, due October 2018)
    • 2.7 (EOL, January 2020)
    Current maintained versions
    DjangoCon US 2018
 @mariatta
    

    View Slide

11. SCENARIO: BUG IN 3.6
    Bug in f-strings
    DjangoCon US 2018
 @mariatta
    

    View Slide

12. SCENARIO: BUG IN 3.6
    Bug in f-strings
    3.6
    DjangoCon US 2018
 @mariatta
    

    View Slide

13. SCENARIO: BUG IN 3.6
    Bug in f-strings
    3.6 3.6
    DjangoCon US 2018
 @mariatta
    

    View Slide

14. SCENARIO: BUG IN 3.6
    3.6
    Bug in f-strings
    3.6
    DjangoCon US 2018
 @mariatta
    

    View Slide

15. SCENARIO: BUG IN 3.6
    Bug in f-strings
    DjangoCon US 2018
 @mariatta
    

    View Slide

16. SCENARIO: BUG IN 3.6
    Bug in f-strings
    3.6
    3.7
    3.8
    DjangoCon US 2018
 @mariatta
    

    View Slide

17. SCENARIO: BUG IN 3.6
    Bug in f-strings
    3.8
    3.6
    3.7
    3.8
    DjangoCon US 2018
 @mariatta
    

    View Slide

18. SCENARIO: BUG IN 3.6
    Bug in f-strings
    3.8
    3.6
    3.7
    3.8
    DjangoCon US 2018
 @mariatta
    

    View Slide

19. SCENARIO: BUG IN 3.6
    Bug in f-strings
    3.8
    3.6
    3.7
    3.8
    3.6
    3.7
    DjangoCon US 2018
 @mariatta
    

    View Slide

20. SCENARIO: BUG IN 3.6
    Bug in f-strings
    3.8
    3.6
    3.7
    3.8
    3.6
    3.7
    DjangoCon US 2018
 @mariatta
    

    View Slide

21. CHERRY-PICKING
    $ git fetch upstream
    $ git checkout -b backport-3384d38d-3.7 upstream/3.7
    $ git cherry-pick -x 3384d38d51a2c3450e742175db5d6d638fa5d2eb
    $ git push origin backport-3384d38d-3.7
    $ git checkout master
    $ git branch -D backport-3384d38d-3.7
    $ git checkout -b backport-3384d38d-3.6 upstream/3.6
    $ git cherry-pick -x 3384d38d51a2c3450e742175db5d6d638fa5d2eb
    $ git push origin backport-3384d38d-3.6
    $ git checkout master
    $ git branch -D backport-3384d38d-3.6
    3.7
    3.6
    DjangoCon US 2018
 @mariatta
    

    View Slide

22. CHERRY_PICKER.PY
    $ pip install cherry_picker
    DjangoCon US 2018
 @mariatta
    

    View Slide

23. CHERRY_PICKER.PY
    $ pip install cherry_picker
    $ cherry_picker 3384d38d51a2c3450e742175db5d6d638fa5d2eb 3.7 3.6
    DjangoCon US 2018
 @mariatta
    

    View Slide

24. Problems:
    DjangoCon US 2018
 @mariatta
    

    View Slide

25. Problems:
    DjangoCon US 2018
 @mariatta
    

    View Slide

26. Problems:
    DjangoCon US 2018
 @mariatta
    

    View Slide

27. Problems:
    DjangoCon US 2018
 @mariatta
    

    View Slide

28. Problems:
    DjangoCon US 2018
 @mariatta
    

    View Slide

29. DON’T BE A ROBOT
    Build the Bot
    DjangoCon US 2018
 @mariatta
    

    View Slide

30. GITHUB WEBHOOKS
    Pull request (opened, closed, labeled)
    Status (pending, success, failure)
    Pull request review (submitted, dismissed)
    …
    Events
    GitHub
    Webservice
    Docs: https://developer.github.com/webhooks
    DjangoCon US 2018
 @mariatta
    

    View Slide

31. GITHUB APIS
    Create a pull request
    Merge a pull request
    Comment on a pull request
    Apply labels to pull request
    …
    APIs
    
    Webservice
    GitHub
    Docs: https://developer.github.com/v3
    DjangoCon US 2018
 @mariatta
    

    View Slide

32. PR merged event (webhook)
    
    Open pull request (REST API)
    Run cherry_picker.py
    DjangoCon US 2018
 @mariatta
    

    View Slide

33. PR merged event (webhook)
    
    Open pull request (REST API)
    Run cherry_picker.py
    Clone CPython repo
    DjangoCon US 2018
 @mariatta
    

    View Slide

34. PR merged event (webhook)
    
    Open pull request (REST API)
    Run cherry_picker.py
    Clone CPython repo
    2 minutes
    12-15 seconds
    DjangoCon US 2018
 @mariatta
    

    View Slide

35. PR merged event (webhook)
    
    Open pull request (REST API)
    Run cherry_picker.py
    Clone CPython repo
    2 minutes
    12-15 seconds
    timesout after 30 seconds
    DjangoCon US 2018
 @mariatta
    

    View Slide

36. PR merged event (webhook)
    
    Open pull request (REST API)
    Run cherry_picker.py
    Clone CPython repo
    2 minutes
    12-15 seconds
    timesout after 30 seconds
    DjangoCon US 2018
 @mariatta
    

    View Slide

37. PR merged event (webhook)
    
    Open pull request (REST API)
    Run cherry_picker.py
    Clone CPython repo
    Webservice
    (aiohttp)
    Background task
    (celery)
    Start a task
    DjangoCon US 2018
 @mariatta
    

    View Slide

38. PR merged event (webhook)
    
    Open pull request (REST API)
    Run cherry_picker.py
    Clone CPython repo
    Webservice
    (aiohttp)
    Background task
    (celery)
    miss-islington
    Start a task
    DjangoCon US 2018
 @mariatta
    

    View Slide

39. miss-islington
    • open-source https://github.com/python/miss-islington
    • Python 3.6 (because f-strings)
    • aiohttp: async web server/client
    • gidgethub: async Python library for GitHub
    DjangoCon US 2018
 @mariatta
    

    View Slide

40. miss-islington
    @router.register("pull_request", action="closed")
    async def backport_pr(event, gh, *args, **kwargs):
    if event.data["pull_request"]["merged"]:
    commit_hash = event.data["pull_request"]["merge_commit_sha"]
    ...
    branches = [
    label["name"].split()[-1]
    for label in pr_labels
    if label["name"].startswith("needs backport to”)
    ]
    for branch in branches:
    ...
    tasks.backport_task.delay(commit_hash, branch, ...)
    source: https://github.com/python/miss-islington/blob/master/miss_islington/backport_pr.py
    DjangoCon US 2018
 @mariatta
    

    View Slide

41. First PR
    https://twitter.com/ChristianHeimes/status/905218281937575936
    DjangoCon US 2018
 @mariatta
    

    View Slide

42. Problems:
    DjangoCon US 2018
 @mariatta
    

    View Slide

43. Problems:
    ✅ ✅ ✅ ✅
    DjangoCon US 2018
 @mariatta
    

    View Slide

44. Problems:
    ✅ ✅ ✅ ✅
    DjangoCon US 2018
 @mariatta
    

    View Slide

45. Problem: Complicated Workflow
    Bug or feature?
    CLA signed?
    Tests passed?
    Needs an issue?
    Needs news entry?
    News entry added?
    Needs backport? How? Who? When?
    First time contributor?
    Commit message?
    DjangoCon US 2018
 @mariatta
    

    View Slide

46. DjangoCon US 2018
 @mariatta
    

    View Slide

47. DjangoCon US 2018
 @mariatta
    

    View Slide

48. DjangoCon US 2018
 @mariatta
    

    View Slide

49. PR status event (webhook)
    Get the combined status for a ref (REST API)
    (status of a Git commit changes)
    Post a comment in PR (REST API)
    2. https://developer.github.com/v3/repos/statuses/#get-the-combined-status-for-a-specific-ref
    2
    1
    3
    1. https://developer.github.com/v3/activity/events/types/#statusevent
    3. https://developer.github.com/v3/issues/comments/#create-a-comment
    DjangoCon US 2018
 @mariatta
    

    View Slide

50. PR status event (webhook)
    Get the combined status for a ref (REST API)
    (status of a Git commit changes)
    Post a comment in PR (REST API)
    Which PR is this?
    2. https://developer.github.com/v3/repos/statuses/#get-the-combined-status-for-a-specific-ref
    2
    1
    3
    1. https://developer.github.com/v3/activity/events/types/#statusevent
    3. https://developer.github.com/v3/issues/comments/#create-a-comment
    DjangoCon US 2018
 @mariatta
    

    View Slide

51. GITHUB SEARCH ISSUES API
    https://developer.github.com/v3/search/#search-issues
    DjangoCon US 2018
 @mariatta
    

    View Slide

52. GITHUB SEARCH ISSUES API
    async def get_pr_for_commit(gh, sha):
    prs_for_commit = await gh.getitem(
    f"/search/issues/?q=type:pr+repo:python/cpython+sha:{sha}"
    )
    source: https://github.com/python/miss-islington/blob/master/miss_islington/util.py
    https://developer.github.com/v3/search/#search-issues
    DjangoCon US 2018
 @mariatta
    

    View Slide

53. PR status event (webhook)
    Get the combined status for a ref (REST API)
    Post a comment in PR (REST API)
    Search PR containing commit (REST API)
    DjangoCon US 2018
 @mariatta
    

    View Slide

54. DjangoCon US 2018
 @mariatta
    

    View Slide

55. ✅
    ✅
    ✅
    ✅
    ✅
    DjangoCon US 2018
 @mariatta
    

    View Slide

56. Problem: Complicated Workflow
    Bug or feature?
    CLA signed?
    Tests passed?
    Needs an issue?
    Needs news entry?
    News entry added?
    Needs backport? How? Who? When?
    First time contributor?
    Commit message?
    DjangoCon US 2018
 @mariatta
    

    View Slide

57. https://mail.python.org/pipermail/python-committers/2018-January/005174.html
    DjangoCon US 2018
 @mariatta
    

    View Slide

58. Merge the PR
    pr_number = 5590
    await gh.put(f"/repos/python/cpython/pulls/{pr_number}/merge",
    data={"commit_title": "Fix typo in f-strings docs",
    "commit_message": "Shorten the comment to: 'using integer format specifier'"),
    "sha": sha,
    "merge_method": "squash"
    }
    )
    DjangoCon US 2018
 @mariatta
    

    View Slide

59. Merge the PR
    pr_number = 5590
    await gh.put(f"/repos/python/cpython/pulls/{pr_number}/merge",
    data={"commit_title": "Fix typo in f-strings docs",
    "commit_message": "Shorten the comment to: 'using integer format specifier'"),
    "sha": sha,
    "merge_method": "squash"
    }
    )
    DjangoCon US 2018
 @mariatta
    

    View Slide

60. DjangoCon US 2018
 @mariatta
    

    View Slide

61. DjangoCon US 2018
 @mariatta
    

    View Slide

62. DjangoCon US 2018
 @mariatta
    

    View Slide

63. Don’t Be a Robot
    BUILD THE BOT
    DjangoCon US 2018
 @mariatta
    

    View Slide

64. black out
    Bot that formats code with black
    DjangoCon US 2018
 @mariatta
    

    View Slide

65. black out
    Bot that formats code with black
    https://github.com/mariatta/black_out
    DjangoCon US 2018
 @mariatta
    

    View Slide

66. Pyup-automerge
    Automatically merge pyup-bot pull requests
    DjangoCon US 2018
 @mariatta
    

    View Slide

67. Pyup-automerge
    Automatically merge pyup-bot pull requests
    DjangoCon US 2018
 @mariatta
    

    View Slide

68. Automatically add CoC
    Automatically add CoC to newly created repo in
    organization
    DjangoCon US 2018
 @mariatta
    

    View Slide

69. OOOS bot
    Out of open source auto reply bot
    DjangoCon US 2018
 @mariatta
    

    View Slide

70. RHYMES WITH HAPPIER!
    OOOS bot
    Out of open source auto reply bot
    DjangoCon US 2018
 @mariatta
    

    View Slide

71. Close all PRs
    DjangoCon US 2018
 @mariatta
    

    View Slide

72. Close all PRs
    PR opened/reopened event
    
    Close pull request
    DjangoCon US 2018
 @mariatta
    

    View Slide

73. Close all PRs
    PR opened/reopened event
    
    Close pull request
    @router.register("pull_request", action="opened")
    @router.register("pull_request", action="reopened")
    async def close_pr(event, gh, *args, **kwargs):
    await gh.patch(event.data["pull_request"]["url"],
    data= {"state": "closed"}
    )
    Source: https://github.com/Mariatta/close-all-pr/blob/master/close-all-pr/close_pr.py
    DjangoCon US 2018
 @mariatta
    

    View Slide

74. Don’t Be a Robot;
    Build the Bot
    https://github-bot-tutorial.readthedocs.io/en/latest/
    DjangoCon US 2018
 @mariatta
    

    View Slide

75. Contact mariatta@python.org

    Follow on twitter @mariatta
    Thank you!
    DjangoCon US 2018
 @mariatta
    PS: Please fill in Python Developers Survey http://bit.ly/pds2018-django
    

    View Slide