Practical Data Science with Python and Jupyter

Python Brasil 2019 keynote in Ribeirao Preto. Python and Jupyter provide the tools and libraries for Practical Data Science.

What is Practical Data Science? Practical Data Science uses data responsibly to solve real world issues, such as climate change and disease, and improve human lives.

The keynote gives a state of Python and Jupyter in 2019. It outlines three major steps for Practical Data Science: Start, Explore, Deploy. Weather and climate are used as evidence of the impact and importance of data science predictions.

Transcript

1. @WillingCarol
   Practical Data Science with
   Python and Jupyter
   Carol Willing
   Python Brasil
   October 25, 2019
   

   View Slide

2. @WillingCarol
   Hi! I'm Carol.
   • Python Steering Council
   • Core Developer
   • PSF Fellow
   • PSF Director (Former)
   • Frank Willison Award 2019
   2
   

   View Slide

3. @WillingCarol 3
   Core maintainer
   Papermill, Scrapbook, Bookstore, Commuter
   Steering Council, Core Developer
   JupyterHub, BinderHub, mybinder.org
   I love creating tools which
   educate and empower
   people.
   

   View Slide

4. @WillingCarol
   Practical
   Data Science
   4
   

   View Slide

5. @WillingCarol 5
   Using data responsibly to
   solve real world issues and
   improve human lives
   

   View Slide

6. @WillingCarol 6
   A Story
   Python
   Jupyter
   Practical
   Data Science
   

   View Slide

7. @WillingCarol 7
   San Diego, CA
   

   View Slide

8. @WillingCarol 8
   

   View Slide

9. @WillingCarol 9
   

   View Slide

10. @WillingCarol 10
    Tokyo
    

    View Slide

11. @WillingCarol 11
    

    View Slide

12. Copyright: 2019 European
    Union, contains modified
    Copernicus Sentinel data
    2019, processed by
    EUMETSAT
    Super Typhoon
    Hagibis
    View of Super Typhoon
    Hagibis south-west of
    Japan, as captured by
    the Copernicus
    Sentinel-3 satellite on
    08 October at 00:16
    UTC.
    

    View Slide

13. Title Typhoon Hagibis
    Released 10/10/2019 4:45 pm
    Copyright contains modified Copernicus
    Sentinel data (2019), processed by ESA,
    CC BY-SA 3.0 IGO
    

    View Slide

14. Source:Twitter
    

    View Slide

15. A sign is partially submerged as the Tama River floods during Typhoon Hagibis.
    Source:Getty Images
    Source:Japan Times
    

    View Slide

16. @WillingCarol 16
    

    View Slide

17. @WillingCarol 17
    

    View Slide

18. @WillingCarol
    Preparation
    Evacuation
    Safety
    18
    

    View Slide

19. Title: Typhoon Bualoi
    10/24/2019
    Source: JTWC/SATOPS
    

    View Slide

20. @WillingCarol 20
    Python
    

    View Slide

21. @WillingCarol
    Practical data science
    grew up around
    

    View Slide

22. @WillingCarol
    Python - 2019
    Governance
    Python 3.8
    Farewell to Python 2
    22
    

    View Slide

23. @WillingCarol 23
    Python's Growth in 2019
    Web
    DevOps
    Systems
    Testing
    Science
    Data Science
    Embedded systems
    Education
    Micropython,
    CircuitPython,
    Raspberry Pi
    

    View Slide

24. @WillingCarol
    Governance
    Post-BDFL
    24
    

    View Slide

25. @WillingCarol
    PEP 8016
    The Steering Council Model
    25
    

    View Slide

26. @WillingCarol
    Steering Council
    26
    Barry Warsaw
    Brett Cannon
    Carol Willing
    Guido van Rossum
    Nick Coghlan
    

    View Slide

27. @WillingCarol
    PEP 13 Python Language Governance
    27
    ‣ Quality and Stability
    ‣ Contributing accessible, inclusive, sustainable
    ‣ Core team and PSF relationship
    ‣ Decision making processes for PEPs
    ‣ Seek consensus
    

    View Slide

28. @WillingCarol
    Core Development
    28
    

    View Slide

29. @WillingCarol
    Python 3.8
    29
    https://www.python.org/downloads/
    https://www.python.org/dev/peps/pep-0569/
    Schedule and Content
    Try it
    

    View Slide

30. @WillingCarol 30
    Positional only parameters
    Assignment operator (walrus operator)
    "equals" specifier added to f-string (debugging)
    PEP 587: Python Initialization Configuration
    PEP 574: pickle protocol 5
    Python 3.8
    

    View Slide

31. @WillingCarol 31
    

    View Slide

32. @WillingCarol 32
    

    View Slide

33. @WillingCarol
    Farewell to Python 2
    33
    

    View Slide

34. @WillingCarol 34
    Jupyter
    

    View Slide

35. @WillingCarol 35
    jupyter.org
    

    View Slide

36. @WillingCarol 36
    2014
    Now,
    5 years
    later...
    36
    

    View Slide

37. 

    Millions of
    Notebooks
    https://github.com/trending/jupyter-notebook
    Over 5 million
    on GitHub
    

    View Slide

38. jupyter.org
    

    View Slide

39. @WillingCarol 39
    ‣ Growth
    ‣ ACM Award
    ‣ Industry adoption
    ‣ Creative uses
    ‣ Open Source Book
    https://www.youtube.com/watch?v=qbtDVdEr8SY
    

    View Slide

40. @WillingCarol
    JupyterLab
    40
    

    View Slide

41. @WillingCarol
    ecosystem
    41
    

    View Slide

42. @WillingCarol 42
    Practical Data Science
    Source: xkcd
    

    View Slide

43. @WillingCarol 43
    Start
    Deploy
    Explore
    Practical Data Science
    

    View Slide

44. @WillingCarol
    Start
    44
    Step 1
    

    View Slide

45. @WillingCarol 45
    Binder 2.0 blog post
    elifesciences: Share
    your interactive
    research environment
    Nature article about
    Binder
    45
    mybinder.org
    Try it in the
    browser
    

    View Slide

46. 46
    jupyter.org demo
    

    View Slide

47. 47
    jupyter.org demo
    

    View Slide

48. @WillingCarol
    Install Python
    48
    Use the OS Package Manager
    Use a Distribution - Anaconda, miniconda
    Download from Python.org
    

    View Slide

49. @WillingCarol
    Install Libraries
    49
    Use a Distribution
    such as Anaconda
    Use pip
    Use miniconda, conda,
    and conda-forge
    

    View Slide

50. @WillingCarol 50
    python3 -m venv myenv
    source myenv/bin/activate
    pip install jupyter
    jupyter notebook
    

    View Slide

51. @WillingCarol 51
    conda create -n myenv python=3.7
    conda activate myenv
    conda install jupyter -c conda-forge
    jupyter notebook
    

    View Slide

52. @WillingCarol
    Choose your Tools
    52
    

    View Slide

53. @WillingCarol
    JupyterLab
    53
    

    View Slide

54. @WillingCarol 54
    ReactJS front end
    nteract
    nteract.io
    

    View Slide

55. @WillingCarol
    VS Code
    55
    PyCharm
    

    View Slide

56. @WillingCarol
    Start
    56
    Try it in the browser
    Install Python
    Install Libraries
    Choose your tools
    Step 1
    

    View Slide

57. @WillingCarol
    Explore
    57
    Step 2
    

    View Slide

58. @WillingCarol
    Start with your interests
    I usually create music
    projects when exploring new
    languages.
    58
    https://web.mit.edu/music21/
    

    View Slide

59. @WillingCarol 59
    Try a library's tutorial
    

    View Slide

60. @WillingCarol 60
    

    View Slide

61. @WillingCarol 61
    ipyvolume
    https://towardsdatascience.com/multivolume-
    rendering-in-jupyter-with-ipyvolume-cross-
    language-3d-visualization-64389047634a
    

    View Slide

62. @WillingCarol 62
    

    View Slide

63. @WillingCarol 63
    github.com/napari/napari
    napari
    https://ilovesymposia.com/2019/10/24/introducing-napari-
    a-fast-n-dimensional-image-viewer-in-python/
    

    View Slide

64. @WillingCarol
    Learning resources
    64
    Online Books with
    Jupyter Notebooks
    Conferences, pyvideo.org
    and YouTube
    Research papers, blogs
    Source: O'Reilly
    

    View Slide

65. @WillingCarol 65
    Ten Simple Rules
    for Reproducible
    Research in
    Jupyter Notebooks
    Adam Rule et al.
    https://github.com/jupyter-guide/
    ten-rules-jupyter
    https://github.com/jupyter-guide/
    jupyter-guide
    

    View Slide

66. @WillingCarol
    Learn AI online
    66
    

    View Slide

67. @WillingCarol
    Community
    67
    User groups
    Meetups
    PyLadies
    Carpentries
    Photo: Python Sul
    

    View Slide

68. @WillingCarol
    What's new
    68
    Talk Python to Me
    Tracking Jupyter Newsletter
    https://tinyletter.com/TrackingJupyter/archive
    Open Source Directions
    GitHub Trending
    Follow projects on Social Media
    

    View Slide

69. @WillingCarol
    Explore
    69
    Start with your interests
    Try a tutorial
    Learning resources
    Community
    What's new
    Step 2
    

    View Slide

70. @WillingCarol
    Deploy
    70
    Step 3
    

    View Slide

71. @WillingCarol
    Notebooks to web
    71
    https://blog.jupyter.org/and-
    voil%C3%A0-f6a2c08a4a93
    

    View Slide

72. @WillingCarol 72
    nteract
    Papermill
    Scrapbook
    Bookstore
    Commuter
    Production data at scale
    72
    https://medium.com/netflix-techblog/notebook-innovation-591ee3221233
    Bookstore
    

    View Slide

73. @WillingCarol 73
    Papermill - parameterize / run
    Scrapbook - recording / reading
    Bookstore - store notebooks
    Commuter - share notebooks
    Production data at scale
    73
    

    View Slide

74. @WillingCarol 74
    Enterprise data workflows
    74
    

    View Slide

75. @WillingCarol 75
    https://medium.com/dagster-io/dagster-0-6-0-impossible-princess-898b459375e0
    Pipelines
    

    View Slide

76. zero-to-jupyterhub.readthedocs.io
    

    View Slide

77. https://blog.jupyter.org/simpler-authentication-for-small-
    scale-jupyterhubs-with-nativeauthenticator-999534c77a09
    Native Authenticator
    Thank you Leticia
    

    View Slide

78. @WillingCarol 78 78
    Deploy
    binders
    

    View Slide

79. 79
    Juliette Taka
    

    View Slide

80. @WillingCarol 80
    From a
    phone in
    the park!
    80
    

    View Slide

81. @WillingCarol
    Pangeo
    81
    https://pangeo.io
    

    View Slide

82. @WillingCarol 82
    

    View Slide

83. @WillingCarol 83
    https://simexp.github.io/vcog_hps_ad_book/intro.html
    Jupyter Book
    Binder
    Jupyter
    pandas
    scipy
    scikit learn
    matplotlib
    numpy
    seaborn
    Canadian Open Neuroscience Platform
    

    View Slide

84. @WillingCarol 84
    Practical Data Science
    How?
    

    View Slide

85. @WillingCarol
    Start
    85
    

    View Slide

86. @WillingCarol
    Ask why
    86
    

    View Slide

87. @WillingCarol
    Plan
    Execute
    Change
    87
    https://jupyterhub-team-compass.readthedocs.io
    https://github.com/jupyterhub/team-compass
    

    View Slide

88. @WillingCarol
    Share binders.
    Foster research.
    88
    

    View Slide

89. @WillingCarol
    Build Community
    89
    Python Brasil official photo.
    The sign represents Python in BSL (Brazilian Sign Language),
    made by Amanda and Sávio at Python Sul 2018
    https://pyfound.blogspot.com/2019/02/python-brasil-people-technology.html
    PyLadiesBRConf official photo
    PyLadiesBRConf official photo
    

    View Slide

90. @WillingCarol 90
    Python Brasil
    Practical Data Science
    

    View Slide

91. @WillingCarol 91
    Python Brasil
    Changing the
    world together
    

    View Slide

92. @WillingCarol 92
    Thank you
    https://speakerdeck.com/willingc
    @WillingCarol
    

    View Slide

93. @WillingCarol 93
    Join the
    

    View Slide

94. @WillingCarol 94
    Questions
    https://speakerdeck.com/willingc
    

    View Slide

95. @WillingCarol 95
    Karaoke
    Time
    Lorena, This is for you.
    https://www.youtube.com/watch?v=m41cVv3tChE
    

    View Slide

96. @WillingCarol 96
    Thank you Python Brasil
    Project Jupyter Team
    Core Python Team
    PSF
    

    View Slide

97. @WillingCarol
    Attributions
    97
    Attributions on slides.
    Photos
    [7-10] Carol Willing and Linnea Willing
    [38] The Carpentries, Tracy Teal, Bérénice Batut
    [17] Godzilla
    By Toho Company Ltd. (東宝株式会社, Tōhō Kabushiki-kaisha) © 1954 - movie
    poster made by Toho Company Ltd. (東宝株式会社, Tōhō Kabushiki-kaisha),
    Public Domain, https://commons.wikimedia.org/w/index.php?curid=3648684
    

    View Slide