Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Static Type Analysis for Robust Data Products @ PyData London 2017

Static Type Analysis for Robust Data Products @ PyData London 2017

Slides for my talk at PyData London 2017:
https://pydata.org/london2017/schedule/presentation/20/

As a dynamically typed language, Python is an extremely flexible tool that allows to write code quickly and concisely. This flexibility makes Python a popular tool for R&D and prototyping, but what about bringing Data Science in production? When comparing Python to statically typed languages, one of the downsides is that many type-related errors are not captured until runtime.

This talk discusses the steps taken by the Python community to promote static type analysis, in particular the semantic definition of type hints and the adoption of mypy as type checking tool.

The audience will learn about static typing for Python, its pros and cons, and how to adopt static type analysis in your workflow. Since the focus is on building and deploying data products, static type analysis is discussed as a means to improve the robustness of your data products.

Marco Bonzanini

May 06, 2017
Tweet

More Decks by Marco Bonzanini

Other Decks in Programming

Transcript

  1. >>> '1' + 1 Traceback (most recent call last): File

    "<stdin>", line 1, in <module> TypeError: Can't convert 'int' object
 to str implicitly
  2. >>> 1.0 == 1 == True True >>> 1 +

    True 2 >>> 10 * False 0
  3. >>> '1' * 2 '11' >>> '1' + 2 Traceback

    (most recent call last): File "<stdin>", line 1, in <module> TypeError: Can't convert 'int' object to str implicitly
  4. If it looks like a duck, swims like a duck,

    and quacks like a duck, then it probably is a duck. — somebody on the Web
  5. “It’s easier to ask forgiveness
 than it is to get

    permission” EAFP principle — Grace Hopper
  6. try: dog.quack() # if the dog quacks # it’s still

    a duck except AttributeError: dog.woof_woof() EAFP + Duck Typing
  7. How Can You Live without Static Types? •Catch errors before

    runtime •Code documentation •Support for IDEs
  8. How Can You Live without Static Types? •Catch errors before

    runtime •Code documentation •Support for IDEs •(compiler optimisations)
  9. How Can You Live without Static Types? •Catch errors before

    runtime •Code documentation •Support for IDEs •(compiler optimisations)
  10. Problems many of us have • New hires • Refactoring

    • Poor documentation • Not enough tests
  11. def do_stuff(a: int, b: int) -> str: ... return something

    PEP 3107 — Function Annotations
 (since Python 3.0)
  12. def do_stuff(a: int, b: int) -> str: ... return something

    PEP 3107 — Function Annotations
 (since Python 3.0) (annotations are ignored by the interpreter)
  13. typing module: semantically coherent PEP 484 — Type Hints
 (since

    Python 3.5) (annotations still ignored by the interpreter)
  14. Example from typing import List, Dict def do_stuff(a: int) ->

    Dict: b = [] # type: List[int] for x in range(a): b.append(x) return b
  15. Gradual Typing • From dynamic to static overnight? • Any

    reduces the friction • Improving code understanding
  16. Supported Types • from typing import … • List, Dict,

    Tuple, … • Iterable, Optional, Union, Any, … • … and more • Built-in types and custom objects
  17. Python Requirements • typing: since Python 3.5 • mypy runs

    on Python 3.3+ • Using Python 2.7? Annotations in comments
  18. When to run it # pre-flight-checks-in-your-ci-server.sh flake8 myprogram # linter

    pytest myprogram # unit tests MYPYPATH=./stubs # static analysis mypy myprogram
  19. Third-party libraries • Stubs: interface definition in *.pyi • mypy

    --follow-imports silent <myprogram> • --follow-imports {normal, skip, error}
  20. Summary • From script to mature codebase • Better understanding

    of your codebase • Life easier with heterogeneous teams
  21. mypy references • http://www.mypy-lang.org/ • http://mypy.readthedocs.io/ Images: • Rubber ducks:

    https://en.wikipedia.org/wiki/File:Rubber_ducks.jpg • The Thinker: https://pixabay.com/en/the-thinker-rodin-museum-thinker-1431333/ • Skull and bones: https://commons.wikimedia.org/wiki/File:Skull_and_crossbones.svg • Scrum: https://commons.wikimedia.org/wiki/File:Scrum_Italy_New_Zealand.jpg • Alberto Sordi / spaghetti: https://it.wikipedia.org/wiki/File:Un_americano_a_Roma_-_maccheroni.jpg