PyCon India keynote, presented September 19, 2021.
Main sections:
1. The state of static typing in Python 2. The four modes of typing 3. The central role of typing.Protocol 4. Typing limits and how to address them
© 2021 ThoughtworksType hints,protocols,and good sense
View Slide
© 2021 Thoughtworks 2
© 2021 ThoughtworksPython’s static type hints are ready for use,but they are not perfect.Understand the caveats and workarounds.Avoiding them is the best solutionin some cases.3
© 2021 ThoughtworksFluent Python,First EditionPublished in 9 languages:1. English2. Portuguese3. Japanese4. Korean5. Simplified Chinese6. Traditional Chinese7. French8. Polish9. Russian4
© 2021 ThoughtworksFluent Python,Second Edition● Covers Python 3.9 and 3.10,including pattern matching● Extensive coverage of type hints,with many examples● Updated coverage of async/await,with examples in asyncio, FastAPIand Curio● Draft available now atOReilly.com:https://bit.ly/FluPy2e2021 Q4, est.)5
© 2021 Thoughtworks 6Who am I to talkabout this?Using Python professionally since1998, at first I considered static typinga bad fit for the language.typing.Protocol addressed mymain concern.I am now a happy user,with moderation.
© 2021 Thoughtworks 7Who am I to talkabout this?Using Python professionally since1998, at first I considered static typinga bad fit for the language.typing.Protocol addressed mymain concern.I am now a happy user,with moderation.
© 2021 Thoughtworks 8Who am I to talkabout this?Using Python professionally since1998, at first I considered static typinga bad fit for the language.typing.Protocol addressed mymain concern.I am now a happy user,with moderation.
© 2021 Thoughtworks 9Researched type hintsto write about themWho am I to talkabout type hints?Read lots of PEPs, messages, issuereports; interacted with some keyproponents of type hints in Python;studied similar features in otherlanguages—to write than 100 pagesand dozens of examples in FluentPython, Second Edition.Contributed bug fixesimproving type hints ontypeshedtypeshed is the official repositoryof type hints used by all typechecking tools, covering thestandard library and otherpackages.Reorganized thedocumentation of Python’styping moduleProposed and implemented newoverall organization to typingmodule docs in the Python 3.9standard library, in collaborationwith Guido van Rossum.Using Python professionally since1998, at first I considered static typinga bad fit for the language.typing.Protocol addressed mymain concern.I am now a happy user,with moderation.
© 2021 Thoughtworks 10The State ofStatic Typingin Python1.The four modesof typing2.The central role oftyping.Protocol3.Typing limitsand how toaddress them4.10© 2021 ThoughtworksAgenda
© 2021 Thoughtworks1.The State of Static Typingin Python11
© 2021 ThoughtworksA few type declarations...1212© 2021 Thoughtworks
© 2021 Thoughtworks...and functions with type hints1313© 2021 Thoughtworks
© 2021 ThoughtworksStatic v. Dynamic Typing1414© 2021 Thoughtworksstatictypingdynamictyping
© 2021 ThoughtworksStatic v. Dynamic TypingRUNTIMECHECKINGA matter of when1515© 2021 ThoughtworksSTATICCHECKINGstatictypingdynamictyping
© 2021 ThoughtworksGradual typingNot a binary choice, but a continuum1616© 2021 Thoughtworksstatictypingdynamictypinggradualtyping
© 2021 ThoughtworksOptional at All LevelsType hints may be omitted and/ortype checking disabled on single lines,functions and entire packages.Python Adopted aGradual Type System17The Default type is AnyAny is consistent with all other types.It is more general than object, andis understood to implement all interfaces.Does Not Catch Errors at RuntimeType hints are not used at runtime,only by code analysis tools such aslinters, IDEs, and dedicated typecheckers.Does Not Increase PerformanceCurrent Python implementationsdo not use type hints to optimizebytecode or machine code generation.
© 2021 Thoughtworks 18TypeScriptby MicrosoftThe most successfulJavaScript dialect to date.Dartby GoogleThe language of theFlutter SDK.Hackby FacebookPHP dialect supported by theJIT-enabled HHVM runtime.Other Languages with Gradual Typing
© 2021 ThoughtworksPython will remain adynamically typedlanguage, and theauthors have no desireto ever make typehints mandatory,even by convention.19Guido van Rossum, Jukka Lehtosalo, Łukasz Langain PEP 484Type Hints19© 2021 Thoughtworks
© 2021 ThoughtworksThe best feature ofgradual typing:type hints are alwaysoptional.2020© 2021 Thoughtworks
© 2021 ThoughtworksHow type hints are used21Run timeImport timeCI pipelineLintingWriting codeIDE features● Autocomplete● Instant warnings (wiggly underlines)● Refactoring assistanceStand-alone type checkers(same tools used for linting)Explicit type checking and services● explicit checks withisinstance() or issubclass()● Single dispatch functions● Data validation (e.g. pydantic)● API generation (e.g. FastAPIStand-alone type checkers● Mypy● Pytype● Pyre● Pyright CLIData class builders● @dataclasses.dataclass● typing.NamedTuple metaclass
© 2021 ThoughtworksCompanies with very large Python code basesinvested heavily in type hints, and report theywere also useful to migrate from Python 2.7.22DropboxLarge scale users...Employer of typingPEP authors.Released: Mypy.FacebookEmployer of typingPEP authors.Released Pyre.GoogleReleased Pytype.MicrosoftEmployer of typingPEP authors.Released Pyright.JetBrainsCreated proprietarytype checkerembedded inPyCharm IDE....and IDE vendors
© 2021 ThoughtworksPython DevelopersSurvey, October 202023Source: https://bit.ly/PySurvey2020Produced by thePython Software Foundationand JetBrains
© 2021 Thoughtworks 24“At least sometimes”79% use code linters78% write tests71% use type hints
© 2021 Thoughtworks2.The four modesof typing25
© 2021 ThoughtworksDuck typing26
© 2021 ThoughtworksDon’t check whether itis-a duck:check whether itquacks-like-a duck,walks-like-a duck, etc,depending on exactlywhat subsetofduck-like behavior youneed...27Alex Martelli in comp-lang-python, 20000726"polymorphism (was Re: Type checking in python?"27© 2021 Thoughtworks
© 2021 ThoughtworksTyping MapRUNTIMECHECKING2828© 2021 ThoughtworksSTATICCHECKINGSTRUCTURAL TYPESNOMINAL TYPESstatictypingducktyping
© 2021 ThoughtworksTyping MapRUNTIMECHECKING2929© 2021 ThoughtworksSTATICCHECKINGSTRUCTURAL TYPESNOMINAL TYPES“Pyt onst le”“Javst le”statictypingducktyping
© 2021 ThoughtworksTyping MapRUNTIMECHECKING3030© 2021 ThoughtworksSTATICCHECKINGSTRUCTURAL TYPESNOMINAL TYPESsu portedby ABstatictypingducktypinggoosetyping
© 2021 Thoughtworks 31
© 2021 ThoughtworksTyping MapRUNTIMECHECKING3232© 2021 ThoughtworksstatictypingducktypingSTATICCHECKINGSTRUCTURAL TYPESNOMINAL TYPESgoosetypingstaticduck typingsu ported bytyping.Proto lsu portedby AB
© 2021 ThoughtworksTyping Map: languagesRUNTIMECHECKING3333© 2021 ThoughtworksSTATICCHECKINGSTRUCTURAL TYPESNOMINAL TYPESPythonTypeScriptJavaScriptSmalltalkPython ≥ 2.6TypeScriptGoPython ≥ 3.8TypeScriptGoPython ≥ 3.5TypeScriptGoJavastatictypingducktypinggoosetypingstaticduck typing
© 2021 Thoughtworks3.The central role oftyping.Protocol34
© 2021 Thoughtworkstyping.Protocol allows (static) duck typing3535© 2021 Thoughtworks
© 2021 Thoughtworkstyping.Protocol allows (static) duck typing3636© 2021 Thoughtworks
© 2021 Thoughtworkstyping.Protocol allows (static) duck typing3737© 2021 Thoughtworks
© 2021 Thoughtworks 38
© 2021 Thoughtworks 39
© 2021 Thoughtworksmedian_low: fixed code4040© 2021 Thoughtworksetc.
© 2021 ThoughtworksUses of the SupportsLessThan ProtocolStub files for Python 3.9 standard library on typeshed4141builtins: list.sortmaxminsortedstatistics: median_lowmedian_highfunctools: cmp_to_keybisect: bisect_leftbisect_rightinsort_leftinsort_rightheapq: nlargestnsmallestos.path: commonprefix
© 2021 ThoughtworksSupport duck typing with type hintsThe essence of Python’sData Model and standard libraryUse typing.Protocolto build Pythonic APIs42Follow theInterface Segregation PrincipleClient code should not be forced todepend on methods it does not usePrefer narrow protocolsSingle method protocols should be themost common. Sometimes, twomethods. Rarely more.
© 2021 ThoughtworksSupport duck typing with type hintsThe essence of Python’sData Model and standard libraryUse typing.Protocolto build Pythonic APIs43Follow theInterface Segregation PrincipleClient code should not be forced todepend on methods it does not usePrefer narrow protocolsSingle method protocols should be themost common. Sometimes, twomethods. Rarely more.
© 2021 ThoughtworksSupport duck typing with type hintsThe essence of Python’sData Model and standard libraryUse typing.Protocolto build Pythonic APIs44Follow theInterface Segregation PrincipleClient code should not be forced todepend on methods it does not usePrefer narrow protocolsSingle method protocols should be themost common. Sometimes, twomethods. Rarely more.
© 2021 Thoughtworks4.Typing limits andhow to address them4545© 2021 Thoughtworks
© 2021 ThoughtworksType hints are onlyhelpful if they are usedto enhance thought,not replace it.46Variation on a phrase by Brian Marick, inHow to Misuse Code Coveragehttps://bit.ly/miscodecover 46© 2021 Thoughtworks
© 2021 ThoughtworksImagine a programming environmentthat refuses to build a programunless there is 100% test coverage.47
© 2021 ThoughtworksFalse Positives and False NegativesThe static type checkingecosystem—tools and annotations—are still immature and beneath thequality of Python itself.Static Typing isNo Silver Bullet48No support for common, useful dataconstraints—even simple ones“Quantity must be int > 0”“Airport code must be str of length 3”“Email address must not be empty str”Type hints are generally unsuitable tocheck business rules.We need testing and runtime checkingto verify business rules and algorithms.
© 2021 Thoughtworks 49
© 2021 Thoughtworks 50
© 2021 Thoughtworks 51
© 2021 ThoughtworksComplexity5252© 2021 Thoughtworks
© 2021 ThoughtworksA parade of typing-related PEPs53PEP Python Enhancement Proposal22 and counting...No other aspect ofPython has more PEPs.
© 2021 ThoughtworksThe max() built-in function54Flexible and easy to use, but very hard to annotate
© 2021 Thoughtworks 55
© 2021 Thoughtworks 56faneg t ve!
© 2021 Thoughtworksmax: old type hints5757© 2021 Thoughtworks
© 2021 Thoughtworksmax: fixed type hints5858© 2021 Thoughtworks
© 2021 Thoughtworksmaximplemented inPython, for testing5959© 2021 Thoughtworks
© 2021 Thoughtworks 606026Lines of code to implement allthe documented functionality,with 2 constants, no imports29Lines of code for type hints:7 imports, 4 definitions, and6 overloaded signatures
© 2021 ThoughtworksNo support for convenientcoding idioms, like unpackingconfig(**settings)Static type checkerscan’t handle theexpressive powerof Python61Limited support for advancedfeatures and abstractionsProperties, descriptors,metaclasses, metaprogrammingtechniques in generalType checkers lag behindFalse reports and crashes whenchecking new features, sometimesmore than a year after a Pythonrelease
© 2021 ThoughtworksGood sense6262© 2021 Thoughtworks
© 2021 ThoughtworksAny code you can writein Python, you can testin Python.6363© 2021 Thoughtworks
© 2021 ThoughtworksIf a Python programhas adequate unittests, it can be asrobust as a C++, Java,or C# program withadequate unit tests(although the tests inPython will be fasterto write).64Bruce Eckel, author of Thinking in C, On Java 8,Atomic Scala, and Atomic Kotlin in The Best OfSoftware Writing anthology by Joel Spolsky (ed.) 64© 2021 Thoughtworks
© 2021 ThoughtworksCompared with unittests, type hints helpdetect a different andsmaller set of bugs.Static type checkingcomplements butcannot replaceautomated testing.6565© 2021 Thoughtworks
© 2021 ThoughtworksPython is a dynamiclanguage withpowerful abstractionsand advancedmetaprogrammingfeatures.That’s why we like it.6666© 2021 Thoughtworks
© 2021 ThoughtworksTeams can only avoidthe downsides of typehints if they are notmandatory everywhere.6767© 2021 Thoughtworks
© 2021 ThoughtworksEmbrace GradualTyping with MypySome options help with gradual typing.Others are unhelpful:they force static typing everywhere.68Helpful for gradual typing:--disallow-incomplete-defs--check-untyped-defs--warn-unused-ignoresUnhelpful for gradual typing:--strict--disallow-any-* 3 options)--disallow-untyped-* 2 options)
© 2021 ThoughtworksStatic typing is valuable in certain contextsFor professional software development teams,the benefits of static typing surpass the cost.But remember that Python is widely used inmany other contexts.Main takeaways69
© 2021 ThoughtworksStatic typing is valuable in certain contextsFor professional software development teams,the benefits of static typing surpass the cost.But remember that Python is widely used inmany other contexts.Main takeaways70Use Protocols to enable Pythonic APIsWithout the support for static duck typingprovided by typing.Protocol, widespreaduse of type hints may lead to inflexible andverbose codebases.Define and use narrow protocols as needed.
© 2021 ThoughtworksStatic typing is valuable in certain contextsFor professional software development teams,the benefits of static typing surpass itssignificant cost. But remember that Python iswidely used in many other contexts.Main takeaways71Use Protocols to enable Pythonic APIsWithout the support for static duck typingprovided by typing.Protocol, widespreaduse of type hints may lead to inflexible andverbose codebases.Define and use narrow protocols as needed.Don’t make type hints mandatory everywhereDevelopers should be free to leave portions ofthe code without type hints, to avoid thedownsides and limitations of the static typesystem and the tools that enforce it.
© 2021 ThoughtworksBeing optional is not abug or limitation ofPython type hints.It’s the feature thatgives us the powerto cope with theinherent complexities,annoyances, flaws,and limitations ofstatic types.7272© 2021 Thoughtworks
© 2021 ThoughtworksBeing optional is not abug or limitation ofPython type hints.It’s the feature thatgives us the powerto cope with theinherent complexities,annoyances, flaws,and limitations ofstatic types.7373© 2021 Thoughtworks
© 2021 ThoughtworksThank you!Luciano RamalhoPrincipal Consultant[email protected]Twitter: @ramalhoorg74