The Python You Don’t Know

“You know Python. You don’t know this Python.”

Python is highly touted as “batteries-included,” referring to the extensive features provided by the standard library, so Python programmers can pick up the tool already around, and focus straight onto the real problem.

Transcript

1. The Python You Don’t Know
   

   View Slide

2. https://pypi.org
   

   View Slide

3. PYTHON COMMUNITY
   

   View Slide

4. View Slide

5. BATTERIES
   INCLUDED
   

   View Slide

6. WHAT IF I TOLD YOU
   PYTHON HAS IT BUILT-IN
   

   View Slide

7. import antigravity
   https://xkcd.com/353/
   

   View Slide

8. The Obvious
   • import this
   • python -m json.tool
   • python -m http.server
   • python -m zipfile
   • ...
   

   View Slide

9. Me
   • Call me TP
   • Follow @uranusjr
   • https://uranusjr.com
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. http://macdown.uranusjr.com
    

    View Slide

14. www. .com
    

    View Slide

15. Do Things Better
    

    View Slide

16. Do Things Better
    • Nothing you couldn’t do
    • Can be done easier, better
    

    View Slide

17. pathlib
    • Path object
    • Neutral or OS-specific
    • Path & file operations
    • open, os, os.path
    • glob
    

    View Slide

18. >>> import pathlib
    >>> base_dir = pathlib.Path(__file__).parent
    >>> base_dir / 'data' / 'main.cfg'
    PosixPath('data/main.cfg')
    >>> conf = base_dir.joinpath('data', 'main.cfg')
    >>> str(conf)
    'data/main.cfg'
    >>> str(log_dir.resolve())
    '/User/uranusjr/my_project/log/project.log'
    

    View Slide

19. glob
    >>> for path in base_dir.glob('**/*.py'):
    ... print(path)
    ...
    MacDown/Code/Dependency/build.py
    Tools/build_for_release.py
    Tools/import_translations.py
    

    View Slide

20. OS-independent
    >>> str(PureWindowsPath('foo', 'bar'))
    foo\bar
    >>> str(PurePosixPath('foo', 'bar'))
    foo/bar
    

    View Slide

21. https://docs.python.org/3/library/logging.handlers.html
    

    View Slide

22. enum
    • Augment common enum pattern
    • Metaclass magic
    • Useful decorators
    

    View Slide

23. >>> class Color:
    ... RED = 1
    ... GREEN = 2
    ... BLUE = 3
    >>> print(Color.RED)
    1
    >>> print(type(Color.RED))
    
    

    View Slide

24. >>> import enum
    >>> class Color(enum.Enum):
    ... RED = 1
    ... GREEN = 2
    ... BLUE = 3
    >>> print(Color.RED)
    Color.RED
    >>> print(type(Color.RED))
    
    

    View Slide

25. >>> print(Color.RED.value)
    1
    >>> print(Color.RED.name)
    RED
    >>> isinstance(Color.RED, Color)
    True
    >>> Color['BLUE']
    Color.BLUE
    >>> Color(1)
    Color.GREEN
    

    View Slide

26. No Duplicate Keys
    >>> class Shape(enum.Enum):
    ... SQUARE = 'square'
    ... SQUARE = '█'
    Traceback (most recent call last):
    ...
    TypeError: Attempted to reuse key: 'SQUARE'
    

    View Slide

27. Value Alias Are Fine
    >>> class Shape(enum.Enum):
    ... SQUARE = '█'
    ... RECTANGLE = '█'
    >>> Shape.SQUARE == Shape.RECTANGLE
    True
    

    View Slide

28. Unless You Say No
    >>> @enum.unique
    ... class Shape(enum.Enum):
    ... SQUARE = '█'
    ... RECTANGLE = '█'
    Traceback (most recent call last):
    ...
    ValueError: duplicate values found in
    : SQUARE -> RECTANGLE
    

    View Slide

29. More
    • enum.IntEnum
    • enum.Flag
    • enum.IntFlag
    • Functional API
    • …
    

    View Slide

30. Do Things Better
    • Nothing you couldn’t do
    • Can be done easier, better
    • More type-safety?
    

    View Slide

31. Write Better Code
    

    View Slide

32. “More Correct”
    • Program safety
    • Cross-platform
    • User friendliness
    

    View Slide

33. collections.abc
    • ABC = Abstract Base Classes
    • Object-oriented interface for
    feature checking
    

    View Slide

34. def normalize_data(data):
    if isinstance(data, list):
    return data
    return [data]
    

    View Slide

35. def normalize_data(data):
    if isinstance(data, (list, tuple)):
    return data
    return [data]
    

    View Slide

36. def normalize_data(data):
    if hasattr(data, '__iter__'):
    return data
    return [data]
    

    View Slide

37. def normalize_data(data):
    if (hasattr(data, '__iter__') and
    hasattr(data, '__len__')):
    return data
    return [data]
    

    View Slide

38. def normalize_data(data):
    if (hasattr(data, '__iter__') and
    hasattr(data, '__len__') and
    hasattr(data, '__contains__')):
    return data
    return [data]
    

    View Slide

39. def normalize_data(data):
    if isinstance(data, __________):
    return data
    return [data]
    

    View Slide

40. from collections.abc import Collection
    def normalize_data(data):
    if isinstance(data, Collection):
    return data
    return [data]
    

    View Slide

41. ABC
    • Metaclass magic!
    • Represents a set of features
    • Does not require inheritance
    

    View Slide

42. class Emptiness:
    def __contains__(self, x):
    return False
    def __iter__(self):
    return iter([])
    def __len__(self):
    return 0
    emptiness = Emptiness()
    isinstance(emptiness, Collection)
    # True!
    

    View Slide

43. class Emptiness(Collection):
    def __contains__(self, x):
    return False
    def __iter__(self):
    return iter([])
    def __len__(self):
    return 0
    emptiness = Emptiness()
    isinstance(emptiness, Collection)
    # True!
    

    View Slide

44. class Emptiness(Collection):
    def __contains__(self, x):
    return False
    def __iter__(self):
    return iter([])
    # Forgot to implement __len__
    emptiness = Emptiness()
    # TypeError: Can't instantiate abstract class
    # Emptiness with abstract methods __len__
    

    View Slide

45. https://docs.python.org/3/library/collections.abc.html
    

    View Slide

46. contextlib
    • You should know @contextmanager
    • class AbstractContextManager
    

    View Slide

47. f = open('output.txt')
    f.write('Hello!')
    f.close()
    

    View Slide

48. with open('output.txt') as f:
    f.write('Hello!')
    

    View Slide

49. from pymongo import MongoClient
    c = MongoClient(MONGODB_URL)
    c['mycol'].find_one({})
    c.close()
    

    View Slide

50. from contextlib import closing
    from pymongo import MongoClient
    with closing(MongoClient(MONGODB_URL)) as c:
    c['mycol'].find_one({})
    

    View Slide

51. try:
    os.remove('process.pid')
    except (FileNotFoundError, PermissionError):
    pass
    

    View Slide

52. from contextlib import suppress
    with suppress(FileNotFoundError, PermissionError):
    os.remove('process.pid')
    

    View Slide

53. with connect(server, user, password, 'tempdb') as conn:
    with conn.cursor(as_dict=True) as cursor:
    cursor.execute(
    'SELECT * FROM persons WHERE salesrep=%s',
    'John Doe'
    )
    for row in cursor:
    print('ID={}, Name={}'.format(
    row['id'], row['name'],
    ))
    http://pymssql.org/en/stable/pymssql_examples.html
    

    View Slide

54. View Slide

55. import contextlib
    with contextlib.ExitStack() as es:
    conn = es.enter_context(connect(...))
    cursor = es.enter_context(conn.cursor())
    do_things(cursor)
    

    View Slide

56. contextlib
    • redirect_stdout
    • redirect_stderr
    • ContextDecorator
    

    View Slide

57. http://click.pocoo.org/
    

    View Slide

58. “More Correct”
    • Program safety
    • Cross-platform
    • User friendliness
    

    View Slide

59. Write Better Code
    Functional
    Special Edition!
    

    View Slide

60. https://github.com/kennethreitz/pipenv/pull/344/
    

    View Slide

61. Avoid Lambdas
    • Limited introspection
    • No function name (anonymousness!)
    • No local scope
    • Difficult to make sense in traceback
    

    View Slide

62. https://www.artima.com/weblogs/viewpost.jsp?thread=98196
    

    View Slide

63. Lambda Alternatives
    • Comprehensions go a long way
    • If you need a function, just def one
    • But maybe you don’t!
    

    View Slide

64. Local Closures
    def job_callback(loop):
    print('Finish job in', loop)
    def run_app():
    ...
    loop.call_later(
    lambda: job_callback(loop),
    )
    

    View Slide

65. import functools
    def job_callback(loop):
    print('Finish job in', loop)
    def run_app():
    ...
    loop.call_later(functools.partial(
    job_callback, loop,
    ))
    

    View Slide

66. Semantically Functional
    correct_obj_iter = (
    obj
    for obj in object_list
    if obj.correct
    )
    

    View Slide

67. correct_obj_iter = filter(
    lambda obj: obj.correct,
    object_list,
    )
    

    View Slide

68. import operator
    correct_obj_iter = filter(
    operator.attrgetter('correct'),
    object_list,
    )
    

    View Slide

69. result_iter = map(
    lambda data: data.validate(),
    data_list,
    )
    

    View Slide

70. import operator
    result_iter = map(
    operator.methodcaller('validate'),
    data_list,
    )
    

    View Slide

71. # Something like this.
    pair_list = [
    (2, 3), (19, 8), (42, 1),
    ]
    summed_iter = map(
    lambda x, y: x + y,
    pair_list,
    )
    

    View Slide

72. import operator
    pair_list = [
    (2, 3), (19, 8), (42, 1),
    ]
    summed_iter = map(
    operator.add,
    pair_list,
    )
    

    View Slide

73. pair_list = [
    (2, 3), (19, 8), (42, 1),
    ]
    summed_iter = map(
    sum,
    pair_list,
    )
    

    View Slide

74. import math
    pair_list = [
    (2, 3), (19, 8), (42, 1),
    ]
    summed_iter = map(
    math.sum,
    pair_list,
    )
    

    View Slide

75. list_of_lists = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9],
    ]
    joined_list = functools.reduce(
    lambda s, x: s + x,
    list_of_lists,
    initializer=list,
    )
    

    View Slide

76. list_of_lists = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9],
    ]
    joined_list = functools.reduce(
    operator.add,
    list_of_lists,
    initializer=list,
    )
    

    View Slide

77. list_of_lists = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9],
    ]
    joined_list = list(
    itertool.chain.from_iterable(
    list_of_list,
    ),
    )
    

    View Slide

78. Lambda Alternatives
    • functools.partial
    • operator for builtin operators
    • itertools to handle iterables
    

    View Slide

79. Useful When Needed
    

    View Slide

80. getpass
    • getpass.getpass()
    • “input() without echo”
    • Not really (but better)
    

    View Slide

81. difflib
    • Generates diff — diff-ing
    • Restores input — patching
    • Utilities for content comparison
    

    View Slide

82. textwrap
    • Break text into lines
    • Indent, dedent, tab expansion
    • Simple hyphenation & trimming
    

    View Slide

83. xml.*
    • “Mom, I don’t need lxml!”
    • Generate and parse XML
    • Different parsers (e.g. expat)
    

    View Slide

84. re
    • Wait, you should know this
    • Unless you don’t really!
    

    View Slide

85. http://lucumr.pocoo.org/2015/11/18/pythons-hidden-re-gems/
    

    View Slide

86. https://github.com/python/cpython/blob/master/Lib/re.py
    

    View Slide

87. def s_ident(scanner, token): return token
    def s_operator(scanner, token): return "op%s" % token
    def s_float(scanner, token): return float(token)
    def s_int(scanner, token): return int(token)
    scanner = re.Scanner([
    (r"[a-zA-Z_]\w*", s_ident), (r"\d+\.\d*", s_float),
    (r"\d+", s_int), (r"=|\+|-|\*|/", s_operator),
    (r"\s+", None),
    ])
    print(scanner.scan("sum = 3*foo + 312.50 + bar"))
    # (['sum', 'op=', 3, 'op*', 'foo', 'op+', 312.5, 'op+', 'bar'], '')
    

    View Slide

88. The sre Modules
    

    View Slide

89. The sre Modules
    • sre_constants defines constants
    • sre_parse defines a pattern and how to parse
    string input into one
    • sre_compile provides an interface to convert
    a pattern list to internal format
    

    View Slide

90. Go Third-Party
    

    View Slide

91. View Slide

92. asyncore
    • Wait, does anyone actually
    still use it?
    • I don’t need to tell you
    about asyncio
    • Wait that’s built-in too
    

    View Slide

93. getopt & optparse
    • argparse is fine
    • click
    • docopt
    

    View Slide

94. urllib.request
    • Use Requests
    • No good SSL support
    • Don’t use httplib either
    • urllib.parse is fine
    

    View Slide

95. xml.*
    • Wait didn’t you just say…
    • Not good for parsing
    • defusedxml
    • Need to beware of vulnerabilities
    in any case
    

    View Slide

96. venv
    • Virtualenv: marginally better
    • pew
    • pipenv
    • pex
    

    View Slide

97. https://gist.github.com/datagrok/2199506
    

    View Slide

98. Final Words
    • There are more!
    • Read the docs as a novel
    • Read the source
    

    View Slide

99. ܅ͷ஌Βͳ͍ 1ZUIPO
    

    View Slide

100. LIVE LONG
     AND PROSPER
     

     View Slide