$30 off During Our Annual Pro Sale. View Details »

The Python You Don’t Know

The Python You Don’t Know

“You know Python. You don’t know this Python.”

Python is highly touted as “batteries-included,” referring to the extensive features provided by the standard library, so Python programmers can pick up the tool already around, and focus straight onto the real problem.

Tzu-ping Chung

August 26, 2017
Tweet

More Decks by Tzu-ping Chung

Other Decks in Programming

Transcript

  1. The Python You Don’t Know

    View Slide

  2. https://pypi.org

    View Slide

  3. PYTHON COMMUNITY

    View Slide

  4. View Slide

  5. BATTERIES
    INCLUDED

    View Slide

  6. WHAT IF I TOLD YOU
    PYTHON HAS IT BUILT-IN

    View Slide

  7. import antigravity
    https://xkcd.com/353/

    View Slide

  8. The Obvious
    • import this
    • python -m json.tool
    • python -m http.server
    • python -m zipfile
    • ...

    View Slide

  9. Me
    • Call me TP
    • Follow @uranusjr
    • https://uranusjr.com

    View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. http://macdown.uranusjr.com

    View Slide

  14. www. .com

    View Slide

  15. Do Things Better

    View Slide

  16. Do Things Better
    • Nothing you couldn’t do
    • Can be done easier, better

    View Slide

  17. pathlib
    • Path object
    • Neutral or OS-specific
    • Path & file operations
    • open, os, os.path
    • glob

    View Slide

  18. >>> import pathlib
    >>> base_dir = pathlib.Path(__file__).parent
    >>> base_dir / 'data' / 'main.cfg'
    PosixPath('data/main.cfg')
    >>> conf = base_dir.joinpath('data', 'main.cfg')
    >>> str(conf)
    'data/main.cfg'
    >>> str(log_dir.resolve())
    '/User/uranusjr/my_project/log/project.log'

    View Slide

  19. glob
    >>> for path in base_dir.glob('**/*.py'):
    ... print(path)
    ...
    MacDown/Code/Dependency/build.py
    Tools/build_for_release.py
    Tools/import_translations.py

    View Slide

  20. OS-independent
    >>> str(PureWindowsPath('foo', 'bar'))
    foo\bar
    >>> str(PurePosixPath('foo', 'bar'))
    foo/bar

    View Slide

  21. https://docs.python.org/3/library/logging.handlers.html

    View Slide

  22. enum
    • Augment common enum pattern
    • Metaclass magic
    • Useful decorators

    View Slide

  23. >>> class Color:
    ... RED = 1
    ... GREEN = 2
    ... BLUE = 3
    >>> print(Color.RED)
    1
    >>> print(type(Color.RED))

    View Slide

  24. >>> import enum
    >>> class Color(enum.Enum):
    ... RED = 1
    ... GREEN = 2
    ... BLUE = 3
    >>> print(Color.RED)
    Color.RED
    >>> print(type(Color.RED))

    View Slide

  25. >>> print(Color.RED.value)
    1
    >>> print(Color.RED.name)
    RED
    >>> isinstance(Color.RED, Color)
    True
    >>> Color['BLUE']
    Color.BLUE
    >>> Color(1)
    Color.GREEN

    View Slide

  26. No Duplicate Keys
    >>> class Shape(enum.Enum):
    ... SQUARE = 'square'
    ... SQUARE = '█'
    Traceback (most recent call last):
    ...
    TypeError: Attempted to reuse key: 'SQUARE'

    View Slide

  27. Value Alias Are Fine
    >>> class Shape(enum.Enum):
    ... SQUARE = '█'
    ... RECTANGLE = '█'
    >>> Shape.SQUARE == Shape.RECTANGLE
    True

    View Slide

  28. Unless You Say No
    >>> @enum.unique
    ... class Shape(enum.Enum):
    ... SQUARE = '█'
    ... RECTANGLE = '█'
    Traceback (most recent call last):
    ...
    ValueError: duplicate values found in
    : SQUARE -> RECTANGLE

    View Slide

  29. More
    • enum.IntEnum
    • enum.Flag
    • enum.IntFlag
    • Functional API
    • …

    View Slide

  30. Do Things Better
    • Nothing you couldn’t do
    • Can be done easier, better
    • More type-safety?

    View Slide

  31. Write Better Code

    View Slide

  32. “More Correct”
    • Program safety
    • Cross-platform
    • User friendliness

    View Slide

  33. collections.abc
    • ABC = Abstract Base Classes
    • Object-oriented interface for
    feature checking

    View Slide

  34. def normalize_data(data):
    if isinstance(data, list):
    return data
    return [data]

    View Slide

  35. def normalize_data(data):
    if isinstance(data, (list, tuple)):
    return data
    return [data]

    View Slide

  36. def normalize_data(data):
    if hasattr(data, '__iter__'):
    return data
    return [data]

    View Slide

  37. def normalize_data(data):
    if (hasattr(data, '__iter__') and
    hasattr(data, '__len__')):
    return data
    return [data]

    View Slide

  38. def normalize_data(data):
    if (hasattr(data, '__iter__') and
    hasattr(data, '__len__') and
    hasattr(data, '__contains__')):
    return data
    return [data]

    View Slide

  39. def normalize_data(data):
    if isinstance(data, __________):
    return data
    return [data]

    View Slide

  40. from collections.abc import Collection
    def normalize_data(data):
    if isinstance(data, Collection):
    return data
    return [data]

    View Slide

  41. ABC
    • Metaclass magic!
    • Represents a set of features
    • Does not require inheritance

    View Slide

  42. class Emptiness:
    def __contains__(self, x):
    return False
    def __iter__(self):
    return iter([])
    def __len__(self):
    return 0
    emptiness = Emptiness()
    isinstance(emptiness, Collection)
    # True!

    View Slide

  43. class Emptiness(Collection):
    def __contains__(self, x):
    return False
    def __iter__(self):
    return iter([])
    def __len__(self):
    return 0
    emptiness = Emptiness()
    isinstance(emptiness, Collection)
    # True!

    View Slide

  44. class Emptiness(Collection):
    def __contains__(self, x):
    return False
    def __iter__(self):
    return iter([])
    # Forgot to implement __len__
    emptiness = Emptiness()
    # TypeError: Can't instantiate abstract class
    # Emptiness with abstract methods __len__

    View Slide

  45. https://docs.python.org/3/library/collections.abc.html

    View Slide

  46. contextlib
    • You should know @contextmanager
    • class AbstractContextManager

    View Slide

  47. f = open('output.txt')
    f.write('Hello!')
    f.close()

    View Slide

  48. with open('output.txt') as f:
    f.write('Hello!')

    View Slide

  49. from pymongo import MongoClient
    c = MongoClient(MONGODB_URL)
    c['mycol'].find_one({})
    c.close()

    View Slide

  50. from contextlib import closing
    from pymongo import MongoClient
    with closing(MongoClient(MONGODB_URL)) as c:
    c['mycol'].find_one({})

    View Slide

  51. try:
    os.remove('process.pid')
    except (FileNotFoundError, PermissionError):
    pass

    View Slide

  52. from contextlib import suppress
    with suppress(FileNotFoundError, PermissionError):
    os.remove('process.pid')

    View Slide

  53. with connect(server, user, password, 'tempdb') as conn:
    with conn.cursor(as_dict=True) as cursor:
    cursor.execute(
    'SELECT * FROM persons WHERE salesrep=%s',
    'John Doe'
    )
    for row in cursor:
    print('ID={}, Name={}'.format(
    row['id'], row['name'],
    ))
    http://pymssql.org/en/stable/pymssql_examples.html

    View Slide

  54. View Slide

  55. import contextlib
    with contextlib.ExitStack() as es:
    conn = es.enter_context(connect(...))
    cursor = es.enter_context(conn.cursor())
    do_things(cursor)

    View Slide

  56. contextlib
    • redirect_stdout
    • redirect_stderr
    • ContextDecorator

    View Slide

  57. http://click.pocoo.org/

    View Slide

  58. “More Correct”
    • Program safety
    • Cross-platform
    • User friendliness

    View Slide

  59. Write Better Code
    Functional
    Special Edition!

    View Slide

  60. https://github.com/kennethreitz/pipenv/pull/344/

    View Slide

  61. Avoid Lambdas
    • Limited introspection
    • No function name (anonymousness!)
    • No local scope
    • Difficult to make sense in traceback

    View Slide

  62. https://www.artima.com/weblogs/viewpost.jsp?thread=98196

    View Slide

  63. Lambda Alternatives
    • Comprehensions go a long way
    • If you need a function, just def one
    • But maybe you don’t!

    View Slide

  64. Local Closures
    def job_callback(loop):
    print('Finish job in', loop)
    def run_app():
    ...
    loop.call_later(
    lambda: job_callback(loop),
    )

    View Slide

  65. import functools
    def job_callback(loop):
    print('Finish job in', loop)
    def run_app():
    ...
    loop.call_later(functools.partial(
    job_callback, loop,
    ))

    View Slide

  66. Semantically Functional
    correct_obj_iter = (
    obj
    for obj in object_list
    if obj.correct
    )

    View Slide

  67. correct_obj_iter = filter(
    lambda obj: obj.correct,
    object_list,
    )

    View Slide

  68. import operator
    correct_obj_iter = filter(
    operator.attrgetter('correct'),
    object_list,
    )

    View Slide

  69. result_iter = map(
    lambda data: data.validate(),
    data_list,
    )

    View Slide

  70. import operator
    result_iter = map(
    operator.methodcaller('validate'),
    data_list,
    )

    View Slide

  71. # Something like this.
    pair_list = [
    (2, 3), (19, 8), (42, 1),
    ]
    summed_iter = map(
    lambda x, y: x + y,
    pair_list,
    )

    View Slide

  72. import operator
    pair_list = [
    (2, 3), (19, 8), (42, 1),
    ]
    summed_iter = map(
    operator.add,
    pair_list,
    )

    View Slide

  73. pair_list = [
    (2, 3), (19, 8), (42, 1),
    ]
    summed_iter = map(
    sum,
    pair_list,
    )

    View Slide

  74. import math
    pair_list = [
    (2, 3), (19, 8), (42, 1),
    ]
    summed_iter = map(
    math.sum,
    pair_list,
    )

    View Slide

  75. list_of_lists = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9],
    ]
    joined_list = functools.reduce(
    lambda s, x: s + x,
    list_of_lists,
    initializer=list,
    )

    View Slide

  76. list_of_lists = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9],
    ]
    joined_list = functools.reduce(
    operator.add,
    list_of_lists,
    initializer=list,
    )

    View Slide

  77. list_of_lists = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9],
    ]
    joined_list = list(
    itertool.chain.from_iterable(
    list_of_list,
    ),
    )

    View Slide

  78. Lambda Alternatives
    • functools.partial
    • operator for builtin operators
    • itertools to handle iterables

    View Slide

  79. Useful When Needed

    View Slide

  80. getpass
    • getpass.getpass()
    • “input() without echo”
    • Not really (but better)

    View Slide

  81. difflib
    • Generates diff — diff-ing
    • Restores input — patching
    • Utilities for content comparison

    View Slide

  82. textwrap
    • Break text into lines
    • Indent, dedent, tab expansion
    • Simple hyphenation & trimming

    View Slide

  83. xml.*
    • “Mom, I don’t need lxml!”
    • Generate and parse XML
    • Different parsers (e.g. expat)

    View Slide

  84. re
    • Wait, you should know this
    • Unless you don’t really!

    View Slide

  85. http://lucumr.pocoo.org/2015/11/18/pythons-hidden-re-gems/

    View Slide

  86. https://github.com/python/cpython/blob/master/Lib/re.py

    View Slide

  87. def s_ident(scanner, token): return token
    def s_operator(scanner, token): return "op%s" % token
    def s_float(scanner, token): return float(token)
    def s_int(scanner, token): return int(token)
    scanner = re.Scanner([
    (r"[a-zA-Z_]\w*", s_ident), (r"\d+\.\d*", s_float),
    (r"\d+", s_int), (r"=|\+|-|\*|/", s_operator),
    (r"\s+", None),
    ])
    print(scanner.scan("sum = 3*foo + 312.50 + bar"))
    # (['sum', 'op=', 3, 'op*', 'foo', 'op+', 312.5, 'op+', 'bar'], '')

    View Slide

  88. The sre Modules

    View Slide

  89. The sre Modules
    • sre_constants defines constants
    • sre_parse defines a pattern and how to parse
    string input into one
    • sre_compile provides an interface to convert
    a pattern list to internal format

    View Slide

  90. Go Third-Party

    View Slide

  91. View Slide

  92. asyncore
    • Wait, does anyone actually
    still use it?
    • I don’t need to tell you
    about asyncio
    • Wait that’s built-in too

    View Slide

  93. getopt & optparse
    • argparse is fine
    • click
    • docopt

    View Slide

  94. urllib.request
    • Use Requests
    • No good SSL support
    • Don’t use httplib either
    • urllib.parse is fine

    View Slide

  95. xml.*
    • Wait didn’t you just say…
    • Not good for parsing
    • defusedxml
    • Need to beware of vulnerabilities
    in any case

    View Slide

  96. venv
    • Virtualenv: marginally better
    • pew
    • pipenv
    • pex

    View Slide

  97. https://gist.github.com/datagrok/2199506

    View Slide

  98. Final Words
    • There are more!
    • Read the docs as a novel
    • Read the source

    View Slide

  99. ܅ͷ஌Βͳ͍ 1ZUIPO

    View Slide

  100. LIVE LONG
    AND PROSPER

    View Slide