The Python You Don’t Know

The Python You Don’t Know

“You know Python. You don’t know this Python.”

Python is highly touted as “batteries-included,” referring to the extensive features provided by the standard library, so Python programmers can pick up the tool already around, and focus straight onto the real problem.

9dafad54b5b4f360b7aae5f482bc1c91?s=128

Tzu-ping Chung

August 26, 2017
Tweet

Transcript

  1. The Python You Don’t Know

  2. https://pypi.org

  3. PYTHON COMMUNITY

  4. None
  5. BATTERIES INCLUDED

  6. WHAT IF I TOLD YOU PYTHON HAS IT BUILT-IN

  7. import antigravity https://xkcd.com/353/

  8. The Obvious • import this • python -m json.tool •

    python -m http.server • python -m zipfile • ...
  9. Me • Call me TP • Follow @uranusjr • https://uranusjr.com

  10. None
  11. None
  12. None
  13. http://macdown.uranusjr.com

  14. www. .com

  15. Do Things Better

  16. Do Things Better • Nothing you couldn’t do • Can

    be done easier, better
  17. pathlib • Path object • Neutral or OS-specific • Path

    & file operations • open, os, os.path • glob
  18. >>> import pathlib >>> base_dir = pathlib.Path(__file__).parent >>> base_dir /

    'data' / 'main.cfg' PosixPath('data/main.cfg') >>> conf = base_dir.joinpath('data', 'main.cfg') >>> str(conf) 'data/main.cfg' >>> str(log_dir.resolve()) '/User/uranusjr/my_project/log/project.log'
  19. glob >>> for path in base_dir.glob('**/*.py'): ... print(path) ... MacDown/Code/Dependency/build.py

    Tools/build_for_release.py Tools/import_translations.py
  20. OS-independent >>> str(PureWindowsPath('foo', 'bar')) foo\bar >>> str(PurePosixPath('foo', 'bar')) foo/bar

  21. https://docs.python.org/3/library/logging.handlers.html

  22. enum • Augment common enum pattern • Metaclass magic •

    Useful decorators
  23. >>> class Color: ... RED = 1 ... GREEN =

    2 ... BLUE = 3 >>> print(Color.RED) 1 >>> print(type(Color.RED)) <class 'int'>
  24. >>> import enum >>> class Color(enum.Enum): ... RED = 1

    ... GREEN = 2 ... BLUE = 3 >>> print(Color.RED) Color.RED >>> print(type(Color.RED)) <enum 'Color'>
  25. >>> print(Color.RED.value) 1 >>> print(Color.RED.name) RED >>> isinstance(Color.RED, Color) True

    >>> Color['BLUE'] Color.BLUE >>> Color(1) Color.GREEN
  26. No Duplicate Keys >>> class Shape(enum.Enum): ... SQUARE = 'square'

    ... SQUARE = '█' Traceback (most recent call last): ... TypeError: Attempted to reuse key: 'SQUARE'
  27. Value Alias Are Fine >>> class Shape(enum.Enum): ... SQUARE =

    '█' ... RECTANGLE = '█' >>> Shape.SQUARE == Shape.RECTANGLE True
  28. Unless You Say No >>> @enum.unique ... class Shape(enum.Enum): ...

    SQUARE = '█' ... RECTANGLE = '█' Traceback (most recent call last): ... ValueError: duplicate values found in <enum 'Shape'>: SQUARE -> RECTANGLE
  29. More • enum.IntEnum • enum.Flag • enum.IntFlag • Functional API

    • …
  30. Do Things Better • Nothing you couldn’t do • Can

    be done easier, better • More type-safety?
  31. Write Better Code

  32. “More Correct” • Program safety • Cross-platform • User friendliness

  33. collections.abc • ABC = Abstract Base Classes • Object-oriented interface

    for feature checking
  34. def normalize_data(data): if isinstance(data, list): return data return [data]

  35. def normalize_data(data): if isinstance(data, (list, tuple)): return data return [data]

  36. def normalize_data(data): if hasattr(data, '__iter__'): return data return [data]

  37. def normalize_data(data): if (hasattr(data, '__iter__') and hasattr(data, '__len__')): return data

    return [data]
  38. def normalize_data(data): if (hasattr(data, '__iter__') and hasattr(data, '__len__') and hasattr(data,

    '__contains__')): return data return [data]
  39. def normalize_data(data): if isinstance(data, __________): return data return [data]

  40. from collections.abc import Collection def normalize_data(data): if isinstance(data, Collection): return

    data return [data]
  41. ABC • Metaclass magic! • Represents a set of features

    • Does not require inheritance
  42. class Emptiness: def __contains__(self, x): return False def __iter__(self): return

    iter([]) def __len__(self): return 0 emptiness = Emptiness() isinstance(emptiness, Collection) # True!
  43. class Emptiness(Collection): def __contains__(self, x): return False def __iter__(self): return

    iter([]) def __len__(self): return 0 emptiness = Emptiness() isinstance(emptiness, Collection) # True!
  44. class Emptiness(Collection): def __contains__(self, x): return False def __iter__(self): return

    iter([]) # Forgot to implement __len__ emptiness = Emptiness() # TypeError: Can't instantiate abstract class # Emptiness with abstract methods __len__
  45. https://docs.python.org/3/library/collections.abc.html

  46. contextlib • You should know @contextmanager • class AbstractContextManager

  47. f = open('output.txt') f.write('Hello!') f.close()

  48. with open('output.txt') as f: f.write('Hello!')

  49. from pymongo import MongoClient c = MongoClient(MONGODB_URL) c['mycol'].find_one({}) c.close()

  50. from contextlib import closing from pymongo import MongoClient with closing(MongoClient(MONGODB_URL))

    as c: c['mycol'].find_one({})
  51. try: os.remove('process.pid') except (FileNotFoundError, PermissionError): pass

  52. from contextlib import suppress with suppress(FileNotFoundError, PermissionError): os.remove('process.pid')

  53. with connect(server, user, password, 'tempdb') as conn: with conn.cursor(as_dict=True) as

    cursor: cursor.execute( 'SELECT * FROM persons WHERE salesrep=%s', 'John Doe' ) for row in cursor: print('ID={}, Name={}'.format( row['id'], row['name'], )) http://pymssql.org/en/stable/pymssql_examples.html
  54. None
  55. import contextlib with contextlib.ExitStack() as es: conn = es.enter_context(connect(...)) cursor

    = es.enter_context(conn.cursor()) do_things(cursor)
  56. contextlib • redirect_stdout • redirect_stderr • ContextDecorator

  57. http://click.pocoo.org/

  58. “More Correct” • Program safety • Cross-platform • User friendliness

  59. Write Better Code Functional Special Edition!

  60. https://github.com/kennethreitz/pipenv/pull/344/

  61. Avoid Lambdas • Limited introspection • No function name (anonymousness!)

    • No local scope • Difficult to make sense in traceback
  62. https://www.artima.com/weblogs/viewpost.jsp?thread=98196

  63. Lambda Alternatives • Comprehensions go a long way • If

    you need a function, just def one • But maybe you don’t!
  64. Local Closures def job_callback(loop): print('Finish job in', loop) def run_app():

    ... loop.call_later( lambda: job_callback(loop), )
  65. import functools def job_callback(loop): print('Finish job in', loop) def run_app():

    ... loop.call_later(functools.partial( job_callback, loop, ))
  66. Semantically Functional correct_obj_iter = ( obj for obj in object_list

    if obj.correct )
  67. correct_obj_iter = filter( lambda obj: obj.correct, object_list, )

  68. import operator correct_obj_iter = filter( operator.attrgetter('correct'), object_list, )

  69. result_iter = map( lambda data: data.validate(), data_list, )

  70. import operator result_iter = map( operator.methodcaller('validate'), data_list, )

  71. # Something like this. pair_list = [ (2, 3), (19,

    8), (42, 1), ] summed_iter = map( lambda x, y: x + y, pair_list, )
  72. import operator pair_list = [ (2, 3), (19, 8), (42,

    1), ] summed_iter = map( operator.add, pair_list, )
  73. pair_list = [ (2, 3), (19, 8), (42, 1), ]

    summed_iter = map( sum, pair_list, )
  74. import math pair_list = [ (2, 3), (19, 8), (42,

    1), ] summed_iter = map( math.sum, pair_list, )
  75. list_of_lists = [ [1, 2, 3], [4, 5, 6], [7,

    8, 9], ] joined_list = functools.reduce( lambda s, x: s + x, list_of_lists, initializer=list, )
  76. list_of_lists = [ [1, 2, 3], [4, 5, 6], [7,

    8, 9], ] joined_list = functools.reduce( operator.add, list_of_lists, initializer=list, )
  77. list_of_lists = [ [1, 2, 3], [4, 5, 6], [7,

    8, 9], ] joined_list = list( itertool.chain.from_iterable( list_of_list, ), )
  78. Lambda Alternatives • functools.partial • operator for builtin operators •

    itertools to handle iterables
  79. Useful When Needed

  80. getpass • getpass.getpass() • “input() without echo” • Not really

    (but better)
  81. difflib • Generates diff — diff-ing • Restores input —

    patching • Utilities for content comparison
  82. textwrap • Break text into lines • Indent, dedent, tab

    expansion • Simple hyphenation & trimming
  83. xml.* • “Mom, I don’t need lxml!” • Generate and

    parse XML • Different parsers (e.g. expat)
  84. re • Wait, you should know this • Unless you

    don’t really!
  85. http://lucumr.pocoo.org/2015/11/18/pythons-hidden-re-gems/

  86. https://github.com/python/cpython/blob/master/Lib/re.py

  87. def s_ident(scanner, token): return token def s_operator(scanner, token): return "op%s"

    % token def s_float(scanner, token): return float(token) def s_int(scanner, token): return int(token) scanner = re.Scanner([ (r"[a-zA-Z_]\w*", s_ident), (r"\d+\.\d*", s_float), (r"\d+", s_int), (r"=|\+|-|\*|/", s_operator), (r"\s+", None), ]) print(scanner.scan("sum = 3*foo + 312.50 + bar")) # (['sum', 'op=', 3, 'op*', 'foo', 'op+', 312.5, 'op+', 'bar'], '')
  88. The sre Modules

  89. The sre Modules • sre_constants defines constants • sre_parse defines

    a pattern and how to parse string input into one • sre_compile provides an interface to convert a pattern list to internal format
  90. Go Third-Party

  91. None
  92. asyncore • Wait, does anyone actually still use it? •

    I don’t need to tell you about asyncio • Wait that’s built-in too
  93. getopt & optparse • argparse is fine • click •

    docopt
  94. urllib.request • Use Requests • No good SSL support •

    Don’t use httplib either • urllib.parse is fine
  95. xml.* • Wait didn’t you just say… • Not good

    for parsing • defusedxml • Need to beware of vulnerabilities in any case
  96. venv • Virtualenv: marginally better • pew • pipenv •

    pex
  97. https://gist.github.com/datagrok/2199506

  98. Final Words • There are more! • Read the docs

    as a novel • Read the source
  99. ܅ͷ஌Βͳ͍ 1ZUIPO

  100. LIVE LONG AND PROSPER