The Python You Don’t Know

“You know Python. You don’t know this Python.”

Python is highly touted as “batteries-included,” referring to the extensive features provided by the standard library, so Python programmers can pick up the tool already around, and focus straight onto the real problem.

Transcript

1. The Python You Don’t Know
2. https://pypi.org
3. PYTHON COMMUNITY
5. BATTERIES INCLUDED
6. WHAT IF I TOLD YOU PYTHON HAS IT BUILT-IN
7. import antigravity https://xkcd.com/353/
8. The Obvious • import this • python -m json.tool • python -m http.server • python -m zipfile • ...
9. Me • Call me TP • Follow @uranusjr • https://uranusjr.com
13. http://macdown.uranusjr.com
14. www. .com
15. Do Things Better
16. Do Things Better • Nothing you couldn’t do • Can be done easier, better
17. pathlib • Path object • Neutral or OS-specific • Path & file operations • open, os, os.path • glob
18. >>> import pathlib >>> base_dir = pathlib.Path(__file__).parent >>> base_dir / 'data' / 'main.cfg' PosixPath('data/main.cfg') >>> conf = base_dir.joinpath('data', 'main.cfg') >>> str(conf) 'data/main.cfg' >>> str(log_dir.resolve()) '/User/uranusjr/my_project/log/project.log'
19. glob >>> for path in base_dir.glob('**/*.py'): ... print(path) ... MacDown/Code/Dependency/build.py Tools/build_for_release.py Tools/import_translations.py
20. OS-independent >>> str(PureWindowsPath('foo', 'bar')) foo\bar >>> str(PurePosixPath('foo', 'bar')) foo/bar
21. https://docs.python.org/3/library/logging.handlers.html
22. enum • Augment common enum pattern • Metaclass magic • Useful decorators
23. >>> class Color: ... RED = 1 ... GREEN = 2 ... BLUE = 3 >>> print(Color.RED) 1 >>> print(type(Color.RED)) <class 'int'>
24. >>> import enum >>> class Color(enum.Enum): ... RED = 1 ... GREEN = 2 ... BLUE = 3 >>> print(Color.RED) Color.RED >>> print(type(Color.RED)) <enum 'Color'>
25. >>> print(Color.RED.value) 1 >>> print(Color.RED.name) RED >>> isinstance(Color.RED, Color) True >>> Color['BLUE'] Color.BLUE >>> Color(1) Color.GREEN
26. No Duplicate Keys >>> class Shape(enum.Enum): ... SQUARE = 'square' ... SQUARE = '█' Traceback (most recent call last): ... TypeError: Attempted to reuse key: 'SQUARE'
27. Value Alias Are Fine >>> class Shape(enum.Enum): ... SQUARE = '█' ... RECTANGLE = '█' >>> Shape.SQUARE == Shape.RECTANGLE True
28. Unless You Say No >>> @enum.unique ... class Shape(enum.Enum): ... SQUARE = '█' ... RECTANGLE = '█' Traceback (most recent call last): ... ValueError: duplicate values found in <enum 'Shape'>: SQUARE -> RECTANGLE
29. More • enum.IntEnum • enum.Flag • enum.IntFlag • Functional API • …
30. Do Things Better • Nothing you couldn’t do • Can be done easier, better • More type-safety?
31. Write Better Code
32. “More Correct” • Program safety • Cross-platform • User friendliness
33. collections.abc • ABC = Abstract Base Classes • Object-oriented interface for feature checking
34. def normalize_data(data): if isinstance(data, list): return data return [data]
35. def normalize_data(data): if isinstance(data, (list, tuple)): return data return [data]
36. def normalize_data(data): if hasattr(data, '__iter__'): return data return [data]
37. def normalize_data(data): if (hasattr(data, '__iter__') and hasattr(data, '__len__')): return data return [data]
38. def normalize_data(data): if (hasattr(data, '__iter__') and hasattr(data, '__len__') and hasattr(data, '__contains__')): return data return [data]
39. def normalize_data(data): if isinstance(data, __________): return data return [data]
40. from collections.abc import Collection def normalize_data(data): if isinstance(data, Collection): return data return [data]
41. ABC • Metaclass magic! • Represents a set of features • Does not require inheritance
42. class Emptiness: def __contains__(self, x): return False def __iter__(self): return iter([]) def __len__(self): return 0 emptiness = Emptiness() isinstance(emptiness, Collection) # True!
43. class Emptiness(Collection): def __contains__(self, x): return False def __iter__(self): return iter([]) def __len__(self): return 0 emptiness = Emptiness() isinstance(emptiness, Collection) # True!
44. class Emptiness(Collection): def __contains__(self, x): return False def __iter__(self): return iter([]) # Forgot to implement __len__ emptiness = Emptiness() # TypeError: Can't instantiate abstract class # Emptiness with abstract methods __len__
45. https://docs.python.org/3/library/collections.abc.html
46. contextlib • You should know @contextmanager • class AbstractContextManager
47. f = open('output.txt') f.write('Hello!') f.close()
48. with open('output.txt') as f: f.write('Hello!')
49. from pymongo import MongoClient c = MongoClient(MONGODB_URL) c['mycol'].find_one({}) c.close()
50. from contextlib import closing from pymongo import MongoClient with closing(MongoClient(MONGODB_URL)) as c: c['mycol'].find_one({})
51. try: os.remove('process.pid') except (FileNotFoundError, PermissionError): pass
52. from contextlib import suppress with suppress(FileNotFoundError, PermissionError): os.remove('process.pid')
53. with connect(server, user, password, 'tempdb') as conn: with conn.cursor(as_dict=True) as cursor: cursor.execute( 'SELECT * FROM persons WHERE salesrep=%s', 'John Doe' ) for row in cursor: print('ID={}, Name={}'.format( row['id'], row['name'], )) http://pymssql.org/en/stable/pymssql_examples.html
55. import contextlib with contextlib.ExitStack() as es: conn = es.enter_context(connect(...)) cursor = es.enter_context(conn.cursor()) do_things(cursor)
56. contextlib • redirect_stdout • redirect_stderr • ContextDecorator
57. http://click.pocoo.org/
58. “More Correct” • Program safety • Cross-platform • User friendliness
59. Write Better Code Functional Special Edition!
60. https://github.com/kennethreitz/pipenv/pull/344/
61. Avoid Lambdas • Limited introspection • No function name (anonymousness!) • No local scope • Difficult to make sense in traceback
62. https://www.artima.com/weblogs/viewpost.jsp?thread=98196
63. Lambda Alternatives • Comprehensions go a long way • If you need a function, just def one • But maybe you don’t!
64. Local Closures def job_callback(loop): print('Finish job in', loop) def run_app(): ... loop.call_later( lambda: job_callback(loop), )
65. import functools def job_callback(loop): print('Finish job in', loop) def run_app(): ... loop.call_later(functools.partial( job_callback, loop, ))
66. Semantically Functional correct_obj_iter = ( obj for obj in object_list if obj.correct )
67. correct_obj_iter = filter( lambda obj: obj.correct, object_list, )
68. import operator correct_obj_iter = filter( operator.attrgetter('correct'), object_list, )
69. result_iter = map( lambda data: data.validate(), data_list, )
70. import operator result_iter = map( operator.methodcaller('validate'), data_list, )
71. # Something like this. pair_list = [ (2, 3), (19, 8), (42, 1), ] summed_iter = map( lambda x, y: x + y, pair_list, )
72. import operator pair_list = [ (2, 3), (19, 8), (42, 1), ] summed_iter = map( operator.add, pair_list, )
73. pair_list = [ (2, 3), (19, 8), (42, 1), ] summed_iter = map( sum, pair_list, )
74. import math pair_list = [ (2, 3), (19, 8), (42, 1), ] summed_iter = map( math.sum, pair_list, )
75. list_of_lists = [ [1, 2, 3], [4, 5, 6], [7, 8, 9], ] joined_list = functools.reduce( lambda s, x: s + x, list_of_lists, initializer=list, )
76. list_of_lists = [ [1, 2, 3], [4, 5, 6], [7, 8, 9], ] joined_list = functools.reduce( operator.add, list_of_lists, initializer=list, )
77. list_of_lists = [ [1, 2, 3], [4, 5, 6], [7, 8, 9], ] joined_list = list( itertool.chain.from_iterable( list_of_list, ), )
78. Lambda Alternatives • functools.partial • operator for builtin operators • itertools to handle iterables
79. Useful When Needed
80. getpass • getpass.getpass() • “input() without echo” • Not really (but better)
81. difflib • Generates diff — diff-ing • Restores input — patching • Utilities for content comparison
82. textwrap • Break text into lines • Indent, dedent, tab expansion • Simple hyphenation & trimming
83. xml.* • “Mom, I don’t need lxml!” • Generate and parse XML • Different parsers (e.g. expat)
84. re • Wait, you should know this • Unless you don’t really!
85. http://lucumr.pocoo.org/2015/11/18/pythons-hidden-re-gems/
86. https://github.com/python/cpython/blob/master/Lib/re.py
87. def s_ident(scanner, token): return token def s_operator(scanner, token): return "op%s" % token def s_float(scanner, token): return float(token) def s_int(scanner, token): return int(token) scanner = re.Scanner([ (r"[a-zA-Z_]\w*", s_ident), (r"\d+\.\d*", s_float), (r"\d+", s_int), (r"=|\+|-|\*|/", s_operator), (r"\s+", None), ]) print(scanner.scan("sum = 3*foo + 312.50 + bar")) # (['sum', 'op=', 3, 'op*', 'foo', 'op+', 312.5, 'op+', 'bar'], '')
88. The sre Modules
89. The sre Modules • sre_constants defines constants • sre_parse defines a pattern and how to parse string input into one • sre_compile provides an interface to convert a pattern list to internal format
90. Go Third-Party
92. asyncore • Wait, does anyone actually still use it? • I don’t need to tell you about asyncio • Wait that’s built-in too
93. getopt & optparse • argparse is fine • click • docopt
94. urllib.request • Use Requests • No good SSL support • Don’t use httplib either • urllib.parse is fine
95. xml.* • Wait didn’t you just say… • Not good for parsing • defusedxml • Need to beware of vulnerabilities in any case
96. venv • Virtualenv: marginally better • pew • pipenv • pex
97. https://gist.github.com/datagrok/2199506
98. Final Words • There are more! • Read the docs as a novel • Read the source
99. ܅ͷ஌Βͳ͍ 1ZUIPO
100. LIVE LONG AND PROSPER