Tweet
Tweet

Slide 1

Slide 1 text

The Python You Don’t Know

Slide 2

Slide 2 text

https://pypi.org

Slide 3

Slide 3 text

PYTHON COMMUNITY

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

BATTERIES INCLUDED

Slide 6

Slide 6 text

WHAT IF I TOLD YOU PYTHON HAS IT BUILT-IN

Slide 7

Slide 7 text

import antigravity https://xkcd.com/353/

Slide 8

Slide 8 text

The Obvious • import this • python -m json.tool • python -m http.server • python -m zipfile • ...

Slide 9

Slide 9 text

Me • Call me TP • Follow @uranusjr • https://uranusjr.com

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

http://macdown.uranusjr.com

Slide 14

Slide 14 text

www. .com

Slide 15

Slide 15 text

Do Things Better

Slide 16

Slide 16 text

Do Things Better • Nothing you couldn’t do • Can be done easier, better

Slide 17

Slide 17 text

pathlib • Path object • Neutral or OS-specific • Path & file operations • open, os, os.path • glob

Slide 18

Slide 18 text

>>> import pathlib >>> base_dir = pathlib.Path(__file__).parent >>> base_dir / 'data' / 'main.cfg' PosixPath('data/main.cfg') >>> conf = base_dir.joinpath('data', 'main.cfg') >>> str(conf) 'data/main.cfg' >>> str(log_dir.resolve()) '/User/uranusjr/my_project/log/project.log'

Slide 19

Slide 19 text

glob >>> for path in base_dir.glob('**/*.py'): ... print(path) ... MacDown/Code/Dependency/build.py Tools/build_for_release.py Tools/import_translations.py

Slide 20

Slide 20 text

OS-independent >>> str(PureWindowsPath('foo', 'bar')) foo\bar >>> str(PurePosixPath('foo', 'bar')) foo/bar

Slide 21

Slide 21 text

https://docs.python.org/3/library/logging.handlers.html

Slide 22

Slide 22 text

enum • Augment common enum pattern • Metaclass magic • Useful decorators

Slide 23

Slide 23 text

>>> class Color: ... RED = 1 ... GREEN = 2 ... BLUE = 3 >>> print(Color.RED) 1 >>> print(type(Color.RED))

Slide 24

Slide 24 text

>>> import enum >>> class Color(enum.Enum): ... RED = 1 ... GREEN = 2 ... BLUE = 3 >>> print(Color.RED) Color.RED >>> print(type(Color.RED))

Slide 25

Slide 25 text

>>> print(Color.RED.value) 1 >>> print(Color.RED.name) RED >>> isinstance(Color.RED, Color) True >>> Color['BLUE'] Color.BLUE >>> Color(1) Color.GREEN

Slide 26

Slide 26 text

No Duplicate Keys >>> class Shape(enum.Enum): ... SQUARE = 'square' ... SQUARE = '█' Traceback (most recent call last): ... TypeError: Attempted to reuse key: 'SQUARE'

Slide 27

Slide 27 text

Value Alias Are Fine >>> class Shape(enum.Enum): ... SQUARE = '█' ... RECTANGLE = '█' >>> Shape.SQUARE == Shape.RECTANGLE True

Slide 28

Slide 28 text

Unless You Say No >>> @enum.unique ... class Shape(enum.Enum): ... SQUARE = '█' ... RECTANGLE = '█' Traceback (most recent call last): ... ValueError: duplicate values found in : SQUARE -> RECTANGLE

Slide 29

Slide 29 text

More • enum.IntEnum • enum.Flag • enum.IntFlag • Functional API • …

Slide 30

Slide 30 text

Do Things Better • Nothing you couldn’t do • Can be done easier, better • More type-safety?

Slide 31

Slide 31 text

Write Better Code

Slide 32

Slide 32 text

“More Correct” • Program safety • Cross-platform • User friendliness

Slide 33

Slide 33 text

collections.abc • ABC = Abstract Base Classes • Object-oriented interface for feature checking

Slide 34

Slide 34 text

def normalize_data(data): if isinstance(data, list): return data return [data]

Slide 35

Slide 35 text

def normalize_data(data): if isinstance(data, (list, tuple)): return data return [data]

Slide 36

Slide 36 text

def normalize_data(data): if hasattr(data, '__iter__'): return data return [data]

Slide 37

Slide 37 text

def normalize_data(data): if (hasattr(data, '__iter__') and hasattr(data, '__len__')): return data return [data]

Slide 38

Slide 38 text

def normalize_data(data): if (hasattr(data, '__iter__') and hasattr(data, '__len__') and hasattr(data, '__contains__')): return data return [data]

Slide 39

Slide 39 text

def normalize_data(data): if isinstance(data, __________): return data return [data]

Slide 40

Slide 40 text

from collections.abc import Collection def normalize_data(data): if isinstance(data, Collection): return data return [data]

Slide 41

Slide 41 text

ABC • Metaclass magic! • Represents a set of features • Does not require inheritance

Slide 42

Slide 42 text

class Emptiness: def __contains__(self, x): return False def __iter__(self): return iter([]) def __len__(self): return 0 emptiness = Emptiness() isinstance(emptiness, Collection) # True!

Slide 43

Slide 43 text

class Emptiness(Collection): def __contains__(self, x): return False def __iter__(self): return iter([]) def __len__(self): return 0 emptiness = Emptiness() isinstance(emptiness, Collection) # True!

Slide 44

Slide 44 text

class Emptiness(Collection): def __contains__(self, x): return False def __iter__(self): return iter([]) # Forgot to implement __len__ emptiness = Emptiness() # TypeError: Can't instantiate abstract class # Emptiness with abstract methods __len__

Slide 45

Slide 45 text

https://docs.python.org/3/library/collections.abc.html

Slide 46

Slide 46 text

contextlib • You should know @contextmanager • class AbstractContextManager

Slide 47

Slide 47 text

f = open('output.txt') f.write('Hello!') f.close()

Slide 48

Slide 48 text

with open('output.txt') as f: f.write('Hello!')

Slide 49

Slide 49 text

from pymongo import MongoClient c = MongoClient(MONGODB_URL) c['mycol'].find_one({}) c.close()

Slide 50

Slide 50 text

from contextlib import closing from pymongo import MongoClient with closing(MongoClient(MONGODB_URL)) as c: c['mycol'].find_one({})

Slide 51

Slide 51 text

try: os.remove('process.pid') except (FileNotFoundError, PermissionError): pass

Slide 52

Slide 52 text

from contextlib import suppress with suppress(FileNotFoundError, PermissionError): os.remove('process.pid')

Slide 53

Slide 53 text

with connect(server, user, password, 'tempdb') as conn: with conn.cursor(as_dict=True) as cursor: cursor.execute( 'SELECT * FROM persons WHERE salesrep=%s', 'John Doe' ) for row in cursor: print('ID={}, Name={}'.format( row['id'], row['name'], )) http://pymssql.org/en/stable/pymssql_examples.html

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

import contextlib with contextlib.ExitStack() as es: conn = es.enter_context(connect(...)) cursor = es.enter_context(conn.cursor()) do_things(cursor)

Slide 56

Slide 56 text

contextlib • redirect_stdout • redirect_stderr • ContextDecorator

Slide 57

Slide 57 text

http://click.pocoo.org/

Slide 58

Slide 58 text

“More Correct” • Program safety • Cross-platform • User friendliness

Slide 59

Slide 59 text

Write Better Code Functional Special Edition!

Slide 60

Slide 60 text

https://github.com/kennethreitz/pipenv/pull/344/

Slide 61

Slide 61 text

Avoid Lambdas • Limited introspection • No function name (anonymousness!) • No local scope • Difficult to make sense in traceback

Slide 62

Slide 62 text

https://www.artima.com/weblogs/viewpost.jsp?thread=98196

Slide 63

Slide 63 text

Lambda Alternatives • Comprehensions go a long way • If you need a function, just def one • But maybe you don’t!

Slide 64

Slide 64 text

Local Closures def job_callback(loop): print('Finish job in', loop) def run_app(): ... loop.call_later( lambda: job_callback(loop), )

Slide 65

Slide 65 text

import functools def job_callback(loop): print('Finish job in', loop) def run_app(): ... loop.call_later(functools.partial( job_callback, loop, ))

Slide 66

Slide 66 text

Semantically Functional correct_obj_iter = ( obj for obj in object_list if obj.correct )

Slide 67

Slide 67 text

correct_obj_iter = filter( lambda obj: obj.correct, object_list, )

Slide 68

Slide 68 text

import operator correct_obj_iter = filter( operator.attrgetter('correct'), object_list, )

Slide 69

Slide 69 text

result_iter = map( lambda data: data.validate(), data_list, )

Slide 70

Slide 70 text

import operator result_iter = map( operator.methodcaller('validate'), data_list, )

Slide 71

Slide 71 text

# Something like this. pair_list = [ (2, 3), (19, 8), (42, 1), ] summed_iter = map( lambda x, y: x + y, pair_list, )

Slide 72

Slide 72 text

import operator pair_list = [ (2, 3), (19, 8), (42, 1), ] summed_iter = map( operator.add, pair_list, )

Slide 73

Slide 73 text

pair_list = [ (2, 3), (19, 8), (42, 1), ] summed_iter = map( sum, pair_list, )

Slide 74

Slide 74 text

import math pair_list = [ (2, 3), (19, 8), (42, 1), ] summed_iter = map( math.sum, pair_list, )

Slide 75

Slide 75 text

list_of_lists = [ [1, 2, 3], [4, 5, 6], [7, 8, 9], ] joined_list = functools.reduce( lambda s, x: s + x, list_of_lists, initializer=list, )

Slide 76

Slide 76 text

list_of_lists = [ [1, 2, 3], [4, 5, 6], [7, 8, 9], ] joined_list = functools.reduce( operator.add, list_of_lists, initializer=list, )

Slide 77

Slide 77 text

list_of_lists = [ [1, 2, 3], [4, 5, 6], [7, 8, 9], ] joined_list = list( itertool.chain.from_iterable( list_of_list, ), )

Slide 78

Slide 78 text

Lambda Alternatives • functools.partial • operator for builtin operators • itertools to handle iterables

Slide 79

Slide 79 text

Useful When Needed

Slide 80

Slide 80 text

getpass • getpass.getpass() • “input() without echo” • Not really (but better)

Slide 81

Slide 81 text

difflib • Generates diff — diff-ing • Restores input — patching • Utilities for content comparison

Slide 82

Slide 82 text

textwrap • Break text into lines • Indent, dedent, tab expansion • Simple hyphenation & trimming

Slide 83

Slide 83 text

xml.* • “Mom, I don’t need lxml!” • Generate and parse XML • Different parsers (e.g. expat)

Slide 84

Slide 84 text

re • Wait, you should know this • Unless you don’t really!

Slide 85

Slide 85 text

http://lucumr.pocoo.org/2015/11/18/pythons-hidden-re-gems/

Slide 86

Slide 86 text

https://github.com/python/cpython/blob/master/Lib/re.py

Slide 87

Slide 87 text

def s_ident(scanner, token): return token def s_operator(scanner, token): return "op%s" % token def s_float(scanner, token): return float(token) def s_int(scanner, token): return int(token) scanner = re.Scanner([ (r"[a-zA-Z_]\w*", s_ident), (r"\d+\.\d*", s_float), (r"\d+", s_int), (r"=|\+|-|\*|/", s_operator), (r"\s+", None), ]) print(scanner.scan("sum = 3*foo + 312.50 + bar")) # (['sum', 'op=', 3, 'op*', 'foo', 'op+', 312.5, 'op+', 'bar'], '')

Slide 88

Slide 88 text

The sre Modules

Slide 89

Slide 89 text

The sre Modules • sre_constants defines constants • sre_parse defines a pattern and how to parse string input into one • sre_compile provides an interface to convert a pattern list to internal format

Slide 90

Slide 90 text

Go Third-Party

Slide 91

Slide 91 text

No content

Slide 92

Slide 92 text

asyncore • Wait, does anyone actually still use it? • I don’t need to tell you about asyncio • Wait that’s built-in too

Slide 93

Slide 93 text

getopt & optparse • argparse is fine • click • docopt

Slide 94

Slide 94 text

urllib.request • Use Requests • No good SSL support • Don’t use httplib either • urllib.parse is fine

Slide 95

Slide 95 text

xml.* • Wait didn’t you just say… • Not good for parsing • defusedxml • Need to beware of vulnerabilities in any case

Slide 96

Slide 96 text

venv • Virtualenv: marginally better • pew • pipenv • pex

Slide 97

Slide 97 text

https://gist.github.com/datagrok/2199506

Slide 98

Slide 98 text

Final Words • There are more! • Read the docs as a novel • Read the source

Slide 99

Slide 99 text

܅ͷ஌Βͳ͍ 1ZUIPO

Slide 100

Slide 100 text

LIVE LONG AND PROSPER