Making with Python 3

Invited keynote. PyCon Taiwan 2013. Taipei.

Transcript

1. Making with Python 3
   David Beazley (@dabeaz)
   PyCon Taiwan 2013
   

   View Slide

2. Hello! 你好
   Thanks for inviting me!
   

   View Slide

3. Question 1:
   Do you like Python programming?
   >>> print('Hello World')
   Hello World!
   >>>
   

   View Slide

4. Question 2:
   Do you like making things?
   

   View Slide

5. Question 2:
   Do you like making things?
   Of course you do!
   

   View Slide

6. Making Things is Fun!
   It's how I got into Python programming
   It's about solving day-to-day problems
   

   View Slide

7. Supercomputing
   Scriptable Scientific Software
   beazley@cs.uchicago.edu
   Sample Session
   Python
   

   View Slide

8. Data hacking
   Toys
   

   View Slide

9. Data hacking example
   Find worst area
   of Chicago to
   ride my bike
   Based on number
   of open potholes
   in street
   

   View Slide

10. Science
    

    View Slide

11. Plotting (in wood)
    

    View Slide

12. Actually, Toys
    

    View Slide

13. Actually, Toys
    

    View Slide

14. Premise: IT Should Be Fun!
    Programming that is
    

    View Slide

15. Question: Is Python 3 Fun?
    Does it have the same "feel" as Python 2?
    Can you solve problems with it?
    Can you make things with it?
    Is it something that you want to use?
    

    View Slide

16. Some History
    Python 3000 discussed for a long time
    Python 3 first release: December, 2008
    Expectation: Might take >5 years for users
    

    View Slide

17. Brief Summary
    Python 3 "fixes" the design flaws in Python 2
    But breaks backwards compatibility
    

    View Slide

18. Poll
    Are you using Python 3?
    

    View Slide

19. My Interest
    I like Python (language and community)
    Want to see it improve
    However, I'm only a user (it's a tool)
    I am not a core developer
    

    View Slide

20. Using Python 3
    Have been using Python 3 for various
    personal projects since about 2010
    Trying to use it for all new projects
    However, mostly just playing around
    

    View Slide

21. Book Writing
    (2009) (2013)
    

    View Slide

22. Python Cookbook
    Just finished Python Cookbook, 3rd Ed.
    A deep dive into Python 3.3
    No regard for backwards compatibility
    Be as modern as possible
    

    View Slide

23. Thinking about Python 3
    Python 3 looks almost the same
    However, it's quite different in many ways
    I like it a lot
    But, not for reasons you might think
    

    View Slide

24. Python 2 < "Three"
    

    View Slide

25. Please Explain
    >>> 2 < "Three"
    True
    >>>
    

    View Slide

26. Please Explain
    >>> x = 10
    >>> y = 11
    >>> avgxy = (x + y) / 2
    >>> avgxy
    10
    >>>
    Or this...
    

    View Slide

27. Please Explain
    >>> s = "Hello"
    >>> t = u"World"
    >>> s + t
    u'HelloWorld'
    >>>
    And this...
    

    View Slide

28. Please Explain
    >>> name = "ACME"
    >>> price = 123.45
    >>> print name, price
    ACME 123.45
    >>> print name + ',' + str(price)
    ACME,123.45
    >>>
    Ugh...
    

    View Slide

29. Please Explain
    >>> name = "ACME"
    >>> price = 123.45
    >>> print name, price
    ACME 123.45
    >>> print '%s,%0.2f'%(name, price)
    ACME,123.45
    >>>
    Ugh...
    

    View Slide

30. Please Explain
    >>> name = "ACME"
    >>> price = 123.45
    >>> print name, price
    ACME 123.45
    >>> print ','.join([name,str(price)])
    ACME,123.45
    >>>
    Ugh...
    

    View Slide

31. Please Explain
    s = ["N/A", "12", "34"]
    try:
    v = int(s[n])
    except ValueError, IndexError:
    v = 0.0
    Finally...
    

    View Slide

32. Only a Small Sample
    Python 2 has many "features" like this
    Has grown organically
    Not always clean or consistent
    

    View Slide

33. Language Evolution
    Good ideas often take time to mature
    Sometimes in unexpected ways
    

    View Slide

34. Example: Iteration
    Custom iterators == awesome
    >>> c = countdown(5)
    >>> for x in c:
    ... print x
    ...
    5
    4
    3
    2
    1
    >>>
    

    View Slide

35. Example: Iteration
    Early days: hacking __getitem__()
    class countdown:
    def __init__(self, start):
    self.n = start
    def __getitem__(self, index):
    if index >= self.n:
    raise IndexError()
    return self.n - index
    

    View Slide

36. Example: Iteration
    Iteration protocol
    class countdown:
    def __init__(self, start):
    self.n = start
    def __iter__(self):
    return self
    def next(self):
    if self.n <= 0:
    raise StopIteration()
    n = self.n
    self.n -= 1
    return n
    

    View Slide

37. Example: Iteration
    Generators
    def countdown(n):
    while n > 0:
    yield n
    n -= 1
    It's now something quite elegant
    

    View Slide

38. Getting Walled In
    Sometimes a good idea can only go so far
    Limited by existing language features
    Constrained by backwards compatibility
    

    View Slide

39. The Story of Sets
    print ' , '.join([name, str(shares), str(price)])
    

    View Slide

40. Before Sets
    Dictionaries with dummy values
    a = {
    'Guido' : None,
    'Barry' : None,
    'Tim' : None,
    'Paula' : None
    }
    You write clumsy code that manipulates keys
    

    View Slide

41. Before Sets
    def in_common(a, b):
    return [ key for key in a
    if key in b ]
    

    View Slide

42. After Sets
    A lot of code gets simplified
    a = set(['Guido', 'Barry',
    'Tim', 'Paula'])
    b = set(['Dave', 'Paula',
    'Thomas', 'Lewis'])
    all = a | b
    in_common = a & b
    a_not_b = a - b
    

    View Slide

43. Sets and Dicts
    There is a close relationship
    Sets : Unordered collection of items
    Dicts : Unordered set of keys mapped to values
    Underlying implementation almost identical
    

    View Slide

44. Sets and Dicts
    Yet, sets and dicts don't play together
    a = {
    'Guido': 1,
    'Barry': 2,
    'Tim': 3,
    'Paula': 4
    }
    b = {
    'Dave': 5,
    'Paula': 6,
    'Thomas': 7,
    'Lewis': 8
    }
    Find keys in common
    >>> a.keys() & b.keys()
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: unsupported operand type(s) for &: 'list'
    and 'list'
    >>>
    

    View Slide

45. Sets and Dicts
    Yet, sets and dicts don't play together
    a = {
    'Guido': 1,
    'Barry': 2,
    'Tim': 3,
    'Paula': 4
    }
    b = {
    'Dave': 5,
    'Paula': 6,
    'Thomas': 7,
    'Lewis': 8
    }
    Find keys in common (workaround)
    >>> set(a.keys()) & set(b.keys())
    set(['Paula'])
    >>>
    

    View Slide

46. Sets and Dicts
    It all just "works" in Python 3 though
    >>> a.keys() & b.keys()
    {'Paula'}
    >>>
    Hey, cool!
    

    View Slide

47. Python 3 In a Nutshell
    It's a more polished Python
    All of the parts just fit together better
    Think of it as a more finely tuned machine
    Made possible by breaking backwards compat
    

    View Slide

48. Python 3 Examples
    Comparison
    >>> 2 < "Three"
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: unorderable types: int() <
    str()
    >>>
    

    View Slide

49. Python 3 Examples
    Division
    >>> x = 10
    >>> y = 11
    >>> avgxy = (x + y) / 2
    >>> avgxy
    10.5
    >>>
    

    View Slide

50. Python 3 Examples
    Printing
    >>> name = "ACME"
    >>> price = 123.45
    >>> print(name, price)
    ACME 123.45
    >>> print(name, price, sep=',')
    ACME,123.45
    >>>
    

    View Slide

51. Cool Stuff
    Unpacking
    >>> values = ('Dave', 37, 'Chicago')
    >>> name, *rest = values
    >>> name
    'Dave'
    >>> rest
    [37, 'Chicago']
    >>>
    

    View Slide

52. Cool Stuff
    Keyword-only arguments
    def recv(maxbytes, *, block=True):
    ...
    recv(1024, block=False) # OK
    recv(1024, False) # ERROR
    

    View Slide

53. Cool Stuff
    Generator subroutines
    def countdown(n):
    while n > 0:
    yield n
    n -= 1
    def spam():
    yield from countdown(5)
    

    View Slide

54. Cool Stuff
    Generator subroutines are basis for this...
    

    View Slide

55. Commentary
    In Python 3, I find myself using fewer "hacks"
    and workarounds (basically none)
    Again, the parts just fit together a bit better
    It's nice
    

    View Slide

56. More Control
    

    View Slide

57. Features for Frameworks
    There are a lot of power features
    Mostly aimed at more "serious" development
    Libraries/Frameworks
    

    View Slide

58. Exception Chains
    try:
    raise ValueError("Must be > 0")
    except Exception as e:
    raise RuntimeError("Failed") from e
    

    View Slide

59. Exception Chains
    Traceback (most recent call last):
    File "", line 2, in
    ValueError: Must be > 0
    The above exception was the direct cause
    of the following exception:
    Traceback (most recent call last):
    File "", line 4, in
    RuntimeError: Failed
    

    View Slide

60. Metaclasses
    from abc import ABCMeta, abstractmethod
    class Stream(metaclass=ABCMeta):
    @abstractmethod
    def recv(self, maxsize):
    pass
    @abstractmethod
    def send(self, data):
    pass
    

    View Slide

61. Metaclasses
    from enum import Enum
    class Colors(Enum):
    red = 1
    blue = 2
    green = 3
    yellow = 4
    magenta = 5
    Coming : Python 3.4 (PEP 435)
    Uses some serious metaclass magic
    

    View Slide

62. Import Hooks
    Look at sys.meta_path sometime...
    Can take over the import statement
    Can create custom importers
    Diabolical!
    

    View Slide

63. C Interoperability
    Standardized interface for exposing memory
    regions and arrays (memoryviews)
    Gets array processing libraries (e.g., numpy),
    I/O libraries, and related tools to play nice
    

    View Slide

64. Commentary
    Building big frameworks isn't about "fun"
    But, correctness matters
    Python 3 helps a lot (fewer "hacks")
    A different talk though...
    

    View Slide

65. Annoyances
    

    View Slide

66. Unicode/Bytes
    It's a huge headache (still)
    Rules of thumb:
    All text input must be decoded
    All text output must be encoded
    Python 3 forces you to be precise about it
    There are semantic differences
    

    View Slide

67. Strings vs. Bytes
    >>> s = "Hello"
    >>> s[0]
    'H'
    >>> s[0:4]
    'Hell'
    >>>
    Major headaches for people porting libraries
    >>> s = b"Hello"
    >>> s[0]
    72
    >>> s[0:4]
    b'Hell'
    >>>
    

    View Slide

68. Serialization Woes
    >>> s = "Hello"
    >>> json.dumps(s)
    '"Hello"'
    >>> pickle.dumps(s)
    b'\x80\x03X\x05\x00\x00\x00Helloq\x00.
    >>>
    Not entirely consistent across libraries
    Text
    Bytes
    Frankly, a mess
    

    View Slide

69. Unicode Insanity
    >>> f = open('jalape\xf1o.txt', 'w')
    >>> g = open(b'jalape\xf1o.txt', 'w')
    >>> import os
    >>> os.listdir('.')
    ['jalape%F1o.txt', 'jalapeño.txt']
    >>> os.listdir(b'.')
    [b'jalape%F1o.txt', b'jalapen\xcc\x83o.txt
    >>>
    Really crazy things in system calls
    

    View Slide

70. Error Messages
    >>> data = b"Hello World"
    >>> 'Hell' in data
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: Type str doesn't support
    the buffer API
    >>>
    WhAt?!?
    

    View Slide

71. New Style Formatting
    print("{0} has {1} messages".format('Da
    Always results in lines too wide in editor
    Kind of silly complaint (I know)
    

    View Slide

72. What to Make of Python 3?
    Overall, a really nice language
    Better appreciated by doing new projects
    Especially if you're unchained from past
    Porting still a potential problem (sigh)
    

    View Slide

73. But is it Fun?
    On the whole, I think so
    More so if not porting old code
    Kind of fun to play with advanced features
    

    View Slide

74. Recent Fun
    Using Python 3 to control a CNC mill
    Stepper
    Motors (3)
    Arduino w/
    Grblshield
    Laptop
    (USB)
    Rotary
    cutter
    

    View Slide

75. Software
    import serial
    ser = serial.Serial(
    '/dev/tty.usbmodem641', 9600)
    def command(cmd):
    ser.send(cmd.encode('ascii')+b'\n')
    resp = ser.readline()
    if resp != b'Ok\n':
    raise RuntimeError(resp)
    It's just serial ports... use pyserial
    Simple command/response protocol
    

    View Slide

76. GCode
    G1 Z10
    G1 X0 Y0
    G1 Z-2
    G1 X50 Y10
    G1 X20 Y40
    G1 X0 Y0
    G1 Z0
    Movement controlled by simple commands
    It's a lot like plotting graphics
    (0,0)
    (50,10)
    (20,40)
    

    View Slide

77. Whirling Knives
    You're in the physical world
    Plotting with 25000 RPM end mill
    

    View Slide

78. Toy Making (for kids)
    

    View Slide

79. A Thought
    Much of the joy in making things is in using
    precision tools that work
    This is the real reason to try Python 3
    Also: An opportunity
    

    View Slide

80. That is All!
    Thanks for inviting me!
    Hope I've inspired you to look at Python 3
    Twitter : @dabeaz
    Questions?
    

    View Slide