Alex Martelli - Exception and error handling in Python 2 and Python 3

Handling errors and exceptions optimally is crucial in solid Python programs. Some technical details have changed in Python 3, and the talk covers those, but the core of "best practices" is quite enduring, and the talk focuses on presenting and explaining them.

https://us.pycon.org/2016/schedule/presentation/2093/

Transcript

1. ©2016 Google -- aleax@google.com
   
   Exceptions & error handling
   in Python 2 and Python 3
   http://www.aleax.it/pycon16.pdf
   

   View Slide

2. Python in a Nutshell 3rd ed
   This talk cover parts of Chapter 5 of the
   
   Early Release e-book version
   
   50% off: http:/
   /www.oreilly.com/go/python45
   
   disc.code TS2016; 40% on paper book pre-order
   2
   DRM-free e-book
   
   epub, mobi, PDF…
   
   copy it to all of
   
   your devices!
   Send us feedback
   
   while we can still do
   
   major changes!
   
   

   View Slide

3. Exceptions: not always errors
   >>> 1/0
   Traceback (most recent call last):
   File "", line 1, in
   ZeroDivisionError: division by zero
   >>> next(iter([]))
   Traceback (most recent call last):
   File "", line 1, in
   StopIteration
   >>>
   Throughout, I use exc to mean exception
   3
   

   View Slide

4. The try statement
   try: ...block...
   
   [except (type, type...) [as x]:] ...
   
   (0+ except clauses, narrowest first)
   
   [else:] ...
   
   (optional: execute without exception guard
   
   iff no exc in block; must have 1+ except)
   
   [finally:] ...
   
   (optional: execute unconditionally at end;
   
   no break, return [continue forbidden])
   4
   

   View Slide

5. The raise statement
   raise exception_object
   must be an instance of BaseException
   
   (in v2, could be a subclass -- avoid that!)
   
   raise
   must be in an except clause (or a function
   
   called, directly or not, from one)
   
   re-raises the exception being handled
   5
   

   View Slide

6. When to raise and why
   def cross_product(seq1, seq2):
   if not seq1 or not seq2:
   raise ValueError('empty seq arg')
   return [(x1, x2) for x1 in seq1
   for x2 in seq2]
   Note: no duplicate checks of errors that
   Python itself checks anyway, e.g seq1 or
   seq2 not being iterable (that will presumably
   give a TypeError, which is probably fine).
   6
   

   View Slide

7. Exceptions wrapping (v3)
   v3 only (upgrade to v3, already!-)
   
   traceback is held by the exception object
   
   exc.with_traceback(tb) gives a copy
   of exc with a different traceback
   
   last exc caught is __context__ of new one
   
   raise new_one from x sets __cause__
   to x, which is None or exception instance
   7
   

   View Slide

8. >>> def inverse(x):
   ... try: return 1/x
   ... except ZeroDivisionError as err:
   ... raise ValueError() from err
   >>> inverse(0)
   Traceback (most recent call last):
   File "", line 2, in inverse
   ZeroDivisionError: division by zero
   The above exception was the direct cause of the
   following exception:
   Traceback (most recent call last):
   File "", line 4, in inverse
   ValueError
   >>> try: print('inverse is', inverse(0))
   ... except ValueError: print('no inverse there')
   no inverse there
   8
   

   View Slide

9. exc __context__ in v3
   try: 1/0
   except ZeroDivisionError:
   1+'x'
   Traceback (most recent call last):
   File "", line 1, in
   ZeroDivisionError: division by zero
   During handling of the above exception,
   another exception occurred:
   Traceback (most recent call last):
   File "", line 3, in
   TypeError: unsupported operand type(s) for +:
   'int' and 'str'
   9
   v2 would hide 1st exc; v3 shows both!
   

   View Slide

10. the with statement
    with x [as y]: ...block...
    
    ...is roughly the same as...:
    
    y = x.__enter__()
    _ok = True
    try: ...block...
    except:
    _ok = False
    if not x.__exit__(*sys.exc_info()):
    raise
    finally:
    if _ok: x.__exit__(None, None, None)
    10
    

    View Slide

11. Making a context manager
    @contextlib.contextmanager
    def a_context_manager(args_if_any):
    # __init__ code if any
    try:
    # __enter__ code
    yield some_result
    except (handled, types) as exc:
    # __exit__ w/exception
    # maybe `raise` again if needed
    finally:
    # unconditional __exit__ code
    11
    

    View Slide

12. Exceptions in generators
    caller may use generator.throw(exc)
    
    like raise exc at generator's yield
    
    in v2 may pass type, value, traceback
    
    typically used as...:
    
    try: result = yield previous
    except GeneratorExit: clean-up
    
    caller may use generator.close()
    
    which is just like:
    generator.throw(GeneratorExit())
    12
    

    View Slide

13. exception propagation
    exceptions "bubble up" through the stack of
    call frames (callee to caller to ...)
    
    until caught in an except clause
    
    or __exit__ in a context manager
    
    (as finally clauses or context managers
    are "bubbled through", they execute)
    
    if never caught, all the way to the top
    
    where sys.excepthook may act
    
    ...but essentially just reporting/logging
    
    lastly, atexit-registered functions
    13
    

    View Slide

14. exceptions hierarchy (v2)
    BaseException
    
    Exception
    
    StandardError
    
    ...many types/subtypes...
    
    EnvironmentError
    
    IOError, OSError
    
    StopIteration
    
    Warning
    
    GeneratorExit
    
    KeyboardInterrupt
    
    SystemExit
    14
    

    View Slide

15. exceptions hierarchy (v3)
    BaseException
    
    Exception
    
    ...many types/subtypes...
    
    OSError (AKA: IOError, EnvironmentError)
    
    ...subtypes, e.g FileNotFoundError...
    
    StopIteration
    
    Warning
    
    GeneratorExit
    
    KeyboardInterrupt
    
    SystemExit
    15
    

    View Slide

16. OSError subclasses (v3)
    def read_or_def(path, default):
    try:
    with open(path) as f: return f.read()
    except IOError as e:
    if e.errno == errno.ENOENT:
    return default
    raise
    def read_or_def(path, default):
    try:
    with open(path) as f:return f.read()
    except FileNotFoundError:
    return default
    16
    

    View Slide

17. custom exceptions
    best practice: have your module or package
    define a custom exc class:
    
    class Error(Exception): "docstring"
    
    this lets client code easily catch errors
    specific to your module or package
    
    also use multiple inheritance to define your
    module's versions of standard exceptions:
    
    class MyTE(Error, TypeError): "doc"
    
    this also lets client code easily catch (e.g)
    TypeErrors wherever they come from
    17
    

    View Slide

18. Strategies: LBYL, EAFP
    Look Before You Leap:
    
    check that all preconditions are met
    
    if not, raise exc (or otherwise give error)
    
    if met, perform op (no exc expected)
    
    Easier to Ask Forgiveness than Permission
    
    just try performing the operation
    
    Python catches any prereq violations
    
    and raises exc on your behalf
    
    optionally catch to transform/enrich
    18
    

    View Slide

19. LBYL problems
    duplicates work Python performs anyway to
    check preconditions
    
    obscures code clarity due to structure:
    
    check, raise if it fails
    
    ...(repeat N times)…
    
    actual useful work (only at the end)
    
    some checks might erroneously be omitted
    
    resulting in unexpected exceptions
    
    things (e.g filesystem) may change at any
    time (inc. between checks and operation!)
    19
    

    View Slide

20. LBYL vs EAFP
    20
    def read_or_default(path, default):
    # spot the many problems...:
    if os.path.exists(path):
    with open(path) as f: return f.read()
    else:
    return default
    def read_or_default(path, default):
    try:
    with open(path) as f: return f.read()
    except FileNotFoundError:
    return default
    

    View Slide

21. how to EAFP right
    def trycall(obj, attr, default, *a, **k):
    try: return getattr(obj, attr)(*a, **k)
    except AttributeError: return default
    def trycall(obj, attr, default, *a, **k):
    try: method = getattr(obj, attr)
    except AttributeError: return default
    else: return method(*a, **k)
    21
    Keep it narrow: DON'T guard too many operations
    within a try clause!
    

    View Slide

22. Errors in large programs
    consider all possible causes of errors...:
    
    bugs in your code, or libraries you use
    
    cover those with unit tests
    
    mismatches between libraries' prereqs and
    your understanding of them
    
    cover those with integration tests
    
    invalid inputs to your code
    
    great use for try/except!
    
    remember: let Python do most checks!
    
    invalid environment/config: ditto
    22
    

    View Slide

23. Case-by-case handling
    the info you (the coder) need (to fix bugs
    etc) is NOT the same the user needs (to
    remedy invalid inputs/environment/etc)
    
    think of the user: everything else follows
    
    never show the user a traceback
    
    they can't do anything with it!
    
    archive it, send to yourself (w/perm!), ...
    
    design user error messages with care
    
    focus on what they can/should do NOW!
    
    if feasible, restart the program (maybe with
    snapshot/restore; ideally with a "watchdog")
    23
    

    View Slide

24. logging
    Python stdlib's logging package can be very
    rich and complex if used to the fullest
    
    worth your effort! logs is how you debug
    (and optimize, etc) esp. server programs
    
    ensure your logs are machine-parsable,
    design log-parsing scripts carefully
    
    when in doubt, default to logging all info
    
    program state, environment, inputs, ...
    
    don't log just errors: logging.info can give
    you precious "base-case" comparisons too
    24
    

    View Slide

25. avoid assert
    although assert seems an attractive way to
    check inputs and environment...
    
    ...it's NOT: it's an "attractive nuisance"!
    
    becomes no-op when you run optimized
    
    but inputs &c can be wrong even then!
    
    often duplicates checks Python performs
    
    it's usually a sub-case of LBYL...
    
    use ONLY for sanity checks on internal state
    (and as "executable docs") while you're
    developing and debugging your program!
    25
    

    View Slide

26. ? !
    Q & A
    http://www.aleax.it/pycon16.pdf
    26
    http:/
    /www.oreilly.com/go/python45
    discount code: TS2016
    

    View Slide