When is an exception not an exception? Using Python's warnings

If your code encounters a big problem, then you probably want to raise an exception. But what should your code do if it finds a small problem, one that shouldn't be ignored, but that doesn't merit an exception? Python's answer to this question is warnings.

In this talk, I'll introduce Python's warnings, close cousins to exceptions but still distinct from them. We'll see how you can generate warnings, and what happens when you do. But then we'll dig deeper, looking at how you can filter and redirect warnings, telling Python which types of warnings you want to see, and which you want to hide. We'll also see how you can get truly fancy, turning some warnings into (potentially fatal) exceptions.

After this talk, you'll be able to take advantage of Python's warning system, letting your users know when something is wrong without having to choose between "print" and a full-blown exception.

Transcript

1. When is an exception not an exception? Warnings in Python Reuven M. Lerner • @reuvenmlerner PyCon 2021 • reuven@lerner.co.il 1
2. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il 2
3. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il 3
4. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il A low-fuel light for our software • Non-fatal when we’re running our program • Annoying and persistent enough to get us to change • “If you don’t change soon, bad things will happen!” 4
5. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Warnings • Introduced in PEP 230 • Back in November of 2000 • First added to Python 2.1 • A main motivation: Warn developers what’s going to change in Python 3, so that they have time to change 5
6. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il You’ve probably seen warnings already >>> from collections import Mapping <stdin>:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working 6
7. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Do we really need warnings? 7
8. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Exceptions: Good • They are a separate communications channel • We can use them to indicate that something unusual has happened (cellphone metaphor) • We can trap them • We can distinguish between them • We can (usually) decide whether or not to ignore them 8
9. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Exceptions: Bad • If you don’t catch an exception, the program ends • No, it’s not a “crash,” but it might as well be • In some other languages, you have to trap for any exceptions that might be raised. • In Python, any exception might be raised at any time. Using exceptions to warn developers would force more error trapping. 9
10. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Why not use “print”? • Not serious or scary enough • Output might get mixed up with regular program output • And yes, you could send it to stderr, but even so… • Cannot be fi ltered or trapped in a standard way 10
11. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il How can we use warnings? 11
12. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Example • I maintain a Python function, “hello”: def hello(name): return f'Hello, {name}!' • I plan to change the function, such that it takes a list of inputs, rather than a single string. 12
13. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Add a warning • Before this change goes into e ff ect, I add a warning to the function, telling them that things will be changing: import warnings def hello(name): warnings.warn( ‘Coming soon: pass a sequence of strings.') return f'Hello, {name}!' You need to import “warnings” Call the “warn” function This message is shown to the user 13
14. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il When I run the program from hello import hello print(hello('world')) $ ./usehello.py hello.py:7: UserWarning: Coming soon: pass a sequence of strings. warnings.warn( Hello, world! 14
15. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il stderr, not stdout • Warning output is sent to stderr • Which means that you’ll still see it, even if you redirect stdout • (Unless you redirect stderr, of course) $ ./usehello.py > hello.txt hello.py:7: UserWarning: Coming soon: pass a sequence of strings. warnings.warn( 15
16. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Plan ahead • Remember that warnings are to warn in advance of a potentially breaking change • You’ll want to give your users some time to transition • So warnings should come several versions before • This requires planning ahead, which isn’t always easy (but will be appreciated by your users) 16
17. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il UserWarning • “UserWarning” is a “category,” similar to an exception class • Semantic power • Automated detection and fi ltering • In fact, warning categories are exception classes! >>> UserWarning.__bases__ (<class 'Warning'>,) >>> Warning.__bases__ (<class 'Exception'>,) 17
18. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Warning categories • Python comes with a number, including: • Warning • UserWarning • DeprecationWarning • SyntaxWarning • RuntimeWarning • PendingDeprecationWarning 18
19. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Specifying a category • Just pass a warning category as the second argument to “warnings.warn”: import warnings def hello(name): warnings.warn( 'Coming soon: pass a sequence of strings.', DeprecationWarning) return f'Hello, {name}!' 19
20. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Why doesn’t it appear? • If you actually run the code on the previous slide, you won’t see any warnings. • That’s because DeprecationWarning is fi ltered out by default — so it won’t appear. • We’ll talk more about fi ltering later. 20
21. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Custom warnings • When raising exceptions, you should use your own, custom classes rather than built-in exception classes. • Similarly, it’s a good idea to use your own custom warning categories. • However, your new warning should probably subclass one of the existing warning types, so that it’ll be fi ltered appropriately. 21
22. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Custom warning import warnings class ArgsChangingWarning(UserWarning): pass def hello(name): warnings.warn( 'Coming soon: pass a sequence of strings.', ArgsChangingWarning) return f'Hello, {name}!' 22
23. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il What happens when we’re warned? • We’ve already seen that a warning will be sent to stderr • But we can actually customize what happens to warnings • Moreover, we can customize what happens to particular categories of warnings 23
24. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Warnings fi lter • Python’s “warnings fi lter” lets us specify what should be done with warnings. • We can specify not only what should be done for a given category of warning, fi lter based on the message contents, the module. 24
25. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il The default fi lter • By default, warnings print the fi rst time they appear in a given fi le, on a given line. • Meaning: • If you encounter the same call to “warnings.warn” multiple times, you’ll only see one message. • But if the same warning appears in multiple places in the code, you’ll see multiple messages, once for each call to “warnings.warn” 25
26. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Example from hello import hello print(hello('world 1')) print(hello('world 2')) 26
27. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Result $ ./usehello2.py hello.py:9: ArgsChangingWarning: Coming soon: pass a sequence of strings. warnings.warn( Hello, world 1! Hello, world 2! We are warned once But the function was called twice 27
28. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Wait a second… $ ./usehello2.py hello.py:9: ArgsChangingWarning: Coming soon: pass a sequence of strings. warnings.warn( Hello, world 1! Hello, world 2! But it’s pretty obvious the warning was generated by “warnings.warn”, right? We got the warning on line 9 of hello.py 28
29. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Setting “stacklevel” • The “stacklevel” parameter in “warnings.warn” is an integer indicating what function should be mentioned in the printed warning message. • By default, stacklevel=1, meaning the call to “warnings.warn” itself. • It’s pretty common to say “stacklevel=2”, so that whoever called “warnings.warn” is shown. 29
30. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il import warnings class ArgsChangingWarning(UserWarning): pass def hello(name): warnings.warn( 'Coming soon: pass a sequence of strings.', ArgsChangingWarning, 2) return f'Hello, {name}!' The fi nal (optional) argument, 2, means that the call to “hello” will be printed in the warning 30
31. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Sure enough… $ ./usehello.py usehello.py:5: ArgsChangingWarning: Coming soon: pass a sequence of strings. print(hello('world')) Hello, world! 31
32. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Warning actions • When Python encounters a warning, it can take one of six actions: • “default” — what we’ve seen, namely print a warning once per combination of module + line number • “error” — raise an exception • “ignore” — ignore the warning • “always” — always print, even if it’s the same warning, from the same line of code • “module” — only print once per module, regardless of line number • “once” — only print once, no matter where the warning was raised 32
33. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Assigning actions to warning categories • Let’s say we want “ArgsChangingWarning” to be displayed every time it occurs, no matter what • “Every time” is the the “always” action. • How do we set it? 33
34. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il -W switch • -Waction • Each action starts with a di ff erent letter, so you can abbreviate! 34
35. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Setting via the command line python3 -Walways ./usehello2.py /Users/reuven/Conferences/PyCon US/2021/warnings/hello.py:9: ArgsChangingWarning: Coming soon: pass a sequence of strings. warnings.warn( Hello, world 1! /Users/reuven/Conferences/PyCon US/2021/warnings/hello.py:9: ArgsChangingWarning: Coming soon: pass a sequence of strings. warnings.warn( Hello, world 2! 35
36. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Speci fi c actions for speci fi c categories • You can fi lter in fi ve di ff erent ways: • Action (we’ve seen this) • Message (regexp match of the case-insensitive message start) • Category • Module (name) • Line number • Pass these, separated by colons, to -W. • You can pass -W multiple times, for multiple fi lters 36
37. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Examples • To give DeprecationWarning (but nothing else) an “always” action, regardless of message: -Waction::DeprecationWarning • To make DeprecationWarnings visible, but only if the message starts with an “a”: -Waction:a:DeprecationWarning • Make UnicodeWarning into an exception: -Werror::UnicodeWarning 37
38. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Default fi lters default::DeprecationWarning:__main__ ignore::DeprecationWarning ignore::PendingDeprecationWarning ignore::ImportWarning ignore::ResourceWarning 38
39. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il PYTHONWARNINGS • Another option: set the “PYTHONWARNINGS” environment variable • Examples: export PYTHONWARNINGS=e::DeprecationWarning export PYTHONWARNINGS=d::ResourceWarning 39
40. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Our warning • These cannot trap for custom warning categories! • Fortunately, we can do it from within Python: • warnings. fi lterwarnings, which lets us specify all fi ve fi lter elements • warnings.simple fi lter, which lets us specify the action, category, and line number • warnings.resetwarnings, which resets these settings 40
41. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Setting fi lters in code import warnings from hello import hello, ArgsChangingWarning warnings.simplefilter('default', ArgsChangingWarning) print(hello('world 1')) print(hello('world 2')) Import our custom category Set default behavior for our custom category 41
42. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il catch_warnings • You can temporarily change the warning fi lters with the “catch_warnings” context manager. • For example, you could turn o ff all warnings: with warnings.catch_warnings(): warnings.simplefilter(‘ignore’) poorly_behaved_function() 42
43. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Where should we use warnings? 43
44. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Warnings in modules • Is a module going away, or making extensive changes? • You can put “warnings.warn” at the top of a module • Remember, DeprecationWarning is ignored by default 44
45. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il Noticing common mistakes • If people commonly call a function with the wrong arguments, use a warning to point them in the right direction. • The “pandas” library has SettingWithCopyWarning • It notices you’re assigning to a copy, and warns that this is a bad idea • scikit-learn used to warn you if you tried to run “predict” on a 1-dimensional list; now it has an exception. • Python 2 would warn you if you used “global” after a variable assignment in a function; now it raises an exception. 45
46. Reuven M. Lerner • PyCon 2021 @reuvenmlerner • https://lerner.co.il 46
47. Reuven M. Lerner • PyCon Israel 2021 @reuvenmlerner • https://lerner.co.il Questions? Comments? • Contact me! • reuven@lerner.co.il • Twitter: @reuvenmlerner • https://lerner.co.il • Join 20k others on my weekly “Better developers” list: • https://BetterDevelopersWeekly.com/ 47