It's time for datetime

Working with time is not a trivial challenge. Python includes a native module in the standard library to work with it but datetime keeps being together with unicode a common source of errors. This often leads to the widespread of many other libraries in the attempt of easing the work of working with datetime. Datetime is one of those API that looks easy to use but given the many concepts around time, is it easy to get backfired if the developer has not solid knowledge about the them.

In this talk we will overview the main concepts about timestamps represented through datetime objects, the limitations on the standard library and some simple steps to try to avoid the common mistakes that everyone can fall into.

Naive datetimes (which the datetime API works by default with) are a great tool to represent calendar times, but when talking about timestamps (focus of this talk) timezones is n essential part of it and the datetime module can be tricky to use for that use cases.

We will also speak about different standards of time, time zones, Daylight Saving Times, leap seconds, serialization and datetime arithmetics.

The talk will be focused on giving the foundations that everyone knows to be able to understand and work efficiently and without making painful mistakes when dealing with time related algorithms.

Video: https://www.youtube.com/watch?v=2BRdKf6WYIQ

Transcript

1. © 2017 Bloomberg Finance L.P. All rights reserved. It’s time for datetime PyCon 2017 May 20, 2017 Mario Corchero Senior Software Developer mcorcherojim@bloomberg.net @mariocj89
2. © 2017 Bloomberg Finance L.P. All rights reserved. Agenda • Background o Time standards o Time-zones o DST • Common issues and how to evade them • Links and references • Questions • Extra secret material
3. © 2017 Bloomberg Finance L.P. All rights reserved. Finding a time standard
4. © 2017 Bloomberg Finance L.P. All rights reserved. Finding a time standard All you need is a reference. A constant recurrent event.
5. © 2017 Bloomberg Finance L.P. All rights reserved. Finding a time standard Finding the constant recurrent event UT1
6. © 2017 Bloomberg Finance L.P. All rights reserved. Finding a time standard Finding the constant recurrent event TAI
7. © 2017 Bloomberg Finance L.P. All rights reserved.
8. © 2017 Bloomberg Finance L.P. All rights reserved. Finding a time standard UTC
9. © 2017 Bloomberg Finance L.P. All rights reserved. 06:00 UTC+05:00 == 11:00 UTC 06:00 UTC-05:00 == 01:00 UTC UTC offsets UTC offsets
10. © 2017 Bloomberg Finance L.P. All rights reserved. Time zones
11. © 2017 Bloomberg Finance L.P. All rights reserved. Daylight Saving Time
12. © 2017 Bloomberg Finance L.P. All rights reserved. Timestamps vs wall time
13. © 2017 Bloomberg Finance L.P. All rights reserved. Common issues
14. © 2017 Bloomberg Finance L.P. All rights reserved. Using local time import datetime as dt x = dt.datetime.now() send_to_mercedes(x)
15. © 2017 Bloomberg Finance L.P. All rights reserved. Using local time Mercedes receives: '2017-04-09T09:48:11.657367' ?????
16. © 2017 Bloomberg Finance L.P. All rights reserved. The issue A time without an offset is not an absolute point in time. >>> import datetime as dt >>> x = dt.datetime.now() >>> send_to_mercedes(x) 2017-04-10T15:52:48.283302 Using local time
17. © 2017 Bloomberg Finance L.P. All rights reserved. Tip Just set the timezone. >>> import datetime as dt >>> x = dt.datetime.now(dt.timezone.utc) >>> send_to_mercedes(x) 2017-04-10T13:56:44.238834+00:00 Using local time
18. © 2017 Bloomberg Finance L.P. All rights reserved. No more issues Using local time All is clear now! 2017-04-10T13:56:44.238834+00:00
19. © 2017 Bloomberg Finance L.P. All rights reserved. Using time zones with DST filename creation date 2017-10-28_at_00_london 3h ago 2017-10-28_at_01_london 1h ago 2017-10-28_at_02_london Minutes ago >>> now = dt.datetime.now(gettz(“Europe/Madrid”)) >>> file_name = f"{now.date().isoformat()}_at_{now.hour}_london” >>> print(file_name) 2017-04-10_at_14
20. © 2017 Bloomberg Finance L.P. All rights reserved. >>> dt.datetime(2017, 1, 1, tzinfo=gettz(“Europe/Madrid”)).isoformat() '2017-01-01T00:00:00+01:00’ >>> dt.datetime(2017, 7, 1, tzinfo=gettz(“Europe/Madrid”)).isoformat() '2017-07-01T00:00:00+02:00' The issue A time zone can have more than one offset. Which makes time not monotonic and ascending for some time zones. DST
21. © 2017 Bloomberg Finance L.P. All rights reserved. Tip If working with timestamps, use UTC+00:00. >>> dt.datetime(2017, 1, 1, tzinfo=dt.timezone.utc).isoformat() '2017-01-01T00:00:00+00:00’ >>> dt.datetime(2017, 7, 1, tzinfo=dt.timezone.utc).isoformat() '2017-07-01T00:00:00+00:00' DST
22. © 2017 Bloomberg Finance L.P. All rights reserved. Tip 2 If you need to return in a different timezone, convert on the borders. >>> dt.datetime(2017, 7, 1, tzinfo=dt.timezone.utc).astimezone(gettz(“Europe/Madrid”)) '2017-07-01T02:00:00+02:00’ >>> dt.datetime(2017, 6, 1, tzinfo=dt.timezone.utc).astimezone(gettz(“Europe/Madrid”)) '2017-01-01T01:00:00+01:00' DST
23. © 2017 Bloomberg Finance L.P. All rights reserved. No more issues DST >>> now = dt.datetime.now(dt.timezone.utc) >>> file_name = f"{now.date().isoformat()}_at_{now.hour}_utc” >>> print(file_name) 2017-04-10_at_14_utc UTC FTW
24. © 2017 Bloomberg Finance L.P. All rights reserved. Changing time zones >>> opening_date = dt.datetime(year, month, day, hour, tzinfo=gettz(“Asia/Istanbul”)) >>> opening_date = opening_date.astimezone(dt.timezone.utc) # 3 months from now
25. © 2017 Bloomberg Finance L.P. All rights reserved. Changing time zones Breaking news! UPS!
26. © 2017 Bloomberg Finance L.P. All rights reserved. The issue You cannot map a future wall time to a timestamp. Changing time zones
27. © 2017 Bloomberg Finance L.P. All rights reserved. Tip 1 Keep your time zone database updated. Changing time zones
28. © 2017 Bloomberg Finance L.P. All rights reserved. Tip 2 If working with future wall times, don’t “resolve them” Changing time zones
29. © 2017 Bloomberg Finance L.P. All rights reserved. No more issues If working with future wall times, don’t “resolve them” until you need them Future wall times are quite tricky! Changing time zones
30. © 2017 Bloomberg Finance L.P. All rights reserved. No more issues (kind of) 9-Feb-2017 Mongolia Scraps Daylight Saving Time 8-Sep-2016 Turkey Scraps Daylight Saving Time 14-Nov-2016 Russia Changes Another Time Zone 4-Jul-2016 Egypt Scraps Return of Daylight Saving Time 5-Jul-2016 Russia Changes Another Time Zone in Siberia 18-Apr-2016 Venezuela Time Changes 12-Mar-2016 No Clock Change in Haiti This Year 17-Mar-2016 Azerbaijan Cancels Daylight Saving Time Changing time zones
31. © 2017 Bloomberg Finance L.P. All rights reserved. Date and time serialization >>> now = dt.datetime.now(dt.timezone.utc) >>> json.dumps(now) TypeError: Object of type 'datetime' is not JSON serializable Let me send this datetime to my JavaScript frontend in JSON
32. © 2017 Bloomberg Finance L.P. All rights reserved. The issue You cannot just send a datetime in JSON. Serialization
33. © 2017 Bloomberg Finance L.P. All rights reserved. Tip Use ISO 8601 Serialization >>> dt.datetime.now(dt.timezone.utc).isoformat() '2017-04-23T13:10:01.098238+00:00'
34. © 2017 Bloomberg Finance L.P. All rights reserved. Tip 2 Use integers for an epoch based time standard. Serialization
35. © 2017 Bloomberg Finance L.P. All rights reserved. Tip 3 Use object and hooks to serialize in JSON. { "__type__": "datetime", "year": 2017, "month": 4, "day": 19, "hour": 22, "minute": 32, "second": 44, "microsecond": 778735, "tz": "UTC” } json.dumps(now, default=dt_to_json) json.loads(input_json, object_hook=json_to_dt) Serialization
36. © 2017 Bloomberg Finance L.P. All rights reserved. No more issues My UI now works with proper times! new Date('2017-04-23T13:10:01.098238+00:00') Sun Apr 23 2017 14:10:01 GMT+0100 (GMT Daylight Time)) * *I hope you can forgive me for writing JS in a Python conference! Serialization
37. © 2017 Bloomberg Finance L.P. All rights reserved. No more issues with Json? RFC 7159: “Numeric values that cannot be represented in the grammar below (such as Infinity and NaN) are not permitted.” json.dumps(float("inf")) ‘Infinite’ Serialization
38. © 2017 Bloomberg Finance L.P. All rights reserved. Time arithmetic
39. © 2017 Bloomberg Finance L.P. All rights reserved. Time arithmetic I set 24 hours, but 25 has passed. What has happened?
40. © 2017 Bloomberg Finance L.P. All rights reserved. The issue Time arithmetic can be ambiguous. >>> today = dt.datetime(2017, 10, 29, tzinfo=gettz("Europe/Madrid")) >>> tomorrow = today + dt.timedelta(days=1) >>> tomorrow.astimezone(dt.timezone.utc) - today.astimezone(dt.timezone.utc) datetime.timedelta(1, 3600) Arithmetic
41. © 2017 Bloomberg Finance L.P. All rights reserved. Tip Stop and think about the real meaning of the operation. This tennis ball has absolutely nothing to do with what I am speaking about J Arithmetic
42. © 2017 Bloomberg Finance L.P. All rights reserved. No more issues I will always use UTC when performing arithmetic with absolute times. Arithmetic
43. © 2017 Bloomberg Finance L.P. All rights reserved. Reviewing the tips • Always use time zones. • Use dateutil/pytz to handle time zones. • Always use UTC when working with timestamps. • Remember that a day is not always made of 24 hours. • Keep your time zone database up to date. • Always test your code against situations such as DST changes.
44. © 2017 Bloomberg Finance L.P. All rights reserved. Libraries worth mentioning • Pytz • dateutil • arrow/pendulum • astropy • freezegun
45. © 2017 Bloomberg Finance L.P. All rights reserved. Links and references • Pluto’s Cave: Allegory of the cave • PEP495: Local Time Disambiguation (fold attribute) • NTP: Network Time Protocol • Google Smeared Time: A time standard with smeared leap seconds • ISO8601: Standard for time representation using strings • Working with datetime blogpost: https://bloom.bg/2qAWBxq • The Twins Paradox: Time is relative (not relevant for Python, though!) • GPS Time: How is time handled within GPS systems
46. © 2017 Bloomberg Finance L.P. All rights reserved. PyConES17
47. © 2017 Bloomberg Finance L.P. All rights reserved. Bloomberg
48. © 2017 Bloomberg Finance L.P. All rights reserved. Questions?
49. © 2017 Bloomberg Finance L.P. All rights reserved. UTC vs. GMT
50. © 2017 Bloomberg Finance L.P. All rights reserved. Pytz vs. dateutil
51. © 2017 Bloomberg Finance L.P. All rights reserved. But wait, what clock is my computer using? NTP
52. © 2017 Bloomberg Finance L.P. All rights reserved. Smeared time TAI Unsmeared UTC Smeared time 2016-12-31 14:00:35.000000 2016-12-31 13:59:59.000000 2016-12-31 13:59:59.000000 2016-12-31 14:00:36.000000 2016-12-31 14:00:00.000000 2016-12-31 14:00:00.000000 2016-12-31 14:00:37.000014 2016-12-31 14:00:01.000014 2016-12-31 14:00:01.000000 2017-01-01 00:00:34.499972 2016-12-31 23:59:58.499972 2016-12-31 23:59:58.000000 2017-01-01 00:00:35.499986 2016-12-31 23:59:59.499986 2016-12-31 23:59:59.000000 2017-01-01 00:00:36.000000 2016-12-31 23:59:60.000000 2016-12-31 23:59:59.500007 2017-01-01 00:00:36.500000 2016-12-31 23:59:60.500000 2017-01-01 00:00:00.000000 2017-01-01 00:00:37.000000 2017-01-01 00:00:00.000000 2017-01-01 00:00:00.499993 2017-01-01 00:00:37.500014 2017-01-01 00:00:00.500014 2017-01-01 00:00:01.000000 2017-01-01 00:00:38.500028 2017-01-01 00:00:01.500028 2017-01-01 00:00:02.000000 2017-01-01 10:00:35.999986 2017-01-01 09:59:58.999986 2017-01-01 09:59:59.000000 2017-01-01 10:00:37.000000 2017-01-01 10:00:00.000000 2017-01-01 10:00:00.000000 2017-01-01 10:00:38.000000 2017-01-01 10:00:01.000000 2017-01-01 10:00:01.000000
53. © 2017 Bloomberg Finance L.P. All rights reserved.