Python 3 Migration

How we at CMB migrated a codebase of more than a million lines of code successfully from Python 2 to Python 3

Transcript

1. PYTHON 3 MIGRATION ANN PAUL ann@coffeemeetsbagel.com 1
2. 2 LOC 1 MILLION FILES 7000 LIBRARIES 70
3. WHY UPGRADE ? 3
4. 4 TYPE HINTS F STRINGS ASYNCIO UNICODE IMPROVEMENTS EXCEPTION CHAINING 4
5. TIMELINE 5 5
6. 6 MONTH 1 MONTH 2 2 WEEKS 3RD PARTY LIBRARIES _________________ 70 50 CODE COMPATIBILITY _________________ six, future, 2to3 TEST DEPLOY MONITOR _________________ iterate! 6
7. CHALLENGES 7 7
8. 1. INCOMPATIBLE PICKLED OBJECTS 8 8
9. Python 2.7.13 (default, Dec 19 2016, 14:18:49) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin >>> import pickle >>> set_cache('cached_pickle', pickle.dumps(obj, fix_imports=True)) 9 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1 9 Python 3.6.4 (default, Dec 19 2016, 14:20:49) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin >>> import pickle >>> obj = pickle.loads(get_cache('cached_pickle'))
10. 2. INCONSISTENT RESULTS FROM hash() 10 10
11. Python 3.6.4 (default, Dec 19 2016, 14:18:49) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin >>> hash('secret') -4264008114590981298 11 11 Python 3.6.4 (default, Dec 19 2016, 14:20:49) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin >>> hash('secret') -4139676931567953040 # Different hash
12. Python 3.6.4 (default, Dec 19 2016, 14:18:49) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin >>> import hashlib >>> int(hashlib.md5(b'secret').hexdigest(), 16) 252103622270029474919056516158423780772 12 12 Python 3.6.4 (default, Dec 19 2016, 14:20:49) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin >>> import hashlib >>> int(hashlib.md5(b'secret').hexdigest(), 16) 252103622270029474919056516158423780772
13. BONUS! 13 13
14. AUTO DECODING 14 Django 1.9 • HttpResponse.json() redis-py 2.6 • decode_responses 14
15. CONCLUSION 15 15
16. ROLLOUT TEST, MONITOR AND SLOWLY ROLLOUT REMOVE UPGRADE AND TRIM 3RD PARTY LIBRARIES REFACTOR MAKE CODEBASE PYTHON 2 AND 3 COMPATIBLE ❖ http://python-future.org/compatible_idioms.html ❖ https://docs.python.org/3/howto/pyporting.html 16
17. THANKS ! ANN PAUL ann@coffeemeetsbagel.com 17 17