Slide 1

Slide 1 text

Refactoring Python: Why and how to restructure your code Brett Slatkin @haxor onebigfluke.com 2016-05-30T11:30-07:00

Slide 2

Slide 2 text

● What, When, Why, How ● Strategies ○ Extract Variable & Function ○ Extract Class & Move Fields ○ Move Field gotchas ● Follow-up ● Bonus ○ Extract Closure Agenda

Slide 3

Slide 3 text

Repeatedly reorganizing and rewriting code until it's obvious* to a new reader. What is refactoring? * See Clean Code by Robert Martin

Slide 4

Slide 4 text

● In advance ● For testing ● "Don't repeat yourself" ● Brittleness ● Complexity When do you refactor?

Slide 5

Slide 5 text

What's the difference between good and great programmers? (anecdotally) Me usually Great Time spent 0 100% Good Writing & testing Refactoring Style & docs

Slide 6

Slide 6 text

1. Identify bad code 2. Improve it 3. Run tests 4. Fix and improve tests 5. Repeat How do you refactor?

Slide 7

Slide 7 text

How do you refactor in practice? ● Rename, split, move ● Simplify ● Redraw boundaries

Slide 8

Slide 8 text

The canonical reference (1999)

Slide 9

Slide 9 text

But...

Slide 10

Slide 10 text

But... it's for Java programmers

Slide 11

Slide 11 text

The more recent version (2009)

Slide 12

Slide 12 text

But...

Slide 13

Slide 13 text

But... it's for Ruby programmers

Slide 14

Slide 14 text

How do you refactor Python? Image © Hans Hillewaert Creative Commons Attribution-Share Alike 4.0 International

Slide 15

Slide 15 text

Strategies

Slide 16

Slide 16 text

● Thorough tests ● Quick tests ● Source control ● Willing to make mistakes Prerequisites

Slide 17

Slide 17 text

Extract Variable & Extract Function

Slide 18

Slide 18 text

MONTHS = ('January', 'February', ...) def what_to_eat(month): if (month.lower().endswith('r') or month.lower().endswith('ary')): print('%s: oysters' % month) elif 8 > MONTHS.index(month) > 4: print('%s: tomatoes' % month) else: print('%s: asparagus' % month) When should you eat certain foods?

Slide 19

Slide 19 text

>>> what_to_eat('November') what_to_eat('July') what_to_eat('March') When should you eat certain foods? November: oysters July: tomatoes March: asparagus

Slide 20

Slide 20 text

Before if (month.lower().endswith('r') or month.lower().endswith('ary')): print('%s: oysters' % month) elif 8 > MONTHS.index(month) > 4: print('%s: tomatoes' % month) else: print('%s: asparagus' % month)

Slide 21

Slide 21 text

lowered = month.lower() ends_in_r = lowered.endswith('r') ends_in_ary = lowered.endswith('ary') index = MONTHS.index(month) summer = 8 > index > 4 if ends_in_r or ends_in_ary: print('%s: oysters' % month) elif summer: print('%s: tomatoes' % month) else: print('%s: asparagus' % month) After: Extract variables

Slide 22

Slide 22 text

def oysters_good(month): lowered = month.lower() return ( lowered.endswith('r') or lowered.endswith('ary')) def tomatoes_good(month): index = MONTHS.index(month) return 8 > index > 4 Extract variables into functions

Slide 23

Slide 23 text

Before if (month.lower().endswith('r') or month.lower().endswith('ary')): print('%s: oysters' % month) elif 8 > MONTHS.index(month) > 4: print('%s: tomatoes' % month) else: print('%s: asparagus' % month)

Slide 24

Slide 24 text

After: Using functions if oysters_good(month): print('%s: oysters' % month) elif tomatoes_good(month): print('%s: tomatoes' % month) else: print('%s: asparagus' % month)

Slide 25

Slide 25 text

After: Using functions with variables time_for_oysters = oysters_good(month) time_for_tomatoes = tomatoes_good(month) if time_for_oysters: print('%s: oysters' % month) elif time_for_tomatoes: print('%s: tomatoes' % month) else: print('%s: asparagus' % month)

Slide 26

Slide 26 text

def oysters_good(month): lowered = month.lower() return ( lowered.endswith('r') or lowered.endswith('ary')) def tomatoes_good(month): index = MONTHS.index(month) return 8 > index > 4 These functions will get complicated

Slide 27

Slide 27 text

class OystersGood: def __init__(self, month): lowered = month.lower() self.r = lowered.endswith('r') self.ary = lowered.endswith('ary') self._result = self.r or self.ary def __bool__(self): # aka __nonzero__ return self._result Extract variables into classes

Slide 28

Slide 28 text

class TomatoesGood: def __init__(self, month): self.index = MONTHS.index(month) self._result = 8 > index > 4 def __bool__(self): # aka __nonzero__ return self._result Extract variables into classes

Slide 29

Slide 29 text

time_for_oysters = oysters_good(month) time_for_tomatoes = tomatoes_good(month) if time_for_oysters: print('%s: oysters' % month) elif time_for_tomatoes: print('%s: tomatoes' % month) else: print('%s: asparagus' % month) Before: Using functions

Slide 30

Slide 30 text

After: Using classes time_for_oysters = OystersGood(month) time_for_tomatoes = TomatoesGood(month) if time_for_oysters: # Calls __bool__ print('%s: oysters' % month) elif time_for_tomatoes: # Calls __bool__ print('%s: tomatoes' % month) else: print('%s: asparagus' % month)

Slide 31

Slide 31 text

test = OystersGood('November') assert test assert test.r assert not test.ary test = OystersGood('July') assert not test assert not test.r assert not test.ary Extracting classes facilitates testing

Slide 32

Slide 32 text

Things to remember ● Extract variables and functions to improve readability ● Extract variables into classes to improve testability ● Use __bool__ to indicate a class is a paper trail

Slide 33

Slide 33 text

Extract Class & Move Fields

Slide 34

Slide 34 text

Keeping track of your pets class Pet: def __init__(self, name): self.name = name

Slide 35

Slide 35 text

>>> pet = Pet('Gregory the Gila') print(pet.name) Keeping track of your pets Gregory the Gila

Slide 36

Slide 36 text

Keeping track of your pet's age class Pet: def __init__(self, name, age): self.name = name self.age = age

Slide 37

Slide 37 text

>>> pet = Pet('Gregory the Gila', 3) print('%s is %d years old' % (pet.name, pet.age)) Keeping track of your pet's age Gregory the Gila is 3 years old

Slide 38

Slide 38 text

class Pet: def __init__(self, name, age): self.name = name self.age = age self.treats_eaten = 0 def give_treats(self, count): self.treats_eaten += count Keeping track of your pet's treats

Slide 39

Slide 39 text

>>> pet = Pet('Gregory the Gila', 3) pet.give_treats(2) print('%s ate %d treats' % (pet.name, pet.treats_eaten)) Keeping track of your pet's treats Gregory the Gila ate 2 treats

Slide 40

Slide 40 text

class Pet: def __init__(self, name, age, *, has_scales=False, lays_eggs=False, drinks_milk=False): self.name = name self.age = age self.treats_eaten = 0 self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk Keeping track of your pet's needs

Slide 41

Slide 41 text

class Pet: def __init__(self, ...): ... def give_treats(self, count): .. @property def needs_heat_lamp(self): return ( self.has_scales and self.lays_eggs and not self.drinks_milk) Keeping track of your pet's needs

Slide 42

Slide 42 text

>>> pet = Pet('Gregory the Gila', 3, has_scales=True, lays_eggs=True) print('%s needs a heat lamp? %s' % (pet.name, pet.needs_heat_lamp)) Keeping track of your pet's needs Gregory the Gila needs a heat lamp? True

Slide 43

Slide 43 text

class Pet: def __init__(self, name, age, *, has_scales=False, lays_eggs=False, drinks_milk=False): self.name = name self.age = age self.treats_eaten = 0 self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk It's getting complicated

Slide 44

Slide 44 text

1. Add an improved interface ○ Maintain backwards compatibility ○ Issue warnings for old usage 2. Migrate old usage to new usage ○ Run tests to verify correctness ○ Fix and improve broken tests 3. Remove code for old interface How do you redraw boundaries?

Slide 45

Slide 45 text

import warnings warnings.warn('Helpful message') ● Default: Print messages to stderr ● Force warnings to become exceptions: python -W error your_code.py What are warnings?

Slide 46

Slide 46 text

Before class Pet: def __init__(self, name, age, *, has_scales=False, lays_eggs=False, drinks_milk=False): self.name = name self.age = age self.treats_eaten = 0 self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk

Slide 47

Slide 47 text

After: Extract Animal from Pet class Animal: def __init__(self, *, has_scales=False, lays_eggs=False, drinks_milk=False): self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk

Slide 48

Slide 48 text

Before class Pet: def __init__(self, name, age, *, has_scales=False, lays_eggs=False, drinks_milk=False): ...

Slide 49

Slide 49 text

After: Add / intro parameter object class Pet: def __init__(self, name, age, animal=None, **kwargs): ...

Slide 50

Slide 50 text

class Pet: def __init__(self, name, age, animal=None, **kwargs): if kwargs and animal is not None: raise TypeError('Mixed usage') if animal is None: warnings.warn('Should use Animal') animal = Animal(**kwargs) self.animal = animal self.name = name self.age = age self.treats_eaten = 0 After: Backwards compatible

Slide 51

Slide 51 text

>>> Mixed usage raises exception animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', 3, animal, has_scales=False) Traceback ... TypeError: Mixed usage

Slide 52

Slide 52 text

>>> pet = Pet('Gregory the Gila', 3, has_scales=True, lays_eggs=True) Old constructor works, but warns UserWarning: Should use Animal

Slide 53

Slide 53 text

>>> My pet is Gregory the Gila New constructor usage doesn't warn animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', 3, animal) print('My pet is %s' % pet.name)

Slide 54

Slide 54 text

class Pet: def __init__(self, name, age, *, has_scales=False, lays_eggs=False, drinks_milk=False): ... self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk Before: Fields on self

Slide 55

Slide 55 text

class Pet: ... @property def has_scales(self): warnings.warn('Use animal attribute') return self.animal.has_scales @property def lays_eggs(self): ... @property def drinks_milk(self): ... After: Move fields to inner object

Slide 56

Slide 56 text

>>> Old attributes issue a warning UserWarning: Use animal attribute Gregory the Gila has scales? True animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', 3, animal) print('%s has scales? %s' % (pet.name, pet.has_scales))

Slide 57

Slide 57 text

>>> New attributes don't warn Gregory the Gila has scales? True animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', 3, animal) print('%s has scales? %s' % (pet.name, pet.animal.has_scales))

Slide 58

Slide 58 text

class Pet: def __init__(self, ...): ... def give_treats(self, count): .. @property def needs_heat_lamp(self): return ( self.has_scales and self.lays_eggs and not self.drinks_milk) Before: Helpers access self

Slide 59

Slide 59 text

class Pet: def __init__(self, ...): ... def give_treats(self, count): .. @property def needs_heat_lamp(self): return ( self.animal.has_scales and self.animal.lays_eggs and not self.animal.drinks_milk) After: Helpers access inner object

Slide 60

Slide 60 text

>>> Existing helper usage doesn't warn Gregory the Gila needs a heat lamp? True animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', 3, animal) print('%s needs a heat lamp? %s' % (pet.name, pet.needs_heat_lamp))

Slide 61

Slide 61 text

● Split classes using optional arguments to __init__ ● Use @property to move methods and fields between classes ● Issue warnings in old code paths to find their occurrences Things to remember

Slide 62

Slide 62 text

Move Field gotchas

Slide 63

Slide 63 text

class Animal: def __init__(self, *, has_scales=False, lays_eggs=False, drinks_milk=False): ... class Pet: def __init__(self, name, age, animal): ... Before: Is this obvious?

Slide 64

Slide 64 text

class Animal: def __init__(self, age=None, *, has_scales=False, lays_eggs=False, drinks_milk=False): ... class Pet: def __init__(self, name, animal): ... After: Move age to Animal

Slide 65

Slide 65 text

class Animal: def __init__(self, age=None, *, has_scales=False, lays_eggs=False, drinks_milk=False): if age is None: warnings.warn('age not specified') self.age = age self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk After: Constructor with optional age

Slide 66

Slide 66 text

class Pet: def __init__(self, name, age, animal): ... Before: Pet constructor with age

Slide 67

Slide 67 text

After: Pet constructor with optional age class Pet: def __init__(self, name, maybe_age, maybe_animal=None): ...

Slide 68

Slide 68 text

class Pet: def __init__(self, name, maybe_age, maybe_animal=None): if maybe_animal is not None: warnings.warn('Put age on animal') self.animal = maybe_animal self.animal.age = maybe_age else: self.animal = maybe_age ... After: Pet constructor with optional age

Slide 69

Slide 69 text

class Pet: def __init__(self, name, maybe_age, maybe_animal=None): ... def give_treats(self, count): ... @property def age(self): warnings.warn('Use animal.age') return self.animal.age After: Compatibility property age

Slide 70

Slide 70 text

>>> animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', 3, animal) print('%s is %d years old' % (pet.name, pet.age)) After: Old usage has a lot of warnings UserWarning: age not specified UserWarning: Put age on animal UserWarning: Use animal.age Gregory the Gila is 3 years old

Slide 71

Slide 71 text

>>> After: New usage has no warnings Gregory the Gila is 3 years old animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) print('%s is %d years old' % (pet.name, pet.animal.age))

Slide 72

Slide 72 text

animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 Gregory is older than I thought

Slide 73

Slide 73 text

>>> Assigning to age breaks! AttributeError: can't set attribute animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 # Error

Slide 74

Slide 74 text

class Pet: ... @property def age(self): warnings.warn('Use animal.age') return self.animal.age @age.setter def age(self, new_age): warnings.warn('Assign animal.age') self.animal.age = new_age Need a compatibility property setter

Slide 75

Slide 75 text

>>> animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 Old assignment now issues a warning UserWarning: Assign animal.age

Slide 76

Slide 76 text

>>> New assignment doesn't warn Gregory the Gila is 5 years old animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.animal.age = 5 print('%s is %d years old' % (pet.name, pet.animal.age))

Slide 77

Slide 77 text

class Animal: def __init__(self, age, *, has_scales=False, lays_eggs=False, drinks_milk=False): self.age = age self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk ... Finally: age is part of Animal

Slide 78

Slide 78 text

class Pet: def __init__(self, name, animal): self.animal = animal self.name = name self.treats_eaten = 0 ... Finally: Pet has no concept of age

Slide 79

Slide 79 text

animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 print('%s is %d years old' % (pet.name, pet.animal.age)) Again: Gregory is older than I thought

Slide 80

Slide 80 text

>>> Surprise! Old usage is doubly broken Gregory the Gila is 3 years old animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 print('%s is %d years old' % (pet.name, pet.animal.age))

Slide 81

Slide 81 text

>>> Surprise! Old usage is doubly broken Gregory the Gila is 3 years old animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 # No error! print('%s is %d years old' % (pet.name, pet.animal.age))

Slide 82

Slide 82 text

class Pet: ... @property def age(self): raise AttributeError('Use animal') @age.setter def age(self, new_age): raise AttributeError('Use animal') Need defensive property tombstones

Slide 83

Slide 83 text

>>> Now accidental old usage will break Traceback ... AttributeError: Use animal animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 # Error

Slide 84

Slide 84 text

Things to remember ● Use @property.setter to move fields that can be assigned ● Defend against muscle memory with tombstone @propertys

Slide 85

Slide 85 text

Follow-up

Slide 86

Slide 86 text

Links ● PMOTW: Warnings - Doug Hellmann ● Stop Writing Classes - Jack Diederich ● Beyond PEP 8 - Raymond Hettinger

Slide 87

Slide 87 text

● This talk's code & slides: ○ github.com/bslatkin/pycon2016 ● My book: EffectivePython.com ○ Discount today: informit.com/deals ● Me: @haxor and onebigfluke.com Links

Slide 88

Slide 88 text

Appendix

Slide 89

Slide 89 text

Bonus: Extract Closure

Slide 90

Slide 90 text

class Grade: def __init__(self, student, score): self.student = student self.score = score grades = [ Grade('Jim', 92), Grade('Jen', 89), Grade('Ali', 73), Grade('Bob', 96), ] Calculating stats for students

Slide 91

Slide 91 text

def print_stats(grades): total, count, lo, hi = 0, 0, 100, 0 for grade in grades: total += grade.score count += 1 if grade.score < lo: lo = grade.score elif grade.score > hi: hi = grade.score print('Avg: %f, Lo: %f Hi: %f' % (total / count, lo, hi)) Calculating stats for students

Slide 92

Slide 92 text

>>> Calculating stats for students print_stats(grades) Avg: 87.5, Lo: 73.0, Hi: 96.0

Slide 93

Slide 93 text

Before def print_stats(grades): total, count, lo, hi = 0, 0, 100, 0 for grade in grades: total += grade.score count += 1 if grade.score < lo: lo = grade.score elif grade.score > hi: hi = grade.score print('Avg: %f, Lo: %f Hi: %f' % (total / count, lo, hi))

Slide 94

Slide 94 text

After: Extract a stateful closure def print_stats(grades): total, count, lo, hi = 0, 0, 100, 0 def adjust_stats(grade): # Closure ... for grade in grades: adjust_stats(grade) print('Avg: %f, Lo: %f Hi: %f' % (total / count, lo, hi))

Slide 95

Slide 95 text

Stateful closure functions are messy def print_stats(grades): total, count, lo, hi = 0, 0, 100, 0 def adjust_stats(grade): nonlocal total, count, lo, hi total += grade.score count += 1 if grade.score < lo: lo = grade.score elif grade.score > hi: hi = grade.score ...

Slide 96

Slide 96 text

class CalculateStats: def __init__(self): self.total = 0 self.count = 0 self.lo = 100 self.hi = 0 def __call__(self, grade): ... @property def avg(self): ... Instead: Stateful closure class

Slide 97

Slide 97 text

class CalculateStats: def __init__(self): ... def __call__(self, grade): self.total += grade.score self.count += 1 if grade.score < self.lo: self.lo = grade.score elif grade.score > self.hi: self.hi = grade.score Instead: Stateful closure class

Slide 98

Slide 98 text

class CalculateStats: def __init__(self): ... def __call__(self, grade): ... @property def avg(self): return self.total / self.count Instead: Stateful closure class

Slide 99

Slide 99 text

def print_stats(grades): total, count, lo, hi = 0, 0, 100, 0 for grade in grades: total += grade.score count += 1 if grade.score < lo: lo = grade.score elif grade.score > hi: hi = grade.score print('Avg: %f, Lo: %f Hi: %f' % (total / count, lo, hi)) Before

Slide 100

Slide 100 text

Before: Closure function def print_stats(grades): total, count, lo, hi = 0, 0, 100, 0 def adjust_stats(grade): # Closure ... for grade in grades: adjust_stats(grade) print('Avg: %f, Lo: %f Hi: %f' % (total / count, lo, hi))

Slide 101

Slide 101 text

def print_stats(grades): stats = CalculateStats() for grade in grades: stats(grade) print('Avg: %f, Lo: %f Hi: %f' % (stats.avg, stats.lo, stats.hi)) After: Using stateful closure class

Slide 102

Slide 102 text

● Extracting a closure function can make code less clear ● Use __call__ to indicate that a class is just a stateful closure ● Closure classes can be tested independently Things to remember