Synchronizing Objects to YAML using Black Magic

My PyOhio 2015 talk on YORM.

Video:
https://www.youtube.com/watch?v=0woNEKf-wAo

Source:
https://github.com/jacebrowning/yorm

1. Synchronizing Objects to YAML using Black Magic
3. @jacebrowning
4. • Quick Demo • Project Goals • Chosen Implementation • Advanced Python Features • Library Overview • Use Cases • Getting Started
5. Quick Demo
6. class Student: def __init__(self, name, school, number, year=2009): self.name = name self.school = school self.number = number self.year = year self.gpa = 0.0 $ ls (empty)
7. import yorm from yorm.converters import String, Integer, Float @yorm.attr(name=String, year=Integer, gpa=Float) @yorm.sync("students/{self.school}/{self.number}.yml") class Student: ... $ ls (empty)
8. >>> s1 = Student("John Doe", "GVSU", 123) >>> s2 = Student("Jane Doe", "GVSU", 456, year=2014) >>> s1.gpa = 3 $ ls students $ ls students/GVSU 123.yml 456.yml $ cat students/GVSU/123.yml name: John Doe gpa: 3.0 school: GVSU year: 2009
9. >>> s1.gpa 1.8 >>> s1.expelled True $ echo "name: John Doe > gpa: 1.8 > year: 2010 > expelled: true " > students/GVSU/123.yml
10. Project Goals
11. Configuration vs. Data Program Configuration: • System setup parameters • User options Program Data: • Seeded default values • Persistence model storage • Loaded test fixtures
12. Storage Requirements • Automatic • Human-editable • Support multiple and/or offline editors • Compatible with Python builtin types
13. Alternatives Explored • Database ORMs (Active Record)
 • Object serialization
 • JSON => dictionary
14. Code Requirements • Minimal interference with existing code • Automatic synchronization of changes • Type inference on new attributes • Custom formatting for user classes
15. Chosen Implementation
16. + config: applications: - name: slack versions: linux: null mac: Slack.app windows: null - name: iphoto versions: linux: null mac: iPhoto windows: null computers: - address: AA:BB:CC:DD:EE:FF hostname: Jaces-MacBook name: macbook - address: '' hostname: '' name: macbook-pro
17. Class Decorator or Function @yorm.attr(gpa=yorm.converters.Float) @yorm.sync("students/{self.number}.yml") class Student: ... student = Student() student = Student() path = "students/123.yml" attrs = {'gpa': yorm.converters.Float} student = yorm.sync(student, path, attrs)
18. Optimized Formatting attribute_named_aaa: 1 attribute_named_bbb: - python3.3 - python3.4 attribute_named_zzz: serial_numbers: - abc - 456 attributes dictionary text file store dump write
19. Conversation on Load attribute_named_aaa: 3.1 # converted to 3 attribute_named_bbb: - python3.3 - “python3.4” # unnecessary quotes stripped attribute_named_zzz: serial_numbers: - 4.5 # converted to “4.5” # extra whitespace cleaned up file text dictionary attributes read load fetch
20. Type Inference ... attribute_named_zzz: {} new_attribute: 2015 >>> mapped_instance.new_attribute 2015
21. Advanced Python Features
22. Metaclass class ContainerMeta(abc.ABCMeta): def __init__(cls, name, bases, namespace): super().__init__(name, bases, namespace) cls.yorm_attrs = {}
23. Class Decorator def sync_instances(path_format, format_spec=None, attrs=None, **kwargs): ... def decorator(cls): ... return cls return decorator
24. Monkey Patching: Method old_init = obj.__init__ def new_init(self, *args, **kwargs): old_init(self, *args, **kwargs) ... obj.__init__ = new_init
25. Monkey Patching: Class class Mapped(Mappable, obj.__class__): ... obj.__class__ = Mapped
26. Data Model: Setting Attributes def __setattr__(self, name, value): super().__setattr__(name, value) if name.startswith('__'): return mapper = get_mapper(self) if mapper.auto and name in mapper.attrs: mapper.store()
27. Data Model: Getting Attributes def __getattribute__(self, name): if name.startswith('__'): return object.__getattribute__(self, name) try: value = object.__getattribute__(self, name) except AttributeError as exc: missing = True else: missing = False mapper = get_mapper(self) if missing or (name in mapper.attrs and mapper.modified): mapper.fetch() value = object.__getattribute__(self, name) return value
28. Subclass Lookup class Object(Converter): """Base class for immutable types.""" ... def match(data): converters = Object.__subclasses__() for converter in converters: if type(data) == converter.TYPE: return converter ...
29. Library Overview
30. Class Decorator or Function @yorm.attr(gpa=yorm.converters.Float) @yorm.sync("students/{self.number}.yml") class Student: ... student = Student() student = Student() path = "students/123.yml" attrs = {'gpa': yorm.converters.Float} yorm.sync(student, path, attrs)
31. Class Decorator or Function @yorm.attr(gpa=yorm.converters.Float) @yorm.sync("students/{self.number}.yml", auto=False) class Student: ... student = Student() student = Student() path = "students/123.yml" attrs = {'gpa': yorm.converters.Float} yorm.sync(student, path, attrs, auto=False)
32. Update File from Object yorm.update_file(mapped_instance)
33. Update Object from File yorm.update_object(mapped_instance)
34. …or Both yorm.update(mapped_instance)
35. Builtin Types @attr(my_string=String) @attr(my_integer=Integer) @attr(my_float=Float) @attr(my_boolean=Boolean) @sync("path/to/{self.key}.yml") class Sample: def __init__(self, name): self.name = name self.my_string = “Hello, world!” self.my_integer = 42 self.my_float = 20.15 self.my_bool = True
36. Container Types @attr(all=Integer) class IntegerList(List): pass @attr(label=String, status=Boolean) class StatusDictionary(Dictionary): pass
37. Nested Types config: # a Dictionary computers: # with a List of Dictionaries - address: AA:BB:CC:DD:EE:FF # with String values hostname: Jaces-MacBook name: macbook - address: '' hostname: '' name: macbook-pro
38. Custom Types class Datetime(String): DATETIME_FORMAT = “%B %d, %Y" @classmethod def to_data(cls, obj): dt = cls.to_value(obj) text = dt.strftime(cls.DATETIME_FORMAT) return text @classmethod def to_value(cls, obj): dt = datetime.strptime(obj, cls.DATETIME_FORMAT) return dt
39. Use Cases
40. Loading Configuration @yorm.attr(location=yorm.converters.String) @yorm.attr(sources=Sources) @yorm.sync("{self.root}/{self.filename}") class Config(ShellMixin): FILENAMES = ('gdm.yml', '.gdm.yml') def __init__(self, root, filename=FILENAMES[0], location='gdm_sources'): super().__init__() self.root = root self.filename = filename self.location = location self.sources = [] github.com/jacebrowning/gdm/blob/0a4faeaebde3e6d9464c6e45bd68111423b13acb/gdm/config.py#L112-126
41. Loading Configuration def load(root): config = None for filename in os.listdir(root): if filename.lower() in Config.FILENAMES: config = Config(root, filename) log.debug("loaded config: %s", config.path) break return config github.com/jacebrowning/gdm/blob/0a4faeaebde3e6d9464c6e45bd68111423b13acb/gdm/config.py#L177-185
42. Loading Configuration location: .gdm sources: - repo: github.com/kstenerud/iOS-Universal-Framework dir: framework rev: Mk5-end-of-life - repo: github.com/jonreid/XcodeCoverage dir: coverage rev: master link: Tools/XcodeCoverage
43. Synchronizing State @yorm.attr(applications=StatusList) @yorm.attr(counter=yorm.converters.Integer) class ProgramStatus(yorm.converters.AttributeDictionary): """A dictionary of current program status.""" def __init__(self): super().__init__() self.applications = StatusList() self.counter = 0 github.com/jacebrowning/mine/blob/1947625ee6d44ff98cee6dd209a2ad13d86884d8/mine/status.py#L99-108
44. Synchronizing State status: applications: - application: itunes computers: - computer: laptop timestamp: started: 444 stopped: 402 - computer: desktop timestamp: started: 335 stopped: 390 counter: 499
45. Persisting Data $ ls data/games 6lwctiox.yml 9q6m4swa.yml b7mubcu6.yml cx26xhoc.yml @yorm.attr(players=PlayersFileModel) @yorm.attr(turn=yorm.converters.Integer) @yorm.sync("data/games/{self.key}.yml", auto=False) class GameFileModel(domain.Game): def __init__(self, key, players=None, turn=0): super().__init__() self.key = key self.players = players or PlayersFileModel() self.turn = turn github.com/jacebrowning/gridcommand/blob/75bc49f8e6eb0c9f61bdff81d4d9cb6a2c64b09a/gridcommand/stores/game.py#L57-66
46. Getting Started
47. $ pip3 install yorm
48. jacebrowning/yorm github.com / twitter.com / @jacebrowning