Synchronizing Objects to YAML using Black Magic

No content

@jacebrowning

• Quick Demo • Project Goals • Chosen Implementation • Advanced Python Features • Library Overview • Use Cases • Getting Started

Quick Demo

class Student: def __init__(self, name, school, number, year=2009): self.name = name self.school = school self.number = number self.year = year self.gpa = 0.0 $ ls (empty)

import yorm from yorm.converters import String, Integer, Float @yorm.attr(name=String, year=Integer, gpa=Float) @yorm.sync("students/{self.school}/{self.number}.yml") class Student: ... $ ls (empty)

>>> s1 = Student("John Doe", "GVSU", 123) >>> s2 = Student("Jane Doe", "GVSU", 456, year=2014) >>> s1.gpa = 3 $ ls students $ ls students/GVSU 123.yml 456.yml $ cat students/GVSU/123.yml name: John Doe gpa: 3.0 school: GVSU year: 2009

>>> s1.gpa 1.8 >>> s1.expelled True $ echo "name: John Doe > gpa: 1.8 > year: 2010 > expelled: true " > students/GVSU/123.yml

Project Goals

Configuration vs. Data Program Configuration: • System setup parameters • User options Program Data: • Seeded default values • Persistence model storage • Loaded test fixtures

Storage Requirements • Automatic • Human-editable • Support multiple and/or offline editors • Compatible with Python builtin types

Alternatives Explored • Database ORMs (Active Record)
 • Object serialization
 • JSON => dictionary

Code Requirements • Minimal interference with existing code • Automatic synchronization of changes • Type inference on new attributes • Custom formatting for user classes

Chosen Implementation

+ config: applications: - name: slack versions: linux: null mac: Slack.app windows: null - name: iphoto versions: linux: null mac: iPhoto windows: null computers: - address: AA:BB:CC:DD:EE:FF hostname: Jaces-MacBook name: macbook - address: '' hostname: '' name: macbook-pro

Class Decorator or Function @yorm.attr(gpa=yorm.converters.Float) @yorm.sync("students/{self.number}.yml") class Student: ... student = Student() student = Student() path = "students/123.yml" attrs = {'gpa': yorm.converters.Float} student = yorm.sync(student, path, attrs)

Optimized Formatting attribute_named_aaa: 1 attribute_named_bbb: - python3.3 - python3.4 attribute_named_zzz: serial_numbers: - abc - 456 attributes dictionary text file store dump write

Conversation on Load attribute_named_aaa: 3.1 # converted to 3 attribute_named_bbb: - python3.3 - “python3.4” # unnecessary quotes stripped attribute_named_zzz: serial_numbers: - 4.5 # converted to “4.5” # extra whitespace cleaned up file text dictionary attributes read load fetch

Type Inference ... attribute_named_zzz: {} new_attribute: 2015 >>> mapped_instance.new_attribute 2015

Advanced Python Features

Metaclass class ContainerMeta(abc.ABCMeta): def __init__(cls, name, bases, namespace): super().__init__(name, bases, namespace) cls.yorm_attrs = {}

Class Decorator def sync_instances(path_format, format_spec=None, attrs=None, **kwargs): ... def decorator(cls): ... return cls return decorator

Monkey Patching: Method old_init = obj.__init__ def new_init(self, *args, **kwargs): old_init(self, *args, **kwargs) ... obj.__init__ = new_init

Monkey Patching: Class class Mapped(Mappable, obj.__class__): ... obj.__class__ = Mapped

Data Model: Setting Attributes def __setattr__(self, name, value): super().__setattr__(name, value) if name.startswith('__'): return mapper = get_mapper(self) if mapper.auto and name in mapper.attrs: mapper.store()

Data Model: Getting Attributes def __getattribute__(self, name): if name.startswith('__'): return object.__getattribute__(self, name) try: value = object.__getattribute__(self, name) except AttributeError as exc: missing = True else: missing = False mapper = get_mapper(self) if missing or (name in mapper.attrs and mapper.modified): mapper.fetch() value = object.__getattribute__(self, name) return value

Subclass Lookup class Object(Converter): """Base class for immutable types.""" ... def match(data): converters = Object.__subclasses__() for converter in converters: if type(data) == converter.TYPE: return converter ...

Library Overview

Class Decorator or Function @yorm.attr(gpa=yorm.converters.Float) @yorm.sync("students/{self.number}.yml") class Student: ... student = Student() student = Student() path = "students/123.yml" attrs = {'gpa': yorm.converters.Float} yorm.sync(student, path, attrs)

Class Decorator or Function @yorm.attr(gpa=yorm.converters.Float) @yorm.sync("students/{self.number}.yml", auto=False) class Student: ... student = Student() student = Student() path = "students/123.yml" attrs = {'gpa': yorm.converters.Float} yorm.sync(student, path, attrs, auto=False)

Update File from Object yorm.update_file(mapped_instance)

Update Object from File yorm.update_object(mapped_instance)

…or Both yorm.update(mapped_instance)

Builtin Types @attr(my_string=String) @attr(my_integer=Integer) @attr(my_float=Float) @attr(my_boolean=Boolean) @sync("path/to/{self.key}.yml") class Sample: def __init__(self, name): self.name = name self.my_string = “Hello, world!” self.my_integer = 42 self.my_float = 20.15 self.my_bool = True

Container Types @attr(all=Integer) class IntegerList(List): pass @attr(label=String, status=Boolean) class StatusDictionary(Dictionary): pass

Nested Types config: # a Dictionary computers: # with a List of Dictionaries - address: AA:BB:CC:DD:EE:FF # with String values hostname: Jaces-MacBook name: macbook - address: '' hostname: '' name: macbook-pro

Custom Types class Datetime(String): DATETIME_FORMAT = “%B %d, %Y" @classmethod def to_data(cls, obj): dt = cls.to_value(obj) text = dt.strftime(cls.DATETIME_FORMAT) return text @classmethod def to_value(cls, obj): dt = datetime.strptime(obj, cls.DATETIME_FORMAT) return dt

Use Cases

Loading Configuration @yorm.attr(location=yorm.converters.String) @yorm.attr(sources=Sources) @yorm.sync("{self.root}/{self.filename}") class Config(ShellMixin): FILENAMES = ('gdm.yml', '.gdm.yml') def __init__(self, root, filename=FILENAMES[0], location='gdm_sources'): super().__init__() self.root = root self.filename = filename self.location = location self.sources = [] github.com/jacebrowning/gdm/blob/0a4faeaebde3e6d9464c6e45bd68111423b13acb/gdm/config.py#L112-126

Loading Configuration def load(root): config = None for filename in os.listdir(root): if filename.lower() in Config.FILENAMES: config = Config(root, filename) log.debug("loaded config: %s", config.path) break return config github.com/jacebrowning/gdm/blob/0a4faeaebde3e6d9464c6e45bd68111423b13acb/gdm/config.py#L177-185

Loading Configuration location: .gdm sources: - repo: github.com/kstenerud/iOS-Universal-Framework dir: framework rev: Mk5-end-of-life - repo: github.com/jonreid/XcodeCoverage dir: coverage rev: master link: Tools/XcodeCoverage

Synchronizing State @yorm.attr(applications=StatusList) @yorm.attr(counter=yorm.converters.Integer) class ProgramStatus(yorm.converters.AttributeDictionary): """A dictionary of current program status.""" def __init__(self): super().__init__() self.applications = StatusList() self.counter = 0 github.com/jacebrowning/mine/blob/1947625ee6d44ff98cee6dd209a2ad13d86884d8/mine/status.py#L99-108

Synchronizing State status: applications: - application: itunes computers: - computer: laptop timestamp: started: 444 stopped: 402 - computer: desktop timestamp: started: 335 stopped: 390 counter: 499

Persisting Data $ ls data/games 6lwctiox.yml 9q6m4swa.yml b7mubcu6.yml cx26xhoc.yml @yorm.attr(players=PlayersFileModel) @yorm.attr(turn=yorm.converters.Integer) @yorm.sync("data/games/{self.key}.yml", auto=False) class GameFileModel(domain.Game): def __init__(self, key, players=None, turn=0): super().__init__() self.key = key self.players = players or PlayersFileModel() self.turn = turn github.com/jacebrowning/gridcommand/blob/75bc49f8e6eb0c9f61bdff81d4d9cb6a2c64b09a/gridcommand/stores/game.py#L57-66

Getting Started

$ pip3 install yorm

jacebrowning/yorm github.com / twitter.com / @jacebrowning