Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Synchronizing Objects to YAML using Black Magic

Synchronizing Objects to YAML using Black Magic

Eff2cdaa8474520ed29ebc38031fab87?s=128

Jace Browning

August 02, 2015
Tweet

More Decks by Jace Browning

Other Decks in Programming

Transcript

  1. Synchronizing Objects to YAML using Black Magic

  2. None
  3. @jacebrowning

  4. • Quick Demo • Project Goals • Chosen Implementation •

    Advanced Python Features • Library Overview • Use Cases • Getting Started
  5. Quick Demo

  6. class Student: def __init__(self, name, school, number, year=2009): self.name =

    name self.school = school self.number = number self.year = year self.gpa = 0.0 $ ls (empty)
  7. import yorm from yorm.converters import String, Integer, Float @yorm.attr(name=String, year=Integer,

    gpa=Float) @yorm.sync("students/{self.school}/{self.number}.yml") class Student: ... $ ls (empty)
  8. >>> s1 = Student("John Doe", "GVSU", 123) >>> s2 =

    Student("Jane Doe", "GVSU", 456, year=2014) >>> s1.gpa = 3 $ ls students $ ls students/GVSU 123.yml 456.yml $ cat students/GVSU/123.yml name: John Doe gpa: 3.0 school: GVSU year: 2009
  9. >>> s1.gpa 1.8 >>> s1.expelled True $ echo "name: John

    Doe > gpa: 1.8 > year: 2010 > expelled: true " > students/GVSU/123.yml
  10. Project Goals

  11. Configuration vs. Data Program Configuration: • System setup parameters •

    User options Program Data: • Seeded default values • Persistence model storage • Loaded test fixtures
  12. Storage Requirements • Automatic • Human-editable • Support multiple and/or

    offline editors • Compatible with Python builtin types
  13. Alternatives Explored • Database ORMs (Active Record)
 • Object serialization


    • JSON => dictionary
  14. Code Requirements • Minimal interference with existing code • Automatic

    synchronization of changes • Type inference on new attributes • Custom formatting for user classes
  15. Chosen Implementation

  16. + config: applications: - name: slack versions: linux: null mac:

    Slack.app windows: null - name: iphoto versions: linux: null mac: iPhoto windows: null computers: - address: AA:BB:CC:DD:EE:FF hostname: Jaces-MacBook name: macbook - address: '' hostname: '' name: macbook-pro
  17. Class Decorator or Function @yorm.attr(gpa=yorm.converters.Float) @yorm.sync("students/{self.number}.yml") class Student: ... student

    = Student() student = Student() path = "students/123.yml" attrs = {'gpa': yorm.converters.Float} student = yorm.sync(student, path, attrs)
  18. Optimized Formatting attribute_named_aaa: 1 attribute_named_bbb: - python3.3 - python3.4 attribute_named_zzz:

    serial_numbers: - abc - 456 attributes dictionary text file store dump write
  19. Conversation on Load attribute_named_aaa: 3.1 # converted to 3 attribute_named_bbb:

    - python3.3 - “python3.4” # unnecessary quotes stripped attribute_named_zzz: serial_numbers: - 4.5 # converted to “4.5” # extra whitespace cleaned up file text dictionary attributes read load fetch
  20. Type Inference ... attribute_named_zzz: {} new_attribute: 2015 >>> mapped_instance.new_attribute 2015

  21. Advanced Python Features

  22. Metaclass class ContainerMeta(abc.ABCMeta): def __init__(cls, name, bases, namespace): super().__init__(name, bases,

    namespace) cls.yorm_attrs = {}
  23. Class Decorator def sync_instances(path_format, format_spec=None, attrs=None, **kwargs): ... def decorator(cls):

    ... return cls return decorator
  24. Monkey Patching: Method old_init = obj.__init__ def new_init(self, *args, **kwargs):

    old_init(self, *args, **kwargs) ... obj.__init__ = new_init
  25. Monkey Patching: Class class Mapped(Mappable, obj.__class__): ... obj.__class__ = Mapped

  26. Data Model: Setting Attributes def __setattr__(self, name, value): super().__setattr__(name, value)

    if name.startswith('__'): return mapper = get_mapper(self) if mapper.auto and name in mapper.attrs: mapper.store()
  27. Data Model: Getting Attributes def __getattribute__(self, name): if name.startswith('__'): return

    object.__getattribute__(self, name) try: value = object.__getattribute__(self, name) except AttributeError as exc: missing = True else: missing = False mapper = get_mapper(self) if missing or (name in mapper.attrs and mapper.modified): mapper.fetch() value = object.__getattribute__(self, name) return value
  28. Subclass Lookup class Object(Converter): """Base class for immutable types.""" ...

    def match(data): converters = Object.__subclasses__() for converter in converters: if type(data) == converter.TYPE: return converter ...
  29. Library Overview

  30. Class Decorator or Function @yorm.attr(gpa=yorm.converters.Float) @yorm.sync("students/{self.number}.yml") class Student: ... student

    = Student() student = Student() path = "students/123.yml" attrs = {'gpa': yorm.converters.Float} yorm.sync(student, path, attrs)
  31. Class Decorator or Function @yorm.attr(gpa=yorm.converters.Float) @yorm.sync("students/{self.number}.yml", auto=False) class Student: ...

    student = Student() student = Student() path = "students/123.yml" attrs = {'gpa': yorm.converters.Float} yorm.sync(student, path, attrs, auto=False)
  32. Update File from Object yorm.update_file(mapped_instance)

  33. Update Object from File yorm.update_object(mapped_instance)

  34. …or Both yorm.update(mapped_instance)

  35. Builtin Types @attr(my_string=String) @attr(my_integer=Integer) @attr(my_float=Float) @attr(my_boolean=Boolean) @sync("path/to/{self.key}.yml") class Sample: def

    __init__(self, name): self.name = name self.my_string = “Hello, world!” self.my_integer = 42 self.my_float = 20.15 self.my_bool = True
  36. Container Types @attr(all=Integer) class IntegerList(List): pass @attr(label=String, status=Boolean) class StatusDictionary(Dictionary):

    pass
  37. Nested Types config: # a Dictionary computers: # with a

    List of Dictionaries - address: AA:BB:CC:DD:EE:FF # with String values hostname: Jaces-MacBook name: macbook - address: '' hostname: '' name: macbook-pro
  38. Custom Types class Datetime(String): DATETIME_FORMAT = “%B %d, %Y" @classmethod

    def to_data(cls, obj): dt = cls.to_value(obj) text = dt.strftime(cls.DATETIME_FORMAT) return text @classmethod def to_value(cls, obj): dt = datetime.strptime(obj, cls.DATETIME_FORMAT) return dt
  39. Use Cases

  40. Loading Configuration @yorm.attr(location=yorm.converters.String) @yorm.attr(sources=Sources) @yorm.sync("{self.root}/{self.filename}") class Config(ShellMixin): FILENAMES = ('gdm.yml',

    '.gdm.yml') def __init__(self, root, filename=FILENAMES[0], location='gdm_sources'): super().__init__() self.root = root self.filename = filename self.location = location self.sources = [] github.com/jacebrowning/gdm/blob/0a4faeaebde3e6d9464c6e45bd68111423b13acb/gdm/config.py#L112-126
  41. Loading Configuration def load(root): config = None for filename in

    os.listdir(root): if filename.lower() in Config.FILENAMES: config = Config(root, filename) log.debug("loaded config: %s", config.path) break return config github.com/jacebrowning/gdm/blob/0a4faeaebde3e6d9464c6e45bd68111423b13acb/gdm/config.py#L177-185
  42. Loading Configuration location: .gdm sources: - repo: github.com/kstenerud/iOS-Universal-Framework dir: framework

    rev: Mk5-end-of-life - repo: github.com/jonreid/XcodeCoverage dir: coverage rev: master link: Tools/XcodeCoverage
  43. Synchronizing State @yorm.attr(applications=StatusList) @yorm.attr(counter=yorm.converters.Integer) class ProgramStatus(yorm.converters.AttributeDictionary): """A dictionary of current

    program status.""" def __init__(self): super().__init__() self.applications = StatusList() self.counter = 0 github.com/jacebrowning/mine/blob/1947625ee6d44ff98cee6dd209a2ad13d86884d8/mine/status.py#L99-108
  44. Synchronizing State status: applications: - application: itunes computers: - computer:

    laptop timestamp: started: 444 stopped: 402 - computer: desktop timestamp: started: 335 stopped: 390 counter: 499
  45. Persisting Data $ ls data/games 6lwctiox.yml 9q6m4swa.yml b7mubcu6.yml cx26xhoc.yml @yorm.attr(players=PlayersFileModel)

    @yorm.attr(turn=yorm.converters.Integer) @yorm.sync("data/games/{self.key}.yml", auto=False) class GameFileModel(domain.Game): def __init__(self, key, players=None, turn=0): super().__init__() self.key = key self.players = players or PlayersFileModel() self.turn = turn github.com/jacebrowning/gridcommand/blob/75bc49f8e6eb0c9f61bdff81d4d9cb6a2c64b09a/gridcommand/stores/game.py#L57-66
  46. Getting Started

  47. $ pip3 install yorm

  48. jacebrowning/yorm github.com / twitter.com / @jacebrowning