Share
Embed
Experiences in Framework Design
Frameworks are useful for more than just web applications. Frameworks, like libraries, provide reusable code. The main difference? You call a library; a framework calls you.
I've written a few frameworks, including https://github.com/xBrite/armrest (a REST client framework that enables command-line interaction and acceptance testing). I will share principles, design patterns, techniques, and anti-patterns that I have found helpful in creating frameworks, which may help you to create your own framework.
George V. Reilly
May 09, 2018
Transcript
31 slides
-
EXPERIENCES IN FRAMEWORK DESIGN George V. Reilly @georgevreilly Puget Sound
Programming Python: PuPPy 2018-05-09 -
LIBRARY VS FRAMEWORK ➤ You call a library. ➤ Whereas,
a framework calls you. Your code Framework Library You use Calls you Uses -
FRAMEWORKS ➤ Use a library; adopt a framework. ➤ Inversion
of Control ➤ “A framework embodies some abstract design, with more behavior built in. In order to use it you need to insert your behavior into various places in the framework either by subclassing or by plugging in your own classes. The framework's code then calls your code at these points.” — https://martinfowler.com/bliki/InversionOfControl.html (emphasis mine) ➤ Framework drives the message/event loop. -
APPLICATION FRAMEWORKS ➤ Application Frameworks constrain app’s fundamental structure ➤
Web Backend: Flask, Django, Rails, Express, Laravel ➤ JavaScript Frontend: Angular, React, Backbone ➤ REST: Falcon, APIStar ➤ GUI: wxPython, PyQT, urwid ➤ CLI: Cement, Cliff ➤ Machine Learning: TensorFlow, PyTorch ➤ Network: Twisted, asyncio ➤ Serverless: Serverless, Zappa, Chalice ➤ Misc: Scrapy, SimPy -
TOPIC FRAMEWORKS ➤ Topic Frameworks control an aspect of an
app: ➤ Testing: unittest, Nose, PyTest ➤ Packaging: setuptools, npm, gems ➤ ORM: SQLAlchemy ➤ Plugins: Stevedore, jQuery UI ➤ ArmRest: REST Client Framework ➤ Several Topic Frameworks can be used in one app. -
ADVANTAGES OF USING FRAMEWORKS ➤ Get complex functionality working in
short time. ➤ Structure & Conventions promote shared understanding. ➤ Well-tested, reusable code. ➤ Framework design informed by domain expertise. ➤ Community understands framework. -
DISADVANTAGES OF ADOPTING A FRAMEWORK ➤ Learning curve. ➤ Framework
abstractions obfuscate. ➤ Superficial understanding of framework problem space. ➤ May not let you customize the pieces you care about. ➤ Opinionated Straitjacket: One size doesn’t fit all. ➤ Hard to replace framework with something else. ➤ Tight coupling between framework and application. ➤ Framework magic may be hard to understand and to debug. -
ARMREST’S GENESIS ➤ ArmRest is a Python REST client framework
that enables command-line interaction and acceptance testing. ➤ Needed a good client to interact with our REST APIs. ➤ Needed richer command-line tool than Curl or HTTPie. ➤ Needed Integration and Acceptance tests. ➤ Needed to glue microservices together. ➤ Wanted typed interfaces for APIs. ➤ https://github.com/xBrite/armrest ➤ No documentation or examples yet ☹ -
TRIPLE, PARALLEL CLASS HIERARCHIES TodoListBaseApi TaskApi ReminderApi ResourceApi TodoListBaseApiCli TaskApiCli
ReminderApiCli ApiCli TodoListBaseTestApi TaskTestApi ReminderTestApi TestApi -
VERSION API from .todo_base import (TodoListBaseApi, TodoListBaseApiCli, TodoListBaseTestApi) class VersionApi(TodoListBaseApi):
Path = "/version/" AuthRequired = False Operations = ('get',) class VersionApiCli(TodoListBaseApiCli): ApiClass = VersionApi SubparserCommandName = 'version' class VersionTestApi(TodoListBaseTestApi): TestApiClass = VersionApi def get_version(self): return self.api.get().data -
VERSIONTESTAPI INTEGRATION TEST class VersionIntegrationTests(unittest.TestCase, TodoListBaseTestApi): def test_min_build_version(self): # Testrunner
configured base URL for environment test_api = VersionTestApi() version_data = test_api.get_version() self.assertGreater(version_data["build"], 17) -
REMINDERAPICLI EXAMPLE $ todoapi reminder --create --subject "Rehearse PuPPy talk"
\ --when 2018-05-09T1400 class ReminderApiCli(TodoListBaseApiCli): ApiClass = ReminderApi SubparserCommandName = 'reminder' @classmethod def add_subparser_args(cls, sp, **kwargs): super(ReminderApiCli, cls).add_subparser_args(sp, **kwargs) sp.add_argument( '--subject', help="Short description of reminder") sp.add_argument( '--when', help="When the reminder should be sent") @classmethod def make_request_body(cls, namespace): when = dateutil.parser.parse(namespace.when) return dict(subject=namespace.subject, when=format_date(when)) -
SOFTWARE DESIGN PATTERNS ➤ A design pattern is a general,
reusable solution to a commonly occurring problem within a given context. ➤ Formalized best practices. ➤ Will talk about: ➤ Bridge pattern ➤ Template Method pattern -
THE BRIDGE PATTERN & SEPARATION OF CONCERNS ➤ Originally, ResourceApi
and ApiCli were one class ➤ A tangled “god object” class ➤ Violated Single Responsibility Principle ➤ ResourceApi: REST API, HTTP , serialization, auth ➤ ApiCli: argparse wrapper for ResourceApi; subcommands ➤ For Separation of Concerns, I found the seams and teased ApiCli out. ➤ Added TestApi: integration testing with ResourceApi ➤ ApiCli and TestApi are clients of ResourceApi ➤ Bridge Design Pattern separates orthogonal hierarchies using composition, promoting cohesion. Here, each subclass uses its counterpart. ➤ (Bridge is frequently used with non-parallel hierarchies.) -
BRIDGE PATTERN: PARALLEL, ORTHOGONAL HIERARCHIES TodoListBaseApi TaskApi ReminderApi ResourceApi TodoListBaseApiCli
TaskApiCli ReminderApiCli ApiCli TodoListBaseTestApi TaskTestApi ReminderTestApi TestApi -
TEMPLATE METHOD DESIGN PATTERN ➤ Template Method: Define the skeleton
of an algorithm in the organizing method in a base class. ➤ (Nothing to do with templates such as Jinja or Mustache.) ➤ Provide overridable or abstract methods (hooks) to implement algorithm. ➤ Algorithm’s implementation is pluggable. ➤ Organizing method calls hook methods to do the work. ➤ Derived classes supply hook implementations. -
TEMPLATE METHOD PATTERN: BASE ALGORITHM class ResourceApi: def fetch(self, verb,
url, payload, headers=None): # algorithm: fetch a resource from REST API headers = self.make_request_headers(headers) headers = self.add_auth(headers) wire_payload = self.serialize_content(payload) ... def make_request_headers(self, headers): # Pretend no useful default implementation return headers or {} def add_auth(self, headers): # default implementation (Basic Auth) not shown def serialize_content(self, payload): return json.dumps(payload) -
TEMPLATE METHOD PATTERN: HOOK IMPLEMENTATIONS class MySvc(ResourceApi): def add_auth(self, headers):
headers = super(MySvc, self).add_auth(headers) headers['foo'] = 'bar' # additional customization (not shown) return headers def serialize_content(self, payload): # new implementation owing nothing to base return payload.SerializeToString() # Protobuf -
SEPARATION OF POLICY & MECHANISM ➤ Class Inheritance and the
Template Method Design Pattern support Separation of Policy and Mechanism. ➤ Base class supplies general mechanisms to control behavior ➤ Derived classes supply policies to determine behavior ➤ URL of target service ➤ Authentication ➤ Error handling ➤ Timeouts, backoffs, retries -
COMMONALITY-VARIABILITY ANALYSIS ➤ Commonality-Variability Analysis involves: ➤ identifying common abstractions
and variations, ➤ relationships between them, ➤ assigning them responsibilities, ➤ and then linking them together. ➤ The Commonalities (concepts) belong in the base class ➤ Put Variabilities (concrete implementations) in derived classes -
DEEPENING HIERARCHIES ➤ ArmRest initially worked with only one service
➤ e.g., ReminderApi derived directly from ResourceApi ➤ Introduced intermediate service-level classes ➤ e.g., TodoListBaseApi and CalendarBaseApi ➤ Before open sourcing ArmRest: ➤ separated company-specific features from ResourceApi ➤ moved ResourceApi et al into new tree ➤ moved company-specific code into intermediate shim class import ResourceApi as _ResourceApi class ResourceApi(_ResourceApi): # organization-specific overrides -
ISOLATE EARLY TO OPEN SOURCE ➤ If I had isolated
the organization-specific code sooner, ArmRest could have been open sourced much earlier. ➤ The organization-agnostic base code should have been in its own repository. ➤ Sharper boundaries should have been drawn. ➤ Costs of isolating and open sourcing: ➤ Features coordinated across several repos. ➤ Changes may break strangers’ code. Better testing required. ➤ Popular tools will demand your time and support. -
WHY & WHEN TO CREATE A FRAMEWORK ➤ Identify that
you’re repeatedly reusing the same boilerplate code, pattern, and architecture in multiple apps or topics. ➤ Minimum 3 instances, with more expected ➤ Analyze: extract structure, concepts, generalities, abstractions ➤ Identify where to allow variations ➤ Rinse and Repeat: ➤ Apply framework to new instances ➤ Identify further commonalities and variabilities ➤ Refactor -
WHEN NOT TO EXTRACT A FRAMEWORK ➤ Libraries are easier
to extract and reuse. ➤ Short-lived apps. ➤ Effort unlikely to pay off. ➤ Additional framework complexity obfuscates apps. ➤ See also: Framework Disadvantages slide.