Upgrade to Pro — share decks privately, control downloads, hide ads and more …

API Design for Libraries by Chris McDonough

API Design for Libraries by Chris McDonough

PyCon 2013

March 17, 2013
Tweet

More Decks by PyCon 2013

Other Decks in Technology

Transcript

  1. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 1/16
    API Design for Libraries
    Authors: Chris McDonough
    Agendaless Consulting
    Date: 03/15/2013 (PyCon 2013)
    Who Am I
    Came to Python via Zope in 1999. Worked at Digital Creations (aka Zope Corporation) until 2003. Now a
    consultant with Agendaless Consulting.
    Primary author of Pyramid web framework, Supervisor UNIX process control system, Deform form
    system, Repoze collection of middleware, and other unmentionables. Contributor to Zope, WebOb, and
    other OSS projects.
    I Care About Your Feelings
    During this talk, I call out antipattern examples from actual projects, including my own.
    If I use code from one of your projects as an antipattern example, it doesn't mean I don't like you.
    This talk is impossible to give without showing negative examples.
    I'm lazy and the best negative examples are those that already exist.
    Libs, Frameworks, Apps
    Application: Maintains lots of state, can use global state with abandon.
    Framework: No or little state, but lots of callbacks.
    Library: Maintains none or little of its own state, no or few callbacks.
    A web framework instance is often fed to a global mainloop, but that doesn't mean it should use globals
    with abandon. Even then if the framework doesn't use global state, with a little care, two framework
    instances can live in the same process. Some frameworks mutate or require global state (IMO
    inappropriately).

    View Slide

  2. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 2/16
    Guidelines
    This talk covers four guidelines.
    If you follow these guidelines, your library will be useful for strangers in both the scenarios you expect and
    in ones you don't.
    The importance of the guidelines increases with the number of users whom might reuse your code and your
    social distance from those users.
    Guidelines
    #1: Global State is Precious
    #2: Don't Design Exclusively For Convenience
    #3: Avoid Knobs on Knobs
    #4: Composition Usually Beats Inheritance
    #1: Global State is Precious
    Avoid unnecessary mutation of global (module-level) state when your library is imported.
    Avoid requiring that other people mutate global state to use your library. Ex: telling people to set an
    environment variable or call a function which mutates global state to use your library.
    If your library mutates global state when it's imported or you require people to mutate global state to
    use it, it's not really a library, it's kinda more like an application.
    Registration During Import
    Importing m
    u
    l
    t
    i
    p
    r
    o
    c
    e
    s
    s
    i
    n
    g from the standard library causes an atexit function to be registered at
    module scope:
    d
    e
    f _
    e
    x
    i
    t
    _
    f
    u
    n
    c
    t
    i
    o
    n
    (
    )
    :
    # .
    .
    . e
    l
    i
    d
    e
    d .
    .
    .
    f
    o
    r p i
    n a
    c
    t
    i
    v
    e
    _
    c
    h
    i
    l
    d
    r
    e
    n
    (
    )
    :
    i
    f p
    .
    _
    d
    a
    e
    m
    o
    n
    i
    c
    :
    i
    n
    f
    o
    (
    '
    c
    a
    l
    l
    i
    n
    g t
    e
    r
    m
    i
    n
    a
    t
    e
    (
    ) f
    o
    r d
    a
    e
    m
    o
    n %
    s
    '
    , p
    .
    n
    a
    m
    e
    )
    p
    .
    _
    p
    o
    p
    e
    n
    .
    t
    e
    r
    m
    i
    n
    a
    t
    e
    (
    )
    # .
    . e
    l
    i
    d
    e
    d .
    .
    .
    a
    t
    e
    x
    i
    t
    .
    r
    e
    g
    i
    s
    t
    e
    r
    (
    _
    e
    x
    i
    t
    _
    f
    u
    n
    c
    t
    i
    o
    n
    )

    View Slide

  3. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 3/16
    Reg. During Import (2)
    From l
    o
    g
    g
    i
    n
    g module:
    _
    h
    a
    n
    d
    l
    e
    r
    L
    i
    s
    t = [
    ] # m
    u
    t
    a
    t
    e
    d b
    y l
    o
    g
    g
    i
    n
    g
    .
    g
    e
    t
    L
    o
    g
    g
    e
    r
    , e
    t
    c
    i
    m
    p
    o
    r
    t a
    t
    e
    x
    i
    t
    d
    e
    f s
    h
    u
    t
    d
    o
    w
    n
    (
    h
    a
    n
    d
    l
    e
    r
    L
    i
    s
    t
    =
    _
    h
    a
    n
    d
    l
    e
    r
    L
    i
    s
    t
    )
    :
    f
    o
    r h i
    n h
    a
    n
    d
    l
    e
    r
    L
    i
    s
    t
    [
    :
    ]
    :
    # .
    .
    .
    a
    t
    e
    x
    i
    t
    .
    r
    e
    g
    i
    s
    t
    e
    r
    (
    s
    h
    u
    t
    d
    o
    w
    n
    )
    Why is This Bad?
    Unexpected. Registration of an a
    t
    e
    x
    i
    t function is a mutation of global state that results solely from an
    import of a module, whether or not you actually use any APIs from the module.
    Unnecessary. Both m
    u
    l
    t
    i
    p
    r
    o
    c
    e
    s
    s
    i
    n
    g and l
    o
    g
    g
    i
    n
    g choose to manage global state. But neither
    really needs to register an a
    t
    e
    x
    i
    t function until there's any nondefault state to clean up.
    Your program will behave differently at shutdown if you cause m
    u
    l
    t
    i
    p
    r
    o
    c
    e
    s
    s
    i
    n
    g or l
    o
    g
    g
    i
    n
    g to be
    imported, or if you import a third-party module that happens to import one of them (you might not even
    know).
    It's convenient until your process shutdown starts spewing errors that you can't figure out at unit test exit
    time. Then it's pretty inconvenient. Example: seemingly random error message at shutdown time if you
    attempt to use the l
    o
    g
    g
    i
    n
    g
    .
    H
    a
    n
    d
    l
    e
    r class independent of the rest of the framework.
    Ctor Globals Mutation
    Mutating a global registry as the result of an object constructor (again from l
    o
    g
    g
    i
    n
    g
    ).
    _
    h
    a
    n
    d
    l
    e
    r
    s = {
    } #
    r
    e
    p
    o
    s
    i
    t
    o
    r
    y o
    f h
    a
    n
    d
    l
    e
    r
    s
    _
    h
    a
    n
    d
    l
    e
    r
    L
    i
    s
    t = [
    ] # o
    r
    d
    e
    r
    e
    d l
    i
    s
    t o
    f h
    a
    n
    d
    l
    e
    r
    s
    c
    l
    a
    s
    s H
    a
    n
    d
    l
    e
    r
    (
    F
    i
    l
    t
    e
    r
    e
    r
    )
    :
    d
    e
    f _
    _
    i
    n
    i
    t
    _
    _
    (
    s
    e
    l
    f
    , l
    e
    v
    e
    l
    =
    N
    O
    T
    S
    E
    T
    )
    :
    # .
    . e
    l
    i
    d
    e
    d c
    o
    d
    e .
    .
    .
    _
    h
    a
    n
    d
    l
    e
    r
    s
    [
    s
    e
    l
    f
    ] = 1
    _
    h
    a
    n
    d
    l
    e
    r
    L
    i
    s
    t
    .
    i
    n
    s
    e
    r
    t
    (
    0
    , s
    e
    l
    f
    )
    # .
    . e
    l
    i
    d
    e
    d c
    o
    d
    e .
    .

    View Slide

  4. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 4/16
    Ctor Globals Mutation (2)
    From a
    s
    y
    n
    c
    o
    r
    e (at least it lets you choose the m
    a
    p
    ):
    s
    o
    c
    k
    e
    t
    _
    m
    a
    p = {
    }
    c
    l
    a
    s
    s d
    i
    s
    p
    a
    t
    c
    h
    e
    r
    :
    d
    e
    f _
    _
    i
    n
    i
    t
    _
    _
    (
    s
    e
    l
    f
    , s
    o
    c
    k
    =
    N
    o
    n
    e
    , m
    a
    p
    =
    N
    o
    n
    e
    )
    :
    i
    f m
    a
    p i
    s N
    o
    n
    e
    :
    s
    e
    l
    f
    .
    _
    m
    a
    p = s
    o
    c
    k
    e
    t
    _
    m
    a
    p
    e
    l
    s
    e
    :
    s
    e
    l
    f
    .
    _
    m
    a
    p = m
    a
    p
    i
    f s
    o
    c
    k
    :
    # .
    . s
    e
    t
    _
    s
    o
    c
    k
    e
    t m
    u
    t
    a
    t
    e
    s t
    h
    e m
    a
    p .
    .
    .
    s
    e
    l
    f
    .
    s
    e
    t
    _
    s
    o
    c
    k
    e
    t
    (
    s
    o
    c
    k
    , m
    a
    p
    )
    What's Wrong With This?
    Side effect of globals mutation makes out-of-context reuse of the class more difficult than necessary.
    You can't make an instance without mutating global state.
    If you really must do this, create an alternative "application" API for instance construction which
    constructs an instance and then mutates a global, but let the library be just a library.
    Makes unit testing hard (need to clean up module global state).
    Calls for Side-Effects
    Users of the l
    o
    g
    g
    i
    n
    g module are encouraged to do this:
    i
    m
    p
    o
    r
    t l
    o
    g
    g
    i
    n
    g
    l
    o
    g
    g
    i
    n
    g
    .
    b
    a
    s
    i
    c
    C
    o
    n
    f
    i
    g
    (
    )
    l
    o
    g
    g
    i
    n
    g
    .
    a
    d
    d
    L
    e
    v
    e
    l
    N
    a
    m
    e
    (
    1
    7
    5
    , '
    O
    H
    N
    O
    E
    S
    '
    )
    Calls for Side-Effects (2)
    m
    i
    m
    e
    t
    y
    p
    e
    s module maintains a global registry:
    i
    m
    p
    o
    r
    t m
    i
    m
    e
    t
    y
    p
    e
    s
    m
    i
    m
    e
    t
    y
    p
    e
    s
    .
    i
    n
    i
    t
    (
    )
    m
    i
    m
    e
    t
    y
    p
    e
    s
    .
    a
    d
    d
    _
    t
    y
    p
    e
    (
    '
    t
    e
    x
    t
    /
    f
    o
    o
    '
    , '
    .
    f
    o
    o
    '
    )

    View Slide

  5. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 5/16
    Calls for Side-Effects (3)
    From the Python Braintree payment gateway API:
    i
    m
    p
    o
    r
    t b
    r
    a
    i
    n
    t
    r
    e
    e
    b
    r
    a
    i
    n
    t
    r
    e
    e
    .
    C
    o
    n
    f
    i
    g
    u
    r
    a
    t
    i
    o
    n
    .
    c
    o
    n
    f
    i
    g
    u
    r
    e
    (
    b
    r
    a
    i
    n
    t
    r
    e
    e
    .
    E
    n
    v
    i
    r
    o
    n
    m
    e
    n
    t
    .
    S
    a
    n
    d
    b
    o
    x
    ,
    m
    e
    r
    c
    h
    a
    n
    t
    _
    i
    d
    =
    "
    u
    s
    e
    _
    y
    o
    u
    r
    _
    m
    e
    r
    c
    h
    a
    n
    t
    _
    i
    d
    "
    ,
    p
    u
    b
    l
    i
    c
    _
    k
    e
    y
    =
    "
    u
    s
    e
    _
    y
    o
    u
    r
    _
    p
    u
    b
    l
    i
    c
    _
    k
    e
    y
    "
    ,
    p
    r
    i
    v
    a
    t
    e
    _
    k
    e
    y
    =
    "
    u
    s
    e
    _
    y
    o
    u
    r
    _
    p
    r
    i
    v
    a
    t
    e
    _
    k
    e
    y
    "
    )
    What's Wrong with This?
    The l
    o
    g
    g
    i
    n
    g
    , m
    i
    m
    e
    t
    y
    p
    e
    s and b
    r
    a
    i
    n
    t
    r
    e
    e APIs encourage users to mutate their module's
    global state by exposing APIs that have return values that nobody cares about.
    Introduces responsibility, chronology, and idempotency confusion. Who is responsible for calling
    this? When should they call it? Can it be called more than once? If it is called more than once, what
    happens when it's called the second time?
    l
    o
    g
    g
    i
    n
    g maintains a global registry as a dictionary at module scope. Calling b
    a
    s
    i
    c
    C
    o
    n
    f
    i
    g
    (
    ) is
    effectively a structured monkeypatch of l
    o
    g
    g
    i
    n
    g module state. Same for a
    d
    d
    L
    e
    v
    e
    l
    N
    a
    m
    e
    . Logging
    classes know about this global state and use it. The m
    i
    m
    e
    t
    y
    p
    e
    s module API maintains a global registry
    too. Same deal with Braintree.
    Not-Really-Configuration
    From the l
    o
    g
    g
    i
    n
    g package:
    #
    # r
    a
    i
    s
    e
    E
    x
    c
    e
    p
    t
    i
    o
    n
    s i
    s u
    s
    e
    d t
    o s
    e
    e i
    f e
    x
    c
    e
    p
    t
    i
    o
    n
    s d
    u
    r
    i
    n
    g
    # h
    a
    n
    d
    l
    i
    n
    g s
    h
    o
    u
    l
    d b
    e p
    r
    o
    p
    a
    g
    a
    t
    e
    d
    #
    r
    a
    i
    s
    e
    E
    x
    c
    e
    p
    t
    i
    o
    n
    s = 1
    What's Wrong With This?
    Nothing, if it's only for the benefit/convenience of the library developer himself.
    But if it's presence is advertised to users, it's pseudo-configuration; there's no way to change it
    without monkeypatching the module.
    The setting is global. No way to use separate settings per process.

    View Slide

  6. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 6/16
    Inversion of Config Control
    Django s
    e
    t
    t
    i
    n
    g
    s
    .
    p
    y
    :
    # D
    j
    a
    n
    g
    o s
    e
    t
    t
    i
    n
    g
    s f
    o
    r m
    y
    s
    i
    t
    e p
    r
    o
    j
    e
    c
    t
    .
    D
    E
    B
    U
    G = T
    r
    u
    e
    T
    E
    M
    P
    L
    A
    T
    E
    _
    D
    E
    B
    U
    G = D
    E
    B
    U
    G
    A
    D
    M
    I
    N
    S = (
    # (
    '
    Y
    o
    u
    r N
    a
    m
    e
    '
    , '
    y
    o
    u
    r
    _
    e
    m
    a
    i
    l
    @
    e
    x
    a
    m
    p
    l
    e
    .
    c
    o
    m
    '
    )
    ,
    )
    What's Wrong With This?
    The library/framework wants to import values from this module.
    But since it's Python, the author of the settings code will be tempted to import stuff from the
    library/framework. Extremely high likelihood of circular import problems (framework imports
    settings, settings imports framework).
    The settings are global. No way to use separate settings per process.
    OK at Module Scope
    A non-circular import of another module or global.
    Assignment of a variable name in the module to some constant value.
    The addition of a function via a def statement.
    The addition of a class via a class statement.
    Control flow which handles conditionals for platform-specific failure handling of the above.
    Anything else will usually end in tears.
    Solutions
    Think of an application bootstrapping in two phases.
    Before i
    f _
    _
    n
    a
    m
    e
    _
    _ =
    = '
    _
    _
    m
    a
    i
    n
    _
    _
    '
    :
    . Do nothing.
    As the result of i
    f _
    _
    n
    a
    m
    e
    _
    _ =
    = '
    _
    _
    m
    a
    i
    n
    _
    _
    '
    . Do everything.

    View Slide

  7. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 7/16
    Downsides
    Downside for m
    u
    l
    t
    i
    p
    r
    o
    c
    e
    s
    s
    i
    n
    g to ditch its global state maintenance: its API won't match that of
    t
    h
    r
    e
    a
    d
    i
    n
    g
    .
    Downside for l
    o
    g
    g
    i
    n
    g
    : streams related to the same handler might interleave.
    Downside for m
    i
    m
    e
    t
    y
    p
    e
    s
    : might need to reparse system mimetype files.
    But, But..
    You can always create library code such that it mutates no global state, but then, as necessary, create a
    convenience application module which integrates the library stuff and mutates some global state on behalf
    of its users. This makes the library code reusable, and if someone wants to use the application code, they
    can.
    Restraint Under Pressure
    Example of restraint under obvious pressure for convenience and magic from the Python
    s
    c
    h
    e
    d
    .
    s
    c
    h
    e
    d
    u
    l
    e
    r library class:
    "
    "
    "
    E
    a
    c
    h i
    n
    s
    t
    a
    n
    c
    e o
    f t
    h
    i
    s c
    l
    a
    s
    s m
    a
    n
    a
    g
    e
    s i
    t
    s o
    w
    n q
    u
    e
    u
    e
    .
    N
    o m
    u
    l
    t
    i
    -
    t
    h
    r
    e
    a
    d
    i
    n
    g i
    s i
    m
    p
    l
    i
    e
    d
    ; y
    o
    u a
    r
    e s
    u
    p
    p
    o
    s
    e
    d t
    o h
    a
    c
    k t
    h
    a
    t
    y
    o
    u
    r
    s
    e
    l
    f
    , o
    r u
    s
    e a s
    i
    n
    g
    l
    e i
    n
    s
    t
    a
    n
    c
    e p
    e
    r a
    p
    p
    l
    i
    c
    a
    t
    i
    o
    n
    .
    "
    "
    "
    s
    c
    h
    e
    d
    u
    l
    e
    r = s
    c
    h
    e
    d
    .
    s
    c
    h
    e
    d
    u
    l
    e
    r
    (
    )
    d
    e
    f d
    o
    (
    a
    r
    g
    )
    : p
    r
    i
    n
    t a
    r
    g
    s
    c
    h
    e
    d
    u
    l
    e
    r
    .
    e
    n
    t
    e
    r
    (
    3
    0
    , 0
    , d
    o
    , 1
    )
    s
    c
    h
    e
    d
    u
    l
    e
    r
    .
    r
    u
    n
    (
    )
    Quote
    "This method of turning your code inside out is the secret to solving what appear to be hopelessly state-
    oriented problems in a purely functional style. Push the statefulness to a higher level and let the caller worry
    about it. Keep doing that as much as you can, and you'll end up with the bulk of the code being purely
    functional." -- http://prog21.dadgum.com/131.html

    View Slide

  8. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 8/16
    #2: Avoid Convenience
    Avoid convenience (magical) features such as thread local access until you've finished creating the
    inconvenient (nonmagical) version.
    Expose the inconvenient version as a set of APIs and make the convenience features optional, through a
    separate set of APIs.
    You can always add convenience to a library, you can never remove it.
    Stacked Object Proxies
    Pylons offers importable r
    e
    q
    u
    e
    s
    t and r
    e
    s
    p
    o
    n
    s
    e objects ("stacked object proxies"):
    f
    r
    o
    m p
    y
    l
    o
    n
    s i
    m
    p
    o
    r
    t r
    e
    q
    u
    e
    s
    t
    , r
    e
    s
    p
    o
    n
    s
    e
    f
    r
    o
    m p
    y
    l
    o
    n
    s
    .
    c
    o
    n
    t
    r
    o
    l
    l
    e
    r
    s i
    m
    p
    o
    r
    t B
    a
    s
    e
    C
    o
    n
    t
    r
    o
    l
    l
    e
    r
    c
    l
    a
    s
    s C
    o
    n
    t
    r
    o
    l
    l
    e
    r
    (
    B
    a
    s
    e
    C
    o
    n
    t
    r
    o
    l
    l
    e
    r
    )
    :
    d
    e
    f o
    k
    (
    s
    e
    l
    f
    )
    :
    i
    f r
    e
    q
    u
    e
    s
    t
    .
    p
    a
    r
    a
    m
    s
    .
    g
    e
    t
    (
    '
    o
    k
    '
    )
    :
    r
    e
    s
    p
    o
    n
    s
    e
    .
    b
    o
    d
    y = '
    o
    k
    '
    e
    l
    s
    e
    :
    r
    e
    s
    p
    o
    n
    s
    e
    .
    b
    o
    d
    y = '
    n
    o
    t o
    k
    '
    r
    e
    t
    u
    r
    n r
    e
    s
    p
    o
    n
    s
    e
    Flask's Context Locals
    Flask has the same concept for its r
    e
    q
    u
    e
    s
    t
    :
    f
    r
    o
    m f
    l
    a
    s
    k i
    m
    p
    o
    r
    t r
    e
    q
    u
    e
    s
    t
    @
    a
    p
    p
    .
    r
    o
    u
    t
    e
    (
    '
    /
    l
    o
    g
    i
    n
    '
    , m
    e
    t
    h
    o
    d
    s
    =
    [
    '
    P
    O
    S
    T
    '
    , '
    G
    E
    T
    '
    ]
    )
    d
    e
    f l
    o
    g
    i
    n
    (
    )
    :
    e
    r
    r
    o
    r = N
    o
    n
    e
    i
    f r
    e
    q
    u
    e
    s
    t
    .
    m
    e
    t
    h
    o
    d =
    = '
    P
    O
    S
    T
    '
    :
    i
    f v
    a
    l
    i
    d
    _
    l
    o
    g
    i
    n
    (
    r
    e
    q
    u
    e
    s
    t
    .
    f
    o
    r
    m
    [
    '
    u
    s
    e
    r
    n
    a
    m
    e
    '
    ]
    ,
    r
    e
    q
    u
    e
    s
    t
    .
    f
    o
    r
    m
    [
    '
    p
    a
    s
    s
    w
    o
    r
    d
    '
    ]
    )
    :
    r
    e
    t
    u
    r
    n l
    o
    g
    _
    t
    h
    e
    _
    u
    s
    e
    r
    _
    i
    n
    (
    r
    e
    q
    u
    e
    s
    t
    .
    f
    o
    r
    m
    [
    '
    u
    s
    e
    r
    n
    a
    m
    e
    '
    ]
    )
    e
    l
    s
    e
    :
    e
    r
    r
    o
    r = '
    I
    n
    v
    a
    l
    i
    d u
    s
    e
    r
    n
    a
    m
    e
    /
    p
    a
    s
    s
    w
    o
    r
    d
    '

    View Slide

  9. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 9/16
    What's Wrong With This?
    Things that are not logically global (r
    e
    q
    u
    e
    s
    t and/or r
    e
    s
    p
    o
    n
    s
    e
    ) are obtained via an import.
    Two levels of magic: proxy that accesses a thread-local when asked for an attribute.
    Encourages inappropriate coupling of non-web-context code to a web context (e.g. "model" modules
    start to i
    m
    p
    o
    r
    t r
    e
    q
    u
    e
    s
    t
    ).
    Makes unit testing harder than it needs to be, because proxy objects need to be initialized.
    Instead
    Design a framework so its users receive an argument (e.g. r
    e
    q
    u
    e
    s
    t
    ) and suggest to them that they
    pass derivations (e.g. r
    e
    q
    u
    e
    s
    t
    .
    G
    E
    T
    [
    '
    f
    i
    r
    s
    t
    _
    n
    a
    m
    e
    '
    ]
    ) around. It's less convenient for
    consumers. It's usually also the right thing to do in library and framework code.
    You can always create an (optional) convenience API that allows your library's consumers to elide
    the passing of state, but you can never remove a "mandatory" convenience feature from a library.
    Remember that people will want to use your stuff to compose larger systems, and your assumptions about
    environment may not fit there.
    Convenience != Cleanliness
    The assumption: "clean" == "is maximally convenient for the case I presume this code is going to be
    used in"
    The reality: "clean" == "maximally understandable; without surprises or exceptions".
    The fewer limiting assumptions made by the library, the fewer surprises it will have and the more
    understandable it will be.
    Ex: thread-local state management doesn't work in async systems without magical intervention.
    #3: Avoid Knobs on Knobs
    A "knob" is a replaceable component in a framework or library.
    When a replaceable component itself offers a knob, this is the "knobs on knobs" pattern.

    View Slide

  10. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 10/16
    Pyramid Authn Policy
    From p
    y
    r
    a
    m
    i
    d
    , the use of an authentication policy knob on knob:
    # .
    .
    . i
    m
    p
    o
    r
    t
    s e
    l
    i
    d
    e
    d .
    .
    .
    G
    R
    O
    U
    P
    S = {
    '
    f
    r
    e
    d
    '
    :
    [
    '
    e
    d
    i
    t
    o
    r
    s
    '
    ]
    }
    d
    e
    f g
    r
    o
    u
    p
    f
    i
    n
    d
    e
    r
    (
    u
    s
    e
    r
    i
    d
    , r
    e
    q
    u
    e
    s
    t
    )
    :
    r
    e
    t
    u
    r
    n G
    R
    O
    U
    P
    S
    .
    g
    e
    t
    (
    u
    s
    e
    r
    i
    d
    )
    p
    o
    l = A
    u
    t
    h
    T
    k
    t
    A
    u
    t
    h
    e
    n
    t
    i
    c
    a
    t
    i
    o
    n
    P
    o
    l
    i
    c
    y
    (
    c
    a
    l
    l
    b
    a
    c
    k
    =
    g
    r
    o
    u
    p
    f
    i
    n
    d
    e
    r
    )
    c
    o
    n
    f
    i
    g = C
    o
    n
    f
    i
    g
    u
    r
    a
    t
    o
    r
    (
    )
    c
    o
    n
    f
    i
    g
    .
    s
    e
    t
    _
    a
    u
    t
    h
    e
    n
    t
    i
    c
    a
    t
    i
    o
    n
    _
    p
    o
    l
    i
    c
    y
    (
    p
    o
    l
    )
    Why Is This Bad?
    We're actually dealing with two separate frameworks.
    Pyramid C
    o
    n
    f
    i
    g
    u
    r
    a
    t
    o
    r
    .
    s
    e
    t
    _
    a
    u
    t
    h
    e
    n
    t
    i
    c
    a
    t
    i
    o
    n
    _
    p
    o
    l
    i
    c
    y accepts an authentication
    policy.
    A
    u
    t
    h
    T
    k
    t
    A
    u
    t
    h
    e
    n
    t
    i
    c
    a
    t
    i
    o
    n
    P
    o
    l
    i
    c
    y
    , an authentication policy, is itself a miniframework, that
    accepts a callback.
    People don't understand when or why to replace "the big thing" when there's a "little thing" inside the
    big thing that's also replaceable.
    There's the Pyramid configurator s
    e
    t
    _
    a
    u
    t
    h
    e
    n
    t
    i
    c
    a
    t
    i
    o
    n
    _
    p
    o
    l
    i
    c
    y method, which accepts something
    that adheres to the "authentication policy interface" (the interface requires a number of methods).
    AuthTktAuthenticationPolicy implements this interface.
    But AuthTktAuthenticationPolicy is also its own mini-framework, accepting a c
    a
    l
    l
    b
    a
    c
    k constructor
    argument, which must be a callable that accepts a userid and a request, and which must return a sequence of
    groups.
    This was done with the intent of avoiding documentation that tells people to subclass
    AuthTktAuthenticationPolicy, preferring to tell them to compose something together by passing a callback
    function to the policy's constructor.
    Solution
    Remove or hide a knob.
    Would probably be less confusing and more straightforward in this case to tell folks to subclass
    AuthTktAuthenticationPolicy and override e.g. a f
    i
    n
    d
    _
    g
    r
    o
    u
    p
    s method in the subclass instead of
    passing in a callback.

    View Slide

  11. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 11/16
    #4: Composing>Inheriting
    Offering up superclasses "from on high" in a library or framework is often a bad idea.
    Composition usually beats inheritance (although not always).
    The Yo-Yo Problem
    http://en.wikipedia.org/wiki/Yo-yo_problem
    " ... occurs when a programmer has to read and understand a program whose inheritance graph is so long
    and complicated that the programmer has to keep flipping between many different class definitions in order
    to follow the control flow of the program..."
    Yo-Yo Problem (Cont'd)
    Almost every Zope object visible from the ZMI inherits from this base class:
    c
    l
    a
    s
    s I
    t
    e
    m
    (
    B
    a
    s
    e
    ,
    R
    e
    s
    o
    u
    r
    c
    e
    ,
    C
    o
    p
    y
    S
    o
    u
    r
    c
    e
    ,
    T
    a
    b
    s
    ,
    T
    r
    a
    v
    e
    r
    s
    a
    b
    l
    e
    ,
    O
    w
    n
    e
    d
    ,
    U
    n
    d
    o
    S
    u
    p
    p
    o
    r
    t
    ,
    )
    :
    "
    "
    "
    A c
    o
    m
    m
    o
    n b
    a
    s
    e c
    l
    a
    s
    s f
    o
    r s
    i
    m
    p
    l
    e
    , n
    o
    n
    -
    c
    o
    n
    t
    a
    i
    n
    e
    r
    o
    b
    j
    e
    c
    t
    s
    .
    "
    "
    "
    Codependency
    The "specialization interface" of a superclass can be hard to document and it's very easy to get
    wrong.
    Encapsulation is rarely honored when inheritance is used, so changes to a parent class will almost
    always break some number of existing subclasses whose implementers weren't paying attention to the
    specialization interface when they originally inherited from your library's superclass.
    The superclass may start simple, initially created to handle one or two particular cases of reuse via
    inheritance, but over time, as more folks add to its specialization interface, it will need to do more
    delegation, and may be required to become very abstract.
    When the superclass reaches a high level of abstraction, it may not be obvious what the purpose of the class
    is or why it's implemented as it is. A potential maintainer of the superclass may need to gain a detailed
    understanding of the implementation of in-the-wild subclasses in order to change the superclass. This can
    scare off potential contributors.

    View Slide

  12. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 12/16
    Codependency (Cont'd)
    The superclass may assume a particular state outcome from a combination of method calls. The
    expected outcome of such an operation is hard to explain and difficult for a subclass to enforce. It
    may need to change over time, breaking existing subclasses in hard-to-predict ways.
    Subclasses may unknowingly coopt and change the meaning of superclass instance or class variables.
    Smells
    From "Inheritance Considered Harmful" on http://www.midmarsh.co.uk
    Subclasses which override methods used by other inherited methods.
    A subclass which extends inherited methods using s
    u
    p
    e
    r
    . Other inherited methods may rely on the
    extended method.
    Subclass which uses or changes the state of "private" instance variables or calls/overrides methods
    not part of the specialization interface.
    [1] Which are thus reliant on the behaviour and results of the overridden methods.
    Alternatives to Inheritance
    Composition
    Event systems
    Composition
    Instead of telling folks to override a method of a library superclass via inheritance, you can tell them
    to pass in a component object to a library class constructor.
    The interaction between a component and your library code is "composition".
    When a library or framework uses a component, the only dependency between the library code and
    the component is the component's interface.
    A component represents the custom logic that would have otherwise gone in a method of a subclass.
    The library code will only have visibility into the component via its interface. The component needn't have
    any visibility into the library code at all (but often does).
    It's less likely that a component author will rely on non-API implementation details of the library than it
    would be if he was required to subclass a library parent class. The potential distraction of the ability to
    customize every aspect of the behavior of the system by overriding methods is removed.
    A clear contract makes it feasible to change the implementation of both the library and the component with
    reduced fear of breaking an integration of the two.

    View Slide

  13. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 13/16
    Inheritance Example
    Here's an example of providing a class built to be specialized via inheritance:
    c
    l
    a
    s
    s T
    V
    R
    e
    m
    o
    t
    e
    (
    o
    b
    j
    e
    c
    t
    )
    :
    d
    e
    f _
    _
    i
    n
    i
    t
    _
    _
    (
    s
    e
    l
    f
    )
    :
    s
    e
    l
    f
    .
    c
    h
    a
    n
    n
    e
    l = 0
    d
    e
    f i
    n
    c
    r
    e
    m
    e
    n
    t
    _
    c
    h
    a
    n
    n
    e
    l
    (
    s
    e
    l
    f
    )
    :
    s
    e
    l
    f
    .
    c
    h
    a
    n
    n
    e
    l +
    = 1
    d
    e
    f c
    l
    i
    c
    k
    (
    s
    e
    l
    f
    , b
    u
    t
    t
    o
    n
    _
    n
    a
    m
    e
    )
    :
    r
    a
    i
    s
    e N
    o
    t
    I
    m
    p
    l
    e
    m
    e
    n
    t
    e
    d
    E
    r
    r
    o
    r
    Composition Ex. (Cont'd)
    Here's an example of using the TVRemote class:
    f
    r
    o
    m t
    v i
    m
    p
    o
    r
    t T
    V
    R
    e
    m
    o
    t
    e
    c
    l
    a
    s
    s M
    y
    R
    e
    m
    o
    t
    e
    (
    T
    V
    R
    e
    m
    o
    t
    e
    )
    :
    d
    e
    f c
    l
    i
    c
    k
    (
    s
    e
    l
    f
    , b
    u
    t
    t
    o
    n
    _
    n
    a
    m
    e
    )
    :
    i
    f b
    u
    t
    t
    o
    n
    _
    n
    a
    m
    e =
    = '
    b
    l
    u
    e
    '
    :
    s
    e
    l
    f
    .
    i
    n
    c
    r
    e
    m
    e
    n
    t
    _
    c
    h
    a
    n
    n
    e
    l
    (
    )
    r
    e
    m
    o
    t
    e = M
    y
    R
    e
    m
    o
    t
    e
    (
    )
    r
    e
    m
    o
    t
    e
    .
    c
    l
    i
    c
    k
    (
    '
    b
    l
    u
    e
    '
    )
    Composition Ex. (Cont'd)
    Here's an example of a library class built to be specialized via composition instead of inheritance:
    c
    l
    a
    s
    s T
    V
    R
    e
    m
    o
    t
    e
    (
    o
    b
    j
    e
    c
    t
    )
    :
    d
    e
    f _
    _
    i
    n
    i
    t
    _
    _
    (
    s
    e
    l
    f
    , b
    u
    t
    t
    o
    n
    s
    )
    :
    s
    e
    l
    f
    .
    c
    h
    a
    n
    n
    e
    l = 0
    s
    e
    l
    f
    .
    b
    u
    t
    t
    o
    n
    s = b
    u
    t
    t
    o
    n
    s
    d
    e
    f i
    n
    c
    r
    e
    m
    e
    n
    t
    _
    c
    h
    a
    n
    n
    e
    l
    (
    s
    e
    l
    f
    )
    :
    s
    e
    l
    f
    .
    c
    h
    a
    n
    n
    e
    l +
    = 1
    d
    e
    f c
    l
    i
    c
    k
    (
    s
    e
    l
    f
    , b
    u
    t
    t
    o
    n
    _
    n
    a
    m
    e
    )
    :
    s
    e
    l
    f
    .
    b
    u
    t
    t
    o
    n
    s
    .
    c
    l
    i
    c
    k
    (
    s
    e
    l
    f
    , b
    u
    t
    t
    o
    n
    _
    n
    a
    m
    e
    )

    View Slide

  14. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 14/16
    Composition Ex. (Cont'd)
    Here's an example of someone using the library class we built for composition:
    f
    r
    o
    m t
    v i
    m
    p
    o
    r
    t T
    V
    R
    e
    m
    o
    t
    e
    c
    l
    a
    s
    s B
    u
    t
    t
    o
    n
    s
    (
    o
    b
    j
    e
    c
    t
    )
    :
    d
    e
    f c
    l
    i
    c
    k
    (
    s
    e
    l
    f
    , r
    e
    m
    o
    t
    e
    , b
    u
    t
    t
    o
    n
    _
    n
    a
    m
    e
    )
    :
    i
    f b
    u
    t
    t
    o
    n
    _
    n
    a
    m
    e =
    = '
    b
    l
    u
    e
    '
    :
    r
    e
    m
    o
    t
    e
    .
    i
    n
    c
    r
    e
    m
    e
    n
    t
    _
    c
    h
    a
    n
    n
    e
    l
    (
    )
    b
    u
    t
    t
    o
    n
    s = B
    u
    t
    t
    o
    n
    s
    (
    )
    r
    e
    m
    o
    t
    e = T
    V
    R
    e
    m
    o
    t
    e
    (
    b
    u
    t
    t
    o
    n
    s
    )
    r
    e
    m
    o
    t
    e
    .
    c
    l
    i
    c
    k
    (
    '
    b
    l
    u
    e
    '
    )
    Composition (Cont'd)
    Composition is "take it or leave it" customizability. It's a good choice when a problem and interaction is
    well-defined and well-understood (and, if you're writing a library for other people to use, this should, by
    definition, be true). But it can be limiting in requirements-sparse environments where the problem is not yet
    well-defined or well-understood.
    It can be easier to use inheritance in a system where you control the horizontal and vertical while you're
    working out exactly what the relationship between objects should be. If you control the horizontal and
    vertical, you can always later switch from inheritance to composition once the problem is fully understood
    and people begin to want to reuse your code.
    Event Systems
    Specialized kind of composition.
    For example, instead of adding a o
    n
    _
    m
    o
    d
    i
    f
    i
    c
    a
    t
    i
    o
    n method of a class, and requiring that people
    subclass the class to override the method, have the would-be-superclass send an event to an event
    system. The event system can be subscribed to by system extenders as necessary.
    Event Systems (Cont'd)
    This is more flexible than subclassing too, because there's more than one entry point to extending
    behavior: an event can be subscribed to by any number of prospective listeners instead of just one.
    But systems reliant on event handling can be a bitch to understand and debug due to action-at-a-
    distance.

    View Slide

  15. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 15/16
    Event System Example
    c
    l
    a
    s
    s B
    u
    t
    t
    o
    n
    P
    r
    e
    s
    s
    (
    o
    b
    j
    e
    c
    t
    )
    :
    d
    e
    f _
    _
    i
    n
    i
    t
    _
    _
    (
    s
    e
    l
    f
    , r
    e
    m
    o
    t
    e
    , b
    n
    a
    m
    e
    )
    s
    e
    l
    f
    .
    r
    e
    m
    o
    t
    e = r
    e
    m
    o
    t
    e
    s
    e
    l
    f
    .
    b
    n
    a
    m
    e = b
    n
    a
    m
    e
    c
    l
    a
    s
    s T
    V
    R
    e
    m
    o
    t
    e
    (
    o
    b
    j
    e
    c
    t
    )
    :
    d
    e
    f _
    _
    i
    n
    i
    t
    _
    _
    (
    s
    e
    l
    f
    , e
    v
    e
    n
    t
    _
    s
    y
    s
    t
    e
    m
    )
    :
    s
    e
    l
    f
    .
    c
    h
    a
    n
    n
    e
    l = 0
    s
    e
    l
    f
    .
    e
    v
    e
    n
    t
    _
    s
    y
    s
    t
    e
    m = e
    v
    e
    n
    t
    _
    s
    y
    s
    t
    e
    m
    d
    e
    f c
    l
    i
    c
    k
    (
    s
    e
    l
    f
    , b
    n
    a
    m
    e
    )
    :
    s
    e
    l
    f
    .
    e
    v
    e
    n
    t
    _
    s
    y
    s
    t
    e
    m
    .
    n
    o
    t
    i
    f
    y
    (
    B
    u
    t
    t
    o
    n
    P
    r
    e
    s
    s
    (
    s
    e
    l
    f
    , b
    n
    a
    m
    e
    )
    )
    e
    v
    e
    n
    t
    _
    s
    y
    s
    t
    e
    m = E
    v
    e
    n
    t
    S
    y
    s
    t
    e
    m
    (
    )
    d
    e
    f s
    u
    b
    s
    c
    r
    i
    b
    e
    r
    (
    e
    v
    e
    n
    t
    )
    :
    i
    f e
    v
    e
    n
    t
    .
    b
    n
    a
    m
    e =
    = '
    b
    l
    u
    e
    '
    : e
    v
    e
    n
    t
    .
    r
    e
    m
    o
    t
    e
    .
    i
    n
    c
    r
    e
    m
    e
    n
    t
    _
    c
    h
    a
    n
    n
    e
    l
    (
    )
    e
    v
    e
    n
    t
    _
    s
    y
    s
    t
    e
    m
    .
    s
    u
    b
    s
    c
    r
    i
    b
    e
    (
    B
    u
    t
    t
    o
    n
    P
    r
    e
    s
    s
    , s
    u
    b
    s
    c
    r
    i
    b
    e
    r
    )
    r
    e
    m
    o
    t
    e = T
    V
    R
    e
    m
    o
    t
    e
    (
    e
    v
    e
    n
    t
    _
    s
    y
    s
    t
    e
    m
    )
    r
    e
    m
    o
    t
    e
    .
    c
    l
    i
    c
    k
    (
    '
    b
    l
    u
    e
    '
    )
    When To Offer Superclass
    When the behavior is absolutely fundamental to the spirit and intent of the library or framework (e.g.
    ZODB's P
    e
    r
    s
    i
    s
    t
    e
    n
    t
    ). Parent classes offered as slight variations on a theme (e.g. Django class-based
    views shipped as niceties) are not fundamental.
    A superclass offered by your library should almost always be abstract.
    Composition is harder for people to wrap their brains around.
    When a user inherits from a concrete parent class, he's usually inheriting from something that you haven't
    really designed for specialization, and it's likely that neither you nor he will be completely clear on what the
    specialization interface actually is. High likelihood for future breakage.
    I wish I had used inheritance in the case of an AuthTktAuthenticationPolicy instead of composition because
    I would have had to answer fewer questions about it. Python programmers will always understand the
    mechanics of inheritance better than whatever composition API you provide.
    Guidelines
    #1: Global State is Precious
    #2: Don't Design Exclusively For Convenience
    #3: Avoid Knobs on Knobs
    #4: Composition Usually Beats Inheritance

    View Slide

  16. 3/17/13 API Design for Libraries
    file://localhost/Volumes/Untitled/Uploaded/apidesign/presentation.html 16/16
    Contact Info
    Chris McDonough, Agendaless Consulting
    @chrismcdonough on Twitter
    "mcdonc" on Freenode IRC

    View Slide