Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Purely Functional Programming in Python: Pure Fun

Purely Functional Programming in Python: Pure Fun

This is an introduction to the concepts of purely functional programming in Python. I go over the concepts and techniques of FP, and also talk about specific libraries you can use to help you out with FP.

I gave this talk at PyTennessee 2015.

Christopher Armstrong (radix)

February 07, 2015
Tweet

Other Decks in Programming

Transcript

  1. pure fun
    in python
    yeah
    Christopher Armstrong
    https://github.com/radix/effect
    @radix
    - Introduction

    - Twisted, if you know it

    - Effect (talk about later)

    - Rackspace, Auto Scale, we do FP!

    - Go!

    View Slide

  2. i am not going to say it
    monad
    I will not say the word on this slide at any point during this talk.

    View Slide

  3. hundreds of years ago
    http://yellow5.com/pokey/archive/index264.html
    — @stevehavelka
    heard from someone that every great talk starts with a lengthy section on history.

    - Pascal, the first programming language, had two features: functions and procedures.

    - Functions return, procedures had side-effects.

    - we made a distinction then, most modern languages don’t.

    View Slide

  4. “pure”
    Function, n. (plural functions)
    A relation between a set of inputs and a set of
    permissible outputs with the property that each
    input is related to exactly one output.
    -Wikipedia
    - title of this talk is “PURELY functional programming”

    - shouldn’t have to say that.

    - a distinction was lost: “functions” changed meaning.

    -

    - I’ve also heard that every great talk has definition slides.

    -

    - No room for side-effects. Can’t delete money from your bank account.

    - Most don’t know/care when they write funcs or procedures.

    So, what is pure FP? just programming with functions. :-)

    View Slide

  5. not for fun >:(
    - not here to tell you this for fun

    - not just encouraging you to expand your mind

    - people just go back to familiarity

    - YOLO state manipulation and OO design are a big source of problems

    - pure FP seriously improves our code

    View Slide

  6. j/k it’s super fun :D
    - ok, ok, it’s actually really fun to do functional programming!

    - check it out!

    - learn some stuff!

    View Slide

  7. definition #2
    YOLO, v, t.
    Modifying state or performing IO in a function.
    For the rest of this talk, I’ll refer to side-effects as “YOLO”.

    View Slide

  8. why
    1. reason
    2. express
    3. test
    4. perform
    Let me tell you about why FP is good for you.

    - Reason about code in a way impossible with YOLO.

    - express code concisely/meaningfully. abstract mechanisms for composing funcs.

    - WAY easier to test. Call func with args, check result.

    - Opportunity for speed for PFP, parallelism

    View Slide

  9. 1. reason
    "See, I told you they'd listen to Reason.”
    -Fisheye
    (Snow Crash reference)
    - substantial effect on understanding of code

    - greater confidence

    - “equational reasoning”, function call = return value.

    - for snow crash reference

    - an example…

    View Slide

  10. reasoning
    f(a(‘foo’), b(c(‘bar’)))
    - code

    - If these functions YOLO, we need to carefully search for side-effects.

    - If we know they’re pure, it’s much easier. Let me demonstrate.

    View Slide

  11. reasoning
    >>> a(‘foo’)

    1

    >>> c(‘bar’)

    2

    >>> b(2)

    3

    >>> f(1, 3)

    “Everything is awesome!”
    - This is what it’s like to experiment and trace execution.

    - It’s just like when you learned Python. Try stuff out.

    - May seem trite. Surely, anyone can do this kind of stuff with their Python code?

    View Slide

  12. sad REPL syndrome
    foo.bar(3)?
    Foo(dbconn, auth)
    INSERT INTO txn_log (event, owner, value
    mock.return_value = MockAuthenticationResponse()
    def foo(val):

    print stuff
    return stuff
    Q: ever worked on a big project where it’s hard use the REPL?

    “understand foo.bar(3)”. Easy?

    but class needs DB connobj, authobj -> HTTP req,

    and method YOLOs txn logs into DB.

    Have mocks in tests, -> 5 minutes to find & put together

    End up adding prints to func, try reproduce behavior in testing env.

    Happens to me! All the time!

    It’s a tragedy!

    slows dev cycle.

    View Slide

  13. happy REPL
    pure functions
    better factored dependencies
    :-)
    - If most code PF, you CAN use the REPL.

    - Functional code naturally minimizes deps between comps

    - end up w/ wonderfully factored little funcs, easy to evaluate in REPL.

    View Slide

  14. 2. express
    result = map(comp(f, g), inputs)
    not
    result = []

    for input in inputs:

    x = f(g(input))

    result.append(x)
    - FP lets us specify things in a way that exposes *meaning* more clearly

    - lots of FP advocates say that FP style is “say what you want, not how to get it”

    - it’s rhetoric, but has a grain of truth

    - more on this later when I talk about “style”

    View Slide

  15. 3. test
    call a function.
    pass it some arguments.
    check its return value.
    - Call func w/ args, check return. That’s it.

    - Not how most Python code is tested these days.

    - Mocking has become absolute *nightmare*. mocks -> mocks -> mocks

    - need track complex YOLOs

    - Tests too coupled to implementation. very hard to change your code.

    - care about this a lot. Really into unit testing. I like 100% test coverage.

    - I still mock (or parameterize deps), but the mocks are much more tasteful in FP

    View Slide

  16. 4. perform
    PyPy.
    STM.
    (go fast)
    - PyPy. Obviously.

    - Watch PyPy’s STM

    - STM optimistically runs your code concurrently

    - PFP = trivially parallel.

    - STM needs more work, but it’s the future. So much that it hurts. And so is FP!

    - Basically, STM and PFP will kill the GIL.

    View Slide

  17. purity
    FP in the large
    - Let’s talk about HOW we do pure functional programming.

    - Most people focus on stylistic parts of FP

    - FP in the large

    View Slide

  18. don’t change anything
    • list.append(“foo”)
    • self.foo = bar
    • global
    • x = 1; x = 2
    purity
    - don’t change anything

    - seems crazy? like many paradigms, not so hard once you get used to it

    View Slide

  19. lies
    def foo(coolstuff):

    coolstuff = list(coolstuff.items())

    coolstuff.append(('extrathing', 'value'))

    return coolstuff
    purity
    - but those are lies

    - only thing that matters is you don’t *expose* side-effects.

    View Slide

  20. python
    def assoc(d, k, v):

    new = d.copy()

    new[k] = v

    return new
    purity
    - local YOLO often necessary when doing pure FP in Python.

    - This function replaces a value in a dictionary by copying and mutating.

    - Pure FP, but local mutation. Easiest way to do it in this case.

    View Slide

  21. functional data
    purity/data
    toolz
    warehouse.python.org/project/toolz
    - Python has poor built-in data structures for FP.

    - Get stuff done by explicit copy/modify, but it’s inefficient and error-prone.

    - (e.g. nested mutable objects)

    - toolz is a nice lib that supplies useful functions.

    View Slide

  22. toolz
    purity/data
    >>> from toolz.dicttoolz import assoc

    >>> assoc({'a': 1}, 'b', 2)

    {'a': 1, 'b': 2}
    - here’s the assoc function from earlier. pretty much does what it looks like.

    - this is nice, for small data….

    View Slide

  23. big data (structures)
    • Zone.actors : [Actor]
    • Actor.health : int
    • (Actor.mana, Actor.inventory)
    • Actor.attack(target) -> reduce target’s health
    purity/data
    - e.g.: game simulation with big world.

    - thousands of actors, with health/mana/inventory nested inside

    - inventory may contain hundreds of items too

    - Copying the whole zone is insanity! How do we do this efficiently without mutation?

    View Slide

  24. persistent data
    purity/data
    pyrsistent
    warehouse.python.org/project/pyrsistent/
    - wonderful implementation of “persistent” data structures

    - (not on-disk; it means immutable)

    - incredibly, surprisingly efficient

    - shared structure!

    View Slide

  25. linked list
    a = (1, (2, ()))
    b = (0, a)
    purity/data
    - before I explain pyrsistent, an example of the idea of shared structure

    - pushing/popping a linked list is very efficient. Structure is shared.

    View Slide

  26. slow
    • push/pop: O(1)
    • anything else: O(n)
    purity/data
    - linked lists are limited.

    - push/pop is fast, but insert/index is slow.

    - Much more clever data structures exist now, like…

    View Slide

  27. (H)AMT
    purity/data
    (Hash) Array Mapped Tries
    - (HASH) Array Mapped Tries

    - becoming very common

    - used in the Clojure core data types

    - in the pyrsistent library for Python.

    View Slide

  28. (H)AMT
    purity/data
    - array of integers 0-8

    - array is chunked up into the leaves of the trie.

    - Indexing this structure is very essentially O(1), unlike linked lists, which require traversal

    View Slide

  29. (H)AMT
    purity/data
    - let’s replace ‘5’ with “beef”

    - mutation is also essentially O(1) (O(log32 N))!

    - only the nodes on the way to that chunk need to be allocated

    - all shared structure just gets referenced in the new vector.

    - brown is old, blue is new.

    View Slide

  30. zone -> zone
    purity/data
    from pyrsistent import m, v

    zone = m(actors=v(m(health=100, mana=100,

    inventory=v())))

    updated_zone = zone.set_in(

    (‘actors’, 0, ‘health’), 90)
    - Example of pyrsistent. Let’s see how to change someone’s health.

    - What do you do with a new world, though?

    - Well, honestly, to be practical you could just overwrite the old world.

    - Or you can treat event-handlers as functions of world -> world and continually run them in a purely functional way.

    - Magic word here would be FRP, but I’m not going to talk too much about that.

    View Slide

  31. IO
    purity/effects
    - You’ve been biting your tongue.

    - “This is just a bunch of faffing about. Can’t do _real_ programming with pure FP!”

    - Useful programs have to communicate with the outside world.

    - You’re right. But maybe we can still do something useful

    View Slide

  32. functional core,

    imperative shell
    purity/effects
    — Gary Bernhardt
    - Start simple.

    - the majority of your logic should be written as pure functions.

    - wrap the imperative logic that talks to DB or makes HTTP requests

    View Slide

  33. typical IO
    def foo():

    name = input("Name?")

    print(“Oh, %s is a nice name" % (name,))
    purity/effects
    - Here’s a typical IO-using function.

    - It looks tiny, right?

    - Let’s reduce it to its essence. Which parts can be factored out to be pure?

    View Slide

  34. refactoring
    def compliment(name):

    return "Oh, %s is a nice name" % (name,)
    def foo():

    print(compliment(input(“Name?”)))
    purity/effects
    - extract the part that takes the result of first IO and does the pure calculation based on it.

    - This is a trivial example of functional core/imperative shell, but you can extrapolate this idea to a lot of code.

    - In practice, a lot of your functions are already pure, there are a lot that aren’t, and could be.

    - Thinking in these terms will help you factor your apps much better.

    - I encourage you to go back to your code bases, find the side effects, and try to factor them out.

    View Slide

  35. down the rabbit hole
    def foo():

    print(compliment(input(“Name?”)))
    - But can we go deeper!????

    - Here are our side-effecting functions.

    - Could we represent the ideas in this code in a way that maintains purity?

    View Slide

  36. down the rabbit hole
    def foo():

    eff = Input("Name?")
    purity/effects
    @do

    def foo():

    name = yield Input(“Name?”)

    return Print(compliment(n))
    or
    return eff.on(lambda n: Print(compliment(n)))
    - Here’s an idea.

    - represent our side-effects as objects describing *intent* to perform an effect.

    - input became an Input object, print became a Print object.

    - `on` attaches a callback to deal with the eventual result of that IO.

    - This is a liiiiittle but of a tedious syntax, so…

    - Maybe we can take advantage of generator syntax to make it look nicer.

    - This is just like asyncio’s coroutines.

    - twisted also has inlineCallbacks

    View Slide

  37. results
    purity/effects
    @do

    def foo():

    name = yield Input(“Name?”)

    return Print(compliment(n))
    def main():

    perform(foo())
    - So what’s the point?

    - When we call it, we get back an introspectable object.

    - Test it.

    - In the end, actually perform the side-effects with a “perform” function.

    View Slide

  38. wink, nudge
    • Effect, warehouse.python.org/project/effect
    purity/effects
    - With a few minor syntactic differences, this is exactly how Effect works.

    - It’s rather extreme compared to the typical Python program

    - but similar to asyncio, Twisted, etc.

    - Not just for IO-based side-effects. also useful for RNG, mutable data writes, etc.

    View Slide

  39. style
    In the beginning was the function.

    All things issue from it; all things return to it.

    —Tao Te FP
    - That’s all I’m going to say about purity and side-effects.

    - Could say a lot more, but TIME.

    - Now I want to talk about the *style* of functional programming.

    - Lisp, Erlang, OCaml, are all about stylistic FP. They don’t isolate side-effects, but they do love everything-as-functions.

    View Slide

  40. style
    • higher-order functions
    • use functions for control flow
    • don’t reassign
    • be an abstraction astronaut
    style
    - “higher-order” DEFINE

    - Use functions for control flow. Ex, recursion == looping.

    - HO functions are used for control flow and abstraction

    - Don’t reassign variables - already talked about this

    - combine above = abstraction astronaut.

    View Slide

  41. higher-order
    result = map(comp(f, g), inputs)
    not
    result = []

    for input in inputs:

    x = f(g(input))

    result.append(x)
    style
    - I showed this earlier when talking about expressiveness.

    - Seems like a win! 2 HO funcs: map&compose.

    - map: apply a func to all elements in a list.

    - compose: double-whammy. accepts and returns functions.

    View Slide

  42. compose
    h = comp(f, g)
    same as
    def h(*args, **kwargs):

    return f(g(*args, **kwargs))
    style
    - FPers love these tiny abstractions, use them to great effect.

    - not just shorter lexically: avoids mention “arg” to funcs,*conceptually* simpler (“number of referenced names”).

    View Slide

  43. compose
    def comp(f, g):

    return lambda *a, **kw: f(g(*a, **kw))
    or
    from toolz import comp
    style
    We can define compose, or use toolz.

    View Slide

  44. filter/reduce
    >>> filter(lambda x: x % 2 == 0,

    ... [0,1,2])

    [0, 2]
    >>> reduce(operator.add, [1,2,3])

    6
    style
    more HO funcs

    View Slide

  45. list comprehensions
    >>> [x + 1 for x in [0,1,2,3,4]

    ... if x % 2 == 0]

    [1, 3, 5]
    >>> map(lambda x: x + 1,

    ... filter(lambda x: x % 2 == 0,

    … [0,1,2,3,4]))

    [1, 3, 5]
    style
    - syntax sugar for map/filter (not reduce)

    - often nicer > map/filter+lambdas,

    View Slide

  46. map/filter
    >>> map(add1, filter(is_even, [0,1,2]))

    [1, 3]
    style
    map/filter shine w/ named funcs

    View Slide

  47. recursion
    def factorial(n):

    if n == 0:

    return 1

    else:

    return n * factorial(n-1)
    style
    - Not a lot of recursion in Python. Uses up stack space, no TCE.

    View Slide

  48. abstract it away
    def factorial(n):

    if n == 0:

    return 1

    else:

    return n * factorial(n-1)
    vs
    def factorial(n):

    reduce(operator.mul, range(1, n))
    style
    But that implementation is actually pretty gross conceptually as well as performance-wise.

    Reduce is powerful.

    Reduce usually recursive in FPL, Py = iterative. large inputs = no p.

    View Slide

  49. astronaut
    add1all (first:rest) = first + 1 : add1all rest

    add1all [] = []
    or,
    def add1all(inputs):

    if not inputs: return []

    return [inputs[0] + 1] + add1all(inputs[1:])
    style
    - Functional programmers love abstractions.

    - Many patterns of recursion have been identified.

    - In ancient history, people often wrote things like this, and realized a pattern

    View Slide

  50. astronaut
    map f (head:rest) = f head : map f rest

    map f [] = []
    or
    def map(f, inputs):

    if not inputs: return []

    return [f(inputs[0])] + map(f, inputs[1:])
    style
    - Parameterize the per-element calculation and create `map`.

    - map, reduce, filter are all simple recursion recipes, extracted into HO funcs.

    - though recursion is an important part of FP, people prefer not to do it explicitly

    - Using the recipes makes it easier and quicker to understand what the code is doing

    - Even though Python hates recursion, these abstractions are implemented iteratively so we can use the same style.

    View Slide

  51. compose
    functional purity + functional style
    - Coming up on the end!

    - functional purity: idea of not exposing side-effects

    - functional style: use functions for everything!

    - doing pure functional programming leads to

    - easier reading

    - easier adding features/fixing bugs

    - easier refactoring

    - easier testing

    - Go forth and invoke a function!

    View Slide

  52. questions/refs
    • Pyrsistent, warehouse.python.org/project/pyrsistent
    • Toolz, warehouse.python.org/project/toolz
    • Effect, warehouse.python.org/project/effect
    • HAMT, http://hypirion.com/musings/understanding-
    persistent-vector-pt-1
    • Pokey the Penguin, http://yellow5.com/pokey/

    View Slide