How to Serialize a Closure and Get a Million Downloads

Did you know that you cannot serialize a Closure object! Well, that is what the PHP runtime tells you when you try to do it, anyway. Back in 2010, I set out to prove PHP wrong with my SuperClosure project, which allows you to serialize—and then actually unserialize and execute—PHP closures. I’ve learned a lot along the way, especially about about Closures, Reflection, and lexical parsing. I want to tell you the whole story, teach you some of PHP’s dark magic, and show you how some code snippets in a blog post evolved into a quirky open source project that has been downloaded over two million times, despite its usage of the infamous eval().

Transcript

1. How to serialize()
   a \Closure and Get
   a Million Downloads
   by Jeremy Lindblom – @jeremeamia
   

   View Slide

2. How to serialize()
   a \Closure and Get
   3.6 Million Downloads
   by Jeremy Lindblom – @jeremeamia
   

   View Slide

3. Let me tell you about myself...
   

   View Slide

4. @jeremeamia
   Jeremy Lindblom the Software Developer
   

   View Slide

5. @jeremeamia
   Jeremy Lindblom the Software Developer
   AWS SDK for PHP at Amazon
   @awsforphp
   

   View Slide

6. @jeremeamia
   Jeremy Lindblom the Software Developer
   AWS SDK for PHP at Amazon
   @awsforphp
   President of the Seattle PHP User Group
   @seaphp & #pnwphp
   

   View Slide

7. @jeremeamia
   Jeremy Lindblom the Software Developer
   AWS SDK for PHP at Amazon
   @awsforphp
   President of the Seattle PHP User Group
   @seaphp & #pnwphp
   PHPoetry and PHParodies
   @phpbard
   

   View Slide

8. Once upon a time...
   

   View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. Closures
    

    View Slide

15. Closures
    —or—
    Lambdas (λ)
    

    View Slide

16. Closures
    —or—
    Lambdas
    —or—
    Anonymous Functions
    

    View Slide

17. function ($a, $b) {
    return $a + $b;
    };
    Anonymous (a.k.a. no name)
    

    View Slide

18. $add = function ($a, $b) {
    return $a + $b;
    };
    $result = $add(3, 7);
    // 10
    First-class Functions
    

    View Slide

19. $b = 7;
    $add7 = function ($a) use ($b) {
    return $a + $b;
    };
    $result = $add7(3);
    // 10
    Closures
    

    View Slide

20. function getAdder($b) {
    return function ($a) use ($b) {
    return $a + $b;
    }
    }
    $add7 = getAdder(7);
    First-class Functions + Closures
    

    View Slide

21. $nums = [3, 6, 9];
    $res = array_map($add7, $nums);
    // [10, 13, 16]
    Higher-order Functions
    

    View Slide

22. $closure = function () {...};
    So what is this thing?
    

    View Slide

23. is_callable($closure);
    // true
    Closure Objects
    

    View Slide

24. is_callable($closure);
    // true
    is_object($closure);
    // true
    Closure Objects
    

    View Slide

25. is_callable($closure);
    // true
    is_object($closure);
    // true
    Closure Objects
    Like
    __invoke()
    

    View Slide

26. is_callable($closure);
    // true
    is_object($closure);
    // true
    get_class($closure);
    // Closure
    Closure Objects
    

    View Slide

27. is_callable($closure);
    // true
    is_object($closure);
    // true
    get_class($closure);
    // Closure
    Closure Objects
    “This fact used to be considered an implementation
    detail, but it can now be relied upon.” - PHP Manual
    

    View Slide

28. get_class_methods($closure);
    // array()
    Closure Objects (cont.)
    

    View Slide

29. serialize($closure);
    // ???
    Closure Objects (still cont.)
    

    View Slide

30. Fatal error: Uncaught
    exception 'Exception' with
    message 'Serialization of
    'Closure' is not allowed'
    

    View Slide

31. Back to the Story
    

    View Slide

32. View Slide

33. serialize($closure);
    

    View Slide

34. Fatal error: Uncaught
    exception 'Exception' with
    message 'Serialization of
    'Closure' is not allowed'
    

    View Slide

35. Challenge Accepted
    

    View Slide

36. View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. Really ?
    

    View Slide

42. View Slide

43. Thanks, Laravel!
    

    View Slide

44. SuperClosure Today
    

    View Slide

45. How to Get
    a Million Downloads
    

    View Slide

46. ★ Solve a problem
    ★ Find a niche
    ★ Make it easy to use
    How to Get a Million Downloads
    If you build it, they will come
    

    View Slide

47. ★ Build trust in your project
    ○ Address issues promptly
    ○ Follow SemVer
    ★ Build a community
    ○ Become a dependency to a larger project
    ○ Recruit contributors; reduce “bus factor”
    How to Get a Million Downloads (cont.)
    If you maintain it, they will stay
    

    View Slide

48. How to serialize()
    a \Closure
    

    View Slide

49. $fn = function ($a) use ($b, $c) {
    return ($a + $b) * $c;
    };
    What we need to serialize
    We need to know
    the values of these.
    $context
    

    View Slide

50. $fn = function ($a) use ($b, $c) {
    return ($a + $b) * $c;
    };
    What we need to serialize (cont.)
    And we need to
    know all of this.
    $code
    

    View Slide

51. extract($context);
    eval(“\$closure = {$code};”);
    How do we unserialize?
    

    View Slide

52. Interface for Serializing
    class SerializableClosure
    extends Closure
    implements Serializable
    {
    // ...
    }
    

    View Slide

53. Fatal error: Class
    SerializableClosure may
    not inherit from
    final class (Closure)
    

    View Slide

54. View Slide

55. Interface for Serializing (Decorator)
    class SerializableClosure implements Serializable
    {
    private $closure;
    function __construct($closure) {…}
    function __invoke() {…}
    function serialize() {…}
    function unserialize($serialized) {…}
    }
    

    View Slide

56. Context is easiest to get
    $a = 5; $b = 10;
    $fn = function () use ($a, $b) { … };
    $r = new ReflectionFunction($fn);
    $context = $r->getStaticVariables();
    // [‘a’ => 5, ‘b’ => 10]
    

    View Slide

57. Context is easiest to get… maybe
    $a = 5; $b = 10;
    $fn = function () use ($a, $b) {
    static $c = 15;
    };
    // [‘a’ => 5, ‘b’ => 10, ‘c’ => 15]
    

    View Slide

58. OK, How do we get the code?
    $r = new ReflectionFunction($closure);
    $file = $r->getFileName();
    $start = $r->getStartLine();
    $end = $r->getEndLine();
    $code = implode(‘’, array_slice(
    file($file), $start, $end-$start+1));
    

    View Slide

59. token_get_all()
    parses the given source string into
    PHP language tokens using the
    Zend engine's lexical scanner.
    

    View Slide

60. Use the lexer
    $tokens = token_get_all(‘[ [T_OPEN_TAG, '[T_VARIABLE, '$fn', 1],
    [T_WHITESPACE, ' ', 1],
    '=',
    [T_WHITESPACE, ' ', 1],
    [T_FUNCTION, 'function', 1], ...]
    

    View Slide

61. Create a parser
    foreach ($tokens as $token) {
    switch ($step) {
    case 0:
    if ($token[0] === T_FUNCTION) {
    $keep[] = $token; $step++;
    }
    break;
    // More steps...
    }
    }
    

    View Slide

62. Create a parser (cont.)
    ★ Throw out tokens before 1st T_FUNCTION
    ★ Analyze function signature
    ○ Verify names of “used” variables (i.e., the context)
    ★ Keep all tokens within function body
    ○ Track the brace level to help identify the end.
    

    View Slide

63. Token parser pitfalls
    ★ Assumes first T_FUNCTION encountered
    ★ No context outside of function
    ○ What namespace/class is it in?
    ○ What happens to magic constants __CLASS__?
    

    View Slide

64. Use an AST
    ★ AST = Abstract Syntax Tree
    ★ nikic/PHP-Parser is awesome!
    ★ Create tree data structure from entire file
    ○ Traverse to closure, keeping track of scopes
    ○ Replace some nodes with literal values as visited
    

    View Slide

65. Example - Original
    namespace Foo;
    use Bar\Baz;
    class Fizz
    {
    public function buzz() {
    return function (Baz $baz) {
    return $baz->boop(__CLASS__);
    }
    }
    }
    

    View Slide

66. Example - Token-based Result
    function (Baz $baz) {
    return $baz->boop(__CLASS__);
    }
    Code no longer
    knows the FQCN.
    ERROR
    Value is different
    from original function
    WTF
    

    View Slide

67. Example - AST-based Result
    function (Bar\Baz $baz) {
    return $baz->boop(‘Foo\Fizz’);
    }
    FQCN injected into
    code. No loss of
    information.
    Magic constants
    replaced with literal
    values. No loss of
    information.
    

    View Slide

68. AST parser pitfalls
    ★ SLOW. Really slow.
    

    View Slide

69. Parsing Caveats
    ★ Two closures declared on the same line is
    too ambiguous handle. Oh well…
    ★ If closures use (&$vales, &$by, &$ref)
    then those references cannot be restored
    ○ Unless ref is to the closure (e.g., recursive closures)
    

    View Slide

70. Unsolved Mysteries
    ★ If a closure is declared in a trait, how do
    you resolve the value of __CLASS__ to a
    literal value?
    (If you figure this out, please let me know.)
    

    View Slide

71. Oh crap!
    What about bindings?
    (Curse you, PHP 5.4!)
    

    View Slide

72. class Foo {
    private $bar;
    public function baz() {
    return function () {
    return $this->bar;
    };
    }
    }
    Closure Bindings
    Automatically
    creates a closure
    on $this.
    

    View Slide

73. class Foo {
    private $bar;
    public function baz() {
    return function () {
    return 5;
    };
    }
    }
    Closure Bindings
    Automatically
    creates a closure
    on $this
    Even when you
    don’t need it.
    

    View Slide

74. class Foo {
    private $bar;
    public function baz() {
    return function () {
    return 5;
    };
    }
    }
    Closure Bindings
    Automatically
    creates a closure
    on $this.
    Even when you
    don’t need it.
    UNLESS...
    

    View Slide

75. class Foo {
    private $bar;
    public function baz() {
    return static function () {
    return 5;
    };
    }
    }
    Static Closures
    UNLESS...
    You make it static.
    

    View Slide

76. View Slide

77. $foo = new Foo;
    $letters = new ArrayObject(range(‘a’, ‘z’));
    $fn = $foo->baz(); // baz() returns a static closure
    $fn->bindTo($letters);
    “Warning: Cannot bind an
    instance to a static closure”
    Static is forever
    

    View Slide

78. Handling Closure Bindings
    ★ Two parts:
    ○ Object ($this) is the object it is bound to.
    ○ Scope determines what is visible.
    ★ Affects usage of $this, self::, and
    static:: within the closure
    

    View Slide

79. Getting Binding Information
    class Foo { public function bar() {
    return function () { … };
    }}
    $foo = new Foo();
    $fn = $foo->bar();
    $r = new ReflectionFunction($fn);
    $r->getClosureThis(); // $foo
    $r->getClosureScopeClass()->getName(); // “Foo”
    

    View Slide

80. Setting Binding Information
    $newClosure = $closure->bindTo($newThis, $newScope);
    Notes:
    ★ You can omit $newScope, which means “keep the scope
    the same as it is now”.
    ★ Using null for $newThis makes the closure static.
    Remember, static is forever.
    

    View Slide

81. (Un)Serializing Closure Bindings
    ★ Including object and scope in serialization
    ★ Rebind closure after unserialization
    ○ Tricky, because the object can be null in two cases
    ■ It’s static
    ■ It was not declared within a class
    

    View Slide

82. How to unserialize()
    a \Closure and Scare
    the Hell Out of Everyone
    

    View Slide

83. public function unserialize($serialized) {
    // Unserialize and reconstruct the closure.
    $this->data = unserialize($serialized);
    $this->closure = _reconstruct_closure($this->data);
    // Rebind the closure to its former binding and scope.
    if ($this->data['binding'] || $this->data['isStatic']) {
    $this->closure = $this->closure->bindTo(
    $this->data['binding'],
    $this->data['scope']
    );
    }
    }
    

    View Slide

84. And now, the most
    frightening function
    ever written...
    

    View Slide

85. function _reconstruct_closure(array $__data)
    {
    extract($__data['context']);
    if ($__fn = array_search(RECURSION, $__data['context'])) {
    @eval("\${$__fn} = {$__data['code']};");
    $__closure = $$__fn;
    } else {
    @eval("\$__closure = {$__data['code']};");
    }
    return $__closure;
    }
    

    View Slide

86. function _reconstruct_closure(array $__data)
    {
    extract($__data['context']);
    if ($__fn = array_search(RECURSION, $__data['context'])) {
    @eval("\${$__fn} = {$__data['code']};");
    $__closure = $$__fn;
    } else {
    @eval("\$__closure = {$__data['code']};");
    }
    return $__closure;
    }
    

    View Slide

87. function _reconstruct_closure(array $__data)
    {
    extract($__data['context']);
    if ($__fn = array_search(RECURSION, $__data['context'])) {
    @eval("\${$__fn} = {$__data['code']};");
    $__closure = $$__fn;
    } else {
    @eval("\$__closure = {$__data['code']};");
    }
    return $__closure;
    }
    

    View Slide

88. function _reconstruct_closure(array $__data)
    {
    extract($__data['context']);
    if ($__fn = array_search(RECURSION, $__data['context'])) {
    @eval("\${$__fn} = {$__data['code']};");
    $__closure = $$__fn;
    } else {
    @eval("\$__closure = {$__data['code']};");
    }
    return $__closure;
    }
    

    View Slide

89. function _reconstruct_closure(array $__data)
    {
    extract($__data['context']);
    if ($__fn = array_search(RECURSION, $__data['context'])) {
    @eval("\${$__fn} = {$__data['code']};");
    $__closure = $$__fn;
    } else {
    @eval("\$__closure = {$__data['code']};");
    }
    return $__closure;
    }
    

    View Slide

90. The End
    ★ Closures are weird
    ★ Serializing closures “is not allowed”, but you
    can still do it, if you really want to.
    ★ Reflection is awesome; so is PHP-Parser
    ★ Open Source is fun
    

    View Slide

91. How to serialize()
    a \Closure and Get
    a Million Downloads
    by Jeremy Lindblom – @jeremeamia
    

    View Slide