Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to Serialize a Closure and Get a Million Downloads

How to Serialize a Closure and Get a Million Downloads

Did you know that you cannot serialize a Closure object! Well, that is what the PHP runtime tells you when you try to do it, anyway. Back in 2010, I set out to prove PHP wrong with my SuperClosure project, which allows you to serialize—and then actually unserialize and execute—PHP closures. I’ve learned a lot along the way, especially about about Closures, Reflection, and lexical parsing. I want to tell you the whole story, teach you some of PHP’s dark magic, and show you how some code snippets in a blog post evolved into a quirky open source project that has been downloaded over two million times, despite its usage of the infamous eval().

Jeremy Lindblom

May 21, 2015
Tweet

More Decks by Jeremy Lindblom

Other Decks in Programming

Transcript

  1. How to serialize()
    a \Closure and Get
    a Million Downloads
    by Jeremy Lindblom – @jeremeamia

    View Slide

  2. How to serialize()
    a \Closure and Get
    3.6 Million Downloads
    by Jeremy Lindblom – @jeremeamia

    View Slide

  3. Let me tell you about myself...

    View Slide

  4. @jeremeamia
    Jeremy Lindblom the Software Developer

    View Slide

  5. @jeremeamia
    Jeremy Lindblom the Software Developer
    AWS SDK for PHP at Amazon
    @awsforphp

    View Slide

  6. @jeremeamia
    Jeremy Lindblom the Software Developer
    AWS SDK for PHP at Amazon
    @awsforphp
    President of the Seattle PHP User Group
    @seaphp & #pnwphp

    View Slide

  7. @jeremeamia
    Jeremy Lindblom the Software Developer
    AWS SDK for PHP at Amazon
    @awsforphp
    President of the Seattle PHP User Group
    @seaphp & #pnwphp
    PHPoetry and PHParodies
    @phpbard

    View Slide

  8. Once upon a time...

    View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. View Slide

  14. Closures

    View Slide

  15. Closures
    —or—
    Lambdas (λ)

    View Slide

  16. Closures
    —or—
    Lambdas
    —or—
    Anonymous Functions

    View Slide

  17. function ($a, $b) {
    return $a + $b;
    };
    Anonymous (a.k.a. no name)

    View Slide

  18. $add = function ($a, $b) {
    return $a + $b;
    };
    $result = $add(3, 7);
    // 10
    First-class Functions

    View Slide

  19. $b = 7;
    $add7 = function ($a) use ($b) {
    return $a + $b;
    };
    $result = $add7(3);
    // 10
    Closures

    View Slide

  20. function getAdder($b) {
    return function ($a) use ($b) {
    return $a + $b;
    }
    }
    $add7 = getAdder(7);
    First-class Functions + Closures

    View Slide

  21. $nums = [3, 6, 9];
    $res = array_map($add7, $nums);
    // [10, 13, 16]
    Higher-order Functions

    View Slide

  22. $closure = function () {...};
    So what is this thing?

    View Slide

  23. is_callable($closure);
    // true
    Closure Objects

    View Slide

  24. is_callable($closure);
    // true
    is_object($closure);
    // true
    Closure Objects

    View Slide

  25. is_callable($closure);
    // true
    is_object($closure);
    // true
    Closure Objects
    Like
    __invoke()

    View Slide

  26. is_callable($closure);
    // true
    is_object($closure);
    // true
    get_class($closure);
    // Closure
    Closure Objects

    View Slide

  27. is_callable($closure);
    // true
    is_object($closure);
    // true
    get_class($closure);
    // Closure
    Closure Objects
    “This fact used to be considered an implementation
    detail, but it can now be relied upon.” - PHP Manual

    View Slide

  28. get_class_methods($closure);
    // array()
    Closure Objects (cont.)

    View Slide

  29. serialize($closure);
    // ???
    Closure Objects (still cont.)

    View Slide

  30. Fatal error: Uncaught
    exception 'Exception' with
    message 'Serialization of
    'Closure' is not allowed'

    View Slide

  31. Back to the Story

    View Slide

  32. View Slide

  33. serialize($closure);

    View Slide

  34. Fatal error: Uncaught
    exception 'Exception' with
    message 'Serialization of
    'Closure' is not allowed'

    View Slide

  35. Challenge Accepted

    View Slide

  36. View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. Really ?

    View Slide

  42. View Slide

  43. Thanks, Laravel!

    View Slide

  44. SuperClosure Today

    View Slide

  45. How to Get
    a Million Downloads

    View Slide

  46. ★ Solve a problem
    ★ Find a niche
    ★ Make it easy to use
    How to Get a Million Downloads
    If you build it, they will come

    View Slide

  47. ★ Build trust in your project
    ○ Address issues promptly
    ○ Follow SemVer
    ★ Build a community
    ○ Become a dependency to a larger project
    ○ Recruit contributors; reduce “bus factor”
    How to Get a Million Downloads (cont.)
    If you maintain it, they will stay

    View Slide

  48. How to serialize()
    a \Closure

    View Slide

  49. $fn = function ($a) use ($b, $c) {
    return ($a + $b) * $c;
    };
    What we need to serialize
    We need to know
    the values of these.
    $context

    View Slide

  50. $fn = function ($a) use ($b, $c) {
    return ($a + $b) * $c;
    };
    What we need to serialize (cont.)
    And we need to
    know all of this.
    $code

    View Slide

  51. extract($context);
    eval(“\$closure = {$code};”);
    How do we unserialize?

    View Slide

  52. Interface for Serializing
    class SerializableClosure
    extends Closure
    implements Serializable
    {
    // ...
    }

    View Slide

  53. Fatal error: Class
    SerializableClosure may
    not inherit from
    final class (Closure)

    View Slide

  54. View Slide

  55. Interface for Serializing (Decorator)
    class SerializableClosure implements Serializable
    {
    private $closure;
    function __construct($closure) {…}
    function __invoke() {…}
    function serialize() {…}
    function unserialize($serialized) {…}
    }

    View Slide

  56. Context is easiest to get
    $a = 5; $b = 10;
    $fn = function () use ($a, $b) { … };
    $r = new ReflectionFunction($fn);
    $context = $r->getStaticVariables();
    // [‘a’ => 5, ‘b’ => 10]

    View Slide

  57. Context is easiest to get… maybe
    $a = 5; $b = 10;
    $fn = function () use ($a, $b) {
    static $c = 15;
    };
    // [‘a’ => 5, ‘b’ => 10, ‘c’ => 15]

    View Slide

  58. OK, How do we get the code?
    $r = new ReflectionFunction($closure);
    $file = $r->getFileName();
    $start = $r->getStartLine();
    $end = $r->getEndLine();
    $code = implode(‘’, array_slice(
    file($file), $start, $end-$start+1));

    View Slide

  59. token_get_all()
    parses the given source string into
    PHP language tokens using the
    Zend engine's lexical scanner.

    View Slide

  60. Use the lexer
    $tokens = token_get_all(‘[ [T_OPEN_TAG, '[T_VARIABLE, '$fn', 1],
    [T_WHITESPACE, ' ', 1],
    '=',
    [T_WHITESPACE, ' ', 1],
    [T_FUNCTION, 'function', 1], ...]

    View Slide

  61. Create a parser
    foreach ($tokens as $token) {
    switch ($step) {
    case 0:
    if ($token[0] === T_FUNCTION) {
    $keep[] = $token; $step++;
    }
    break;
    // More steps...
    }
    }

    View Slide

  62. Create a parser (cont.)
    ★ Throw out tokens before 1st T_FUNCTION
    ★ Analyze function signature
    ○ Verify names of “used” variables (i.e., the context)
    ★ Keep all tokens within function body
    ○ Track the brace level to help identify the end.

    View Slide

  63. Token parser pitfalls
    ★ Assumes first T_FUNCTION encountered
    ★ No context outside of function
    ○ What namespace/class is it in?
    ○ What happens to magic constants __CLASS__?

    View Slide

  64. Use an AST
    ★ AST = Abstract Syntax Tree
    ★ nikic/PHP-Parser is awesome!
    ★ Create tree data structure from entire file
    ○ Traverse to closure, keeping track of scopes
    ○ Replace some nodes with literal values as visited

    View Slide

  65. Example - Original
    namespace Foo;
    use Bar\Baz;
    class Fizz
    {
    public function buzz() {
    return function (Baz $baz) {
    return $baz->boop(__CLASS__);
    }
    }
    }

    View Slide

  66. Example - Token-based Result
    function (Baz $baz) {
    return $baz->boop(__CLASS__);
    }
    Code no longer
    knows the FQCN.
    ERROR
    Value is different
    from original function
    WTF

    View Slide

  67. Example - AST-based Result
    function (Bar\Baz $baz) {
    return $baz->boop(‘Foo\Fizz’);
    }
    FQCN injected into
    code. No loss of
    information.
    Magic constants
    replaced with literal
    values. No loss of
    information.

    View Slide

  68. AST parser pitfalls
    ★ SLOW. Really slow.

    View Slide

  69. Parsing Caveats
    ★ Two closures declared on the same line is
    too ambiguous handle. Oh well…
    ★ If closures use (&$vales, &$by, &$ref)
    then those references cannot be restored
    ○ Unless ref is to the closure (e.g., recursive closures)

    View Slide

  70. Unsolved Mysteries
    ★ If a closure is declared in a trait, how do
    you resolve the value of __CLASS__ to a
    literal value?
    (If you figure this out, please let me know.)

    View Slide

  71. Oh crap!
    What about bindings?
    (Curse you, PHP 5.4!)

    View Slide

  72. class Foo {
    private $bar;
    public function baz() {
    return function () {
    return $this->bar;
    };
    }
    }
    Closure Bindings
    Automatically
    creates a closure
    on $this.

    View Slide

  73. class Foo {
    private $bar;
    public function baz() {
    return function () {
    return 5;
    };
    }
    }
    Closure Bindings
    Automatically
    creates a closure
    on $this
    Even when you
    don’t need it.

    View Slide

  74. class Foo {
    private $bar;
    public function baz() {
    return function () {
    return 5;
    };
    }
    }
    Closure Bindings
    Automatically
    creates a closure
    on $this.
    Even when you
    don’t need it.
    UNLESS...

    View Slide

  75. class Foo {
    private $bar;
    public function baz() {
    return static function () {
    return 5;
    };
    }
    }
    Static Closures
    UNLESS...
    You make it static.

    View Slide

  76. View Slide

  77. $foo = new Foo;
    $letters = new ArrayObject(range(‘a’, ‘z’));
    $fn = $foo->baz(); // baz() returns a static closure
    $fn->bindTo($letters);
    “Warning: Cannot bind an
    instance to a static closure”
    Static is forever

    View Slide

  78. Handling Closure Bindings
    ★ Two parts:
    ○ Object ($this) is the object it is bound to.
    ○ Scope determines what is visible.
    ★ Affects usage of $this, self::, and
    static:: within the closure

    View Slide

  79. Getting Binding Information
    class Foo { public function bar() {
    return function () { … };
    }}
    $foo = new Foo();
    $fn = $foo->bar();
    $r = new ReflectionFunction($fn);
    $r->getClosureThis(); // $foo
    $r->getClosureScopeClass()->getName(); // “Foo”

    View Slide

  80. Setting Binding Information
    $newClosure = $closure->bindTo($newThis, $newScope);
    Notes:
    ★ You can omit $newScope, which means “keep the scope
    the same as it is now”.
    ★ Using null for $newThis makes the closure static.
    Remember, static is forever.

    View Slide

  81. (Un)Serializing Closure Bindings
    ★ Including object and scope in serialization
    ★ Rebind closure after unserialization
    ○ Tricky, because the object can be null in two cases
    ■ It’s static
    ■ It was not declared within a class

    View Slide

  82. How to unserialize()
    a \Closure and Scare
    the Hell Out of Everyone

    View Slide

  83. public function unserialize($serialized) {
    // Unserialize and reconstruct the closure.
    $this->data = unserialize($serialized);
    $this->closure = _reconstruct_closure($this->data);
    // Rebind the closure to its former binding and scope.
    if ($this->data['binding'] || $this->data['isStatic']) {
    $this->closure = $this->closure->bindTo(
    $this->data['binding'],
    $this->data['scope']
    );
    }
    }

    View Slide

  84. And now, the most
    frightening function
    ever written...

    View Slide

  85. function _reconstruct_closure(array $__data)
    {
    extract($__data['context']);
    if ($__fn = array_search(RECURSION, $__data['context'])) {
    @eval("\${$__fn} = {$__data['code']};");
    $__closure = $$__fn;
    } else {
    @eval("\$__closure = {$__data['code']};");
    }
    return $__closure;
    }

    View Slide

  86. function _reconstruct_closure(array $__data)
    {
    extract($__data['context']);
    if ($__fn = array_search(RECURSION, $__data['context'])) {
    @eval("\${$__fn} = {$__data['code']};");
    $__closure = $$__fn;
    } else {
    @eval("\$__closure = {$__data['code']};");
    }
    return $__closure;
    }

    View Slide

  87. function _reconstruct_closure(array $__data)
    {
    extract($__data['context']);
    if ($__fn = array_search(RECURSION, $__data['context'])) {
    @eval("\${$__fn} = {$__data['code']};");
    $__closure = $$__fn;
    } else {
    @eval("\$__closure = {$__data['code']};");
    }
    return $__closure;
    }

    View Slide

  88. function _reconstruct_closure(array $__data)
    {
    extract($__data['context']);
    if ($__fn = array_search(RECURSION, $__data['context'])) {
    @eval("\${$__fn} = {$__data['code']};");
    $__closure = $$__fn;
    } else {
    @eval("\$__closure = {$__data['code']};");
    }
    return $__closure;
    }

    View Slide

  89. function _reconstruct_closure(array $__data)
    {
    extract($__data['context']);
    if ($__fn = array_search(RECURSION, $__data['context'])) {
    @eval("\${$__fn} = {$__data['code']};");
    $__closure = $$__fn;
    } else {
    @eval("\$__closure = {$__data['code']};");
    }
    return $__closure;
    }

    View Slide

  90. The End
    ★ Closures are weird
    ★ Serializing closures “is not allowed”, but you
    can still do it, if you really want to.
    ★ Reflection is awesome; so is PHP-Parser
    ★ Open Source is fun

    View Slide

  91. How to serialize()
    a \Closure and Get
    a Million Downloads
    by Jeremy Lindblom – @jeremeamia

    View Slide