How to Serialize a Closure and Get a Million Downloads

How to Serialize a Closure and Get a Million Downloads

Did you know that you cannot serialize a Closure object! Well, that is what the PHP runtime tells you when you try to do it, anyway. Back in 2010, I set out to prove PHP wrong with my SuperClosure project, which allows you to serialize—and then actually unserialize and execute—PHP closures. I’ve learned a lot along the way, especially about about Closures, Reflection, and lexical parsing. I want to tell you the whole story, teach you some of PHP’s dark magic, and show you how some code snippets in a blog post evolved into a quirky open source project that has been downloaded over two million times, despite its usage of the infamous eval().

Ca57a7cfac69ba3abf517470f3770aae?s=128

Jeremy Lindblom

May 21, 2015
Tweet

Transcript

  1. How to serialize() a \Closure and Get a Million Downloads

    by Jeremy Lindblom – @jeremeamia
  2. How to serialize() a \Closure and Get 3.6 Million Downloads

    by Jeremy Lindblom – @jeremeamia
  3. Let me tell you about myself...

  4. @jeremeamia Jeremy Lindblom the Software Developer

  5. @jeremeamia Jeremy Lindblom the Software Developer AWS SDK for PHP

    at Amazon @awsforphp
  6. @jeremeamia Jeremy Lindblom the Software Developer AWS SDK for PHP

    at Amazon @awsforphp President of the Seattle PHP User Group @seaphp & #pnwphp
  7. @jeremeamia Jeremy Lindblom the Software Developer AWS SDK for PHP

    at Amazon @awsforphp President of the Seattle PHP User Group @seaphp & #pnwphp PHPoetry and PHParodies @phpbard
  8. Once upon a time...

  9. None
  10. None
  11. None
  12. None
  13. None
  14. Closures

  15. Closures —or— Lambdas (λ)

  16. Closures —or— Lambdas —or— Anonymous Functions

  17. function ($a, $b) { return $a + $b; }; Anonymous

    (a.k.a. no name)
  18. $add = function ($a, $b) { return $a + $b;

    }; $result = $add(3, 7); // 10 First-class Functions
  19. $b = 7; $add7 = function ($a) use ($b) {

    return $a + $b; }; $result = $add7(3); // 10 Closures
  20. function getAdder($b) { return function ($a) use ($b) { return

    $a + $b; } } $add7 = getAdder(7); First-class Functions + Closures
  21. $nums = [3, 6, 9]; $res = array_map($add7, $nums); //

    [10, 13, 16] Higher-order Functions
  22. $closure = function () {...}; So what is this thing?

  23. is_callable($closure); // true Closure Objects

  24. is_callable($closure); // true is_object($closure); // true Closure Objects

  25. is_callable($closure); // true is_object($closure); // true Closure Objects Like __invoke()

  26. is_callable($closure); // true is_object($closure); // true get_class($closure); // Closure Closure

    Objects
  27. is_callable($closure); // true is_object($closure); // true get_class($closure); // Closure Closure

    Objects “This fact used to be considered an implementation detail, but it can now be relied upon.” - PHP Manual
  28. get_class_methods($closure); // array() Closure Objects (cont.)

  29. serialize($closure); // ??? Closure Objects (still cont.)

  30. Fatal error: Uncaught exception 'Exception' with message 'Serialization of 'Closure'

    is not allowed'
  31. Back to the Story

  32. None
  33. serialize($closure);

  34. Fatal error: Uncaught exception 'Exception' with message 'Serialization of 'Closure'

    is not allowed'
  35. Challenge Accepted

  36. None
  37. None
  38. None
  39. None
  40. None
  41. Really ?

  42. None
  43. Thanks, Laravel!

  44. SuperClosure Today

  45. How to Get a Million Downloads

  46. ★ Solve a problem ★ Find a niche ★ Make

    it easy to use How to Get a Million Downloads If you build it, they will come
  47. ★ Build trust in your project ◦ Address issues promptly

    ◦ Follow SemVer ★ Build a community ◦ Become a dependency to a larger project ◦ Recruit contributors; reduce “bus factor” How to Get a Million Downloads (cont.) If you maintain it, they will stay
  48. How to serialize() a \Closure

  49. $fn = function ($a) use ($b, $c) { return ($a

    + $b) * $c; }; What we need to serialize We need to know the values of these. $context
  50. $fn = function ($a) use ($b, $c) { return ($a

    + $b) * $c; }; What we need to serialize (cont.) And we need to know all of this. $code
  51. extract($context); eval(“\$closure = {$code};”); How do we unserialize?

  52. Interface for Serializing class SerializableClosure extends Closure implements Serializable {

    // ... }
  53. Fatal error: Class SerializableClosure may not inherit from final class

    (Closure)
  54. None
  55. Interface for Serializing (Decorator) class SerializableClosure implements Serializable { private

    $closure; function __construct($closure) {…} function __invoke() {…} function serialize() {…} function unserialize($serialized) {…} }
  56. Context is easiest to get $a = 5; $b =

    10; $fn = function () use ($a, $b) { … }; $r = new ReflectionFunction($fn); $context = $r->getStaticVariables(); // [‘a’ => 5, ‘b’ => 10]
  57. Context is easiest to get… maybe $a = 5; $b

    = 10; $fn = function () use ($a, $b) { static $c = 15; }; // [‘a’ => 5, ‘b’ => 10, ‘c’ => 15]
  58. OK, How do we get the code? $r = new

    ReflectionFunction($closure); $file = $r->getFileName(); $start = $r->getStartLine(); $end = $r->getEndLine(); $code = implode(‘’, array_slice( file($file), $start, $end-$start+1));
  59. token_get_all() parses the given source string into PHP language tokens

    using the Zend engine's lexical scanner.
  60. Use the lexer $tokens = token_get_all(‘<?php ’.$code); [ [T_OPEN_TAG, '<?php

    ', 1], [T_VARIABLE, '$fn', 1], [T_WHITESPACE, ' ', 1], '=', [T_WHITESPACE, ' ', 1], [T_FUNCTION, 'function', 1], ...]
  61. Create a parser foreach ($tokens as $token) { switch ($step)

    { case 0: if ($token[0] === T_FUNCTION) { $keep[] = $token; $step++; } break; // More steps... } }
  62. Create a parser (cont.) ★ Throw out tokens before 1st

    T_FUNCTION ★ Analyze function signature ◦ Verify names of “used” variables (i.e., the context) ★ Keep all tokens within function body ◦ Track the brace level to help identify the end.
  63. Token parser pitfalls ★ Assumes first T_FUNCTION encountered ★ No

    context outside of function ◦ What namespace/class is it in? ◦ What happens to magic constants __CLASS__?
  64. Use an AST ★ AST = Abstract Syntax Tree ★

    nikic/PHP-Parser is awesome! ★ Create tree data structure from entire file ◦ Traverse to closure, keeping track of scopes ◦ Replace some nodes with literal values as visited
  65. Example - Original namespace Foo; use Bar\Baz; class Fizz {

    public function buzz() { return function (Baz $baz) { return $baz->boop(__CLASS__); } } }
  66. Example - Token-based Result function (Baz $baz) { return $baz->boop(__CLASS__);

    } Code no longer knows the FQCN. ERROR Value is different from original function WTF
  67. Example - AST-based Result function (Bar\Baz $baz) { return $baz->boop(‘Foo\Fizz’);

    } FQCN injected into code. No loss of information. Magic constants replaced with literal values. No loss of information.
  68. AST parser pitfalls ★ SLOW. Really slow.

  69. Parsing Caveats ★ Two closures declared on the same line

    is too ambiguous handle. Oh well… ★ If closures use (&$vales, &$by, &$ref) then those references cannot be restored ◦ Unless ref is to the closure (e.g., recursive closures)
  70. Unsolved Mysteries ★ If a closure is declared in a

    trait, how do you resolve the value of __CLASS__ to a literal value? (If you figure this out, please let me know.)
  71. Oh crap! What about bindings? (Curse you, PHP 5.4!)

  72. class Foo { private $bar; public function baz() { return

    function () { return $this->bar; }; } } Closure Bindings Automatically creates a closure on $this.
  73. class Foo { private $bar; public function baz() { return

    function () { return 5; }; } } Closure Bindings Automatically creates a closure on $this Even when you don’t need it.
  74. class Foo { private $bar; public function baz() { return

    function () { return 5; }; } } Closure Bindings Automatically creates a closure on $this. Even when you don’t need it. UNLESS...
  75. class Foo { private $bar; public function baz() { return

    static function () { return 5; }; } } Static Closures UNLESS... You make it static.
  76. None
  77. $foo = new Foo; $letters = new ArrayObject(range(‘a’, ‘z’)); $fn

    = $foo->baz(); // baz() returns a static closure $fn->bindTo($letters); “Warning: Cannot bind an instance to a static closure” Static is forever
  78. Handling Closure Bindings ★ Two parts: ◦ Object ($this) is

    the object it is bound to. ◦ Scope determines what is visible. ★ Affects usage of $this, self::, and static:: within the closure
  79. Getting Binding Information class Foo { public function bar() {

    return function () { … }; }} $foo = new Foo(); $fn = $foo->bar(); $r = new ReflectionFunction($fn); $r->getClosureThis(); // $foo $r->getClosureScopeClass()->getName(); // “Foo”
  80. Setting Binding Information $newClosure = $closure->bindTo($newThis, $newScope); Notes: ★ You

    can omit $newScope, which means “keep the scope the same as it is now”. ★ Using null for $newThis makes the closure static. Remember, static is forever.
  81. (Un)Serializing Closure Bindings ★ Including object and scope in serialization

    ★ Rebind closure after unserialization ◦ Tricky, because the object can be null in two cases ▪ It’s static ▪ It was not declared within a class
  82. How to unserialize() a \Closure and Scare the Hell Out

    of Everyone
  83. public function unserialize($serialized) { // Unserialize and reconstruct the closure.

    $this->data = unserialize($serialized); $this->closure = _reconstruct_closure($this->data); // Rebind the closure to its former binding and scope. if ($this->data['binding'] || $this->data['isStatic']) { $this->closure = $this->closure->bindTo( $this->data['binding'], $this->data['scope'] ); } }
  84. And now, the most frightening function ever written...

  85. function _reconstruct_closure(array $__data) { extract($__data['context']); if ($__fn = array_search(RECURSION, $__data['context']))

    { @eval("\${$__fn} = {$__data['code']};"); $__closure = $$__fn; } else { @eval("\$__closure = {$__data['code']};"); } return $__closure; }
  86. function _reconstruct_closure(array $__data) { extract($__data['context']); if ($__fn = array_search(RECURSION, $__data['context']))

    { @eval("\${$__fn} = {$__data['code']};"); $__closure = $$__fn; } else { @eval("\$__closure = {$__data['code']};"); } return $__closure; }
  87. function _reconstruct_closure(array $__data) { extract($__data['context']); if ($__fn = array_search(RECURSION, $__data['context']))

    { @eval("\${$__fn} = {$__data['code']};"); $__closure = $$__fn; } else { @eval("\$__closure = {$__data['code']};"); } return $__closure; }
  88. function _reconstruct_closure(array $__data) { extract($__data['context']); if ($__fn = array_search(RECURSION, $__data['context']))

    { @eval("\${$__fn} = {$__data['code']};"); $__closure = $$__fn; } else { @eval("\$__closure = {$__data['code']};"); } return $__closure; }
  89. function _reconstruct_closure(array $__data) { extract($__data['context']); if ($__fn = array_search(RECURSION, $__data['context']))

    { @eval("\${$__fn} = {$__data['code']};"); $__closure = $$__fn; } else { @eval("\$__closure = {$__data['code']};"); } return $__closure; }
  90. The End ★ Closures are weird ★ Serializing closures “is

    not allowed”, but you can still do it, if you really want to. ★ Reflection is awesome; so is PHP-Parser ★ Open Source is fun
  91. How to serialize() a \Closure and Get a Million Downloads

    by Jeremy Lindblom – @jeremeamia