How to Serialize a Closure and Get a Million Downloads

Did you know that you cannot serialize a Closure object! Well, that is what the PHP runtime tells you when you try to do it, anyway. Back in 2010, I set out to prove PHP wrong with my SuperClosure project, which allows you to serialize—and then actually unserialize and execute—PHP closures. I’ve learned a lot along the way, especially about about Closures, Reflection, and lexical parsing. I want to tell you the whole story, teach you some of PHP’s dark magic, and show you how some code snippets in a blog post evolved into a quirky open source project that has been downloaded over two million times, despite its usage of the infamous eval().

Transcript

1. How to serialize() a \Closure and Get a Million Downloads by Jeremy Lindblom – @jeremeamia
2. How to serialize() a \Closure and Get 3.6 Million Downloads by Jeremy Lindblom – @jeremeamia
3. Let me tell you about myself...
4. @jeremeamia Jeremy Lindblom the Software Developer
5. @jeremeamia Jeremy Lindblom the Software Developer AWS SDK for PHP at Amazon @awsforphp
6. @jeremeamia Jeremy Lindblom the Software Developer AWS SDK for PHP at Amazon @awsforphp President of the Seattle PHP User Group @seaphp & #pnwphp
7. @jeremeamia Jeremy Lindblom the Software Developer AWS SDK for PHP at Amazon @awsforphp President of the Seattle PHP User Group @seaphp & #pnwphp PHPoetry and PHParodies @phpbard
8. Once upon a time...
14. Closures
15. Closures —or— Lambdas (λ)
16. Closures —or— Lambdas —or— Anonymous Functions
17. function ($a, $b) { return $a + $b; }; Anonymous (a.k.a. no name)
18. $add = function ($a, $b) { return $a + $b; }; $result = $add(3, 7); // 10 First-class Functions
19. $b = 7; $add7 = function ($a) use ($b) { return $a + $b; }; $result = $add7(3); // 10 Closures
20. function getAdder($b) { return function ($a) use ($b) { return $a + $b; } } $add7 = getAdder(7); First-class Functions + Closures
21. $nums = [3, 6, 9]; $res = array_map($add7, $nums); // [10, 13, 16] Higher-order Functions
22. $closure = function () {...}; So what is this thing?
23. is_callable($closure); // true Closure Objects
24. is_callable($closure); // true is_object($closure); // true Closure Objects
25. is_callable($closure); // true is_object($closure); // true Closure Objects Like __invoke()
26. is_callable($closure); // true is_object($closure); // true get_class($closure); // Closure Closure Objects
27. is_callable($closure); // true is_object($closure); // true get_class($closure); // Closure Closure Objects “This fact used to be considered an implementation detail, but it can now be relied upon.” - PHP Manual
28. get_class_methods($closure); // array() Closure Objects (cont.)
29. serialize($closure); // ??? Closure Objects (still cont.)
30. Fatal error: Uncaught exception 'Exception' with message 'Serialization of 'Closure' is not allowed'
31. Back to the Story
33. serialize($closure);
34. Fatal error: Uncaught exception 'Exception' with message 'Serialization of 'Closure' is not allowed'
35. Challenge Accepted
41. Really ?
43. Thanks, Laravel!
44. SuperClosure Today
45. How to Get a Million Downloads
46. ★ Solve a problem ★ Find a niche ★ Make it easy to use How to Get a Million Downloads If you build it, they will come
47. ★ Build trust in your project ◦ Address issues promptly ◦ Follow SemVer ★ Build a community ◦ Become a dependency to a larger project ◦ Recruit contributors; reduce “bus factor” How to Get a Million Downloads (cont.) If you maintain it, they will stay
48. How to serialize() a \Closure
49. $fn = function ($a) use ($b, $c) { return ($a + $b) * $c; }; What we need to serialize We need to know the values of these. $context
50. $fn = function ($a) use ($b, $c) { return ($a + $b) * $c; }; What we need to serialize (cont.) And we need to know all of this. $code
51. extract($context); eval(“\$closure = {$code};”); How do we unserialize?
52. Interface for Serializing class SerializableClosure extends Closure implements Serializable { // ... }
53. Fatal error: Class SerializableClosure may not inherit from final class (Closure)
55. Interface for Serializing (Decorator) class SerializableClosure implements Serializable { private $closure; function __construct($closure) {…} function __invoke() {…} function serialize() {…} function unserialize($serialized) {…} }
56. Context is easiest to get $a = 5; $b = 10; $fn = function () use ($a, $b) { … }; $r = new ReflectionFunction($fn); $context = $r->getStaticVariables(); // [‘a’ => 5, ‘b’ => 10]
57. Context is easiest to get… maybe $a = 5; $b = 10; $fn = function () use ($a, $b) { static $c = 15; }; // [‘a’ => 5, ‘b’ => 10, ‘c’ => 15]
58. OK, How do we get the code? $r = new ReflectionFunction($closure); $file = $r->getFileName(); $start = $r->getStartLine(); $end = $r->getEndLine(); $code = implode(‘’, array_slice( file($file), $start, $end-$start+1));
59. token_get_all() parses the given source string into PHP language tokens using the Zend engine's lexical scanner.
60. Use the lexer $tokens = token_get_all(‘<?php ’.$code); [ [T_OPEN_TAG, '<?php ', 1], [T_VARIABLE, '$fn', 1], [T_WHITESPACE, ' ', 1], '=', [T_WHITESPACE, ' ', 1], [T_FUNCTION, 'function', 1], ...]
61. Create a parser foreach ($tokens as $token) { switch ($step) { case 0: if ($token[0] === T_FUNCTION) { $keep[] = $token; $step++; } break; // More steps... } }
62. Create a parser (cont.) ★ Throw out tokens before 1st T_FUNCTION ★ Analyze function signature ◦ Verify names of “used” variables (i.e., the context) ★ Keep all tokens within function body ◦ Track the brace level to help identify the end.
63. Token parser pitfalls ★ Assumes first T_FUNCTION encountered ★ No context outside of function ◦ What namespace/class is it in? ◦ What happens to magic constants __CLASS__?
64. Use an AST ★ AST = Abstract Syntax Tree ★ nikic/PHP-Parser is awesome! ★ Create tree data structure from entire file ◦ Traverse to closure, keeping track of scopes ◦ Replace some nodes with literal values as visited
65. Example - Original namespace Foo; use Bar\Baz; class Fizz { public function buzz() { return function (Baz $baz) { return $baz->boop(__CLASS__); } } }
66. Example - Token-based Result function (Baz $baz) { return $baz->boop(__CLASS__); } Code no longer knows the FQCN. ERROR Value is different from original function WTF
67. Example - AST-based Result function (Bar\Baz $baz) { return $baz->boop(‘Foo\Fizz’); } FQCN injected into code. No loss of information. Magic constants replaced with literal values. No loss of information.
68. AST parser pitfalls ★ SLOW. Really slow.
69. Parsing Caveats ★ Two closures declared on the same line is too ambiguous handle. Oh well… ★ If closures use (&$vales, &$by, &$ref) then those references cannot be restored ◦ Unless ref is to the closure (e.g., recursive closures)
70. Unsolved Mysteries ★ If a closure is declared in a trait, how do you resolve the value of __CLASS__ to a literal value? (If you figure this out, please let me know.)
71. Oh crap! What about bindings? (Curse you, PHP 5.4!)
72. class Foo { private $bar; public function baz() { return function () { return $this->bar; }; } } Closure Bindings Automatically creates a closure on $this.
73. class Foo { private $bar; public function baz() { return function () { return 5; }; } } Closure Bindings Automatically creates a closure on $this Even when you don’t need it.
74. class Foo { private $bar; public function baz() { return function () { return 5; }; } } Closure Bindings Automatically creates a closure on $this. Even when you don’t need it. UNLESS...
75. class Foo { private $bar; public function baz() { return static function () { return 5; }; } } Static Closures UNLESS... You make it static.
77. $foo = new Foo; $letters = new ArrayObject(range(‘a’, ‘z’)); $fn = $foo->baz(); // baz() returns a static closure $fn->bindTo($letters); “Warning: Cannot bind an instance to a static closure” Static is forever
78. Handling Closure Bindings ★ Two parts: ◦ Object ($this) is the object it is bound to. ◦ Scope determines what is visible. ★ Affects usage of $this, self::, and static:: within the closure
79. Getting Binding Information class Foo { public function bar() { return function () { … }; }} $foo = new Foo(); $fn = $foo->bar(); $r = new ReflectionFunction($fn); $r->getClosureThis(); // $foo $r->getClosureScopeClass()->getName(); // “Foo”
80. Setting Binding Information $newClosure = $closure->bindTo($newThis, $newScope); Notes: ★ You can omit $newScope, which means “keep the scope the same as it is now”. ★ Using null for $newThis makes the closure static. Remember, static is forever.
81. (Un)Serializing Closure Bindings ★ Including object and scope in serialization ★ Rebind closure after unserialization ◦ Tricky, because the object can be null in two cases ▪ It’s static ▪ It was not declared within a class
82. How to unserialize() a \Closure and Scare the Hell Out of Everyone
83. public function unserialize($serialized) { // Unserialize and reconstruct the closure. $this->data = unserialize($serialized); $this->closure = _reconstruct_closure($this->data); // Rebind the closure to its former binding and scope. if ($this->data['binding'] || $this->data['isStatic']) { $this->closure = $this->closure->bindTo( $this->data['binding'], $this->data['scope'] ); } }
84. And now, the most frightening function ever written...
85. function _reconstruct_closure(array $__data) { extract($__data['context']); if ($__fn = array_search(RECURSION, $__data['context'])) { @eval("\${$__fn} = {$__data['code']};"); $__closure = $$__fn; } else { @eval("\$__closure = {$__data['code']};"); } return $__closure; }
86. function _reconstruct_closure(array $__data) { extract($__data['context']); if ($__fn = array_search(RECURSION, $__data['context'])) { @eval("\${$__fn} = {$__data['code']};"); $__closure = $$__fn; } else { @eval("\$__closure = {$__data['code']};"); } return $__closure; }
87. function _reconstruct_closure(array $__data) { extract($__data['context']); if ($__fn = array_search(RECURSION, $__data['context'])) { @eval("\${$__fn} = {$__data['code']};"); $__closure = $$__fn; } else { @eval("\$__closure = {$__data['code']};"); } return $__closure; }
88. function _reconstruct_closure(array $__data) { extract($__data['context']); if ($__fn = array_search(RECURSION, $__data['context'])) { @eval("\${$__fn} = {$__data['code']};"); $__closure = $$__fn; } else { @eval("\$__closure = {$__data['code']};"); } return $__closure; }
89. function _reconstruct_closure(array $__data) { extract($__data['context']); if ($__fn = array_search(RECURSION, $__data['context'])) { @eval("\${$__fn} = {$__data['code']};"); $__closure = $$__fn; } else { @eval("\$__closure = {$__data['code']};"); } return $__closure; }
90. The End ★ Closures are weird ★ Serializing closures “is not allowed”, but you can still do it, if you really want to. ★ Reflection is awesome; so is PHP-Parser ★ Open Source is fun
91. How to serialize() a \Closure and Get a Million Downloads by Jeremy Lindblom – @jeremeamia