Rethinking Extension Development for PHP and HHVM

Presented January 31, 2016 at FOSDEM: https://joind.in/talk/0929c

Presented December 3, 2015 at SymfonyCon Paris: https://joind.in/talk/view/16483

Presented December 1, 2015 at Paris MongoDB User Group: http://www.meetup.com/Paris-MongoDB-User-Group/events/226710915/

Presented November 4, 2015 at eZ Conference: http://conference.ez.no/

Transcript

1. RETHINKING EXTENSION DEVELOPMENT FOR PHP AND HHVM A look at the new MongoDB driver Jeremy Mikola jmikola With special thanks to Derick Rethans and Hannes Magnusson for assisting with slides and working tirelessly to get us to 1.0.0 ὁ
2. One Extension, Three Engines PHP 5 HHVM PHP 7 A History Lesson Rethinking It All A New Architecture Implementation and Userland Current Status and Roadmap
3. History
4. History: 0.8.4 (April 2009) Resources for connections Classes for BSON types Code, Date, Object ID, Regex, BinData Very few functions: m o n g o _ c o n n e c t , m o n g o _ c l o s e , m o n g o _ r e m o v e , m o n g o _ q u e r y , m o n g o _ i n s e r t , m o n g o _ b a t c h _ i n s e r t , m o n g o _ u p d a t e , m o n g o _ h a s _ n e x t , m o n g o _ n e x t , m o n g o _ g r i d f s _ i n i t , m o n g o _ g r i d f s _ s t o r e , m o n g o _ g r i d f i l e _ w r i t e
5. History: 0.8.4 (April 2009) i n c l u d e ' M o n g o . p h p ' ; $ m = n e w M o n g o ( ) ; $ c = $ m - > s e l e c t C o l l e c t i o n ( ' p h p t ' , ' f i n d ' ) ; $ c - > d r o p ( ) ; $ c - > i n s e r t ( a r r a y ( ' f o o ' = > ' b a r ' , ' a ' = > ' b ' , ' b ' = > ' c ' , ) ) ; $ c u r s o r = $ c - > f i n d ( a r r a y ( ' f o o ' = > ' b a r ' ) , a r r a y ( ' a ' = > 1 , ' b ' = > 1 ) ) ; w h i l e ( $ c u r s o r - > h a s N e x t ( ) ) { v a r _ d u m p ( $ c u r s o r - > g e t N e x t ( ) ) ; }
6. History: 0.9.0 (May 2009) Support for PHP 5.3 Reimplement most PHP-based functionality in C Tests are implemented with PHPUnit
7. History: 1.0.0 (September 2009) Query operators don’t have to start with $ anymore: $ m = n e w M o n g o ; $ c = $ m - > s e l e c t C o l l e c t i o n ( ' d b n a m e ' , ' c o l n a m e ' ) ; $ r = $ c - > f i n d ( a r r a y ( ' f i e l d n a m e ' = > a r r a y ( ' $ g t e ' = > 4 2 ) ) ) ; i n i _ s e t ( ' m o n g o . c m d ' , ' @ ' ) ; $ m = n e w M o n g o ; $ c = $ m - > s e l e c t C o l l e c t i o n ( ' d b n a m e ' , ' c o l n a m e ' ) ; $ r = $ c - > f i n d ( a r r a y ( ' f i e l d n a m e ' = > a r r a y ( ' @ g t e ' = > 4 2 ) ) ) ;
8. History: 1.0.0 (September 2009) BSON arrays and documents come back as PHP arrays $ m = n e w M o n g o ; $ c = $ m - > s e l e c t C o l l e c t i o n ( ' d e m o ' , ' t e s t ' ) ; $ c - > d r o p ( ) ; $ c - > i n s e r t ( ( o b j e c t ) a r r a y ( ' f o o ' = > ' b a r ' ) ) ; $ c - > i n s e r t ( a r r a y ( ' f o o ' = > ' b a r ' ) ) ; f o r e a c h ( $ c - > f i n d ( a r r a y ( ' f o o ' = > ' b a r ' ) , a r r a y ( ' _ i d ' = > 0 ) ) a s $ r e s u l t ) { v a r _ d u m p ( $ r e s u l t ) ; } ↓ a r r a y ( 1 ) { [ " f o o " ] = > s t r i n g ( 3 ) " b a r " } a r r a y ( 1 ) { [ " f o o " ] = > s t r i n g ( 3 ) " b a r " }
9. History: 1.2.8 (February 2012) Derick joins as a contractor, after having helped with integer types (November 2011) We no longer use any functionality implemented as PHP classes (everything is C) Connection handling uses connection pools Kristina’s last release
10. History: 1.3.0 (November 2012) Hannes starts helping out (June 2012) Rewritten connection handling and replica set connection management Sane branching in GIT:
11. History: 1.4.0 – 1.6.0 Releases now come together with MongoDB server releases May 2013 – 1.4.0 – MongoDB 2.4 SSL support, Jeremy starts helping out Apr 2014 – 1.5.0 – MongoDB 2.6 SASL support, command cursors, write commands Jan 2015 – 1.6.0 – MongoDB 3.0 Rewritten cursor support, MONGO_METHOD refactoring
12. Current State of the Legacy Driver Former, "great" design choices: Array and object deserialization Positional arguments vs. options arrays i n s e r t ( ) injection of generated Object IDs No clear strategy for adding command helpers Configuration via INI and static properties M o n g o C u r s o r : : $ t i m e o u t , M o n g o C u r s o r : : $ s l a v e O k a y MongoLog and stream context listeners GridFS implementation could get some more love
13. Rethinking It All
14. Goals for a New Driver Support for multiple engines Well-thought-out and minimalistic API Faster to write, easier to maintain Build on top of existing code
15. Which Engines? PHP 5 HHVM PHP 7 PHP 5.4+ HHVM 3.9+ As of 1.1.1
16. Engine: PHP 5 PHP 5.4 – PHP 5.6 Not sure what else to say…
17. Engine: HHVM Alternative PHP runtime from Facebook Evolution of HipHop, which compiled PHP to C++ binaries Implements PHP language spec, supports Hack language Written in C++, OCAML, et al.
18. Engine: PHP 7 Userland code is mostly compatible with PHP 5 Internals are quite different from PHP 5
19. New Architecture
21. libbson l i b b s o n is a new shared library written in C for developers wanting to work with the BSON serialization format. It’s used by l i b m o n g o c It’s used by the drivers directly as well High performance BSON serialization and deserialization Callback API for deserialization
22. libmongoc (mongo-c-driver) l i b m o n g o c is a client library written in C for MongoDB Meant as both a low level driver for applications and higher level languages About two years old Used by the PHP and HHVM drivers, among others
23. Phongo (mongo-php-driver) New MongoDB extension for PHP 5.4+ Backwards compatibility is not preserved Very minimal API Core classes, BSON, exceptions Meant to be built upon by PHP libraries (like the 0.8.4 legacy driver) Uses PHP namespaces
24. On Backwards Compatibility Old extension name is "mongo" New extension name is "mongodb" Both extensions may be loaded simultaneously PECL PHP.net docs
25. Legacy API (mongo.so) $ m = n e w M o n g o C l i e n t ( ' m o n g o d b : / / l o c a l h o s t : 2 7 0 1 7 ' ) ; $ m - > d e m o - > t e s t - > d r o p ( ) ; $ m - > d e m o - > t e s t - > i n s e r t ( [ ' s t r i n g ' = > ' b a r ' , ' n u m b e r _ i ' = > 5 5 , ' n u m b e r _ l ' = > 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 , ' b o o l ' = > t r u e , ' n u l l ' = > n u l l , ' f l o a t ' = > M _ P I , ] ) ; $ c u r s o r = $ m - > d e m o - > t e s t - > f i n d ( ) ; f o r e a c h ( $ c u r s o r a s $ r e s u l t ) { v a r _ d u m p ( $ r e s u l t ) ; }
26. New API (mongodb.so) $ m = n e w M o n g o D B \ D r i v e r \ M a n a g e r ( ' m o n g o d b : / / l o c a l h o s t : 2 7 0 1 7 ' ) ; $ c m d = n e w M o n g o D B \ D r i v e r \ C o m m a n d ( [ ' d r o p ' = > ' t e s t ' ] ) ; $ m - > e x e c u t e C o m m a n d ( ' d e m o ' , $ c m d ) ; $ b u l k = n e w M o n g o D B \ D r i v e r \ B u l k W r i t e ; $ b u l k - > i n s e r t ( [ ' s t r i n g ' = > ' b a r ' , ' n u m b e r _ i ' = > 5 5 , ' n u m b e r _ l ' = > 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 , ' b o o l ' = > t r u e , ' n u l l ' = > n u l l , ' f l o a t ' = > M _ P I , ] ) ; $ m - > e x e c u t e B u l k W r i t e ( ' d e m o . t e s t ' , $ b u l k ) ; $ q u e r y = n e w M o n g o D B \ D r i v e r \ Q u e r y ( [ ] ) ; $ c u r s o r = $ m - > e x e c u t e Q u e r y ( ' d e m o . t e s t ' , $ q u e r y ) ; f o r e a c h ( $ c u r s o r a s $ r e s u l t ) { v a r _ d u m p ( $ r e s u l t ) ; }
27. Hippo’s API Same implementation of Phongo’s public API, but available as an extension for HHVM Drop-in replacement HHVM is C++, so no shared code with Phongo Parts of the extension may be written in Hack HNI: HHVM-Native Interface
28. Serialization Specification Persistence in Hippo and Phongo This document discusses the methods how compound structures (documents, arrays, objects) are persisted through the drivers. And how they are brought back into PHP land. Serialization to BSON Arrays If an array is a packed array — i.e. the keys start at 0 and are sequential without gaps: BSON array. If the array is not packed — i.e. having associative (string) keys, the keys don't start at 0, or when there are gaps:: BSON object A top­level (root) document, always serializes as a BSON document. Examples These serialize as a BSON array: [ 8 , 5 , 2 , 3 ] = > [ 8 , 5 , 2 , 3 ]
29. A Built-in ODM, too! Object serialization This driver includes four public interfaces to facilitate serializing PHP classes to and from BSON. MongoDB\BSON\Type (no methods) MongoDB\BSON\Serializable abstract public method bsonSerialize(); MongoDB\BSON\Unserializable abstract public method bsonUnserialize(array $data); MongoDB\BSON\Persistable extends both BSON\Serializable and BSON\Unserializable Objects that implement the MongoDB\BSON\Type interface get special treatment by the BSON serializer. In general, these objects represent a BSON type that cannot be natively represented in PHP (e.g. MongoDB\BSON\UTCDatetime, MongoDB\BSON\ObjectID) and are explicitly checked for and handled by the driver. MongoDB\BSON\Type should not be implemented directly by userland classes. Userland classes that require custom BSON serialization should utilize the MongoDB\BSON\Serializable interface and implement the b s o n S e r i a l i z e ( ) function, which should return a document (i.e. PHP array or stdClass object) representing the document that should be stored. Furthermore, if the object implements the MongoDB\BSON\Persistable interface, the driver will also inject a MongoDB\BSON\Binary value (with type 0x80 and an internal field name) into the document, which contains the userland object's fully qualified classname. This field can then be used during unserialization to ensure that the BSON document becomes an object of the same class on the way out of the database. During unserialization of a document, if a BSON Binary value (with type 0x80 and within the expected internal field name) is encountered, the driver will peek at the value and attempt to resolve it to a classname (triggering autoloaders if neccesary). If the class name cannot be resolved, nothing magical happens; however, if the class exists, the driver will create a new instance (without invoking the constructor) and invoke its
30. The Implementation
31. Hippo (implementation) An extract from e x t _ m o n g o d b . p h p : < ? h h n a m e s p a c e M o n g o D B \ D r i v e r ; < < _ _ N a t i v e > > p u b l i c f u n c t i o n _ _ c o n s t r u c t ( s t r i n g $ d s n = " l o c a l h o s t " , a r r a y $ o p t i o n s = [ ] , a r r a y $ d r i v e r O p t i o n s = [ ] ) ; < < _ _ N a t i v e > > p u b l i c f u n c t i o n _ _ d e b u g I n f o ( ) : a r r a y ; < < _ _ N a t i v e > > p u b l i c f u n c t i o n e x e c u t e Q u e r y ( s t r i n g $ n a m e s p a c e , Q u e r y $ q u e r y , R e a d P r e f e r e n c e $ r e a d P r e f e r e n c e = n u l l ) : C u r s o r ; < < _ _ N a t i v e > > p u b l i c f u n c t i o n s e l e c t S e r v e r ( R e a d P r e f e r e n c e $ r e a d P r e f e r e n c e ) : S e r v e r ; / / … } ? >
32. Hippo (implementation) An extract from s r c / M o n g o D B / D r i v e r / M a n a g e r . c p p : v o i d H H V M _ M E T H O D ( M o n g o D B D r i v e r M a n a g e r , _ _ c o n s t r u c t , c o n s t S t r i n g & d s n , c o n s t A r r a y & o p t i o { M o n g o D B D r i v e r M a n a g e r D a t a * d a t a = N a t i v e : : d a t a < M o n g o D B D r i v e r M a n a g e r D a t a > ( t h i s _ ) ; m o n g o c _ u r i _ t * u r i ; m o n g o c _ c l i e n t _ t * c l i e n t ; u r i = h i p p o _ m o n g o _ d r i v e r _ m a n a g e r _ m a k e _ u r i ( d s n . c _ s t r ( ) , o p t i o n s ) ; c l i e n t = m o n g o c _ c l i e n t _ n e w _ f r o m _ u r i ( u r i ) ; i f ( ! c l i e n t ) { t h r o w M o n g o D r i v e r : : U t i l s : : t h r o w R u n T i m e E x c e p t i o n ( " F a i l e d t o c r e a t e M a n a g e r f r o m U R } d a t a - > m _ c l i e n t = c l i e n t ; h i p p o _ m o n g o _ d r i v e r _ m a n a g e r _ a p p l y _ s s l _ o p t s ( d a t a - > m _ c l i e n t , d r i v e r O p t i o n s ) ; h i p p o _ m o n g o _ d r i v e r _ m a n a g e r _ a p p l y _ r p ( d a t a - > m _ c l i e n t , o p t i o n s ) ; h i p p o _ m o n g o _ d r i v e r _ m a n a g e r _ a p p l y _ w c ( d a t a - > m _ c l i e n t , o p t i o n s ) ; }
33. Hippo (implementation) An extract from e x t _ m o n g o d b . p h p : < ? h h n a m e s p a c e M o n g o D B \ D r i v e r ; < < _ _ N a t i v e D a t a ( " M o n g o D B D r i v e r R e a d P r e f e r e n c e " ) > > f i n a l c l a s s R e a d P r e f e r e n c e { < < _ _ N a t i v e > > p r i v a t e f u n c t i o n _ s e t R e a d P r e f e r e n c e ( i n t $ r e a d P r e f e r e n c e ) : v o i d ; < < _ _ N a t i v e > > p r i v a t e f u n c t i o n _ s e t R e a d P r e f e r e n c e T a g s ( a r r a y $ t a g S e t s ) : v o i d ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( i n t $ r e a d P r e f e r e n c e , m i x e d $ t a g S e t s = n u l l ) { i f ( $ t a g S e t s ! = = N U L L & & U t i l s : : m u s t B e A r r a y O r O b j e c t ( ' p a r a m e t e r 2 ' , $ t a g S e t s r e t u r n ; } s w i t c h ( $ r e a d P r e f e r e n c e ) { c a s e R e a d P r e f e r e n c e : : R P _ P R I M A R Y : c a s e R e a d P r e f e r e n c e : : R P _ P R I M A R Y _ P R E F E R R E D : c a s e R e a d P r e f e r e n c e : : R P _ S E C O N D A R Y : c a s e R e a d P r e f e r e n c e : : R P _ S E C O N D A R Y _ P R E F E R R E D : c a s e R e a d P r e f e r e n c e : : R P _ N E A R E S T : / / c a l l i n g i n t o N a t i v e $ t h i s - > _ s e t R e a d P r e f e r e n c e ( $ r e a d P r e f e r e n c e ) ; i f ( $ t a g S e t s ) {
34. Hippo vs. Phongo v o i d H H V M _ M E T H O D ( M o n g o D B D r i v e r M a n a g e r , _ _ c o n s t r u c t , c o n s t S t r i n g & d s n , c o n s t A r r a y & o p t i o { / * … * / } P H P _ M E T H O D ( M a n a g e r , _ _ c o n s t r u c t ) { p h p _ p h o n g o _ m a n a g e r _ t * i n t e r n ; z e n d _ e r r o r _ h a n d l i n g e r r o r _ h a n d l i n g ; m o n g o c _ u r i _ t * u r i ; c h a r * u r i _ s t r i n g ; i n t u r i _ s t r i n g _ l e n ; z v a l * o p t i o n s = N U L L ; b s o n _ t b s o n _ o p t i o n s = B S O N _ I N I T I A L I Z E R ; z v a l * d r i v e r O p t i o n s = N U L L ; ( v o i d ) r e t u r n _ v a l u e ; ( v o i d ) r e t u r n _ v a l u e _ p t r ; ( v o i d ) r e t u r n _ v a l u e _ u s e d ; z e n d _ r e p l a c e _ e r r o r _ h a n d l i n g ( E H _ T H R O W , p h o n g o _ e x c e p t i o n _ f r o m _ p h o n g o _ d o m a i n ( P H O N G O _ E R R O i n t e r n = ( p h p _ p h o n g o _ m a n a g e r _ t * ) z e n d _ o b j e c t _ s t o r e _ g e t _ o b j e c t ( g e t T h i s ( ) T S R M L S _ C C ) i f ( z e n d _ p a r s e _ p a r a m e t e r s ( Z E N D _ N U M _ A R G S ( ) T S R M L S _ C C , " s | a ! a ! " , & u r i _ s t r i n g , & u r i _ s t r i z e n d _ r e s t o r e _ e r r o r _ h a n d l i n g ( & e r r o r _ h a n d l i n g T S R M L S _ C C ) ; r e t u r n ; } z e n d _ r e s t o r e _ e r r o r _ h a n d l i n g ( & e r r o r _ h a n d l i n g T S R M L S _ C C ) ; i f ( o p t i o n s ) { z v a l _ t o _ b s o n ( o p t i o n s , P H O N G O _ B S O N _ N O N E , & b s o n _ o p t i o n s , N U L L T S R M L S _ C C ) ; } / * … * /
35. Adding PHP 7 Support to Phongo Order of elements in structs c h a r * vs. z e n d _ s t r i n g zend_hash changes PHP 7 itself was being developed alongside our new driver, which meant frequent internal API changes PHP 7 was released after we shipped 1.0 (PHP 5, HHVM), so support was added 1.1.1
36. A Minimalistic API Does Not a Full Driver Make
37. MongoDB PHP Library (PHPLIB) Userland library sitting atop Phongo and Hippo Depends on e x t - m o n g o d b Implements the convenience methods lacking from the basic extension APIs , command helpers and enumeration Installable via Composer: CRUD specification Collection index $ m k d i r n e w - p r o j e c t & & c d n e w - p r o j e c t $ c o m p o s e r r e q u i r e " m o n g o d b / m o n g o d b = ^ 1 . 0 . 0 "
38. Library Usage r e q u i r e ' v e n d o r / a u t o l o a d . p h p ' ; $ m = n e w M o n g o D B \ C l i e n t ( " m o n g o d b : / / l o c a l h o s t : 2 7 0 1 7 " ) ; $ c = $ m - > s e l e c t C o l l e c t i o n ( ' t e s t ' , ' d e m o ' ) ; $ c - > d r o p ( ) ; $ c - > i n s e r t O n e ( [ ' s t r i n g ' = > ' b a r ' , ' n u m b e r _ i ' = > 5 5 , ' n u m b e r _ l ' = > 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 , ' b o o l ' = > t r u e , ' n u l l ' = > n u l l , ' f l o a t ' = > M _ P I , ] ) ; f o r e a c h ( $ c - > f i n d ( ) a s $ r e s u l t ) { v a r _ d u m p ( $ r e s u l t ) ; }
39. Challenges Along the Way
40. Dependencies and APIs were a Moving Target As the first major driver built atop l i b m o n g o c , there were bugs to be found and fixed PHPLIB required identical Phongo and Hippo APIs, which wasn’t always the case during development Namespace changes along the way Exceptions moved to MongoDB\Driver\Exception Debates over a top-level BSON\ namespace Consolidating classes and methods CommandResult and QueryResult → Cursor Remove single-write helpers on Manager
41. HHVM Extension Writing Woes Many of the internal APIs are not documented Subtle compatibility issues regarding exceptions and constructors Bundling and configuring HHVM does not yet have a PECL equivalent Sharing connections and state across requests Derick’s memoirs: HHVM/HNI Extension Cookbook
42. Current Status
43. PHP and HHVM Drivers Docs: Phongo 1.1.2 released Hippo 1.1.0RC1 released php.net/mongodb Userland Library Docs: 1.0.0 released mongodb.github.io/mongo-php-library
44. Differentiating New from Old New PHP Driver JIRA: PECL: Docs: GitHub: PHPC mongodb PHP.net/mongodb mongodb/mongo-php-driver Legacy PHP Driver JIRA: PECL: Docs: GitHub: PHP mongo PHP.net/mongo mongodb/mongo-php-driver-legacy
45. On the Roadmap Resolve HHVM memory issues and ship 1.1.0 Additional ODM features planned for 1.2.0 GridFS coming to PHPLIB in 1.1.0 Upgrading Doctrine ODM and other OSS libs for legacy apps Mongo PHP Adapter
46. Future Possibilities
47. by François Laupretre PHP Code Service An easy way to mix C and PHP in extensions PECL extension: Supports both PHP 5 and 7 ( ) Opcode caching and minimal loading overhead Classes may be autoloaded Functions/constants registered on RINIT pcs pecl-compat
48. PCS Architecture
49. PCS with MongoDB
50. Thanks! speakerdeck.com/jmikola joind.in/talk/0929c Questions?