Building OpenSky with Symfony2

OpenSky was an early adopter of the Symfony2 framework. Over 2 years have passed and we’ve experienced just about everything you could imagine. Join me and hear about our experiences using Symfony2 in a large production environment and in a lean startup environment.

Transcript

1. Building OpenSky with Symfony2 Friday, February 8, 13
2. Thanks • ServerGrove • Pablo • Kim • Adam • And everyone else who helped organize Friday, February 8, 13
3. @jwage • Director of Technology at OpenSky.com • Before OpenSky, SensioLabs • ServerGrove Partner • Doctrine and Symfony contributor • Over a decade of experience building web applications with various programming languages Friday, February 8, 13
4. What is OpenSky? OpenSky is a social network for shopping that helps people discover, buy and share unique goods that match their individual taste through trusted insider connections. Friday, February 8, 13
5. Who are the insiders? • Bobby Flay • Martha Stewart • Shaq • Alicia Silverstone • Alton Brown • Buddy Valastro (Cake Boss) Range from celebrity chefs, actors, stylists, fashionistas, electronics gurus, etc. Friday, February 8, 13
6. How big is OpenSky? • 2.5 million members • 50 million connections • ~70 employees • Offices in • New York • Nashville • Portland • San Francisco Friday, February 8, 13
7. Technology Overview Graphite statsd Friday, February 8, 13
8. Application Layers Friday, February 8, 13
9. DEVO • PHP Web Application • Domain model and business logic • HTTP API • Sends messages to OSIS through HornetQ • Receives and processes callbacks from OSIS Friday, February 8, 13
10. OSIS • Enterprise Service Bus (ESB) • Mule framework From Wikipedia: An enterprise service bus (ESB) is a software architecture model used for designing and implementing the interaction and communication between mutually interacting software applications in service- oriented architecture (SOA). Friday, February 8, 13
11. What does OSIS do? • Sits behind HornetQ and consumes messages originated either from DEVO or by OSIS itself • Processes messages and does “something” • Interact with a 3rd party API • Chunk message in to smaller messages and put in other queues • Send transactional e-mails • Relay calls back to DEVO Friday, February 8, 13
12. OSIS Example When we receive shipments to process OSIS chunks them in to batches and relays them back to DEVO for payment capture, to mark the item as shipped and DEVO originates another event to OSIS which notifies the customer that the item shipped. Friday, February 8, 13
13. DEVO web1 web3 web5 hornetq hornetq hornetq hornetq cluster varnish/load balancer nginx Request web2 web4 web6 hornetq hornetq hornetq php web nodes OSIS java1 java2 java nodes MongoDB Replica Set Primary secondaries MySQL Master Slave Setup Primary secondaries PHP App ESB Friday, February 8, 13
14. Domain Model From Wikipedia: A domain model in problem solving and software engineering is a conceptual model of all the topics related to a specific problem. It describes the various entities, their attributes, roles, and relationships, plus the constraints that govern the problem domain. Friday, February 8, 13
15. Domain Objects • Plain old PHP objects (POPOs) • Mapped to MongoDB and MySQL using Doctrine • Don’t extend any base persistence class • No dependencies • THIN • Should contain minimal business logic • Merely wrappers around data Friday, February 8, 13
16. Domain Objects /** @ORM\Entity(...) */ class Order { /** @ORM\Id */ protected $id; /** @ODM\ObjectId */ protected $productId; /** * @Gedmo\ReferenceOne( * type="document", * targetDocument="Product", * identifier="productId" * ) */ protected $product; // ... } Cross-database references Mapping information via annotations /** @ODM\Document(...) */ class Product { /** @ODM\Id */ protected $id; // ... } Friday, February 8, 13
17. Domain Objects /** @ODM\Document(...) */ class Seller { /** @ODM\Id */ protected $id; // ... } /** @ODM\Document(...) */ class User { /** @ODM\Id */ protected $id; // ... } Friday, February 8, 13
18. Persistence decoupled from domain objects • Objects have no or light dependencies to other objects. • Only depend on other domain objects. • Makes it easy to test your domain objects in isolation when it is not coupled with persistence. Friday, February 8, 13
19. Domain Managers • They manage the business logic around domain objects. For Example: • OrderManager • ProductManager • FollowManager • UserManager • Used in controllers and other manager classes Friday, February 8, 13
20. Domain Managers class OrderManager { public function createOrder(Order $order); // ... } class ProductManager { public function createProduct(Product $product); // ... } Friday, February 8, 13
21. Domain Managers class FollowManager { public function follow(User $user, Seller $seller); // ... } class UserManager { public function createUser(User $user); // ... } Friday, February 8, 13
22. Using Domain Managers $dm = $container->get('doctrine.odm.mongodb.document_manager'); $followManager = $container->get('seller.follow_manager'); $sellerRepository = $container->get('seller.repository'); $userRepository = $container->get('user.repository'); $user = $userRepository->findOneBySlug('jwage'); $seller = $sellerRepository->findOneBySlug('shaq'); $followManager->follow($user, $seller); $dm->flush(); • Example usage where a User follows a Seller Friday, February 8, 13
23. Using Domain Managers • Usage in a controller • Keep business logic out of controllers. • Delegate work to services (domain managers, libs, etc.) class FollowController { // ... public function follow($sellerSlug) { $seller = $this->sellerRepository->findOneBySlug($sellerSlug); $user = $this->loggedInUserProvider->getLoggedInUser(); $this->followManager->follow($user, $seller); // give the request a response } // ... } Friday, February 8, 13
24. Decoupled Code • The actual work is abstracted away in a nice clean API that the controller can use. class FollowManager { // ... public function follow(User $user, Seller $seller) { $user->followSeller($seller); // ... } } Friday, February 8, 13
25. Decoupled Code • How can I do something else when FollowManager::follow() is called? class SellerManager { // ... public function follow(User $user, Seller $seller) { $user->followSeller($seller); // ... } } Friday, February 8, 13
26. EventDispatcher • Events are notified when domain manager classes perform an operation. For example: • order.created • user.created • product.created • seller.follow Friday, February 8, 13
27. EventDispatcher • Notify an Event when FollowManager::follow() is called class FollowManager { // ... public function follow(User $user, Seller $seller) { $user->followSeller($seller); $this->eventDispatcher->notify(new Event($seller, 'seller.follow', array( 'user' => $user ))); } } Friday, February 8, 13
28. Note about our EventDispatcher • We still utilize the old style of Symfony events where you have a generic Event class with a subject and parameters. • We had already adopted the old style of events before it was changed before Symfony2 was released • Easier to serialize and unserialize the generic Event object Friday, February 8, 13
29. Listening to the Event <service id="app.listener.name" class="App\Listener\UserFollowSellerListener"> <tag name="kernel.event_listener" event="seller.follow" method="onSellerFollow" /> </service> class SellerFollowListener { /** * Listens to 'seller.follow' */ public function onSellerFollow(EventInterface $event) { $seller = $event->getSubject(); $user = $event['user']; // do something } } Friday, February 8, 13
30. Sending messages to HornetQ from PHP • To send messages to HornetQ from PHP we use a library called Fuse Stomp http://stomp.fusesource.org $connection = new Stomp('tcp://localhost:61613'); $connection->connect('username', 'password'); $message = array( 'some' => 'data', 'more' => 'data' ); // send headers to use special HornetQ functionality $headers = array( '_HQ_SCHED_DELIVERY' => 3600 // delay by an hour ); // send the message $connection->send('jms.queue.name', json_encode($message), $headers); Friday, February 8, 13
31. Integrating with Symfony • ClientInterface - what sends the message • MessageInterface - what gets sent • EventForwarderInterface - convert a Symfony Event to a Message that is sent Friday, February 8, 13
32. Integrating with Symfony interface ClientInterface { function connect(); function disconnect(); function send(MessageInterface $message); } interface MessageInterface { function getHeaders(); function getParameters(); function getQueueName(); } interface EventForwarderInterface { function forward(Event $event); } Friday, February 8, 13
33. Integrating with Symfony $client = new Client(new Stomp('tcp://localhost:61613')); $eventForwarder = new EventForwarder($client); $eventForwarder->forward(new Event($seller, 'seller.follow', array( 'user' => $user ))); • Now we have a simple API that lets us forward Symfony Event instances to HornetQ Friday, February 8, 13
34. • From the DIC we can configure an event to be forwarded to hornetq Integrating with Symfony <service id="seller.follow.event_forwarder" class="EventForwarder" scope="container"> <tag name="kernel.event_listener" event="seller.follow" method="forward" /> <argument>jms.queue.opensky.seller</argument> <argument type="service" id="memoryqueue.client" /> </service> Friday, February 8, 13
35. Asynchronous Events • Having HornetQ is a convenient middle man and allows DEVO to behave in asynchronous ways • We’ve added an EventDispatcher::notifyAsync() method that forwards the Event to HornetQ and OSIS slings the event back to DEVO asynchronously. Friday, February 8, 13
36. Asynchronous Events • Imagine we want to use a third party API to check for fraudulent orders but we don’t want the main request to hang while we do this check. • The event is serialized in the EventForwarder and sent to a queue in OSIS which immediately slings it back to DEVO in another request. $eventDispatcher->notifyAsync(new Event($order, 'order.check_fraud')); Friday, February 8, 13
37. Asynchronous Events • A DEVO controller receives the request and reconstructs the original event and notifies it class EventController { // ... public function handle() { $event = $this->getEventFactory()->getReconstructedEvent($request); $this->getEventDispatcher()->notify($event); } } Friday, February 8, 13
38. Asynchronous Events • Now when you listen to order.check_fraud it happens in another process asynchronously. class OrderCheckFraudListener { // ... public function onOrderCheckFraud(EventInterface $event) { $order = $event->getSubject(); // $order instanceof User $this->orderFraudDetection->checkOrder($order); } } <service id="app.listener.name" class="OrderCheckFraudListener"> <tag name="kernel.event_listener" event="order.check_fraud" method="onOrderCheckFraud" /> </service> Friday, February 8, 13
39. Facebook Realtime Updates as Symfony Events Friday, February 8, 13
40. Facebook Realtime Updates • Facebook will post to a callback URL when an event happens. For example if a user adds or removes a friend. Friday, February 8, 13
41. • Facebook posts an array of data like the following to your callback URL Facebook Realtime Updates $change = array( 'object' => 'user', 'entry' => array( array( 'uid' => '1234', 'changed_fields' => array( 'friends' ) ), array( 'uid' => '12345', 'changed_fields' => array( 'friends' ) ) ) ); Friday, February 8, 13
42. • Controller to receive the POST from Facebook Facebook Realtime Updates class RealtimeUpdatesController { public function callbackAction() { $request = $this->getRequest(); $manager = $this->getRealtimeUpdatesManager(); if ($manager->isSubscribeRequest($request)) { if (!$manager->isSubscribeRequestValid($request)) { throw new NotFoundHttpException(); } return new Response($manager->getSubscribeRequestResponse($request)); } else if ($manager->isUpdateRequest($request)) { $manager->handleUpdate($request); return new Response(); } throw new NotFoundHttpException(); } // .. } Friday, February 8, 13
43. • Convert callback data to Symfony events Facebook Realtime Updates class RealtimeUpdatesManager { // ... public function handleUpdate(Request $request) { $callback = $request->request->all(); if (isset($callback['object']) && isset($callback['entry'])) { $eventName = 'facebook.realtime_update.%s.%s'; if ($callback['entry']) { foreach ($callback['entry'] as $entry) { $user = $this->dm->getRepository('MainBundle:User')->findUserByFacebookId($entry['uid']); if ($user === null) { continue; } foreach ($entry['changed_fields'] as $field) { $eventName = sprintf('facebook.realtime_updates.%s.%s', $callback['object'], $field); $this->eventDispatcher->notify(new Event($user, $eventName)); } } } } } } Friday, February 8, 13
44. Facebook Realtime Updates • facebook.realtime_updates.user.friends • facebook.realtime_updates.user.hometown Friday, February 8, 13
45. Facebook Realtime Updates • Listen to the realtime updates and execute PHP code class UserFriendsRealtimeUpdateListener { private $facebookFriendManager; public function __construct(FacebookFriendManager $facebookFriendManager) { $this->facebookFriendManager = $facebookFriendManager; } public function onUserFriendsRealtimeUpdate(EventInterface $event) { $this->facebookFriendManager->updateUserFriends($event->getSubject()); } } <service id="facebook.realtime_updates.user.friends.listener" class="UserFriendsRealtimeUpdateListener" scope="request"> <tag name="kernel.event_listener" event="facebook.realtime_updates.user.friends" method="onUserFriendsRealtimeUpdate" /> <argument type="service" id="facebook.friend_manager" /> </service> Friday, February 8, 13
46. Rules Engine • Ruler • A simple stateless production rules engine for PHP 5.3+ • Written by Justin Hileman • Used to work at OpenSky • https://github.com/bobthecow/ruler Friday, February 8, 13
47. Ruler Usage $rb = new RuleBuilder; $rule = $rb->create( $rb->logicalAnd( $rb['minAge']->greaterThan($rb['age']), $rb['maxAge']->lessThan($rb['age']) ), function() { echo 'Congratulations! You are between the ages of 18 and 25!'; } ); $context = new Context(array( 'minAge' => 18, 'maxAge' => 25, 'age' => function() { return 20; }, )); $rule->execute($context); // "Congratulations! You are between the ages of 18 and 25!" Friday, February 8, 13
48. What does OpenSky use Ruler for? • Issuing rewards to users when an event is notified and certain criteria is met • $$$ Credits for shopping • Points for rewards club • New member levels when point thresholds are reached • Free shipping Friday, February 8, 13
49. Rule Model • Rule • Condition • Reward interface Rule { function execute(Context $context); } interface Condition { function getType(); function buildRule(RuleBuilder $rb); function buildContext(Context $context); } interface Reward { function getType(); function getRecipientUser(Context $context); function issueReward(Context $context); } Friday, February 8, 13
50. Rule class Rule { private $eventName; private $condition; private $reward; // ... public function evaluate(Context $context) { $this->buildContext($context); return $this->buildRule()->evaluate($context); } public function execute(Context $context) { if ($this->evaluate($context)) { $this->issueReward($context); } } protected function issueReward(Context $context) { return $this->reward->issueReward($context); } protected function buildRule() { if (!isset($this->rule)) { $rb = new RuleBuilder(); $rule = $this->condition->buildRule($rb); // ... $this->rule = $rule; } return $this->rule; } protected function buildContext(Context $context) { $this->condition->buildContext($context); // ... } } Friday, February 8, 13
51. Example Reward Rule • On a users first purchase issue them a $5 credit Friday, February 8, 13
52. OrderCondition class OrderCondition extends Condition { private $requireFirstPurchase; // ... public function getType() { return 'order'; } public function buildRule(RuleBuilder $rb) { if ($this->requireFirstPurchase) { $rule = $rb->logicalAnd( $rule, $rb['user.is_first_purchase']->equalTo(true) ); } } public function buildContext(Context $context) { if ($this->requireFirstPurchase) { $context['user.is_first_purchase'] = $context->share(function($context) { return $context['em']->getRepository('MainBundle:Order') ->isUsersFirstOrder($context['order']); }); } } } Friday, February 8, 13
53. CreditReward class CreditReward extends Reward { private $amount; // ... public function getType() { return 'credit'; } public function getRecipientUser(Context $context) { return $context['order']->getUser(); } public function issueReward(Context $context) { $user = $this->getRecipientUser($context); $context['creditManager']->issueCredit($user, $this->amount); } } Friday, February 8, 13
54. Executing the Rule $context = new Context(array( 'em' => $container->get('doctrine.orm.entity_manager'), 'creditManager' => $container->get('user.credit_manager'), 'order' => $order )); $reward = new CreditReward(); $reward->setAmount(5); $orderCondition = new OrderCondition(); $orderCondition->setRequireFirstPurchase(true); $rule = new Rule(); $rule->setEventName('order.created'); $rule->setCondition($orderCondition); $rule->setReward($reward); $rule->execute($context); // user gets $5 credit if it is the users first purchase Friday, February 8, 13
55. Storing Rules in MongoDB • Rules are stored in MongoDB • Backend admin area for adding new rules • Mix and match different combinations of conditions and rewards Friday, February 8, 13
56. Rules and Events • Rules stored in the database are subscribed to specific event names • When an event is notified we can configure the DIC to check the rules database for any rules to evaluate Friday, February 8, 13
57. Rules and Events class RewardRuleListener { // ... public function handleEvent(Event $event) { $rules = $this->findRulesForEvent($event); foreach ($rules as $rule) { $context = $this->createContext($event); $context['rule'] = $rule; $rule->execute($context); } } private function createContext(Event $event) { return new Context(array( // ... )); } // ... } <service id="reward_rule.listener" class="RewardRuleListener" scope="request"> <tag name="kernel.event_listener" event="order.created" method="handleEvent" /> <argument type="service" id="container" /> </service> Friday, February 8, 13
58. Doctrine Lifecycle Events Friday, February 8, 13
59. What are they? • Feature of Doctrine • Doctrine triggers several events internally during the life-time of a domain object • Execute other code when these events occur Friday, February 8, 13
60. What kind of events? • preRemove - before an object is removed • postRemove - after an object is removed • prePersist - before an object is persisted • postPersist - after an object is persisted • preUpdate - before an object is updated • postUpdate - after an object is updated • postLoad - after an object is loaded • loadClassMetadata - when a classes metdata is loaded • onFlush - when the object manager is flushed Friday, February 8, 13
61. Lifecycle Event Callbacks • A lifecycle event callback is defined on the domain object directly and is executed when when the lifecycle event occurs. /** * @Entity */ class User { // ... /** @PrePersist */ public function doSomethingOnPrePersist() { $this->createdAt = new \DateTime(); } } Friday, February 8, 13
62. Listening to Lifecycle Events • Much more powerful than the simple lifecycle callbacks that are defined on the object classes. • Implement re-usable behaviors between different domain object classes. • To register an event listener you have to hook it into the EventManager Friday, February 8, 13
63. Listening to Lifecycle Events $eventManager = $em->getEventManager(); $eventManager->addEventListener(array(Events::preUpdate), new MyEventListener()); $eventManager->addEventSubscriber(new MyEventSubscriber()); • Event listener • Event subscribers Friday, February 8, 13
64. Event Listener class MyEventListener { public function preUpdate(PreUpdateEventArgs $eventArgs) { // ... } } Friday, February 8, 13
65. Event Subscriber • Only difference between listeners and subscribers is that the MyEventSubscriber class provides the array of events it should subscribe to. class MyEventSubscriber { public function preUpdate(PreUpdateEventArgs $eventArgs) { // ... } public function getSubscribedEvents() { return array(Events::preUpdate); } } Friday, February 8, 13
66. Configuring from Symfony <service id="example.listener" class="MyEventListener" public="false"> <tag name="doctrine.odm.mongodb.default_event_listener" event="preUpdate" /> </service> • Registering a listener from the Symfony2 DIC is easy enough Friday, February 8, 13
67. PreUpdateEventArgs • If you noticed, the listener preUpdate() method receives an instance of PreUpdateEventArgs • The PreUpdateEventArgs gives you access to all the fields that have changed for this object with the old and new value Friday, February 8, 13
68. PreUpdateEventArgs • getEntity() - to get access to the actual entity. • getEntityChangeSet() - to get a copy of the changeset array. • hasChangedField($fieldName) - to check if the given field name of the current entity changed. • getOldValue($fieldName) - to access the old value of the field • getNewValue($fieldName) - to access the new value of the field • setNewValue($fieldName, $value) - to change the value of a field to be updated. Friday, February 8, 13
69. PreUpdateEventArgs class NeverAliceOnlyBobListener { public function preUpdate(PreUpdateEventArgs $eventArgs) { if ($eventArgs->getEntity() instanceof User) { if ($eventArgs->hasChangedField('name') && $eventArgs- >getNewValue('name') == 'Alice') { $eventArgs->setNewValue('name', 'Bob'); } } } } • Example demonstrating the API of PreUpdateEventArgs Friday, February 8, 13
70. What does OpenSky use lifecycle events for? • preFlush - record all changesets that occur in our admin control panel and provide a frontend for browsing object changeset history Friday, February 8, 13
71. What does OpenSky use lifecycle events for? • Setting createdAt and updatedAt fields automatically /** @Entity */ class User { /** @PrePersist */ public function initializeCreatedAt() { $this->createdAt = new DateTime(); } /** @PreUpdate */ public function updateUpdatedAt() { $this->updatedAt = new DateTime(); } } Friday, February 8, 13
72. API • We use the JMSSerializerBundle for our API • Written by Johannes Schmitt • https://github.com/schmittjoh/JMSSerializerBundle Friday, February 8, 13
73. API • Use annotations to define the properties on our domain objects that we want to be serialized for our API class Product { // ... /** * @ODM\String * @Assert\NotBlank(message="Please provide a sellable name", groups={"Default", "Publish"}) * @SerializerGroups({"api"}) */ protected $name; // ... } Friday, February 8, 13
74. API class ProductsController extends BaseController { public function readAction($productId) { // ... $serialized = $this->getSerializer()->serialize( $product, 'json', array('api') ); return $this->createResponse(200, $serialized); } // ... } • Create a controller that can serialize the properties we annotated as JSON Friday, February 8, 13
75. API • Using the JMSSerializerBundle allowed us to rapidly build an API with minimal effort • It took 6 weeks from start to finish to build out the API, build an iPhone application and get it submitted to the app store. • Wouldn’t have been possible if we home built everything Friday, February 8, 13
76. Local Test Data • Most common options I see • Use data fixtures - https://github.com/doctrine/ data-fixtures • Restore production database locally • What other options??? Friday, February 8, 13
77. Fixtures • Pros • Easy to start, provides data to test with before you have production data • Flexible, you can add data for any case you need to test • Cons • Tedious and cumbersome to add and keep updated test data for real production scenarios Friday, February 8, 13
78. Restore production database locally • Pros • Gives you all the data you would ever need to test with • Troubleshoot production issues locally with production data • Cons • Security nightmare. Developers downloading customers personal information is no good • Doesn’t scale. You can’t download and restore your whole production database forever Friday, February 8, 13
79. Sanitized Production Slice • At OpenSky we wrote a custom tool that will extract a slice of production data • Used to build regression/QA testing environments. • Developers download and restore the slice locally for development • All personal information is sanitized to ensure customer security. • Slice regenerated from production automatically every morning Friday, February 8, 13
80. • Pros • Contains real production data • It is small, so downloading and restoring is fast • Cons • You have to update the slice tool when something is added to the model and you want the data to be included in the slice Sanitized Production Slice Friday, February 8, 13
81. Database Migrations Types of database migrations we use at OpenSky Friday, February 8, 13
82. Deploy Migrations • Execute migrations on the database before a new version is deployed and made live. • Doctrine migrations library • https://github.com/doctrine/migrations • http://docs.doctrine-project.org/projects/doctrine- migrations/en/latest/index.html Friday, February 8, 13
83. Deploy Migrations • Deploying new code often requires a DB migration • Add new tables • Add new fields • Migrate some data • Rename fields • Remove deprecated data • Anything else you can imagine • Try to make migrations backwards compatible to avoid downtime while migrations are performed and new code deployed Friday, February 8, 13
84. Deploy Migrations $ ./app/console doctrine:migrations:generate class Version20130204114559 extends AbstractMigration { public function up(Schema $schema) { } public function down(Schema $schema) { } } • Generate a new migration • up() - execute operations when migrating • down() - reverse executed operations in up() Friday, February 8, 13
85. Deploy Migrations class Version20130204114559 extends AbstractMigration { public function up(Schema $schema) { $this->addSql('ALTER TABLE stock_items ADD forceSoldout TINYINT(1) NOT NULL DEFAULT 0'); } public function down(Schema $schema) { $this->addSql('ALTER TABLE stock_items DROP COLUMN forceSoldout'); } } Add new column in up() Remove new column in down() • Here is an example where we add a new column to a table in mysql Friday, February 8, 13
86. Deploy Migrations • Checks for new migrations that have not been executed yet • Runs the up() method on any new migrations • Hooked up to fabric deploy process so migrations are automatically executed before new code goes live $ ./app/console doctrine:migrations:migrate Friday, February 8, 13
87. Deploy Migrations • Our migrations live in a standalone git repository • Linked to DEVO with a submodule • Allows database migrations to live outside DEVO. If a migration is needed outside of a code deploy, this allows it. Friday, February 8, 13
88. Eventual Migrations • Eventually migrate data when it is read instead of migrating all data when new code is deployed • No need to migrate data if it’s never read • Eventual Migrations could be called Lazy Migrations Friday, February 8, 13
89. Eventual Migrations • Use Doctrine lifecycle callbacks to modify data when it is read Friday, February 8, 13
90. Eventual Migrations • Initially store a users full name as a single string /** @Entity */ class User { // ... /** * @Column(type="string") */ private $fullName; // ... } Friday, February 8, 13
91. Eventual Migrations • Now imagine later we want the users name split up in to firstName and lastName Friday, February 8, 13
92. Eventual Migrations /** * @Entity * @HasLifecycleCallbacks */ class User { // ... /** * @Column(type="string") * @deprecated */ private $fullName; /** * @Column(type="string") */ private $firstName; /** * @Column(type="string") */ private $lastName; /** @PostLoad */ public function postLoad() { if ($this->fullName) { $e = explode(' ', $this->fullName); $this->firstName = $e[0]; $this->lastName = $e[1]; $this->fullName = null; } } } Friday, February 8, 13
93. Reporting • Reporting is a problem I am sure everyone has faced at some point. Friday, February 8, 13
94. Choices • Model your domain and persistence so that reporting is convenient • Model your domain and persistence for your applications needs only and utilize a data warehouse where the data is transformed in to structures that are convenient for reporting Friday, February 8, 13
95. Wrong Choice • Letting your reporting needs dictate your applications domain model is a road you don’t want to go down • All you end up with is a limited reporting system and a slow production database • Too many indexes • Data stored in ways that are not convenient for the application reading and writing Friday, February 8, 13
96. Right Choice • Use a data warehouse for all your reporting needs From Wikipedia: In computing, a data warehouse is a database used for reporting and data analysis. It is a central repository of data which is created by integrating data from one or more disparate sources. Data warehouses store current as well as historical data and are used for creating trending reports for senior management reporting such as annual and quarterly comparisons. Friday, February 8, 13
97. OpenSky Data • Our data is split up across multiple sources • Production Databases • MySQL • MongoDB • Third Parties • Braintree • Vendornet • Fulfillment Works Friday, February 8, 13
98. Extract Transform and Load From Wikipedia: In computing, Extract, Transform and Load (ETL) refers to a process in database usage and especially in data warehousing that involves: • Extracting data from outside sources • Transforming it to fit operational needs • Loading it into the end target Friday, February 8, 13
99. MySQL Data Warehouse Braintree Vendornet jetstream_mongo opensky_devo atmosphere mongo data mysql slave rollups/aggregates Cron Jobs ETL extract transform and load Production Databases MySQL MongoDB External Data Fulfillment Works flight_deck stored procedures and views DEVO OSIS Applications Warehouse Diagram Friday, February 8, 13
100. Warehouse Databases • jetstream_mongo - mongodb replicated to mysql • jetstream_audit - third party data • opensky_devo - mysql slave • atomosphere - stores historical reports, rollups, aggregates, etc. • flight_deck - No data is stored here. Friday, February 8, 13
101. flight_deck • DEVO and other applications only need access to flight_deck • MySQL views to expose the data needed for dashboards, reports and other reporting user interfaces. • Set of stored procedures that run on cron, updating stats, aggregates, rollups, etc. Friday, February 8, 13
102. statsd • Simple daemon for easy stats aggregation • Built by etsy • https://github.com/etsy/statsd/ Friday, February 8, 13
103. Graphite • Scalable Realtime Graphing • Renders graphs for stats collected via statsd • Monitor trends in application performance, usage, etc. Friday, February 8, 13
104. Integrate with Symfony • https://github.com/liuggio/StatsDClientBundle liuggio_stats_d_client: connection: host: localhost port: 8125 Friday, February 8, 13
105. Integrate with Symfony • Now we can easily track statistics from our code using the provided services. • For example if we want to monitor follow trends for specific sellers $statsdClientFactory = $container->get('liuggio_stats_d_client.factory'); $statsdClientService = $container->get('liuggio_stats_d_client.service') $data = $statsdClientFactory->increment(sprintf('seller.follow.%s', $seller->getSlug())); $statsdClientService->send($data); Friday, February 8, 13
106. View Realtime Data in Graphite Friday, February 8, 13
107. Other uses • Trends, trends, trends • Increment stats for every event dispatched • Track logged in users • Page views • Orders • Registrations • Performance/load time • Basically every action in the system we track Friday, February 8, 13
108. Tactical Operations Dashboard • Graphite provides easy ways to access the data for use in other graphs. • Allows us to pull in data from graphite and display on a centralized tactical operations dashboard Friday, February 8, 13
109. Tactical Operations Dashboard https://github.com/jondot/graphene Friday, February 8, 13
110. Tactical Operations Dashboard • Kept open on a LCD in office for easy monitoring • We know when we get a traffic spike • Allows us to pin point where a bottle neck is in the stack if we experience problems • Checking performance stats before and after a deploy to see if a code change had an impact on performance Friday, February 8, 13
111. Jonathan H. Wage http://twitter.com/jwage http://github.com/jwage Questions? We’re hiring! jwage@opensky.com Friday, February 8, 13