Building OpenSky with Symfony2

Slide 1

Slide 1 text

Building OpenSky with Symfony2 Friday, February 8, 13

Slide 2

Slide 2 text

Thanks • ServerGrove • Pablo • Kim • Adam • And everyone else who helped organize Friday, February 8, 13

Slide 3

Slide 3 text

@jwage • Director of Technology at OpenSky.com • Before OpenSky, SensioLabs • ServerGrove Partner • Doctrine and Symfony contributor • Over a decade of experience building web applications with various programming languages Friday, February 8, 13

Slide 4

Slide 4 text

What is OpenSky? OpenSky is a social network for shopping that helps people discover, buy and share unique goods that match their individual taste through trusted insider connections. Friday, February 8, 13

Slide 5

Slide 5 text

Who are the insiders? • Bobby Flay • Martha Stewart • Shaq • Alicia Silverstone • Alton Brown • Buddy Valastro (Cake Boss) Range from celebrity chefs, actors, stylists, fashionistas, electronics gurus, etc. Friday, February 8, 13

Slide 6

Slide 6 text

How big is OpenSky? • 2.5 million members • 50 million connections • ~70 employees • Ofﬁces in • New York • Nashville • Portland • San Francisco Friday, February 8, 13

Slide 7

Slide 7 text

Technology Overview Graphite statsd Friday, February 8, 13

Slide 8

Slide 8 text

Application Layers Friday, February 8, 13

Slide 9

Slide 9 text

DEVO • PHP Web Application • Domain model and business logic • HTTP API • Sends messages to OSIS through HornetQ • Receives and processes callbacks from OSIS Friday, February 8, 13

Slide 10

Slide 10 text

OSIS • Enterprise Service Bus (ESB) • Mule framework From Wikipedia: An enterprise service bus (ESB) is a software architecture model used for designing and implementing the interaction and communication between mutually interacting software applications in service- oriented architecture (SOA). Friday, February 8, 13

Slide 11

Slide 11 text

What does OSIS do? • Sits behind HornetQ and consumes messages originated either from DEVO or by OSIS itself • Processes messages and does “something” • Interact with a 3rd party API • Chunk message in to smaller messages and put in other queues • Send transactional e-mails • Relay calls back to DEVO Friday, February 8, 13

Slide 12

Slide 12 text

OSIS Example When we receive shipments to process OSIS chunks them in to batches and relays them back to DEVO for payment capture, to mark the item as shipped and DEVO originates another event to OSIS which notiﬁes the customer that the item shipped. Friday, February 8, 13

Slide 13

Slide 13 text

DEVO web1 web3 web5 hornetq hornetq hornetq hornetq cluster varnish/load balancer nginx Request web2 web4 web6 hornetq hornetq hornetq php web nodes OSIS java1 java2 java nodes MongoDB Replica Set Primary secondaries MySQL Master Slave Setup Primary secondaries PHP App ESB Friday, February 8, 13

Slide 14

Slide 14 text

Domain Model From Wikipedia: A domain model in problem solving and software engineering is a conceptual model of all the topics related to a speciﬁc problem. It describes the various entities, their attributes, roles, and relationships, plus the constraints that govern the problem domain. Friday, February 8, 13

Slide 15

Slide 15 text

Domain Objects • Plain old PHP objects (POPOs) • Mapped to MongoDB and MySQL using Doctrine • Don’t extend any base persistence class • No dependencies • THIN • Should contain minimal business logic • Merely wrappers around data Friday, February 8, 13

Slide 16

Slide 16 text

Domain Objects /** @ORM\Entity(...) */ class Order { /** @ORM\Id */ protected $id; /** @ODM\ObjectId */ protected $productId; /** * @Gedmo\ReferenceOne( * type="document", * targetDocument="Product", * identifier="productId" * ) */ protected $product; // ... } Cross-database references Mapping information via annotations /** @ODM\Document(...) */ class Product { /** @ODM\Id */ protected $id; // ... } Friday, February 8, 13

Slide 17

Slide 17 text

Domain Objects /** @ODM\Document(...) */ class Seller { /** @ODM\Id */ protected $id; // ... } /** @ODM\Document(...) */ class User { /** @ODM\Id */ protected $id; // ... } Friday, February 8, 13

Slide 18

Slide 18 text

Persistence decoupled from domain objects • Objects have no or light dependencies to other objects. • Only depend on other domain objects. • Makes it easy to test your domain objects in isolation when it is not coupled with persistence. Friday, February 8, 13

Slide 19

Slide 19 text

Domain Managers • They manage the business logic around domain objects. For Example: • OrderManager • ProductManager • FollowManager • UserManager • Used in controllers and other manager classes Friday, February 8, 13

Slide 20

Slide 20 text

Domain Managers class OrderManager { public function createOrder(Order $order); // ... } class ProductManager { public function createProduct(Product $product); // ... } Friday, February 8, 13

Slide 21

Slide 21 text

Domain Managers class FollowManager { public function follow(User $user, Seller $seller); // ... } class UserManager { public function createUser(User $user); // ... } Friday, February 8, 13

Slide 22

Slide 22 text

Using Domain Managers $dm = $container->get('doctrine.odm.mongodb.document_manager'); $followManager = $container->get('seller.follow_manager'); $sellerRepository = $container->get('seller.repository'); $userRepository = $container->get('user.repository'); $user = $userRepository->findOneBySlug('jwage'); $seller = $sellerRepository->findOneBySlug('shaq'); $followManager->follow($user, $seller); $dm->flush(); • Example usage where a User follows a Seller Friday, February 8, 13

Slide 23

Slide 23 text

Using Domain Managers • Usage in a controller • Keep business logic out of controllers. • Delegate work to services (domain managers, libs, etc.) class FollowController { // ... public function follow($sellerSlug) { $seller = $this->sellerRepository->findOneBySlug($sellerSlug); $user = $this->loggedInUserProvider->getLoggedInUser(); $this->followManager->follow($user, $seller); // give the request a response } // ... } Friday, February 8, 13

Slide 24

Slide 24 text

Decoupled Code • The actual work is abstracted away in a nice clean API that the controller can use. class FollowManager { // ... public function follow(User $user, Seller $seller) { $user->followSeller($seller); // ... } } Friday, February 8, 13

Slide 25

Slide 25 text

Decoupled Code • How can I do something else when FollowManager::follow() is called? class SellerManager { // ... public function follow(User $user, Seller $seller) { $user->followSeller($seller); // ... } } Friday, February 8, 13

Slide 26

Slide 26 text

EventDispatcher • Events are notiﬁed when domain manager classes perform an operation. For example: • order.created • user.created • product.created • seller.follow Friday, February 8, 13

Slide 27

Slide 27 text

EventDispatcher • Notify an Event when FollowManager::follow() is called class FollowManager { // ... public function follow(User $user, Seller $seller) { $user->followSeller($seller); $this->eventDispatcher->notify(new Event($seller, 'seller.follow', array( 'user' => $user ))); } } Friday, February 8, 13

Slide 28

Slide 28 text

Note about our EventDispatcher • We still utilize the old style of Symfony events where you have a generic Event class with a subject and parameters. • We had already adopted the old style of events before it was changed before Symfony2 was released • Easier to serialize and unserialize the generic Event object Friday, February 8, 13

Slide 29

Slide 29 text

Listening to the Event class SellerFollowListener { /** * Listens to 'seller.follow' */ public function onSellerFollow(EventInterface $event) { $seller = $event->getSubject(); $user = $event['user']; // do something } } Friday, February 8, 13

Slide 30

Slide 30 text

Sending messages to HornetQ from PHP • To send messages to HornetQ from PHP we use a library called Fuse Stomp http://stomp.fusesource.org $connection = new Stomp('tcp://localhost:61613'); $connection->connect('username', 'password'); $message = array( 'some' => 'data', 'more' => 'data' ); // send headers to use special HornetQ functionality $headers = array( '_HQ_SCHED_DELIVERY' => 3600 // delay by an hour ); // send the message $connection->send('jms.queue.name', json_encode($message), $headers); Friday, February 8, 13

Slide 31

Slide 31 text

Integrating with Symfony • ClientInterface - what sends the message • MessageInterface - what gets sent • EventForwarderInterface - convert a Symfony Event to a Message that is sent Friday, February 8, 13

Slide 32

Slide 32 text

Integrating with Symfony interface ClientInterface { function connect(); function disconnect(); function send(MessageInterface $message); } interface MessageInterface { function getHeaders(); function getParameters(); function getQueueName(); } interface EventForwarderInterface { function forward(Event $event); } Friday, February 8, 13

Slide 33

Slide 33 text

Integrating with Symfony $client = new Client(new Stomp('tcp://localhost:61613')); $eventForwarder = new EventForwarder($client); $eventForwarder->forward(new Event($seller, 'seller.follow', array( 'user' => $user ))); • Now we have a simple API that lets us forward Symfony Event instances to HornetQ Friday, February 8, 13

Slide 34

Slide 34 text

• From the DIC we can conﬁgure an event to be forwarded to hornetq Integrating with Symfony jms.queue.opensky.seller Friday, February 8, 13

Slide 35

Slide 35 text

Asynchronous Events • Having HornetQ is a convenient middle man and allows DEVO to behave in asynchronous ways • We’ve added an EventDispatcher::notifyAsync() method that forwards the Event to HornetQ and OSIS slings the event back to DEVO asynchronously. Friday, February 8, 13

Slide 36

Slide 36 text

Asynchronous Events • Imagine we want to use a third party API to check for fraudulent orders but we don’t want the main request to hang while we do this check. • The event is serialized in the EventForwarder and sent to a queue in OSIS which immediately slings it back to DEVO in another request. $eventDispatcher->notifyAsync(new Event($order, 'order.check_fraud')); Friday, February 8, 13

Slide 37

Slide 37 text

Asynchronous Events • A DEVO controller receives the request and reconstructs the original event and notiﬁes it class EventController { // ... public function handle() { $event = $this->getEventFactory()->getReconstructedEvent($request); $this->getEventDispatcher()->notify($event); } } Friday, February 8, 13

Slide 38

Slide 38 text

Asynchronous Events • Now when you listen to order.check_fraud it happens in another process asynchronously. class OrderCheckFraudListener { // ... public function onOrderCheckFraud(EventInterface $event) { $order = $event->getSubject(); // $order instanceof User $this->orderFraudDetection->checkOrder($order); } } Friday, February 8, 13

Slide 39

Slide 39 text

Facebook Realtime Updates as Symfony Events Friday, February 8, 13

Slide 40

Slide 40 text

Facebook Realtime Updates • Facebook will post to a callback URL when an event happens. For example if a user adds or removes a friend. Friday, February 8, 13

Slide 41

Slide 41 text

• Facebook posts an array of data like the following to your callback URL Facebook Realtime Updates $change = array( 'object' => 'user', 'entry' => array( array( 'uid' => '1234', 'changed_fields' => array( 'friends' ) ), array( 'uid' => '12345', 'changed_fields' => array( 'friends' ) ) ) ); Friday, February 8, 13

Slide 42

Slide 42 text

• Controller to receive the POST from Facebook Facebook Realtime Updates class RealtimeUpdatesController { public function callbackAction() { $request = $this->getRequest(); $manager = $this->getRealtimeUpdatesManager(); if ($manager->isSubscribeRequest($request)) { if (!$manager->isSubscribeRequestValid($request)) { throw new NotFoundHttpException(); } return new Response($manager->getSubscribeRequestResponse($request)); } else if ($manager->isUpdateRequest($request)) { $manager->handleUpdate($request); return new Response(); } throw new NotFoundHttpException(); } // .. } Friday, February 8, 13

Slide 43

Slide 43 text

• Convert callback data to Symfony events Facebook Realtime Updates class RealtimeUpdatesManager { // ... public function handleUpdate(Request $request) { $callback = $request->request->all(); if (isset($callback['object']) && isset($callback['entry'])) { $eventName = 'facebook.realtime_update.%s.%s'; if ($callback['entry']) { foreach ($callback['entry'] as $entry) { $user = $this->dm->getRepository('MainBundle:User')->findUserByFacebookId($entry['uid']); if ($user === null) { continue; } foreach ($entry['changed_fields'] as $field) { $eventName = sprintf('facebook.realtime_updates.%s.%s', $callback['object'], $field); $this->eventDispatcher->notify(new Event($user, $eventName)); } } } } } } Friday, February 8, 13

Slide 44

Slide 44 text

Facebook Realtime Updates • facebook.realtime_updates.user.friends • facebook.realtime_updates.user.hometown Friday, February 8, 13

Slide 45

Slide 45 text

Facebook Realtime Updates • Listen to the realtime updates and execute PHP code class UserFriendsRealtimeUpdateListener { private $facebookFriendManager; public function __construct(FacebookFriendManager $facebookFriendManager) { $this->facebookFriendManager = $facebookFriendManager; } public function onUserFriendsRealtimeUpdate(EventInterface $event) { $this->facebookFriendManager->updateUserFriends($event->getSubject()); } } Friday, February 8, 13

Slide 46

Slide 46 text

Rules Engine • Ruler • A simple stateless production rules engine for PHP 5.3+ • Written by Justin Hileman • Used to work at OpenSky • https://github.com/bobthecow/ruler Friday, February 8, 13

Slide 47

Slide 47 text

Ruler Usage $rb = new RuleBuilder; $rule = $rb->create( $rb->logicalAnd( $rb['minAge']->greaterThan($rb['age']), $rb['maxAge']->lessThan($rb['age']) ), function() { echo 'Congratulations! You are between the ages of 18 and 25!'; } ); $context = new Context(array( 'minAge' => 18, 'maxAge' => 25, 'age' => function() { return 20; }, )); $rule->execute($context); // "Congratulations! You are between the ages of 18 and 25!" Friday, February 8, 13

Slide 48

Slide 48 text

What does OpenSky use Ruler for? • Issuing rewards to users when an event is notiﬁed and certain criteria is met • $$$ Credits for shopping • Points for rewards club • New member levels when point thresholds are reached • Free shipping Friday, February 8, 13

Slide 49

Slide 49 text

Rule Model • Rule • Condition • Reward interface Rule { function execute(Context $context); } interface Condition { function getType(); function buildRule(RuleBuilder $rb); function buildContext(Context $context); } interface Reward { function getType(); function getRecipientUser(Context $context); function issueReward(Context $context); } Friday, February 8, 13

Slide 50

Slide 50 text

Rule class Rule { private $eventName; private $condition; private $reward; // ... public function evaluate(Context $context) { $this->buildContext($context); return $this->buildRule()->evaluate($context); } public function execute(Context $context) { if ($this->evaluate($context)) { $this->issueReward($context); } } protected function issueReward(Context $context) { return $this->reward->issueReward($context); } protected function buildRule() { if (!isset($this->rule)) { $rb = new RuleBuilder(); $rule = $this->condition->buildRule($rb); // ... $this->rule = $rule; } return $this->rule; } protected function buildContext(Context $context) { $this->condition->buildContext($context); // ... } } Friday, February 8, 13

Slide 51

Slide 51 text

Example Reward Rule • On a users ﬁrst purchase issue them a $5 credit Friday, February 8, 13

Slide 52

Slide 52 text

OrderCondition class OrderCondition extends Condition { private $requireFirstPurchase; // ... public function getType() { return 'order'; } public function buildRule(RuleBuilder $rb) { if ($this->requireFirstPurchase) { $rule = $rb->logicalAnd( $rule, $rb['user.is_first_purchase']->equalTo(true) ); } } public function buildContext(Context $context) { if ($this->requireFirstPurchase) { $context['user.is_first_purchase'] = $context->share(function($context) { return $context['em']->getRepository('MainBundle:Order') ->isUsersFirstOrder($context['order']); }); } } } Friday, February 8, 13

Slide 53

Slide 53 text

CreditReward class CreditReward extends Reward { private $amount; // ... public function getType() { return 'credit'; } public function getRecipientUser(Context $context) { return $context['order']->getUser(); } public function issueReward(Context $context) { $user = $this->getRecipientUser($context); $context['creditManager']->issueCredit($user, $this->amount); } } Friday, February 8, 13

Slide 54

Slide 54 text

Executing the Rule $context = new Context(array( 'em' => $container->get('doctrine.orm.entity_manager'), 'creditManager' => $container->get('user.credit_manager'), 'order' => $order )); $reward = new CreditReward(); $reward->setAmount(5); $orderCondition = new OrderCondition(); $orderCondition->setRequireFirstPurchase(true); $rule = new Rule(); $rule->setEventName('order.created'); $rule->setCondition($orderCondition); $rule->setReward($reward); $rule->execute($context); // user gets $5 credit if it is the users first purchase Friday, February 8, 13

Slide 55

Slide 55 text

Storing Rules in MongoDB • Rules are stored in MongoDB • Backend admin area for adding new rules • Mix and match different combinations of conditions and rewards Friday, February 8, 13

Slide 56

Slide 56 text

Rules and Events • Rules stored in the database are subscribed to specific event names • When an event is notified we can configure the DIC to check the rules database for any rules to evaluate Friday, February 8, 13

Slide 57

Slide 57 text

Rules and Events class RewardRuleListener { // ... public function handleEvent(Event $event) { $rules = $this->findRulesForEvent($event); foreach ($rules as $rule) { $context = $this->createContext($event); $context['rule'] = $rule; $rule->execute($context); } } private function createContext(Event $event) { return new Context(array( // ... )); } // ... } Friday, February 8, 13

Slide 58

Slide 58 text

Doctrine Lifecycle Events Friday, February 8, 13

Slide 59

Slide 59 text

What are they? • Feature of Doctrine • Doctrine triggers several events internally during the life-time of a domain object • Execute other code when these events occur Friday, February 8, 13

Slide 60

Slide 60 text

What kind of events? • preRemove - before an object is removed • postRemove - after an object is removed • prePersist - before an object is persisted • postPersist - after an object is persisted • preUpdate - before an object is updated • postUpdate - after an object is updated • postLoad - after an object is loaded • loadClassMetadata - when a classes metdata is loaded • onFlush - when the object manager is ﬂushed Friday, February 8, 13

Slide 61

Slide 61 text

Lifecycle Event Callbacks • A lifecycle event callback is deﬁned on the domain object directly and is executed when when the lifecycle event occurs. /** * @Entity */ class User { // ... /** @PrePersist */ public function doSomethingOnPrePersist() { $this->createdAt = new \DateTime(); } } Friday, February 8, 13

Slide 62

Slide 62 text

Listening to Lifecycle Events • Much more powerful than the simple lifecycle callbacks that are deﬁned on the object classes. • Implement re-usable behaviors between different domain object classes. • To register an event listener you have to hook it into the EventManager Friday, February 8, 13

Slide 63

Slide 63 text

Listening to Lifecycle Events $eventManager = $em->getEventManager(); $eventManager->addEventListener(array(Events::preUpdate), new MyEventListener()); $eventManager->addEventSubscriber(new MyEventSubscriber()); • Event listener • Event subscribers Friday, February 8, 13

Slide 64

Slide 64 text

Event Listener class MyEventListener { public function preUpdate(PreUpdateEventArgs $eventArgs) { // ... } } Friday, February 8, 13

Slide 65

Slide 65 text

Event Subscriber • Only difference between listeners and subscribers is that the MyEventSubscriber class provides the array of events it should subscribe to. class MyEventSubscriber { public function preUpdate(PreUpdateEventArgs $eventArgs) { // ... } public function getSubscribedEvents() { return array(Events::preUpdate); } } Friday, February 8, 13

Slide 66

Slide 66 text

Conﬁguring from Symfony • Registering a listener from the Symfony2 DIC is easy enough Friday, February 8, 13

Slide 67

Slide 67 text

PreUpdateEventArgs • If you noticed, the listener preUpdate() method receives an instance of PreUpdateEventArgs • The PreUpdateEventArgs gives you access to all the ﬁelds that have changed for this object with the old and new value Friday, February 8, 13

Slide 68

Slide 68 text

PreUpdateEventArgs • getEntity() - to get access to the actual entity. • getEntityChangeSet() - to get a copy of the changeset array. • hasChangedField($fieldName) - to check if the given field name of the current entity changed. • getOldValue($fieldName) - to access the old value of the field • getNewValue($fieldName) - to access the new value of the field • setNewValue($fieldName, $value) - to change the value of a field to be updated. Friday, February 8, 13

Slide 69

Slide 69 text

PreUpdateEventArgs class NeverAliceOnlyBobListener { public function preUpdate(PreUpdateEventArgs $eventArgs) { if ($eventArgs->getEntity() instanceof User) { if ($eventArgs->hasChangedField('name') && $eventArgs- >getNewValue('name') == 'Alice') { $eventArgs->setNewValue('name', 'Bob'); } } } } • Example demonstrating the API of PreUpdateEventArgs Friday, February 8, 13

Slide 70

Slide 70 text

What does OpenSky use lifecycle events for? • preFlush - record all changesets that occur in our admin control panel and provide a frontend for browsing object changeset history Friday, February 8, 13

Slide 71

Slide 71 text

What does OpenSky use lifecycle events for? • Setting createdAt and updatedAt ﬁelds automatically /** @Entity */ class User { /** @PrePersist */ public function initializeCreatedAt() { $this->createdAt = new DateTime(); } /** @PreUpdate */ public function updateUpdatedAt() { $this->updatedAt = new DateTime(); } } Friday, February 8, 13

Slide 72

Slide 72 text

API • We use the JMSSerializerBundle for our API • Written by Johannes Schmitt • https://github.com/schmittjoh/JMSSerializerBundle Friday, February 8, 13

Slide 73

Slide 73 text

API • Use annotations to deﬁne the properties on our domain objects that we want to be serialized for our API class Product { // ... /** * @ODM\String * @Assert\NotBlank(message="Please provide a sellable name", groups={"Default", "Publish"}) * @SerializerGroups({"api"}) */ protected $name; // ... } Friday, February 8, 13

Slide 74

Slide 74 text

API class ProductsController extends BaseController { public function readAction($productId) { // ... $serialized = $this->getSerializer()->serialize( $product, 'json', array('api') ); return $this->createResponse(200, $serialized); } // ... } • Create a controller that can serialize the properties we annotated as JSON Friday, February 8, 13

Slide 75

Slide 75 text

API • Using the JMSSerializerBundle allowed us to rapidly build an API with minimal effort • It took 6 weeks from start to ﬁnish to build out the API, build an iPhone application and get it submitted to the app store. • Wouldn’t have been possible if we home built everything Friday, February 8, 13

Slide 76

Slide 76 text

Local Test Data • Most common options I see • Use data ﬁxtures - https://github.com/doctrine/ data-ﬁxtures • Restore production database locally • What other options??? Friday, February 8, 13

Slide 77

Slide 77 text

Fixtures • Pros • Easy to start, provides data to test with before you have production data • Flexible, you can add data for any case you need to test • Cons • Tedious and cumbersome to add and keep updated test data for real production scenarios Friday, February 8, 13

Slide 78

Slide 78 text

Restore production database locally • Pros • Gives you all the data you would ever need to test with • Troubleshoot production issues locally with production data • Cons • Security nightmare. Developers downloading customers personal information is no good • Doesn’t scale. You can’t download and restore your whole production database forever Friday, February 8, 13

Slide 79

Slide 79 text

Sanitized Production Slice • At OpenSky we wrote a custom tool that will extract a slice of production data • Used to build regression/QA testing environments. • Developers download and restore the slice locally for development • All personal information is sanitized to ensure customer security. • Slice regenerated from production automatically every morning Friday, February 8, 13

Slide 80

Slide 80 text

• Pros • Contains real production data • It is small, so downloading and restoring is fast • Cons • You have to update the slice tool when something is added to the model and you want the data to be included in the slice Sanitized Production Slice Friday, February 8, 13

Slide 81

Slide 81 text

Database Migrations Types of database migrations we use at OpenSky Friday, February 8, 13

Slide 82

Slide 82 text

Deploy Migrations • Execute migrations on the database before a new version is deployed and made live. • Doctrine migrations library • https://github.com/doctrine/migrations • http://docs.doctrine-project.org/projects/doctrine- migrations/en/latest/index.html Friday, February 8, 13

Slide 83

Slide 83 text

Deploy Migrations • Deploying new code often requires a DB migration • Add new tables • Add new ﬁelds • Migrate some data • Rename ﬁelds • Remove deprecated data • Anything else you can imagine • Try to make migrations backwards compatible to avoid downtime while migrations are performed and new code deployed Friday, February 8, 13

Slide 84

Slide 84 text

Deploy Migrations $ ./app/console doctrine:migrations:generate class Version20130204114559 extends AbstractMigration { public function up(Schema $schema) { } public function down(Schema $schema) { } } • Generate a new migration • up() - execute operations when migrating • down() - reverse executed operations in up() Friday, February 8, 13

Slide 85

Slide 85 text

Deploy Migrations class Version20130204114559 extends AbstractMigration { public function up(Schema $schema) { $this->addSql('ALTER TABLE stock_items ADD forceSoldout TINYINT(1) NOT NULL DEFAULT 0'); } public function down(Schema $schema) { $this->addSql('ALTER TABLE stock_items DROP COLUMN forceSoldout'); } } Add new column in up() Remove new column in down() • Here is an example where we add a new column to a table in mysql Friday, February 8, 13

Slide 86

Slide 86 text

Deploy Migrations • Checks for new migrations that have not been executed yet • Runs the up() method on any new migrations • Hooked up to fabric deploy process so migrations are automatically executed before new code goes live $ ./app/console doctrine:migrations:migrate Friday, February 8, 13

Slide 87

Slide 87 text

Deploy Migrations • Our migrations live in a standalone git repository • Linked to DEVO with a submodule • Allows database migrations to live outside DEVO. If a migration is needed outside of a code deploy, this allows it. Friday, February 8, 13

Slide 88

Slide 88 text

Eventual Migrations • Eventually migrate data when it is read instead of migrating all data when new code is deployed • No need to migrate data if it’s never read • Eventual Migrations could be called Lazy Migrations Friday, February 8, 13

Slide 89

Slide 89 text

Eventual Migrations • Use Doctrine lifecycle callbacks to modify data when it is read Friday, February 8, 13

Slide 90

Slide 90 text

Eventual Migrations • Initially store a users full name as a single string /** @Entity */ class User { // ... /** * @Column(type="string") */ private $fullName; // ... } Friday, February 8, 13

Slide 91

Slide 91 text

Eventual Migrations • Now imagine later we want the users name split up in to ﬁrstName and lastName Friday, February 8, 13

Slide 92

Slide 92 text

Eventual Migrations /** * @Entity * @HasLifecycleCallbacks */ class User { // ... /** * @Column(type="string") * @deprecated */ private $fullName; /** * @Column(type="string") */ private $firstName; /** * @Column(type="string") */ private $lastName; /** @PostLoad */ public function postLoad() { if ($this->fullName) { $e = explode(' ', $this->fullName); $this->firstName = $e[0]; $this->lastName = $e[1]; $this->fullName = null; } } } Friday, February 8, 13

Slide 93

Slide 93 text

Reporting • Reporting is a problem I am sure everyone has faced at some point. Friday, February 8, 13

Slide 94

Slide 94 text

Choices • Model your domain and persistence so that reporting is convenient • Model your domain and persistence for your applications needs only and utilize a data warehouse where the data is transformed in to structures that are convenient for reporting Friday, February 8, 13

Slide 95

Slide 95 text

Wrong Choice • Letting your reporting needs dictate your applications domain model is a road you don’t want to go down • All you end up with is a limited reporting system and a slow production database • Too many indexes • Data stored in ways that are not convenient for the application reading and writing Friday, February 8, 13

Slide 96

Slide 96 text

Right Choice • Use a data warehouse for all your reporting needs From Wikipedia: In computing, a data warehouse is a database used for reporting and data analysis. It is a central repository of data which is created by integrating data from one or more disparate sources. Data warehouses store current as well as historical data and are used for creating trending reports for senior management reporting such as annual and quarterly comparisons. Friday, February 8, 13

Slide 97

Slide 97 text

OpenSky Data • Our data is split up across multiple sources • Production Databases • MySQL • MongoDB • Third Parties • Braintree • Vendornet • Fulﬁllment Works Friday, February 8, 13

Slide 98

Slide 98 text

Extract Transform and Load From Wikipedia: In computing, Extract, Transform and Load (ETL) refers to a process in database usage and especially in data warehousing that involves: • Extracting data from outside sources • Transforming it to ﬁt operational needs • Loading it into the end target Friday, February 8, 13

Slide 99

Slide 99 text

MySQL Data Warehouse Braintree Vendornet jetstream_mongo opensky_devo atmosphere mongo data mysql slave rollups/aggregates Cron Jobs ETL extract transform and load Production Databases MySQL MongoDB External Data Fulﬁllment Works ﬂight_deck stored procedures and views DEVO OSIS Applications Warehouse Diagram Friday, February 8, 13

Slide 100

Slide 100 text

Warehouse Databases • jetstream_mongo - mongodb replicated to mysql • jetstream_audit - third party data • opensky_devo - mysql slave • atomosphere - stores historical reports, rollups, aggregates, etc. • ﬂight_deck - No data is stored here. Friday, February 8, 13

Slide 101

Slide 101 text

ﬂight_deck • DEVO and other applications only need access to ﬂight_deck • MySQL views to expose the data needed for dashboards, reports and other reporting user interfaces. • Set of stored procedures that run on cron, updating stats, aggregates, rollups, etc. Friday, February 8, 13

Slide 102

Slide 102 text

statsd • Simple daemon for easy stats aggregation • Built by etsy • https://github.com/etsy/statsd/ Friday, February 8, 13

Slide 103

Slide 103 text

Graphite • Scalable Realtime Graphing • Renders graphs for stats collected via statsd • Monitor trends in application performance, usage, etc. Friday, February 8, 13

Slide 104

Slide 104 text

Integrate with Symfony • https://github.com/liuggio/StatsDClientBundle liuggio_stats_d_client: connection: host: localhost port: 8125 Friday, February 8, 13

Slide 105

Slide 105 text

Integrate with Symfony • Now we can easily track statistics from our code using the provided services. • For example if we want to monitor follow trends for speciﬁc sellers $statsdClientFactory = $container->get('liuggio_stats_d_client.factory'); $statsdClientService = $container->get('liuggio_stats_d_client.service') $data = $statsdClientFactory->increment(sprintf('seller.follow.%s', $seller->getSlug())); $statsdClientService->send($data); Friday, February 8, 13

Slide 106

Slide 106 text

View Realtime Data in Graphite Friday, February 8, 13

Slide 107

Slide 107 text

Other uses • Trends, trends, trends • Increment stats for every event dispatched • Track logged in users • Page views • Orders • Registrations • Performance/load time • Basically every action in the system we track Friday, February 8, 13

Slide 108

Slide 108 text

Tactical Operations Dashboard • Graphite provides easy ways to access the data for use in other graphs. • Allows us to pull in data from graphite and display on a centralized tactical operations dashboard Friday, February 8, 13

Slide 109

Slide 109 text

Tactical Operations Dashboard https://github.com/jondot/graphene Friday, February 8, 13

Slide 110

Slide 110 text

Tactical Operations Dashboard • Kept open on a LCD in ofﬁce for easy monitoring • We know when we get a trafﬁc spike • Allows us to pin point where a bottle neck is in the stack if we experience problems • Checking performance stats before and after a deploy to see if a code change had an impact on performance Friday, February 8, 13

Slide 111

Slide 111 text

Jonathan H. Wage http://twitter.com/jwage http://github.com/jwage Questions? We’re hiring! [email protected] Friday, February 8, 13