Building OpenSky with Symfony2

Building OpenSky
with Symfony2
Friday, February 8, 13

View Slide

Thanks
• ServerGrove
• Pablo
• Kim
• Adam
• And everyone else who helped organize

View Slide

@jwage
• Director of Technology at OpenSky.com
• Before OpenSky, SensioLabs
• ServerGrove Partner
• Doctrine and Symfony contributor
• Over a decade of experience building web
applications with various programming languages

View Slide

What is OpenSky?
OpenSky is a social network for shopping that
helps people discover, buy and share unique
goods that match their individual taste through
trusted insider connections.

View Slide

Who are the insiders?
• Bobby Flay
• Martha Stewart
• Shaq
• Alicia Silverstone
• Alton Brown
• Buddy Valastro (Cake Boss)
Range from celebrity chefs, actors, stylists,
fashionistas, electronics gurus, etc.

View Slide

How big is OpenSky?
• 2.5 million members
• 50 million connections
• ~70 employees
• Ofﬁces in
• New York
• Nashville
• Portland
• San Francisco

View Slide

Technology Overview
Graphite statsd

View Slide

Application Layers

View Slide

DEVO
• PHP Web Application
• Domain model and business logic
• HTTP API
• Sends messages to OSIS through HornetQ
• Receives and processes callbacks from OSIS

View Slide

OSIS
• Enterprise Service Bus (ESB)
• Mule framework
From Wikipedia: An enterprise service bus (ESB) is a
software architecture model used for designing and
implementing the interaction and communication between
mutually interacting software applications in service-
oriented architecture (SOA).

View Slide

What does OSIS do?
• Sits behind HornetQ and consumes messages
originated either from DEVO or by OSIS itself
• Processes messages and does “something”
• Interact with a 3rd party API
• Chunk message in to smaller messages and put in
other queues
• Send transactional e-mails
• Relay calls back to DEVO

View Slide

OSIS Example
When we receive shipments to process OSIS chunks
them in to batches and relays them back to DEVO for
payment capture, to mark the item as shipped and
DEVO originates another event to OSIS which notiﬁes
the customer that the item shipped.

View Slide

DEVO
web1 web3 web5
hornetq hornetq hornetq
hornetq
cluster
varnish/load
balancer
nginx
Request
web2 web4 web6
hornetq hornetq hornetq
php web nodes
OSIS
java1 java2
java nodes
MongoDB Replica Set
Primary
secondaries
MySQL Master Slave Setup
Primary
secondaries
PHP App
ESB

View Slide

Domain Model
From Wikipedia: A domain model in problem
solving and software engineering is a conceptual model
of all the topics related to a speciﬁc problem. It
describes the various entities, their attributes, roles,
and relationships, plus the constraints that govern the
problem domain.

View Slide

Domain Objects
• Plain old PHP objects (POPOs)
• Mapped to MongoDB and MySQL using Doctrine
• Don’t extend any base persistence class
• No dependencies
• THIN
• Should contain minimal business logic
• Merely wrappers around data

View Slide

Domain Objects
/** @ORM\Entity(...) */
class Order
{
/** @ORM\Id */
protected $id;
/** @ODM\ObjectId */
protected $productId;
/**
* @Gedmo\ReferenceOne(
* type="document",
* targetDocument="Product",
* identifier="productId"
* )
*/
protected $product;
// ...
}
Cross-database
references
Mapping
information via
annotations
/** @ODM\Document(...) */
class Product
{
/** @ODM\Id */
protected $id;
// ...
}

View Slide

Domain Objects
class Seller
{
/** @ODM\Id */
protected $id;
// ...
}
class User
{
/** @ODM\Id */
protected $id;
// ...
}

View Slide

Persistence decoupled
from domain objects
• Objects have no or light dependencies to other
objects.
• Only depend on other domain objects.
• Makes it easy to test your domain objects in
isolation when it is not coupled with persistence.

View Slide

Domain Managers
• They manage the business logic around domain
objects. For Example:
• OrderManager
• ProductManager
• FollowManager
• UserManager
• Used in controllers and other manager classes

View Slide

Domain Managers
class OrderManager
{
public function createOrder(Order $order);
// ...
}
class ProductManager
{
public function createProduct(Product $product);
// ...
}

View Slide

Domain Managers
class FollowManager
{
public function follow(User $user, Seller $seller);
// ...
}
class UserManager
{
public function createUser(User $user);
// ...
}

View Slide

Using Domain Managers
$dm = $container->get('doctrine.odm.mongodb.document_manager');
$followManager = $container->get('seller.follow_manager');
$sellerRepository = $container->get('seller.repository');
$userRepository = $container->get('user.repository');
$user = $userRepository->findOneBySlug('jwage');
$seller = $sellerRepository->findOneBySlug('shaq');
$followManager->follow($user, $seller);
$dm->flush();
• Example usage where a User follows a Seller

View Slide

Using Domain Managers
• Usage in a controller
• Keep business logic out of controllers.
• Delegate work to services (domain managers, libs, etc.)
class FollowController
{
// ...
public function follow($sellerSlug)
{
$seller = $this->sellerRepository->findOneBySlug($sellerSlug);
$user = $this->loggedInUserProvider->getLoggedInUser();
$this->followManager->follow($user, $seller);
// give the request a response
}
// ...
}

View Slide

Decoupled Code
• The actual work is abstracted away in a nice clean
API that the controller can use.
class FollowManager
{
// ...
public function follow(User $user, Seller $seller)
{
$user->followSeller($seller);
// ...
}
}

View Slide

Decoupled Code
• How can I do something else when
FollowManager::follow() is called?
class SellerManager
{
// ...
{
// ...
}
}

View Slide

EventDispatcher
• Events are notiﬁed when domain manager classes
perform an operation. For example:
• order.created
• user.created
• product.created
• seller.follow

View Slide

EventDispatcher
• Notify an Event when FollowManager::follow() is called
class FollowManager
{
// ...
{
$this->eventDispatcher->notify(new Event($seller, 'seller.follow', array(
'user' => $user
)));
}
}

View Slide

Note about our
EventDispatcher
• We still utilize the old style of Symfony events
where you have a generic Event class with a subject
and parameters.
• We had already adopted the old style of events
before it was changed before Symfony2 was released
• Easier to serialize and unserialize the generic Event
object

View Slide

Listening to the Event

class SellerFollowListener
{
/**
* Listens to 'seller.follow'
*/
public function onSellerFollow(EventInterface $event)
{
$seller = $event->getSubject();
$user = $event['user'];
// do something
}
}

View Slide

Sending messages to
HornetQ from PHP
• To send messages to HornetQ from PHP we use a
library called Fuse Stomp http://stomp.fusesource.org
$connection = new Stomp('tcp://localhost:61613');
$connection->connect('username', 'password');
$message = array(
'some' => 'data',
'more' => 'data'
);
// send headers to use special HornetQ functionality
$headers = array(
'_HQ_SCHED_DELIVERY' => 3600 // delay by an hour
);
// send the message
$connection->send('jms.queue.name', json_encode($message), $headers);

View Slide

Integrating with Symfony
• ClientInterface - what sends the message
• MessageInterface - what gets sent
• EventForwarderInterface - convert a Symfony Event
to a Message that is sent

View Slide

interface ClientInterface
{
function connect();
function disconnect();
function send(MessageInterface $message);
}
interface MessageInterface
{
function getHeaders();
function getParameters();
function getQueueName();
}
interface EventForwarderInterface
{
function forward(Event $event);
}

View Slide

$client = new Client(new Stomp('tcp://localhost:61613'));
$eventForwarder = new EventForwarder($client);
$eventForwarder->forward(new Event($seller, 'seller.follow', array(
'user' => $user
)));
• Now we have a simple API that lets us forward
Symfony Event instances to HornetQ

View Slide

• From the DIC we can conﬁgure an event to be
forwarded to hornetq

jms.queue.opensky.seller


View Slide

Asynchronous Events
• Having HornetQ is a convenient middle man and
allows DEVO to behave in asynchronous ways
• We’ve added an EventDispatcher::notifyAsync()
method that forwards the Event to HornetQ and
OSIS slings the event back to DEVO asynchronously.

View Slide

Asynchronous Events
• Imagine we want to use a third party API to check
for fraudulent orders but we don’t want the main
request to hang while we do this check.
• The event is serialized in the EventForwarder and
sent to a queue in OSIS which immediately slings it
back to DEVO in another request.
$eventDispatcher->notifyAsync(new Event($order, 'order.check_fraud'));

View Slide

Asynchronous Events
• A DEVO controller receives the request and
reconstructs the original event and notiﬁes it
class EventController
{
// ...
public function handle()
{
$event = $this->getEventFactory()->getReconstructedEvent($request);
$this->getEventDispatcher()->notify($event);
}
}

View Slide

Asynchronous Events
• Now when you listen to order.check_fraud it
happens in another process asynchronously.
class OrderCheckFraudListener
{
// ...
public function onOrderCheckFraud(EventInterface $event)
{
$order = $event->getSubject(); // $order instanceof User
$this->orderFraudDetection->checkOrder($order);
}
}


View Slide

Facebook Realtime
Updates as Symfony Events

View Slide

Facebook Realtime Updates
• Facebook will post to a callback URL when an event
happens. For example if a user adds or removes a
friend.

View Slide

• Facebook posts an array of data like the following to
your callback URL
$change = array(
'object' => 'user',
'entry' => array(
array(
'uid' => '1234',
'changed_fields' => array(
'friends'
)
),
array(
'uid' => '12345',
'changed_fields' => array(
'friends'
)
)
)
);

View Slide

• Controller to receive the POST from Facebook
class RealtimeUpdatesController
{
public function callbackAction()
{
$request = $this->getRequest();
$manager = $this->getRealtimeUpdatesManager();
if ($manager->isSubscribeRequest($request)) {
if (!$manager->isSubscribeRequestValid($request)) {
throw new NotFoundHttpException();
}
return new Response($manager->getSubscribeRequestResponse($request));
} else if ($manager->isUpdateRequest($request)) {
$manager->handleUpdate($request);
return new Response();
}
throw new NotFoundHttpException();
}
// ..
}

View Slide

• Convert callback data to Symfony events
class RealtimeUpdatesManager
{
// ...
public function handleUpdate(Request $request)
{
$callback = $request->request->all();
if (isset($callback['object']) && isset($callback['entry'])) {
$eventName = 'facebook.realtime_update.%s.%s';
if ($callback['entry']) {
foreach ($callback['entry'] as $entry) {
$user = $this->dm->getRepository('MainBundle:User')->findUserByFacebookId($entry['uid']);
if ($user === null) {
continue;
}
foreach ($entry['changed_fields'] as $field) {
$eventName = sprintf('facebook.realtime_updates.%s.%s', $callback['object'], $field);
$this->eventDispatcher->notify(new Event($user, $eventName));
}
}
}
}
}
}

View Slide

• facebook.realtime_updates.user.friends
• facebook.realtime_updates.user.hometown

View Slide

• Listen to the realtime updates and execute PHP code
class UserFriendsRealtimeUpdateListener
{
private $facebookFriendManager;
public function __construct(FacebookFriendManager $facebookFriendManager)
{
$this->facebookFriendManager = $facebookFriendManager;
}
public function onUserFriendsRealtimeUpdate(EventInterface $event)
{
$this->facebookFriendManager->updateUserFriends($event->getSubject());
}
}


View Slide

Rules Engine
• Ruler
• A simple stateless production rules engine for
PHP 5.3+
• Written by Justin Hileman
• Used to work at OpenSky
• https://github.com/bobthecow/ruler

View Slide

Ruler Usage
$rb = new RuleBuilder;
$rule = $rb->create(
$rb->logicalAnd(
$rb['minAge']->greaterThan($rb['age']),
$rb['maxAge']->lessThan($rb['age'])
),
function() {
echo 'Congratulations! You are between the ages of 18 and 25!';
}
);
$context = new Context(array(
'minAge' => 18,
'maxAge' => 25,
'age' => function() {
return 20;
},
));
$rule->execute($context); // "Congratulations! You are between the ages of 18 and 25!"

View Slide

What does OpenSky
use Ruler for?
• Issuing rewards to users when an event is notiﬁed
and certain criteria is met
• $$$ Credits for shopping
• Points for rewards club
• New member levels when point thresholds are
reached
• Free shipping

View Slide

Rule Model
• Rule
• Condition
• Reward
interface Rule
{
function execute(Context $context);
}
interface Condition
{
function getType();
function buildRule(RuleBuilder $rb);
function buildContext(Context $context);
}
interface Reward
{
function getType();
function getRecipientUser(Context $context);
function issueReward(Context $context);
}

View Slide

Rule
class Rule
{
private $eventName;
private $condition;
private $reward;
// ...
public function evaluate(Context $context)
{
$this->buildContext($context);
return $this->buildRule()->evaluate($context);
}
public function execute(Context $context)
{
if ($this->evaluate($context)) {
$this->issueReward($context);
}
}
protected function issueReward(Context $context)
{
return $this->reward->issueReward($context);
}
protected function buildRule()
{
if (!isset($this->rule)) {
$rb = new RuleBuilder();
$rule = $this->condition->buildRule($rb);
// ...
$this->rule = $rule;
}
return $this->rule;
}
protected function buildContext(Context $context)
{
$this->condition->buildContext($context);
// ...
}
}

View Slide

Example Reward Rule
• On a users ﬁrst purchase issue them a $5 credit

View Slide

OrderCondition
class OrderCondition extends Condition
{
private $requireFirstPurchase;
// ...
public function getType()
{
return 'order';
}
public function buildRule(RuleBuilder $rb)
{
if ($this->requireFirstPurchase) {
$rule = $rb->logicalAnd(
$rule,
$rb['user.is_first_purchase']->equalTo(true)
);
}
}
public function buildContext(Context $context)
{
if ($this->requireFirstPurchase) {
$context['user.is_first_purchase'] = $context->share(function($context) {
return $context['em']->getRepository('MainBundle:Order')
->isUsersFirstOrder($context['order']);
});
}
}
}

View Slide

CreditReward
class CreditReward extends Reward
{
private $amount;
// ...
public function getType()
{
return 'credit';
}
public function getRecipientUser(Context $context)
{
return $context['order']->getUser();
}
public function issueReward(Context $context)
{
$user = $this->getRecipientUser($context);
$context['creditManager']->issueCredit($user, $this->amount);
}
}

View Slide

Executing the Rule
$context = new Context(array(
'em' => $container->get('doctrine.orm.entity_manager'),
'creditManager' => $container->get('user.credit_manager'),
'order' => $order
));
$reward = new CreditReward();
$reward->setAmount(5);
$orderCondition = new OrderCondition();
$orderCondition->setRequireFirstPurchase(true);
$rule = new Rule();
$rule->setEventName('order.created');
$rule->setCondition($orderCondition);
$rule->setReward($reward);
$rule->execute($context); // user gets $5 credit if it is the users first purchase

View Slide

Storing Rules in
MongoDB
• Rules are stored in MongoDB
• Backend admin area for adding new rules
• Mix and match different combinations of conditions
and rewards

View Slide

Rules and Events
• Rules stored in the database are subscribed to
specific event names
• When an event is notified we can configure the DIC
to check the rules database for any rules to evaluate

View Slide

Rules and Events
class RewardRuleListener
{
// ...
public function handleEvent(Event $event)
{
$rules = $this->findRulesForEvent($event);
foreach ($rules as $rule) {
$context = $this->createContext($event);
$context['rule'] = $rule;
$rule->execute($context);
}
}
private function createContext(Event $event)
{
return new Context(array(
// ...
));
}
// ...
}


View Slide

Doctrine Lifecycle
Events

View Slide

What are they?
• Feature of Doctrine
• Doctrine triggers several events internally during the
life-time of a domain object
• Execute other code when these events occur

View Slide

What kind of events?
• preRemove - before an object is removed
• postRemove - after an object is removed
• prePersist - before an object is persisted
• postPersist - after an object is persisted
• preUpdate - before an object is updated
• postUpdate - after an object is updated
• postLoad - after an object is loaded
• loadClassMetadata - when a classes metdata is loaded
• onFlush - when the object manager is ﬂushed

View Slide

Lifecycle Event
Callbacks
• A lifecycle event callback is deﬁned on the domain
object directly and is executed when when the
lifecycle event occurs.
/**
* @Entity
*/
class User
{
// ...
/** @PrePersist */
public function doSomethingOnPrePersist()
{
$this->createdAt = new \DateTime();
}
}

View Slide

Listening to Lifecycle
Events
• Much more powerful than the simple lifecycle
callbacks that are deﬁned on the object classes.
• Implement re-usable behaviors between different
domain object classes.
• To register an event listener you have to hook it
into the EventManager

View Slide

Listening to Lifecycle
Events
$eventManager = $em->getEventManager();
$eventManager->addEventListener(array(Events::preUpdate), new MyEventListener());
$eventManager->addEventSubscriber(new MyEventSubscriber());
• Event listener
• Event subscribers

View Slide

Event Listener
class MyEventListener
{
public function preUpdate(PreUpdateEventArgs $eventArgs)
{
// ...
}
}

View Slide

Event Subscriber
• Only difference between listeners and subscribers is
that the MyEventSubscriber class provides the array
of events it should subscribe to.
class MyEventSubscriber
{
{
// ...
}
public function getSubscribedEvents()
{
return array(Events::preUpdate);
}
}

View Slide

Conﬁguring from
Symfony

• Registering a listener from the Symfony2 DIC is easy
enough

View Slide

PreUpdateEventArgs
• If you noticed, the listener preUpdate() method
receives an instance of PreUpdateEventArgs
• The PreUpdateEventArgs gives you access to all the
ﬁelds that have changed for this object with the old
and new value

View Slide

PreUpdateEventArgs
• getEntity() - to get access to the actual entity.
• getEntityChangeSet() - to get a copy of the
changeset array.
• hasChangedField($fieldName) - to check if
the given field name of the current entity changed.
• getOldValue($fieldName) - to access the old
value of the field
• getNewValue($fieldName) - to access the
new value of the field
• setNewValue($fieldName, $value) - to
change the value of a field to be updated.

View Slide

PreUpdateEventArgs
class NeverAliceOnlyBobListener
{
{
if ($eventArgs->getEntity() instanceof User) {
if ($eventArgs->hasChangedField('name') && $eventArgs-
>getNewValue('name') == 'Alice') {
$eventArgs->setNewValue('name', 'Bob');
}
}
}
}
• Example demonstrating the API of
PreUpdateEventArgs

View Slide

What does OpenSky
use lifecycle events for?
• preFlush - record all changesets that occur in our
admin control panel and provide a frontend for
browsing object changeset history

View Slide

What does OpenSky
use lifecycle events for?
• Setting createdAt and updatedAt ﬁelds automatically
/** @Entity */
class User
{
/** @PrePersist */
public function initializeCreatedAt()
{
$this->createdAt = new DateTime();
}
/** @PreUpdate */
public function updateUpdatedAt()
{
$this->updatedAt = new DateTime();
}
}

View Slide

API
• We use the JMSSerializerBundle for our API
• Written by Johannes Schmitt
• https://github.com/schmittjoh/JMSSerializerBundle

View Slide

API
• Use annotations to deﬁne the properties on our
domain objects that we want to be serialized for
our API
class Product
{
// ...
/**
* @ODM\String
* @Assert\NotBlank(message="Please provide a sellable name", groups={"Default",
"Publish"})
* @SerializerGroups({"api"})
*/
protected $name;
// ...
}

View Slide

API
class ProductsController extends BaseController
{
public function readAction($productId)
{
// ...
$serialized = $this->getSerializer()->serialize(
$product, 'json', array('api')
);
return $this->createResponse(200, $serialized);
}
// ...
}
• Create a controller that can serialize the properties
we annotated as JSON

View Slide

API
• Using the JMSSerializerBundle allowed us to rapidly
build an API with minimal effort
• It took 6 weeks from start to ﬁnish to build out the
API, build an iPhone application and get it submitted
to the app store.
• Wouldn’t have been possible if we home built
everything

View Slide

Local Test Data
• Most common options I see
• Use data ﬁxtures - https://github.com/doctrine/
data-ﬁxtures
• Restore production database locally
• What other options???

View Slide

Fixtures
• Pros
• Easy to start, provides data to test with before
you have production data
• Flexible, you can add data for any case you need
to test
• Cons
• Tedious and cumbersome to add and keep
updated test data for real production scenarios

View Slide

Restore production
database locally
• Pros
• Gives you all the data you would ever need to
test with
• Troubleshoot production issues locally with
production data
• Cons
• Security nightmare. Developers downloading
customers personal information is no good
• Doesn’t scale. You can’t download and restore
your whole production database forever

View Slide

Sanitized Production Slice
• At OpenSky we wrote a custom tool that will
extract a slice of production data
• Used to build regression/QA testing environments.
• Developers download and restore the slice locally
for development
• All personal information is sanitized to ensure
customer security.
• Slice regenerated from production automatically
every morning

View Slide

• Pros
• Contains real production data
• It is small, so downloading and restoring is fast
• Cons
• You have to update the slice tool when something
is added to the model and you want the data to
be included in the slice
Sanitized Production Slice

View Slide

Database Migrations
Types of database migrations we use at OpenSky

View Slide

Deploy Migrations
• Execute migrations on the database before a new
version is deployed and made live.
• Doctrine migrations library
• https://github.com/doctrine/migrations
• http://docs.doctrine-project.org/projects/doctrine-
migrations/en/latest/index.html

View Slide

Deploy Migrations
• Deploying new code often requires a DB migration
• Add new tables
• Add new ﬁelds
• Migrate some data
• Rename ﬁelds
• Remove deprecated data
• Anything else you can imagine
• Try to make migrations backwards compatible to
avoid downtime while migrations are performed and
new code deployed

View Slide

Deploy Migrations
$ ./app/console doctrine:migrations:generate
class Version20130204114559 extends AbstractMigration
{
public function up(Schema $schema)
{
}
public function down(Schema $schema)
{
}
}
• Generate a new migration
• up() - execute operations when migrating
• down() - reverse executed operations in up()

View Slide

Deploy Migrations
class Version20130204114559 extends AbstractMigration
{
public function up(Schema $schema)
{
$this->addSql('ALTER TABLE stock_items ADD forceSoldout
TINYINT(1) NOT NULL DEFAULT 0');
}
public function down(Schema $schema)
{
$this->addSql('ALTER TABLE stock_items DROP COLUMN
forceSoldout');
}
}
Add new column in up()
Remove new column in down()
• Here is an example where we add a new column to a
table in mysql

View Slide

Deploy Migrations
• Checks for new migrations that have not been
executed yet
• Runs the up() method on any new migrations
• Hooked up to fabric deploy process so migrations
are automatically executed before new code goes
live
$ ./app/console doctrine:migrations:migrate

View Slide

Deploy Migrations
• Our migrations live in a standalone git repository
• Linked to DEVO with a submodule
• Allows database migrations to live outside DEVO. If
a migration is needed outside of a code deploy, this
allows it.

View Slide

Eventual Migrations
• Eventually migrate data when it is read instead of
migrating all data when new code is deployed
• No need to migrate data if it’s never read
• Eventual Migrations could be called Lazy Migrations

View Slide

Eventual Migrations
• Use Doctrine lifecycle callbacks to modify data
when it is read

View Slide

Eventual Migrations
• Initially store a users full name as a single string
/** @Entity */
class User
{
// ...
/**
* @Column(type="string")
*/
private $fullName;
// ...
}

View Slide

Eventual Migrations
• Now imagine later we want the users name split up
in to ﬁrstName and lastName

View Slide

Eventual Migrations
/**
* @Entity
* @HasLifecycleCallbacks
*/
class User
{
// ...
/**
* @deprecated
*/
private $fullName;
/**
*/
private $firstName;
/**
*/
private $lastName;
/** @PostLoad */
public function postLoad()
{
if ($this->fullName) {
$e = explode(' ', $this->fullName);
$this->firstName = $e[0];
$this->lastName = $e[1];
$this->fullName = null;
}
}
}

View Slide

Reporting
• Reporting is a problem I am sure everyone has faced
at some point.

View Slide

Choices
• Model your domain and persistence so that
reporting is convenient
• Model your domain and persistence for your
applications needs only and utilize a data warehouse
where the data is transformed in to structures that
are convenient for reporting

View Slide

Wrong Choice
• Letting your reporting needs dictate your
applications domain model is a road you don’t want
to go down
• All you end up with is a limited reporting system
and a slow production database
• Too many indexes
• Data stored in ways that are not convenient for
the application reading and writing

View Slide

Right Choice
• Use a data warehouse for all your reporting needs
From Wikipedia: In computing, a data warehouse
is a database used for reporting and data analysis.
It is a central repository of data which is created
by integrating data from one or more disparate
sources. Data warehouses store current as well as
historical data and are used for creating trending
reports for senior management reporting such as
annual and quarterly comparisons.

View Slide

OpenSky Data
• Our data is split up across multiple sources
• Production Databases
• MySQL
• MongoDB
• Third Parties
• Braintree
• Vendornet
• Fulﬁllment Works

View Slide

Extract Transform and Load
From Wikipedia: In computing, Extract, Transform
and Load (ETL) refers to a process in database usage
and especially in data warehousing that involves:
• Extracting data from outside sources
• Transforming it to ﬁt operational needs
• Loading it into the end target

View Slide

MySQL
Data Warehouse
Braintree
Vendornet
jetstream_mongo opensky_devo atmosphere
mongo data mysql slave rollups/aggregates
Cron Jobs
ETL
extract transform and load
Production Databases
MySQL
MongoDB
External Data
Fulﬁllment
Works
ﬂight_deck
stored procedures and views
DEVO
OSIS
Applications
Warehouse Diagram

View Slide

Warehouse Databases
• jetstream_mongo - mongodb replicated to mysql
• jetstream_audit - third party data
• opensky_devo - mysql slave
• atomosphere - stores historical reports, rollups,
aggregates, etc.
• ﬂight_deck - No data is stored here.

View Slide

ﬂight_deck
• DEVO and other applications only need access to
ﬂight_deck
• MySQL views to expose the data needed for
dashboards, reports and other reporting user
interfaces.
• Set of stored procedures that run on cron,
updating stats, aggregates, rollups, etc.

View Slide

statsd
• Simple daemon for easy stats aggregation
• Built by etsy
• https://github.com/etsy/statsd/

View Slide

Graphite
• Scalable Realtime Graphing
• Renders graphs for stats collected via statsd
• Monitor trends in application performance, usage,
etc.

View Slide

Integrate with Symfony
• https://github.com/liuggio/StatsDClientBundle
liuggio_stats_d_client:
connection:
host: localhost
port: 8125

View Slide

Integrate with Symfony
• Now we can easily track statistics from our code
using the provided services.
• For example if we want to monitor follow trends
for speciﬁc sellers
$statsdClientFactory = $container->get('liuggio_stats_d_client.factory');
$statsdClientService = $container->get('liuggio_stats_d_client.service')
$data = $statsdClientFactory->increment(sprintf('seller.follow.%s', $seller->getSlug()));
$statsdClientService->send($data);

View Slide

View Realtime Data in
Graphite

View Slide

Other uses
• Trends, trends, trends
• Increment stats for every event dispatched
• Track logged in users
• Page views
• Orders
• Registrations
• Performance/load time
• Basically every action in the system we track

View Slide

Tactical Operations
Dashboard
• Graphite provides easy ways to access the data for
use in other graphs.
• Allows us to pull in data from graphite and display
on a centralized tactical operations dashboard

View Slide

Tactical Operations
Dashboard
https://github.com/jondot/graphene

View Slide

Tactical Operations
Dashboard
• Kept open on a LCD in ofﬁce for easy monitoring
• We know when we get a trafﬁc spike
• Allows us to pin point where a bottle neck is in the
stack if we experience problems
• Checking performance stats before and after a
deploy to see if a code change had an impact on
performance

View Slide

Jonathan H. Wage
http://twitter.com/jwage
http://github.com/jwage
Questions?
We’re hiring! [email protected]

View Slide

Building OpenSky with Symfony2

Building OpenSky with Symfony2

Jonathan H. Wage

Other Decks in Technology

Featured

Transcript