Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The evolution of a rabbit

Da2d2829b89cde136392973a35b68959?s=47 nealio82
January 21, 2015

The evolution of a rabbit

How our API call queue system evolved from a single 'select rows from the database', through a MySql simple queue and finally to using a dedicated message queue with Rabbit MQ.

Watch a video of the original presentation here: https://www.youtube.com/watch?v=iCg6t5iedEw

Da2d2829b89cde136392973a35b68959?s=128

nealio82

January 21, 2015
Tweet

Transcript

  1. The$evolu)on$of$a$rabbit

  2. None
  3. @nealio82 • I’m%a%PHP%developer • I’ve%been%using%Symfony%since%2011 • I%run%the%TwiAer%product%at%a%major%TwiAer%&%Facebook% adverFsing%partner • I’m%studying%physics%at%the%Open%University

    • I%mainly%tweet%about%science,%nature%&%development
  4. Our$requirement$&$implementa/on

  5. Our$requirement$&$implementa/on • We$are$a$major$social$media$ads$API$consumer • We$work$with$massive$ads$agencies$selling$products$you$use • We$make$thousands$of$API$calls$per$day • We$needed$to$find$a$scalable$solu?on$to$give$us$more$flexibility$ over$the$exis?ng$solu?on

  6. The$first$genera,on

  7. The$first$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs

  8. The$first$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs • Crons&are&running&Symfony&tasks&which&each&perform&one& funcBon

  9. The$first$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs • Crons&are&running&Symfony&tasks&which&each&perform&one& funcBon • Sync&process&spawning&mulBple&curl&requests&to&localhost

  10. The$first$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs • Crons&are&running&Symfony&tasks&which&each&perform&one& funcBon • Sync&process&spawning&mulBple&curl&requests&to&localhost • We&have&a&lot&of&Symfony&tasks&running

  11. 120$cron$tasks$in$total Most%of%them%run%every%minute

  12. SELECT [cols] FROM `adverts_tbl` WHERE `update_flag` = 'U';

  13. SELECT [cols] FROM `adverts_tbl` WHERE `update_flag` = 'U'; Problems: •

    Only&a&certain&amount&can&be&processed&within&a&minute • Might&only&select&a&few&rows&but&need&to&check&hundreds&of& thousands&in&table • Big&data&read&spike&on&DB&&&CPU&spike&every&60&seconds • Data&comparison&performed&on&nonCindexed&column
  14. The$second$genera-on

  15. SELECT [cols] FROM `queue_tbl` q INNER JOIN `adverts_tbl` WHERE q.`update_flag`

    = 'U';
  16. SELECT [cols] FROM `queue_tbl` q INNER JOIN `adverts_tbl` WHERE q.`update_flag`

    = 'U'; Benefits: • Selec&ng)from)fewer)rows) • Joining)on)PKs)&)indexes • Faster!)(but)not)benchmarked)
  17. SELECT [cols] FROM `queue_tbl` q INNER JOIN `adverts_tbl` WHERE q.`update_flag`

    = 'U'; Problems: • Only&a&certain&amount&can&be&processed&within&a&minute • Big&data&read&spike&on&DB&&&CPU&spike&every&60&seconds • Complexity&increases&as&different&queue&types&become&required& (13&queue&tables)
  18. * sync progress table * sync users table * sync

    promoted tweets table * sync targeting table * campaign creation queue table * campaign update / deletion queue table * tweet creation queue table * ads queue table * ads targeting queue * ads account queue * ads tweets queue * ads embedded creatives updates table
  19. None
  20. <?php $campaign->queue()->create(); $campaign->queue()->retrieve(); $campaign->queue()->update(); $campaign->queue()->delete();

  21. <?php use path\to\TwitterQueue; class Campaign { public function queue() {

    return new TwitterQueue('CampaignQueue', 'PromotedTweetId', $this->getId()); } // …
  22. <?php class TwitterQueue { public function __construct($queue_class, $finder_column, $entity_id =

    null) { $this->queue_class = $queue_class; $this->finder_column = $finder_column; if ($entity_id) { return $this->getQueue($entity_id); } $this->makeNewQueue(); return $this; } // …
  23. Inside'a'task fetch [x] campaigns / adverts from database loop rows

    fetch access token & other required info from database perform api call handle response & save back to db
  24. None
  25. The$Rabbit$MQ$solu0on

  26. How$ $Rabbit$MQ$works • Producer)sends)a)message)to)Rabbit • Rabbit)queues)the)message • Consumer)receives)the)message)and)acts)upon)it

  27. <?php $campaign->queue('post'); $campaign->queue('get'); $campaign->queue('put'); $campaign->queue('delete');

  28. <?php use path\to\TwitterQueue; class Campaign { public function queue($action) {

    (new TwitterQueue)->Queue('Campaign', $this->createMessage($action)); } private function createMessage($action) { return json_encode(array( 'access_token' => $this->getAccount()->getAccessToken(), 'call_url' => 'https://ads.twitter.com/campaigns', 'method' => strtoupper($action), 'data' => $this->getURIParams() )); } // …
  29. <?php use PhpAmqpLib\Connection\AMQPConnection; use PhpAmqpLib\Message\AMQPMessage; class TwitterQueue { private $connection,

    $channel; public function __construct() { $this->connection = new AMQPConnection('localhost', 5672, 'guest', 'guest'); $this->channel = $this->connection->channel(); } public function Queue($queue_name, $content) { // prevent queue from auto-deleting with all the falses. $this->channel->queue_declare($queue_name, false, false, false, false); // pass $queue_name as the routing key $this->channel->basic_publish(new AMQPMessage($content), '', $queue_name); } // …
  30. • The%consumer%can%be%wri1en%in%any%language

  31. • The%consumer%can%be%wri1en%in%any%language • Try%to%choose%something%fast

  32. None
  33. None
  34. Benefits: • Queue%processing%load%taken%off%web%server • Number%of%DB%calls%reduced • Queues%processed%con<nually • Consumer%is%fast •

    Simplified%architecture
  35. None
  36. None
  37. None
  38. Re#queueing)messages

  39. None
  40. Implementa)on+Summary !Good!points • Already)removed)13)cron)jobs • Queues)are)processed)con7nuously • Load)moved)away)from)client)facing)servers • Schema)&)code)complexity)reduced

    • Database)load)reduced • Sync)is)much)faster,)especially)for)large)accounts
  41. Implementa)on+Summary Bad$points • Re$modelling,database,in,Sequelize,slowed,down,development

  42. Implementa)on+Summary Bad$points • Re$modelling,database,in,Sequelize,slowed,down,development • We,have,to,maintain,the,DB,model,in,both,Doctrine,and, Sequelize

  43. None
  44. None
  45. None
  46. What’s'next? • Implement)Rabbit)queues)for)campaign)crea6on)and)updates • Implement)Rabbit)for)fetching)reports)from)the)Twi;er)API • Implement)Rabbit)for)the)Facebook)part)of)our)tool • We’ve)already)built)our)iAds)implementa6on)on)top)of)Rabbit

  47. @nealio82