The evolution of a rabbit

How our API call queue system evolved from a single 'select rows from the database', through a MySql simple queue and finally to using a dedicated message queue with Rabbit MQ.

Watch a video of the original presentation here: https://www.youtube.com/watch?v=iCg6t5iedEw

Transcript

1. The$evolu)on$of$a$rabbit
3. @nealio82 • I’m%a%PHP%developer • I’ve%been%using%Symfony%since%2011 • I%run%the%TwiAer%product%at%a%major%TwiAer%&%Facebook% adverFsing%partner • I’m%studying%physics%at%the%Open%University • I%mainly%tweet%about%science,%nature%&%development
4. Our$requirement$&$implementa/on
5. Our$requirement$&$implementa/on • We$are$a$major$social$media$ads$API$consumer • We$work$with$massive$ads$agencies$selling$products$you$use • We$make$thousands$of$API$calls$per$day • We$needed$to$find$a$scalable$solu?on$to$give$us$more$flexibility$ over$the$exis?ng$solu?on
6. The$first$genera,on
7. The$first$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs
8. The$first$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs • Crons&are&running&Symfony&tasks&which&each&perform&one& funcBon
9. The$first$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs • Crons&are&running&Symfony&tasks&which&each&perform&one& funcBon • Sync&process&spawning&mulBple&curl&requests&to&localhost
10. The$first$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs • Crons&are&running&Symfony&tasks&which&each&perform&one& funcBon • Sync&process&spawning&mulBple&curl&requests&to&localhost • We&have&a&lot&of&Symfony&tasks&running
11. 120$cron$tasks$in$total Most%of%them%run%every%minute
12. SELECT [cols] FROM `adverts_tbl` WHERE `update_flag` = 'U';
13. SELECT [cols] FROM `adverts_tbl` WHERE `update_flag` = 'U'; Problems: • Only&a&certain&amount&can&be&processed&within&a&minute • Might&only&select&a&few&rows&but&need&to&check&hundreds&of& thousands&in&table • Big&data&read&spike&on&DB&&&CPU&spike&every&60&seconds • Data&comparison&performed&on&nonCindexed&column
14. The$second$genera-on
15. SELECT [cols] FROM `queue_tbl` q INNER JOIN `adverts_tbl` WHERE q.`update_flag` = 'U';
16. SELECT [cols] FROM `queue_tbl` q INNER JOIN `adverts_tbl` WHERE q.`update_flag` = 'U'; Benefits: • Selec&ng)from)fewer)rows) • Joining)on)PKs)&)indexes • Faster!)(but)not)benchmarked)
17. SELECT [cols] FROM `queue_tbl` q INNER JOIN `adverts_tbl` WHERE q.`update_flag` = 'U'; Problems: • Only&a&certain&amount&can&be&processed&within&a&minute • Big&data&read&spike&on&DB&&&CPU&spike&every&60&seconds • Complexity&increases&as&different&queue&types&become&required& (13&queue&tables)
18. * sync progress table * sync users table * sync promoted tweets table * sync targeting table * campaign creation queue table * campaign update / deletion queue table * tweet creation queue table * ads queue table * ads targeting queue * ads account queue * ads tweets queue * ads embedded creatives updates table
20. <?php $campaign->queue()->create(); $campaign->queue()->retrieve(); $campaign->queue()->update(); $campaign->queue()->delete();
21. <?php use path\to\TwitterQueue; class Campaign { public function queue() { return new TwitterQueue('CampaignQueue', 'PromotedTweetId', $this->getId()); } // …
22. <?php class TwitterQueue { public function __construct($queue_class, $finder_column, $entity_id = null) { $this->queue_class = $queue_class; $this->finder_column = $finder_column; if ($entity_id) { return $this->getQueue($entity_id); } $this->makeNewQueue(); return $this; } // …
23. Inside'a'task fetch [x] campaigns / adverts from database loop rows fetch access token & other required info from database perform api call handle response & save back to db
25. The$Rabbit$MQ$solu0on
26. How$ $Rabbit$MQ$works • Producer)sends)a)message)to)Rabbit • Rabbit)queues)the)message • Consumer)receives)the)message)and)acts)upon)it
27. <?php $campaign->queue('post'); $campaign->queue('get'); $campaign->queue('put'); $campaign->queue('delete');
28. <?php use path\to\TwitterQueue; class Campaign { public function queue($action) { (new TwitterQueue)->Queue('Campaign', $this->createMessage($action)); } private function createMessage($action) { return json_encode(array( 'access_token' => $this->getAccount()->getAccessToken(), 'call_url' => 'https://ads.twitter.com/campaigns', 'method' => strtoupper($action), 'data' => $this->getURIParams() )); } // …
29. <?php use PhpAmqpLib\Connection\AMQPConnection; use PhpAmqpLib\Message\AMQPMessage; class TwitterQueue { private $connection, $channel; public function __construct() { $this->connection = new AMQPConnection('localhost', 5672, 'guest', 'guest'); $this->channel = $this->connection->channel(); } public function Queue($queue_name, $content) { // prevent queue from auto-deleting with all the falses. $this->channel->queue_declare($queue_name, false, false, false, false); // pass $queue_name as the routing key $this->channel->basic_publish(new AMQPMessage($content), '', $queue_name); } // …
30. • The%consumer%can%be%wri1en%in%any%language
31. • The%consumer%can%be%wri1en%in%any%language • Try%to%choose%something%fast
34. Benefits: • Queue%processing%load%taken%off%web%server • Number%of%DB%calls%reduced • Queues%processed%con<nually • Consumer%is%fast • Simplified%architecture
38. Re#queueing)messages
40. Implementa)on+Summary !Good!points • Already)removed)13)cron)jobs • Queues)are)processed)con7nuously • Load)moved)away)from)client)facing)servers • Schema)&)code)complexity)reduced • Database)load)reduced • Sync)is)much)faster,)especially)for)large)accounts
41. Implementa)on+Summary Bad$points • Re$modelling,database,in,Sequelize,slowed,down,development
42. Implementa)on+Summary Bad$points • Re$modelling,database,in,Sequelize,slowed,down,development • We,have,to,maintain,the,DB,model,in,both,Doctrine,and, Sequelize
46. What’s'next? • Implement)Rabbit)queues)for)campaign)crea6on)and)updates • Implement)Rabbit)for)fetching)reports)from)the)Twi;er)API • Implement)Rabbit)for)the)Facebook)part)of)our)tool • We’ve)already)built)our)iAds)implementa6on)on)top)of)Rabbit
47. @nealio82