The evolution of a rabbit

Slide 1

Slide 1 text

The$evolu)on$of$a$rabbit

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

@nealio82 • I’m%a%PHP%developer • I’ve%been%using%Symfony%since%2011 • I%run%the%TwiAer%product%at%a%major%TwiAer%&%Facebook% adverFsing%partner • I’m%studying%physics%at%the%Open%University • I%mainly%tweet%about%science,%nature%&%development

Slide 4

Slide 4 text

Our$requirement$&$implementa/on

Slide 5

Slide 5 text

Our$requirement$&$implementa/on • We$are$a$major$social$media$ads$API$consumer • We$work$with$massive$ads$agencies$selling$products$you$use • We$make$thousands$of$API$calls$per$day • We$needed$to$ﬁnd$a$scalable$solu?on$to$give$us$more$ﬂexibility$ over$the$exis?ng$solu?on

Slide 6

Slide 6 text

The$ﬁrst$genera,on

Slide 7

Slide 7 text

The$ﬁrst$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs

Slide 8

Slide 8 text

The$ﬁrst$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs • Crons&are&running&Symfony&tasks&which&each&perform&one& funcBon

Slide 9

Slide 9 text

The$ﬁrst$genera,on • Cron&jobs&run&every&minute&to&select&[x]&rows&to&send&to&APIs • Crons&are&running&Symfony&tasks&which&each&perform&one& funcBon • Sync&process&spawning&mulBple&curl&requests&to&localhost

Slide 10

Slide 10 text

Slide 11

Slide 11 text

120$cron$tasks$in$total Most%of%them%run%every%minute

Slide 12

Slide 12 text

SELECT [cols] FROM `adverts_tbl` WHERE `update_flag` = 'U';

Slide 13

Slide 13 text

SELECT [cols] FROM `adverts_tbl` WHERE `update_flag` = 'U'; Problems: • Only&a&certain&amount&can&be&processed&within&a&minute • Might&only&select&a&few&rows&but&need&to&check&hundreds&of& thousands&in&table • Big&data&read&spike&on&DB&&&CPU&spike&every&60&seconds • Data&comparison&performed&on&nonCindexed&column

Slide 14

Slide 14 text

The$second$genera-on

Slide 15

Slide 15 text

SELECT [cols] FROM `queue_tbl` q INNER JOIN `adverts_tbl` WHERE q.`update_flag` = 'U';

Slide 16

Slide 16 text

SELECT [cols] FROM `queue_tbl` q INNER JOIN àdverts_tbl` WHERE q.ùpdate_flag` = 'U'; Benefits: • Selec&ng)from)fewer)rows) • Joining)on)PKs)&)indexes • Faster!)(but)not)benchmarked)

Slide 17

Slide 17 text

SELECT [cols] FROM `queue_tbl` q INNER JOIN àdverts_tbl` WHERE q.ùpdate_flag` = 'U'; Problems: • Only&a&certain&amount&can&be&processed&within&a&minute • Big&data&read&spike&on&DB&&&CPU&spike&every&60&seconds • Complexity&increases&as&different&queue&types&become&required& (13&queue&tables)

Slide 18

Slide 18 text

* sync progress table * sync users table * sync promoted tweets table * sync targeting table * campaign creation queue table * campaign update / deletion queue table * tweet creation queue table * ads queue table * ads targeting queue * ads account queue * ads tweets queue * ads embedded creatives updates table

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

queue()->create(); $campaign->queue()->retrieve(); $campaign->queue()->update(); $campaign->queue()->delete();

Slide 21

Slide 21 text

getId()); } // …

Slide 22

Slide 22 text

queue_class = $queue_class; $this->finder_column = $finder_column; if ($entity_id) { return $this->getQueue($entity_id); } $this->makeNewQueue(); return $this; } // …

Slide 23

Slide 23 text

Inside'a'task fetch [x] campaigns / adverts from database loop rows fetch access token & other required info from database perform api call handle response & save back to db

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

The$Rabbit$MQ$solu0on

Slide 26

Slide 26 text

How$ $Rabbit$MQ$works • Producer)sends)a)message)to)Rabbit • Rabbit)queues)the)message • Consumer)receives)the)message)and)acts)upon)it

Slide 27

Slide 27 text

queue('post'); $campaign->queue('get'); $campaign->queue('put'); $campaign->queue('delete');

Slide 28

Slide 28 text

Queue('Campaign', $this->createMessage($action)); } private function createMessage($action) { return json_encode(array( 'access_token' => $this->getAccount()->getAccessToken(), 'call_url' => 'https://ads.twitter.com/campaigns', 'method' => strtoupper($action), 'data' => $this->getURIParams() )); } // …

Slide 29

Slide 29 text

connection = new AMQPConnection('localhost', 5672, 'guest', 'guest'); $this->channel = $this->connection->channel(); } public function Queue($queue_name, $content) { // prevent queue from auto-deleting with all the falses. $this->channel->queue_declare($queue_name, false, false, false, false); // pass $queue_name as the routing key $this->channel->basic_publish(new AMQPMessage($content), '', $queue_name); } // …

Slide 30

Slide 30 text

• The%consumer%can%be%wri1en%in%any%language

Slide 31

Slide 31 text

• The%consumer%can%be%wri1en%in%any%language • Try%to%choose%something%fast

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

Beneﬁts: • Queue%processing%load%taken%oﬀ%web%server • Number%of%DB%calls%reduced • Queues%processed%con

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

Re#queueing)messages

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

Implementa)on+Summary !Good!points • Already)removed)13)cron)jobs • Queues)are)processed)con7nuously • Load)moved)away)from)client)facing)servers • Schema)&)code)complexity)reduced • Database)load)reduced • Sync)is)much)faster,)especially)for)large)accounts

Slide 41

Slide 41 text

Implementa)on+Summary Bad$points • Re$modelling,database,in,Sequelize,slowed,down,development

Slide 42

Slide 42 text

Implementa)on+Summary Bad$points • Re$modelling,database,in,Sequelize,slowed,down,development • We,have,to,maintain,the,DB,model,in,both,Doctrine,and, Sequelize

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

What’s'next? • Implement)Rabbit)queues)for)campaign)crea6on)and)updates • Implement)Rabbit)for)fetching)reports)from)the)Twi;er)API • Implement)Rabbit)for)the)Facebook)part)of)our)tool • We’ve)already)built)our)iAds)implementa6on)on)top)of)Rabbit

Slide 47

Slide 47 text

@nealio82