Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mercari Timeline Personalization powered by Perl in 5minutes

kazeburo
August 20, 2017

Mercari Timeline Personalization powered by Perl in 5minutes

Mercari Timeline Personalization powered by Perl in 5minutes

kazeburo

August 20, 2017
Tweet

More Decks by kazeburo

Other Decks in Technology

Transcript

  1. Q

  2. Solr for TL / Distributed Index App App API MySQL

    blackhole black hole Q4M Solr (master) worker trigger dequeue black hole Q4M Solr (master) worker trigger dequeue black hole Q4M Solr (master) worker trigger dequeue black hole Q4M Solr (master) worker trigger dequeue soft commit in 200 msec Update (realtime) Filtering fresh item and insert to BlackHole table Emulate Pub/Sub by Replication and trigger and Q4M Update only item selected by consistent hashing ৽͍͠঎඼(N೔)ͷΈΛBlackHole tableʹinsert ReplicationͱTriggerͱQ4MͰPub/Sub Emu consistent-hashingͰબ͹Εͨ঎඼ͷΈudpate
  3. • API • Plack, Gazelle, WWW::Form::UrlEncoded::XS, JSON::XS and DBIx::Sunny •

    Worker • Parallel::Prefork, Cache::Memory::Simple
 LWP::UserAgent and Algorithm::ConsistentHash::Ketama
  4. black hole Q4M Solr (master) worker trigger dequeue consul my

    $res = $ua->get(‘http://localhost/v1/health/service/".$SRV.'?passing'); my $ref = JSON::XS::decode_json($res->content); my @list = sort { $a cmp $b } map { $_->{Node}{Address} } @$ref; my $ketama = Algorithm::ConsistentHash::Ketama->new(); $ketama->add_bucket($_ . '_' . $timestamp, 1) for @list; my $s1 = $ketama->hash($item_id); return 1 if $s1 eq $my_ip; my $s2 = $s1; while ($s2 eq $s1) { $s2 = $ketama->hash($item_id.’_'.$i); $i++; } return 1 if $s2 eq $my_ip; return; Get server list from Consul Make Consistent Hash Drawing by consistent-hashing Update Solr