Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mercari Timeline Personalization powered by Perl in 5minutes

700669515ee872152d8b9403c2a0cf8c?s=47 kazeburo
August 20, 2017

Mercari Timeline Personalization powered by Perl in 5minutes

Mercari Timeline Personalization powered by Perl in 5minutes

700669515ee872152d8b9403c2a0cf8c?s=128

kazeburo

August 20, 2017
Tweet

More Decks by kazeburo

Other Decks in Technology

Transcript

  1. None
  2. @kazeburo

  3. Q

  4. What is the difference between two pictures?

  5. Normal Timeline Personalized Timeline(male)

  6. • Timeline personalization • For keeping Retention-Rate • Using ML

    to predict demographic of users
  7. For making Timeline flexible

  8. Very flexible Query

  9. • By the way.. Timeline requires Speed.

  10. •6 sec (40,000,000 docs) •25 sec (160,000,000 docs) • Need

    100x speedup at least!!
  11. Solr for TL / Distributed Index App App API MySQL

    blackhole black hole Q4M Solr (master) worker trigger dequeue black hole Q4M Solr (master) worker trigger dequeue black hole Q4M Solr (master) worker trigger dequeue black hole Q4M Solr (master) worker trigger dequeue soft commit in 200 msec Update (realtime) Filtering fresh item and insert to BlackHole table Emulate Pub/Sub by Replication and trigger and Q4M Update only item selected by consistent hashing ৽͍͠঎඼(N೔)ͷΈΛBlackHole tableʹinsert ReplicationͱTriggerͱQ4MͰPub/Sub Emu consistent-hashingͰબ͹Εͨ঎඼ͷΈudpate
  12. ɹaa //

  13. • API • Plack, Gazelle, WWW::Form::UrlEncoded::XS, JSON::XS and DBIx::Sunny •

    Worker • Parallel::Prefork, Cache::Memory::Simple
 LWP::UserAgent and Algorithm::ConsistentHash::Ketama
  14. black hole Q4M Solr (master) worker trigger dequeue consul my

    $res = $ua->get(‘http://localhost/v1/health/service/".$SRV.'?passing'); my $ref = JSON::XS::decode_json($res->content); my @list = sort { $a cmp $b } map { $_->{Node}{Address} } @$ref; my $ketama = Algorithm::ConsistentHash::Ketama->new(); $ketama->add_bucket($_ . '_' . $timestamp, 1) for @list; my $s1 = $ketama->hash($item_id); return 1 if $s1 eq $my_ip; my $s2 = $s1; while ($s2 eq $s1) { $s2 = $ketama->hash($item_id.’_'.$i); $i++; } return 1 if $s2 eq $my_ip; return; Get server list from Consul Make Consistent Hash Drawing by consistent-hashing Update Solr
  15. None
  16. ߏ૝ GW / ߏங 1೔

  17. None