Perl in Mercari YAPC::Okinawa 2018 ONNASON

Perl in Mercari
YAPC::Okinawa 2018 ONNASON スペシャルセッション

Transcript

1. Perl in Mercari 2018
   @kazeburo

   YAPC::Okinawa 2018 ONNASON
   

   View Slide

2. େ੾ͳ͓஌Βͤ
   

   View Slide

3. memcached UDPϦϑϨΫγϣϯDDoS
   • Πϯλʔωοτʹެ։͞Ε͍ͯΔmemcached 11211/UDPΛར༻ͯ͠UDP ϦϑϨ
   Ϋγϣϯ͕ߦ͑Δ
   • github.com ʹ1.3TbpsͷDDoS߈ܸ͕ൃੜ (2/28)
   • ͘͞ΒVPS͸11211/UDPΛϑΟϧλϦϯά
   • [ॏཁ] memcachedΛάϩʔόϧʹެ։͠ͳ͍
   • UDPΛແޮԽ͢Δɻmemcached -U 0Ͱىಈ

   memcached-1.5.6Ͱ͸σϑΥϧτແޮʹͳΓ·ͨ͠
   

   View Slide

4. View Slide

5. Me
   • ௕໺խ޿
   • PAUSE:KAZEBURO
   • גࣜձࣾϝϧΧϦ

   ϓϦϯγύϧΤϯδχΞ

   Site Reliability Engineering (SRE) νʔϜ
   • BASE, Inc ٕज़ΞυόΠβʔ
   

   View Slide

6. View Slide

7. ϝϧΧϦ
   • ࠃ಺࠷େڃͷϑϦϚΞϓϦ
   • 3෼Ͱ؆୯ʹग़඼
   1) ࣸਅΛࡱΔ
   2) ঎඼৘ใΛهೖ
   3) ग़඼ϘλϯΛԡ͢
   • ҆৺҆શͳܾࡁɾऔҾ
   • ΤεΫϩʔ(͓ۚͷ΍ΓͱΓ͸౰͕ࣾؒʹհࡏ)
   • ಗ໊഑ૹ
   

   View Slide

8. KPI
   μ΢ϯϩʔυ਺
   GMV(૯औҾֹ)
   1ԯDLҎ্(JP+US+UK)
   ݄ؒ100ԯԁҎ্
   ग़඼਺ 1೔100ສ඼Ҏ্
   

   View Slide

9. ϝϧΧϦΛࢧ͑Δ
   αʔόαΠυͷ
   ϓϩάϥϛϯάݴޠ
   

   View Slide

10. PHP 7
    

    View Slide

11. Golang
    https://github.com/golang-samples/gopher-vector
    

    View Slide

12. ϝϧΧϦΛࢧ͑Δϓϩάϥϛϯάݴޠ
    • ϝϧΧϦͷAPI͸΄΅શͯPHPʹΑΓ࣮૷
    • ϚΠΫϩαʔϏε/ϛυϧ΢ΣΞͰ͸ Golang ͕த৺
    • AI/MLͷ෼໺Ͱ͸Python΋࢖ΘΕΔ
    

    View Slide

13. 1FSM͸ϝΠϯϦʔϜͳݴޠͰ͸ͳ͍
    

    View Slide

14. ͕͔ͩ͠͠
    খ͞ͳ Perl εΫϦϓτ͕αʔϏεͷ҆ఆՔಇΛࢧ͑
    ཪํͱͯ͠େن໛ͳτϥϑΟοΫΛॲཧ͍ͯ͠Δɻ
    

    View Slide

15. slacklog
    

    View Slide

16. slacklog = cronlog + Slack notification
    

    View Slide

17. cronlog
    • cronlog ͸kazuho͞Μ࡞੒ͷPerlεΫϦϓτɻҾ਺ͱͯ͠༩͑ΒΕͨίϚϯ
    υΛ࣮ߦ͠ɺͦͷऴྃίʔυ͕ඇθϩͷ৔߹ͷΈɺίϚϯυͷग़ྗ
    (stdout, stderr)Λඪ४ग़ྗʹૹΔ
    • cronͰར༻͢Δ͜ͱͰΤϥʔ࣌ͷΈMAILTOʹϝʔϧૹ৴
    • `5 * * * * cronlog -- ping -n 5 my-server 2>&1`
    • ʮ؂ࢹͱ͸ܧଓతͳςετʯͱ͍͏ݴ༿ΛੜΈग़ͨ͠
    

    View Slide

18. slacklog
    • ίϚϯυͷऴྃίʔυ͕ඇθϩ࣌ʹඪ४ग़ྗ΁ͷૹ৴ʹՃ͑ɺslack΁ͷ௨஌Λߦ͏
    • @channel Λ௥Ճ͢Δ͔ࢦఆՄೳ
    • ىಈͱऴྃ࣌ʹSlack΁ͷ௨஌ɺsyslog΁΋ه࿥
    • hostname,pid ͷ෇Ճ
    • slack΁ͷ௨஌͸ slackboard Λར༻
    • ϝϧΧϦͷ΄΅શͯͷcronͰར༻͍ͯ͠Δ
    

    View Slide

19. slacklog use slackboard
    App Batch MySQL
    App Batch MySQL
    App Batch MySQL
    JP
    US
    UK
    slacklog slacklog slacklog
    slacklog slacklog slacklog
    slacklog slacklog slacklog
    slackborad
    slackborad
    slackborad
    JP
    US
    EU
    HTTP Load
    Balancer
    MySQL
    slacklog slacklog slacklog
    SlackͷAPI keyͷҰݩ؅ཧ
    Buffering/Queueing
    Slack
    https://github.com/cubicdaiya/slackboard
    

    View Slide

20. ࣮ߦྫ
    $ slacklog -c test --log-channel test -- perl -E '$|=1;say "Hello";sleep 3;die'
    [2018-02-25T01:09:51][21066] *hostname* starting: perl -E $|=1;say "Hello";sleep 3;die
    [2018-02-25T01:09:51][21066] Hello
    [2018-02-25T01:09:54][21066] Died at -e line 1.
    [2018-02-25T01:09:54][21066] command exited with code:255
    $
    

    View Slide

21. ࣮ߦྫ(slack)
    ىಈऴྃϩά
    ίϚϯυͷ
    STDOUT
    STDERR
    

    View Slide

22. ಋೖޮՌ
    • batchͷΤϥʔͷݕग़ͱ௨஌ͷଈ࣌ੑΞοϓ
    • ϝʔϧ͸஗͍ɺ΋͘͠͸ΈΒΕͯͳ͍
    • αʔόαΠυΤϯδχΞͷslack channelʹ௚઀ૹ৴ɻ඼࣭ͷ޲্
    • ϩάΛ/dev/nullʹࣺͯͳ͍ӡ༻͕Մೳʹ
    

    View Slide

23. ಋೖޮՌ
    • ʮ͋ͷcronىಈ͍ͯ͠·͔͢ʁʯʮࡢ೔ϦϦʔεͨ͠batchɺਖ਼͘͠ಈ͍
    ͯ·͔͢ʁʯͷΑ͏ͳΤϯδχΞ͔Βͷ֬ೝґཔͷ࡟ݮ
    • τΠϧͷ๾໓
    

    View Slide

24. ঎඼
    Ӿཡ
    ཤྺ
    SERVICE
    

    View Slide

25. Ӿཡཤྺ
    • ͓٬༷ͷ঎඼ӾཡͷཤྺΛαʔόʹ֨ೲ
    • UX޲্ͱͱ΋ʹɺTimeline Personalizationͷσʔλ
    ͱͯ͠΋׆༻
    • ঎඼ͷӾཡ͸਺ઍreq/sec

    PerlͰߏங͞Εͨ ServiceͰॲཧ
    

    View Slide

26. App
    App
    App
    App
    App
    NGINX API(Perl) Q4M
    Worker
    (Perl)
    browse_item_service
    NGINX API(Perl) Q4M
    Worker
    (Perl)
    browse_item_service
    MySQL MySQL
    memcached
    memcached
    memcached
    "ߋ৽"
    "ࢀর" cluster
    (2) DBߋ৽&ཤྺऔಘ
    (3) cache࡞੒
    (1) enqueue
    (1) cacheऔಘ/missͨ͠ΒDB
    App
    App
    App
    App
    App
    memcached
    memcached
    memcached
    NGINX API(Perl) Q4M
    Worker
    (Perl)
    browse_item_service
    DNS-RR by consul
    

    View Slide

27. CPAN modules
    • API
    • Plack, Gazelle
    • Worker
    • Parallel::Prefork
    • ڞ௨
    • JSON::XS, DBD::mysql, Cache::Memcached::Fast::Safe
    Web Application Framework͸ະ࢖༻
    Plack::(Request|Response)΋࢖ͬͯͳ͍
    XS moduleΛ׆༻ͯ͠଎౓޲্
    

    View Slide

28. Performance
    • API/Worker͸Scale Out͕Մೳ
    • GazelleʹΑΔAPIαʔό͸े෼ʹߴ଎
    • ϘτϧωοΫ͸ MySQL ͷϨϓϦέʔγϣϯ஗Ԇ
    • ௨ৗͷϨϓϦέʔγϣϯͰ͸਺ઍ transaction/s Ͱ஗Ԇൃੜ
    • MySQL 5.7ͷ Multi Thread Slave Λར༻ͯ͠΋ 18,000 t/s ͕ݶք
    • Sharding Λॳظઃܭʹ૊ΈࠐΈ
    

    View Slide

29. Performance (newRelic)
    ঎඼͝ͱͷPVΛܭଌ
    ͢Δ Service (Golang)
    Avg time: 3 msec
    Ӿཡཤྺ Service (Perl)
    Avg time: 4 msec
    PHP
    DBͳͲ
    

    View Slide

30. http://tech.mercari.com/entry/2018/02/26/110237
    

    View Slide

31. Կނ
    Perl?
    

    View Slide

32. ͳͥ Perl ?
    • kazeburo͕ॻ͍͔ͨΒ
    • System Call friendly
    ‣ OS ΍ ProcessɺNetwork ͷجૅΛֶͿͷʹద͍ͯ͠Δ
    • (Core Module ͷΈͰ࡞Ε͹) Deploy ͕༰қ
    • PerlͰഓΘΕٕͨज़ͷܧঝ
    ‣ mixi, livedoor (LINE), DeNA, ͸ͯͳ ͳͲେن໛WebαʔϏεΛࢧ͑ͯ(͍Δ|͖ͨ)ݴޠ
    

    View Slide

33. ͜Ε͔Β
    • ϚΠΫϩαʔϏεɾίϯςφத৺ͷInfrustructure΁
    • Serverlessɺk8sʹΑΔࣗಈԽ͞Εͨ Infrastructure
    • OS΍networkͳͲγεςϜΤϯδχΞϦϯάͷ஌͕ࣝ։ൃऀ͔Βԕ͘ͳΔ
    • Golang த৺ɺPerl ΠϯλϓϦλ͕ͳ͍؀ڥ
    

    View Slide

34. ͜Ε͔Β
    • SREݚमͰPerlΛ৮ͬͯ΋Β͏
    • ৽ଔ޲͚ʹ1ϲ݄͔Β1ϲ݄൒ఔ౓ͷSREݚमΛߦ͍ͬͯΔ
    • ͦͷதͰͻͱͭ͸PerlΛ࢖ͬͨ࢓૊Έʹ৮ΕΔػձΛͭ͘Δ
    • slacklogͷىಈϩάΛSlackʹૹΔػೳ͸SREݚमͰ࡞ͬͯ΋Βͬͨ΋ͷ
    • OSɾFile SystemɾProcessɾNetworkͱ͍ͬͨ௿ϨΠϠʔ΁ͷڵຯɾؔ৺

    UNIX ఩ֶʹ৮ΕΔػձ
    • ٕज़ͷఈ্͛ʹඞཁͳݴޠͷҰͭ
    

    View Slide

35. Ҏ্
    careers.mercari.com/job/
    

    View Slide