Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Redisの調査についてとrmlp

Konboi
March 22, 2016

 Redisの調査についてとrmlp

shibuya.go#2 での発表資料にします

Konboi

March 22, 2016
Tweet

More Decks by Konboi

Other Decks in Technology

Transcript

  1. Redisͷௐࠪʹ͍ͭͯͱ
    rmlp
    shibuya.go#2 3/22

    View full-size slide

  2. ࣗݾ঺հ
    • @Konboi
    • ໘ന๏ਓΧϠοΫ
    • ϞόΠϧήʔϜͷ։ൃͱӡ༻

    View full-size slide

  3. ࠓ೔࿩͢ࣄ
    • Redisͷௐࠪ
    • CPU͕
    • ϝϞϦ͕
    • 1୆Ͱ͸
    • rmlpʹ͍ͭͯ

    View full-size slide

  4. ࠓ೔࿩͞ͳ͍ࣄ
    • Redisʹ͍ͭͯͷ࿩
    • Redisͷνϡʔχϯά
    • GoΛ࢖ͬͯͷRedisͷૢ࡞

    View full-size slide

  5. RedisΛۀ຿Ͱ࢖ͬͯΔ
    ͱ͍͏ํ ✋

    View full-size slide

  6. Redisͷௐࠪʹ͍ͭͯ

    View full-size slide

  7. ӡ༻͍ͯ͠ΔҊ݅Ͱ
    RedisͷCPU͕͜ͷ༷ʹ
    ͳͬͨܦݧ͸͋Γ·͔͢

    View full-size slide

  8. ಥൃతʹ͕͋ΔCPUෛՙ

    View full-size slide

  9. ໷ʹͳΔʹͭΕ͕ͯ͋Δ
    CPUෛՙ

    View full-size slide

  10. ೔ʹ೔ʹߴ͘ͳΔCPUෛՙ

    View full-size slide

  11. ͱ͍͏͜ͱͰຊ୊

    View full-size slide

  12. CPUͷෛՙ͕ߴ͍࣌

    View full-size slide

  13. CPUͷෛՙ͕ߴ͍࣌
    • ͳʹ͔ॏ͍ॲཧ͕ߦΘΕ͍ͯΔ֬཰͕ߴ͍Ͱ͢
    • Redis͸γϯάϧεϨουͳͷͰॏ͍ॲཧ͕͋ΔͱͦΕ͕ऴΘ
    Δ·Ͱଞͷॲཧ͸଴ͨ͞ΕͯͭΒ͍
    • ॏ͍ͱ͍ͬͯ਺ϛϦඵͰॲཧ͸ऴΘΔͷͰؾ͖ͮʹ͍͘
    • devͩͱΞΫηε͕গͳ͍ͷͰ࿐ఄ͠ʹ͍͘

    View full-size slide

  14. slow logΛݟ·͠ΐ͏

    View full-size slide

  15. # ࠓճ͸ݕূͷͨΊ slowlogͷ৚݅Λ10microඵʹ͍ͯ͠·͢
    $ redis-cli -h slowlog get
    1) 1) (integer) 34990
    2) (integer) 1457071463
    3) (integer) 11
    4) 1) "slowlog"
    2) "get"
    3) "1"
    ...
    3) 1) (integer) 34988
    2) (integer) 1457071448
    3) (integer) 51
    4) 1) "slowlog"
    2) "get"
    3) "5"

    View full-size slide

  16. ݟํ
    1) 1) (integer) 34990
    2) (integer) 1457071463
    3) (integer) 11
    4) 1) "slowlog"
    2) "get"
    3) "1"
    1) 1) slowlogͷID
    2) ࣮ߦऴྃ࣌ͷtimestamp(UNIXඵ)
    3) ࣮ߦ࣌ؒ(microඵ)
    4) ࣮ߦίϚϯυͱΦϓγϣϯ

    View full-size slide

  17. CPUͷෛՙ͕ߴ͍࣌
    • ͍͍ͩͨ͜ΕͰ஗͍(ෛՙͷߴ͍)ίϚϯυ͕෼͔Δ͸ͣͰ͢
    • ޙ͸֘౰ͷιʔείʔυΛ௚͠·͠ΐ͏
    • ܦݧ্ keys * ΍େྔͷσʔλಉ࢜ͰͷUNION ΍ DIFF Λ͢Δ
    ͱCPUͷෛՙ͕άϯάϯ্͕Γ·͢

    View full-size slide

  18. ೔ʹ೔ʹ૿͑Δσʔλྔ

    View full-size slide

  19. ͜ΕͰ͸͋ͱX೔Ͱ༰ྔ
    ͷݶքΛܴ͑ͯ͠·͏

    View full-size slide

  20. dumpΛऔΓ rdb tool
    Λ࢖ͬͯௐࠪ͠·͠ΐ͏

    View full-size slide

  21. rdb tool ͱ͸
    • sripathikrishnan/redis-rdb-tools
    • Rdbtools is a parser for Redis' dump.rdb files.
    • python੡
    • rdbϑΝΠϧ͔Βσʔλͷ༷ʑͳ৘ใΛग़ྗͰ͖Δπʔϧ

    View full-size slide

  22. αϯϓϧ
    dump͸slaveͳͲ͔ΒऔΓ·͠ΐ͏
    # αʔόʔ͔Βdump
    $ redis-cli -h --rdb dump.rdb
    # memoryͱkeyΛग़ྗ
    $ rdb -c memory dump.rdb > dump_memory.csv
    # memoryͷେ͖͞ॱʹsort
    $ cat dump_memory.csv | awk -F "," '{print $4 "\t" $3'} | sort -n

    View full-size slide

  23. ϝϞϦ
    • ༰ྔॱʹstore͞Ε͍ͯΔσʔλ͕Θ͔ΔͷͰ֘౰ͷσʔλΛ
    ফ͢
    • ඞཁແ͘ͳͬͨσʔλΛ࡟আ͢Δ
    • ܦݧ্ temporary తͳσʔλ͕࢒͍ͬͯΔࣄ͕ଟ͍
    • SET "EX"
    • ͳͲexpire͢Δ࣌ؒΛ͜·Ίʹࢦఆ
    • ফ͜͠Έॲཧ͕࣮૷͞Ε͍ͯΔ͔֬ೝ

    View full-size slide

  24. ͖ͪΜͱফ͜͠ΈΛߦ͏
    Α͏ʹͨ݁͠Ռ

    View full-size slide

  25. ͦΕͰ΋1୆Ͱ͸ݶք͕...

    View full-size slide

  26. ໾ׂ͝ͱʹෳ਺୆ʹ͠Α͏ʂ

    View full-size slide

  27. ໾ׂ͝ͱʹෳ਺୆ʹ͠Α͏
    • session༻
    • cache༻
    • ϥϯΩϯά࡞੒༻
    • ͳͲͳͲ

    View full-size slide

  28. Ͱ΋ͪΐͬͱ଴ͬͯ ✋

    View full-size slide

  29. ͦͷ෼ׂຊ౰ʹޮ͘...?

    View full-size slide

  30. ͦͷ෼ׂຊ౰ʹޮ͘...?
    • $ redis-cli slowlog get Ͱ͸஗͍ίϚϯυ͕Θ͔Δ
    • rdb tool Ͱ৭ʑղੳͰ͖Δ
    • ͜Ε͚ͩͰ͸Ŋ֤keyͷར༻ස౓·Ͱ͸෼͔Βͳ͍
    • ಛఆͷΠϕϯτͷ͚࣌ͩଟ͍͔΋͠Εͳ͍
    • ͋Δkeyͷ࣮ߦ͕΄͔ͷkeyͷ10ഒҎ্͔΋͠Εͳ͍
    • ͳͲ

    View full-size slide

  31. redis-cli monitor

    View full-size slide

  32. redis-cli monitor
    $ redis-cli monitor
    1457077479.354522 [0 127.0.0.1:59247] "INCR" "counter:__rand_int__"
    1457077479.354621 [0 127.0.0.1:59247] "INCR"
    1457077479.352548 [0 127.0.0.1:59245] "SET" "key:__rand_int__" "xxx""counter:__rand_int__"
    1457077479.354852 [0 127.0.0.1:59248] "LPUSH" "mylist" "xxx"
    1457077479.352812 [0 127.0.0.1:59245] "SET" "key:__rand_int__" "xxx"
    1457077479.354522 [0 127.0.0.1:59247] "INCR" "counter:__rand_int__"
    1457077479.354621 [0 127.0.0.1:59247] "INCR" "counter:__rand_int__"
    1457077479.354852 [0 127.0.0.1:59248] "LPUSH" "mylist" "xxx"
    1457077479.353138 [0 127.0.0.1:59245] "SET" "key:__rand_int__" "xxx"
    ݱࡏ࣮ߦ͞Ε͍ͯΔίϚϯυ਺͕෼͔Δ
    एׯෛՙ͕͕͋ΔͷͰؾΛ͚࣮ͭͯߦ͍ͯͩ͘͠͞

    View full-size slide

  33. 1457077479.354852 [0 127.0.0.1:59248] "LPUSH" "mylist" "xxx"
    1457077479.354956 [0 127.0.0.1:59248] "LPUSH" "mylist" "xxx"
    [ ]
    ͳͷͰ2ߦ໨͔Β1ߦͷtimestampΛҾ͚͹1ߦ໨ͷ࣮ߦ࣌ؒ
    ͕Θ͔Δ (ݫີͰ͸ͳ͍)

    View full-size slide

  34. ίϚϯυϥΠϯͰ΍Δͱ
    $ awk 'function fetchcommand() { c =""; for (i = 4; i < NF; i++)
    { c = c " " $i }; c = c " " $NF; return c } NR==1 { time=$1;
    command=fetchcommand() } NR!=1 { print command, $1-time;
    time=$1; command=fetch_command(); }' /tmp/redis.log
    surported by @tkuchiki

    View full-size slide

  35. ίϚϯυʹ͠·ͨ͠

    View full-size slide

  36. ΍ͬͱGoͷ࿩Λ
    ͪΐΖͬͱ͠·͢

    View full-size slide

  37. rmlp
    • Konboi/rmlp
    • Redis Monitor Log Profiler
    • Go੡
    • ઌ΄ͲͷlogΛݟ΍͍͢ܗʹ੔ܗ
    • ίϚϯυͷ ฏۉॱ ࣮ߦॱ ࣮ߦճ਺ॱ ͰιʔτՄೳ

    View full-size slide

  38. $ redis-cli monitor > redis-monitor.log
    $ rmlp -f redis-monitor.log
    Overall Stats
    ==================================
    LineCount 15000
    Commands Rate
    ==================================
    LRANGE 4000
    PING 2000
    ...
    Heavy Commands
    ==================================
    Command Sum(msec)
    LRANGE 0.803558
    PING 0.106445
    ...
    Slowest Calls
    ==================================
    KEY Count Max(msec) Avg(msec)
    PING 2000 0.001511 0.000053
    LRANGE mylist 4000 0.001358 0.000201

    View full-size slide

  39. rmlp
    • खݩ(Mac)Ͱσόοάͯ͠Ϗϧυ͢Ε͹ຊ൪(EC2)Ͱಈ͘࠷ߴ
    • Ͳͷkey͕Ͳͷఔ౓ͷස౓Ͱൃߦ͞Ε͍ͯΔͷ͔೺ѲͰ͖Δ
    • ࿈ଓతʹredis͕ར༻͞Ε͍ͯͳ͍ͱਖ਼֬ͳܭଌ͕Ͱ͖ͳ͍
    • ࣍ͷίϚϯυͷ։࢝࣌ؒΛݩʹ࣮ߦ࣌ؒΛग़͍ͯ͠ΔͨΊ
    • ෼͚Δඞཁੑ͕͋Δͷ͔ͷ൑அࡐྉʹ͸Ͱ͖Δ͔ͳͱ

    View full-size slide

  40. rmlp
    • facebookarchive/redis-faina
    • python੡
    • hirose31/redis-traffic-stats
    • tcpdump࣮ͯ͠ߦ࣌ؒͱස౓Λܭଌ͢Δπʔϧ΋͋Δ

    View full-size slide

  41. ஫ҙࣄ߲
    • ࠓճग़ͨ͠ௐࠪͷྫ͸͋͘·ͰҰྫͰ͢ͷͰଞʹ΋ௐࠪํ๏
    ͸͋Δͱࢥ͍·͢
    • dumpͱ͔͸ෛՙͰ࣮αʔϏε͕Өڹͳ͍ঢ়گͰͦ͏ͧ !

    View full-size slide

  42. շదͳRedisੜ׆Λʂ

    View full-size slide

  43. ! ͝੩ௌ !
    ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View full-size slide