Redisの調査についてとrmlp

Transcript

1. 1.

   Redisͷௐࠪʹ͍ͭͯͱ rmlp shibuya.go#2 3/22

2. 2.

   ࣗݾ঺հ • @Konboi • ໘ന๏ਓΧϠοΫ • ϞόΠϧήʔϜͷ։ൃͱӡ༻

3. 3.

   ࠓ೔࿩͢ࣄ • Redisͷௐࠪ • CPU͕ • ϝϞϦ͕ • 1୆Ͱ͸ •

   rmlpʹ͍ͭͯ
4. 4.

   ࠓ೔࿩͞ͳ͍ࣄ • Redisʹ͍ͭͯͷ࿩ • Redisͷνϡʔχϯά • GoΛ࢖ͬͯͷRedisͷૢ࡞

5. 5.

   RedisΛۀ຿Ͱ࢖ͬͯΔ ͱ͍͏ํ ✋

6. 6.

   Redisͷௐࠪʹ͍ͭͯ

7. 7.

   ͷલʹ

8. 8.

   ӡ༻͍ͯ͠ΔҊ݅Ͱ RedisͷCPU͕͜ͷ༷ʹ ͳͬͨܦݧ͸͋Γ·͔͢

9. 9.

   ಥൃతʹ͕͋ΔCPUෛՙ

10. 10.

    None
11. 11.

    !

12. 12.

    ໷ʹͳΔʹͭΕ͕ͯ͋Δ CPUෛՙ

13. 13.

    None
14. 14.

    !

15. 15.

    ೔ʹ೔ʹߴ͘ͳΔCPUෛՙ

16. 16.

    None
17. 17.

    !!!

18. 18.

    ා͍Ͱ͢Ͷ

19. 19.

    ͱ͍͏͜ͱͰຊ୊

20. 20.

    CPUͷෛՙ͕ߴ͍࣌

21. 21.

    CPUͷෛՙ͕ߴ͍࣌ • ͳʹ͔ॏ͍ॲཧ͕ߦΘΕ͍ͯΔ֬཰͕ߴ͍Ͱ͢ • Redis͸γϯάϧεϨουͳͷͰॏ͍ॲཧ͕͋ΔͱͦΕ͕ऴΘ Δ·Ͱଞͷॲཧ͸଴ͨ͞ΕͯͭΒ͍ • ॏ͍ͱ͍ͬͯ਺ϛϦඵͰॲཧ͸ऴΘΔͷͰؾ͖ͮʹ͍͘ • devͩͱΞΫηε͕গͳ͍ͷͰ࿐ఄ͠ʹ͍͘

22. 22.

    Ͳ͏͢Δ...

23. 23.

    slow logΛݟ·͠ΐ͏

24. 24.

    # ࠓճ͸ݕূͷͨΊ slowlogͷ৚݅Λ10microඵʹ͍ͯ͠·͢ $ redis-cli -h <host> slowlog get <log

    num> 1) 1) (integer) 34990 2) (integer) 1457071463 3) (integer) 11 4) 1) "slowlog" 2) "get" 3) "1" ... 3) 1) (integer) 34988 2) (integer) 1457071448 3) (integer) 51 4) 1) "slowlog" 2) "get" 3) "5"
25. 25.

    ݟํ 1) 1) (integer) 34990 2) (integer) 1457071463 3) (integer)

    11 4) 1) "slowlog" 2) "get" 3) "1" 1) 1) slowlogͷID 2) ࣮ߦऴྃ࣌ͷtimestamp(UNIXඵ) 3) ࣮ߦ࣌ؒ(microඵ) 4) ࣮ߦίϚϯυͱΦϓγϣϯ
26. 26.

    CPUͷෛՙ͕ߴ͍࣌ • ͍͍ͩͨ͜ΕͰ஗͍(ෛՙͷߴ͍)ίϚϯυ͕෼͔Δ͸ͣͰ͢ • ޙ͸֘౰ͷιʔείʔυΛ௚͠·͠ΐ͏ • ܦݧ্ keys * ΍େྔͷσʔλಉ࢜ͰͷUNION

    ΍ DIFF Λ͢Δ ͱCPUͷෛՙ͕άϯάϯ্͕Γ·͢
27. 27.

    ϝϞϦ

28. 28.

    None
29. 29.

    ೔ʹ೔ʹ૿͑Δσʔλྔ

30. 30.

    ͜ΕͰ͸͋ͱX೔Ͱ༰ྔ ͷݶքΛܴ͑ͯ͠·͏

31. 31.

    !!!

32. 32.

    dumpΛऔΓ rdb tool Λ࢖ͬͯௐࠪ͠·͠ΐ͏

33. 33.

    rdb tool ͱ͸ • sripathikrishnan/redis-rdb-tools • Rdbtools is a parser

    for Redis' dump.rdb files. • python੡ • rdbϑΝΠϧ͔Βσʔλͷ༷ʑͳ৘ใΛग़ྗͰ͖Δπʔϧ
34. 34.

    αϯϓϧ dump͸slaveͳͲ͔ΒऔΓ·͠ΐ͏ # αʔόʔ͔Βdump $ redis-cli -h <host> --rdb dump.rdb

    # memoryͱkeyΛग़ྗ $ rdb -c memory dump.rdb > dump_memory.csv # memoryͷେ͖͞ॱʹsort $ cat dump_memory.csv | awk -F "," '{print $4 "\t" $3'} | sort -n
35. 35.

    ϝϞϦ • ༰ྔॱʹstore͞Ε͍ͯΔσʔλ͕Θ͔ΔͷͰ֘౰ͷσʔλΛ ফ͢ • ඞཁແ͘ͳͬͨσʔλΛ࡟আ͢Δ • ܦݧ্ temporary తͳσʔλ͕࢒͍ͬͯΔࣄ͕ଟ͍

    • SET <key> <VALUE> "EX" <EXPIRE_TIME> • ͳͲexpire͢Δ࣌ؒΛ͜·Ίʹࢦఆ • ফ͜͠Έॲཧ͕࣮૷͞Ε͍ͯΔ͔֬ೝ
36. 36.

    ͖ͪΜͱফ͜͠ΈΛߦ͏ Α͏ʹͨ݁͠Ռ

37. 37.

    None
38. 38.

    ! ! !

39. 39.

    ͦΕͰ΋1୆Ͱ͸ݶք͕...

40. 40.

    ໾ׂ͝ͱʹෳ਺୆ʹ͠Α͏ʂ

41. 41.

    ໾ׂ͝ͱʹෳ਺୆ʹ͠Α͏ • session༻ • cache༻ • ϥϯΩϯά࡞੒༻ • ͳͲͳͲ

42. 42.

    Ͱ΋ͪΐͬͱ଴ͬͯ ✋

43. 43.

    ͦͷ෼ׂຊ౰ʹޮ͘...?

44. 44.

    ͦͷ෼ׂຊ౰ʹޮ͘...? • $ redis-cli slowlog get Ͱ͸஗͍ίϚϯυ͕Θ͔Δ • rdb tool

    Ͱ৭ʑղੳͰ͖Δ • ͜Ε͚ͩͰ͸Ŋ֤keyͷར༻ස౓·Ͱ͸෼͔Βͳ͍ • ಛఆͷΠϕϯτͷ͚࣌ͩଟ͍͔΋͠Εͳ͍ • ͋Δkeyͷ࣮ߦ͕΄͔ͷkeyͷ10ഒҎ্͔΋͠Εͳ͍ • ͳͲ
45. 45.

    redis-cli monitor

46. 46.

    redis-cli monitor $ redis-cli monitor 1457077479.354522 [0 127.0.0.1:59247] "INCR" "counter:__rand_int__"

    1457077479.354621 [0 127.0.0.1:59247] "INCR" 1457077479.352548 [0 127.0.0.1:59245] "SET" "key:__rand_int__" "xxx""counter:__rand_int__" 1457077479.354852 [0 127.0.0.1:59248] "LPUSH" "mylist" "xxx" 1457077479.352812 [0 127.0.0.1:59245] "SET" "key:__rand_int__" "xxx" 1457077479.354522 [0 127.0.0.1:59247] "INCR" "counter:__rand_int__" 1457077479.354621 [0 127.0.0.1:59247] "INCR" "counter:__rand_int__" 1457077479.354852 [0 127.0.0.1:59248] "LPUSH" "mylist" "xxx" 1457077479.353138 [0 127.0.0.1:59245] "SET" "key:__rand_int__" "xxx" ݱࡏ࣮ߦ͞Ε͍ͯΔίϚϯυ਺͕෼͔Δ एׯෛՙ͕͕͋ΔͷͰؾΛ͚࣮ͭͯߦ͍ͯͩ͘͠͞
47. 47.

    1457077479.354852 [0 127.0.0.1:59248] "LPUSH" "mylist" "xxx" 1457077479.354956 [0 127.0.0.1:59248] "LPUSH"

    "mylist" "xxx" <command start timestamp> [<DB> <HOST>] <COMMAND> <KEY> ͳͷͰ2ߦ໨͔Β1ߦͷtimestampΛҾ͚͹1ߦ໨ͷ࣮ߦ࣌ؒ ͕Θ͔Δ (ݫີͰ͸ͳ͍)
48. 48.

    ίϚϯυϥΠϯͰ΍Δͱ $ awk 'function fetchcommand() { c =""; for (i

    = 4; i < NF; i++) { c = c " " $i }; c = c " " $NF; return c } NR==1 { time=$1; command=fetchcommand() } NR!=1 { print command, $1-time; time=$1; command=fetch_command(); }' /tmp/redis.log surported by @tkuchiki
49. 49.

    ίϚϯυʹ͠·ͨ͠

50. 50.

    ΍ͬͱGoͷ࿩Λ ͪΐΖͬͱ͠·͢

51. 51.

    rmlp

52. 52.

    rmlp • Konboi/rmlp • Redis Monitor Log Profiler • Go੡

    • ઌ΄ͲͷlogΛݟ΍͍͢ܗʹ੔ܗ • ίϚϯυͷ ฏۉॱ ࣮ߦॱ ࣮ߦճ਺ॱ ͰιʔτՄೳ
53. 53.

    $ redis-cli monitor > redis-monitor.log $ rmlp -f redis-monitor.log Overall

    Stats ================================== LineCount 15000 Commands Rate ================================== LRANGE 4000 PING 2000 ... Heavy Commands ================================== Command Sum(msec) LRANGE 0.803558 PING 0.106445 ... Slowest Calls ================================== KEY Count Max(msec) Avg(msec) PING 2000 0.001511 0.000053 LRANGE mylist 4000 0.001358 0.000201
54. 54.

    rmlp • खݩ(Mac)Ͱσόοάͯ͠Ϗϧυ͢Ε͹ຊ൪(EC2)Ͱಈ͘࠷ߴ • Ͳͷkey͕Ͳͷఔ౓ͷස౓Ͱൃߦ͞Ε͍ͯΔͷ͔೺ѲͰ͖Δ • ࿈ଓతʹredis͕ར༻͞Ε͍ͯͳ͍ͱਖ਼֬ͳܭଌ͕Ͱ͖ͳ͍ • ࣍ͷίϚϯυͷ։࢝࣌ؒΛݩʹ࣮ߦ࣌ؒΛग़͍ͯ͠ΔͨΊ •

    ෼͚Δඞཁੑ͕͋Δͷ͔ͷ൑அࡐྉʹ͸Ͱ͖Δ͔ͳͱ
55. 55.

    rmlp • facebookarchive/redis-faina • python੡ • hirose31/redis-traffic-stats • tcpdump࣮ͯ͠ߦ࣌ؒͱස౓Λܭଌ͢Δπʔϧ΋͋Δ

56. 56.

    ஫ҙࣄ߲ • ࠓճग़ͨ͠ௐࠪͷྫ͸͋͘·ͰҰྫͰ͢ͷͰଞʹ΋ௐࠪํ๏ ͸͋Δͱࢥ͍·͢ • dumpͱ͔͸ෛՙͰ࣮αʔϏε͕Өڹͳ͍ঢ়گͰͦ͏ͧ !

57. 57.

    շదͳRedisੜ׆Λʂ

58. 58.

    ! ͝੩ௌ ! ͋Γ͕ͱ͏͍͟͝·ͨ͠