Redis Bedtime Stories

3 Redis short stories. 30000 requests per second. 3 lessons learned.

Transcript

1. Redis Bedtime Stories @igorwhilef a lse
2. Hi, I'm Igor. St a ff SRE @ GitL a b
5. Agenda • Refresher • Story 1: Microbursts • Story 2: Regression • Story 3: Kubernetes • Takeaways
6. Refresher
7. redis /ˈrɛdɪs/ n. a. remote dictionary service
8. { k => v }
9. array($k => $v)
10. *: 6379
11. SET user:1:session "{...}"
12. GET user:1:session => "{...}"
16. "redis is fast" -- a liar, probably
17. Numbers Everyone Should Know L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 25 ns Main memory reference 100 ns Compress 1K bytes with Zippy 3,000 ns Send 2K bytes over 1 Gbps network 20,000 ns Read 1 MB sequentially from memory 250,000 ns Round trip within same datacenter 500,000 ns Disk seek 10,000,000 ns Read 1 MB sequentially from disk 20,000,000 ns Send packet CA->Netherlands->CA 150,000,000 ns Jeff De a n, LADIS 2009 Keynote
19. Story 1: Microbursts
20. • Clients see periodic performance degradation. • Sometimes it breaches the alert threshold, sometimes it doesn't. • Nothing obvious. • Investigation begins.
21. CPU utilization between 60% and 80%
22. $ sudo perf record \ -p $(pidof redis-server) \ -F 499 \ -g \ -- \ sleep 600
24. $ for i in {1..600} do sleep 1 echo -n "$(date +'%Y-%m-%d %H:%M:%S.%N %Z')" redis-cli info stats \ | grep -w 'instantaneous_ops_per_sec' done M a tt Smiley
25. M a tt Smiley
26. $ sudo tcpdump \ -v \ -G 60 \ -w $(hostname -s).inbound.%Y%m%d_%H%M%S.pcap \ 'dst port 6379' M a tt Smiley
27. $ mergecap -w - pcap-20200325-microburst-sample/*.pcap.gz | tcpflow -r - -I -s -o tcpflow M a tt Smiley
28. $ find tcpflow/ -name '*.06379.findx' \ | xargs -P 8 -n 100 redis_trace_cmd \ > trace_inbound_redis_commands.out M a tt Smiley
29. cache:gitlab:exists?:$PATTERN cache:gitlab:projects/count_service/$NUMBER/$NUMBER/forks_count cache:gitlab:projects/count_service/$NUMBER/$NUMBER/public_open_issues_count cache:gitlab:has_visible_content?:$PATTERN cache:gitlab:root_ref:$PATTERN cache:gitlab:avatar:$PATTERN:$NUMBER cache:gitlab:readme_path:$PATTERN M a tt Smiley
30. M a tt Smiley
31. sum by (controller, action) (rate(gitlab_cache_operations_total{env="gprd",operation="read"}[30s])) {action="GET /api/groups/:id/projects"}
32. json.route=/api/:version/groups/:id/projects
33. 70 QPS 50000 QPS
34. 70 req/s * 100 projects/page * 7 queries per project = 50k QPS
35. Lesson learned: Aggregation intervals hide burstiness
36. Story 2: Regression
37. Let's upgrade, it will be better, right?
38. 5.0.9 => 6.0.10
39. +15% CPU utilization
40. 6.0.10 => 5.0.9
41. $ sudo perf record -ag -F 497 -- sleep 120 $ sudo perf script --header \ | stackcollapse-perf.pl --kernel \ | grep redis-server \ | flamegraph.pl --hash --colors=perl \ > flamegraph.svg
42. M a tt Smiley
43. serveClientsBlockedOnKeyByModule
47. github.com/redis/redis/pull/8689
48. repro.go
49. before after
50. Lesson learned: Sometimes upgrades make things worse
51. Story 3: Kubernetes
52. VMs => k8s
56. dedicated node pool
59. +25% CPU utilization
60. $ sudo perf record
61. M a tt Smiley
62. do_softirq
63. GKE is very smart
65. • When tra ffi c reaches a Kubernetes node, it is handled the same way, regardless of the type of load balancer. • The load balancer is not aware of which nodes in the cluster are running Pods for its Service. • Instead, it balances tra ff i c across all nodes in the cluster, even those not running a relevant Pod.
66. 🤦
67. GKE dataplane v2 & NEGs handle this a little better
68. Lesson learned: Production does not care about your readiness review
69. Takeaways
70. • Story 1: Microbursts • Aggregation intervals hide burstiness • Story 2: Regression • Sometimes upgrades make things worse • Story 3: Kubernetes • Production does not care about your readiness review
71. References • Microburst analysis • gitlab.com/gitlab-com/gl-infra/reliability/-/issues/9420 • Performance regression in BRPOP • github.com/redis/redis/issues/8668 • github.com/redis/redis/pull/8689 • Packet processing overhead on Kubernetes • gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1985 • Further reading • github.com/redis/redis/issues/7071 • about.gitlab.com/blog/2022/11/28/how-we-diagnosed-and-resolved-redis-latency-spikes • gitlab.com/gitlab-com/runbooks/-/blob/master/scripts/redis_trace_cmd.rb
72. Acknowledgements • Matt Smiley • Colleague, friend, sleuth extraordinaire
73. Thanks!