はじめるCassandra

Cassandra Meetup in Tokyo, Summer 2015でお話してきた奴

Transcript

1. ͸͡ΊΔCassandra
2. who are you
3. ͍Θͳ͕͔͚Δ twitter@kakerukaeru ΏͱΓΠϯϑϥԂࣇʹ͋ @CyberAgent Amebaͷೝূɺ՝ۚɺը૾഑৴ج൫ͷ໘౗ݟΔϚϯ Cassandraྺɿ̍೥ऑ... ۓுͯ͠·͢ ΦΤʔʔ!!!!ɹʊʊ_ ɹɹɹ ʊʊ_ʗɹɹ ʋ ɹɹ ʗɹ ʗɹʗ˶ʋ| ɹɹ/ (ƅ)/ɹʗ / ɹ /ɹɹ ŪŊ/Ň˶ʋɻ
4. ࠷ॳʹ
5. ࠓճͷࢿྉ͸
6. For administrator
7. ͩ
8. Α
9. ͸͍
10. ͱ͍͏͜ͱͰ ࠓճͷ͓࿩ͷ಺༰ͱͯ͠͸ ҎԼͷࣄΛॏ఺తʹ͓࿩͠Α͏͔ͱࢥ͍·͢ • ӡ༻্͍ͯ͘͠ͰɺݟΔ΂͖఺ • ఆৗతʹߦ͏΂͖ΦϖϨʔγϣϯ • ॳΊͯCassandraΛӡ༻ͯ͠Έͯͷॴײ
11. agenda
12. agenda • about CyberAgent&Service • why Cassandra • Operation • Build,Monitoring,Backup etc... • about Troubleshooting • ·ͱΊ
13. about CyberAgent
14. blog
15. pigg
16. game
17. Entertainment
18. Entertainment
19. about Cassandra in CyberAgent
20. about Cassandra in CyberAgent • Cassandra Versionɿ1.1.5, 1.2.13 • Production Clusterɿ3 • Production nodesɿ about 150node • Total about qps Read&Writeɿ 50000qps • Total about data sizeɿ15TB
21. about Service
22. use at Cyberagent Smartphone Platform1 1 ϒϥ΢βͷPlatformͳ
23. ͳͷ
24. Ͱ͢
25. ͕
26. ࠓճ͓࿩͢Δͷ͸ɺ
27. طଘͰ͸ͳ͘ ৽͘͠࡞ͬͨClusterͷ͓࿩
28. Ͱ͢
29. For Native App iOS & Android auth payment logging
30. For Native App • ωΟςΟϒΞϓϦ༻ج൫ • ੜ·Εͯ̍೥ऑͷج൫ • ೝূɺ՝ۚɺloggingͷapiΛఏڙ • Cassandraͷ࢖༻͸ओʹid؅ཧͷ෦෼ • idʹඥ෇͚ͯɺ՝ۚˍloggingͳͲͷbackendͷSystemʹܨ ͍͛ͯΔ
31. why Cassandra
32. why Cassandra • ୲౰ʹͳͬͨΒطʹ͋ͬͨ • SPOF͕ͳ͍ • ٸܹͳσʔλ૿ʹ଱͑ΒΕΔscalability • ϊʔυ௥ՃʹΑΔɺεέʔϧΞ΢τ • ฐࣾSmartphone PlatformͰͷӡ༻࣮੷
33. about System
34. Cassandra setting • Versionɿ2.0.8 • Replication Factorɿ3 • Consistency LevelɿQUORUM • use vnodeɿ256 • use CQL,nodejs༻ಠࣗυϥΠό࣮૷ • https://github.com/suguru/cql-client
35. Request • Peak Request • Readɿabout 9,000 qps • Writeɿabout 3,000 qps • Data size • Totalɿ600GB • 1node avgɿ50GB
36. Latency • Readɿavg 2ms • Writeɿavg 0.1ms
37. HW Spec • private Cloud Instance • CPUɿ24core • Memoryɿ94GBɺheap 8GB • Diskʀ1TB • 12node • 1 Cluster
38. HW Spec • ૬౰ʹ᩵୔ͳαʔό • ج൫ͱͯ͠΋͜Ε͔ΒσΧ͘ͳΔ͜ͱΛݟӽͯ͠ͷαʔό • Resourceతʹ͸·ͩ·ͩ༨༟͕͋Δ • nodeݮΒͯ͠΋େৎ෉ͦ • privateCloudͷInstance typeͷϥΠϯφοϓʹΑΓɺ͜ͷ Specʹͳͬͨ
39. about Operation
40. Cassandraશܠ
42. Build
43. Build • Cassandraαʔόͷߏங • Jenkins & ansible • ख࡞ۀ͸Cluster join࣌ͷCassandra ϓϩηεͷىಈͷΈ • vnode(Cassandra ver1.2~)Λ࢖༻ ͍ͯ͠ΔͨΊɺख࡞ۀͰͷtokenͷ ܭࢉˍׂΓ౰͕ͯඞཁͳ͘୯७ͳ ىಈͰOK
44. Monitoring
45. Monitoring • threshold • use sensu • how to check • Community&Original sensu plugin • how to notify • mail & hipchat
46. Monitoring • trend • use OpsCenter • data size&latency • use sensu & influxdb & grafana • how to check • Community&Original sensu plugin
47. Monitoring • check • OS Resource • cpu,memory,disk&nw latency,fd • JVM • heap,gc
48. Monitoring • check • Cassandra • read&write_qps,latency • thread pool • ReadStage • FlushWriter • Compaction • HintedHandoff...
49. Monitoring Ͳ͏΍ͬͯCassandraͷಈ޲Λ௥͏ͷʁʊʁ • CassandraStatusread&writeͷಈ޲Λ௥͏ • Write&ReadStageɺMutationStageɺFlushWrite • Compaction_status • Cassandra_Clusterͷhealth֬ೝ͍ͨ͠ • Gossiptimeowtɺhintedhandoff
50. Operation
51. Operation • repair & cleanup • about 20h / weekly • backup & restore • snapshot & sstableloader • restore CI • ?? h / weekly
52. Operation • repair & cleanup • ϨϓϦΧͷෆ੔߹Λ๷͙ͨΊʹఆظతͳrepairΛ࣮ߦ • σʔλͷ෮׆Λ๷͙ͨΊʹಉ࣌ʹcleanup΋࣮ߦ • ࣮ߦपظ͸ 7days ʻ gc grace seconds(default:10days)2 2 gc grace secondsɿTombstoneͷGarbageCollection࣮ߦ·Ͱͷ࣌ؒ
53. Operation • backup • 2hຖʹ֤nodeͰsnapshotΛ࡞੒͠Swiftʹอଘɻ • restore • test-clusterʹͯఆظతʹrestore͕ग़དྷ͍ͯΔ͔֬ೝ • sstableloaderΛ࢖͍ۭClusterʹdataΛྲྀ͠ࠐ͜Ή3 3 ͨͩsstableloaderΊͬͪΌ͔͔࣌ؒΔ͔Βɺ࣮ࡍͷrestore࡞ۀ͸snapshot௚ஔ͖ͷ෮چʹͳΔ͔΋
54. about Troubleshooting
55. Կ͔͋ͬͨ࣌ʹΑ͘࢖͏nodetool • nodetool status • nodeͷঢ়ଶΛͬ͞͞ͱݟ͍ͨ • nodetool tpstats • ࣮ߦதͷthreadͷ؂ࢹ • nodetool netstats • streamͷ৘ใΛݟΔ
56. Կ͔͋ͬͨ࣌ʹΑ͘࢖͏nodetool • nodetool cfstats • cfຖʹ৘ใΛݟ͍ͨ4 • nodetool disablegossip,disablethrift,disablebinary,flush • disable* : ֤protocolແޮԽ • flushɿmemtable͔Βflushͤ͞Δ 4 CassandraશମͰSlowdownͯ͠Δͷ͔ɺಛఆcfͰ٧·ͬͯΔͷ͔֬ೝ͍ͨ͠ΑͶ
57. Կ͔͋ͬͨ࣌ʹΑ͘࢖͏nodetool ͳͷͰ༧ఆ͍ͯͨ͠nodeͷ࠶ىಈͳͲ͸ԼهΛ࢖ͬͨΓ͢Δ $ nodetool disablegossip && \ nodetool disablethrift && \ nodetool disablebinary && \ nodetool flush && \ /etc/init.d/cassandra restart
58. ͔࣮͠͠ࡍʹ͸ ಥൃతʹnodeʹԿ͔໰୊͕ൃੜͨ͠৔߹ɺ nodetoolͷ݁Ռ͕ฦͬͯ͜ͳ͍ࣄ͕΄ͱΜͲ ͦͷ৔߹͸Ͳ͏͢Δ͔
59. ఘΊͯ࠶ىಈ5 /etc/init.d/cassandra restart 5 ༻๏༻ྔΛकͬͯਖ਼͓͘͠࢖͍͍ͩ͘͞ɻͪΌΜͱlogɺmetricsΛΈͯ൑அͯ͠·͢Αɺɺɺ
60. ͦͷଞɺࠔͬͨ͜ͱ
61. NWো֐6 6 ΋͏طʹා͍Ͱ͢Ͷ
62. ۩ମతʹ͸ʁ
64. ͓͖ͨ͜ͱɺରԠͨ͜͠ͱ • ॠஅ͕ଓ͖L2ϨϕϧͰͷ׬શͳΔஅʹͳΔ • Clusterతʹ͸શnode͕ಠཱͨ͠ঢ়ଶʹɻ • max hint window ms (default:3h)Λ௒͑ͨ(!!)ͷͰhint7ͷ৘ใ ͸શͯഁغ͞ΕϦηοτ͞ΕΔܗʹɻ • NW෮چޙʹશnodeͰrepairΛ͔͚ͯɺClusterͷ෮چʹɻ 7 hint:ଞnode͕μ΢ϯͨ͠ࡍʹॻ͖ࠐ·ΕΔ͸ͣͩͬͨσʔλΛଞͷϨϓϦΧ͕อ࣋͢Δ max_hint_window_ms:↑ ͷhintΛഁغ͢Δ·Ͱͷ࣌ؒ
65. ;Γ͔͑Δͱ • σʔλϩετ͸ແ͠ • ॠஅ͕ଓ͘ܗͰ΋hintΛอ࣋ͯ͠ΔݶΓࣗಈతʹϨϓϦΧͷ ੔߹ੑΛ੔͑Δ͜ͱ͕Մೳ • hint͕ͳͯ͘΋node͑͞௵Εͳ͚Ε͹Clusterͷ෮چ͕Մೳ8 • NWஅʹ΋଱͑ΒΕͨ 8 hint΋ͳ͍node΋μ΢ϯͱͳΔͱɺٹ͑ͳ͍σʔλग़ͯ͘Δɻ౰ͨΓલ͔
66. ͦͷଞઌਓͷ஌ݟ
67. ͦͷଞઌਓͷ஌ݟ9 • slow queryΛݟΔ͜ͱ͕ग़དྷͳ͍ͷͰɺࠔΔલʹΞϓϦଆʹ slow logΛ࣮૷͢Δ • εΩʔϚઃܭେࣄ໰୊ • wide rowΛආ͚Δɻࣄલ෼ׂग़དྷΔͳΒͪΌΜͱ͠Α͏ • Cassandraʹݶͬͨ࿩Ͱͳ͍͚ΕͲ... 9 ઌਓͷي੻ http://www.slideshare.net/oranie/cassandra-summit-jpn-2014-100-node-cluster-admin-operation
68. ·ͱΊ • ࠷௿ݶͷࣄΛ཈͓͚͑ͯ͹ӡ༻͸ָ • CassandraɺŠƂŞūŘż • ઌਓͷ஌ݟΛ͋Γ͕͍ͨͨͩ͘͜͏ • ͦͯࣗ͠෼ͨͪ΋஝ੵͯ͠ڞ༗͠Α͏ɻ • Cassandra CommunityʹߩݙతͳͶ • 1.xxܥͱ͸͓ผΕΛ͠Α͏ɻ͍ͨ͠ʢന໨
69. ͜Ε͔Βͷ͜ͱ • ͜Ε͔ΒͷClusterઃܭ • ͦͷ··Ծ૝ʁ෺ཧʁʊʁ • PITRʹ͍ۙ͜͠ͱɺ͍ͨ͠ɺɺɺ • σʔληϯλʔػೳΛ࢖ͬͯɺBackupઐ༻ͷCluster࡞੒ • Backup͚࣌ͩɺσʔληϯλʔؒͷϨϓϦΛࢭΊͯɺॻ͖ ࠐΈͷͳ͍ঢ়ଶͰBackupΛऔΔͱ͍͏ໝ૝Λ͍ͯ͠Δɻ
70. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ ͳʹ͔͋Ε͹࠙਌ձͷ࣌ʹੋඇʂʢʈТʈʣ