Upgrade to Pro — share decks privately, control downloads, hide ads and more …

はじめるCassandra

kakerukaeru
September 08, 2015
280

 はじめるCassandra

Cassandra Meetup in Tokyo, Summer 2015でお話してきた奴

kakerukaeru

September 08, 2015
Tweet

Transcript

  1. ͩ

  2. Α

  3. agenda • about CyberAgent&Service • why Cassandra • Operation •

    Build,Monitoring,Backup etc... • about Troubleshooting • ·ͱΊ
  4. about Cassandra in CyberAgent • Cassandra Versionɿ1.1.5, 1.2.13 • Production

    Clusterɿ3 • Production nodesɿ about 150node • Total about qps Read&Writeɿ 50000qps • Total about data sizeɿ15TB
  5. ͕

  6. For Native App • ωΟςΟϒΞϓϦ༻ج൫ • ੜ·Εͯ̍೥ऑͷج൫ • ೝূɺ՝ۚɺloggingͷapiΛఏڙ •

    Cassandraͷ࢖༻͸ओʹid؅ཧͷ෦෼ • idʹඥ෇͚ͯɺ՝ۚˍloggingͳͲͷbackendͷSystemʹܨ ͍͛ͯΔ
  7. Cassandra setting • Versionɿ2.0.8 • Replication Factorɿ3 • Consistency LevelɿQUORUM

    • use vnodeɿ256 • use CQL,nodejs༻ಠࣗυϥΠό࣮૷ • https://github.com/suguru/cql-client
  8. Request • Peak Request • Readɿabout 9,000 qps • Writeɿabout

    3,000 qps • Data size • Totalɿ600GB • 1node avgɿ50GB
  9. Build • Cassandraαʔόͷߏங • Jenkins & ansible • ख࡞ۀ͸Cluster join࣌ͷCassandra

    ϓϩηεͷىಈͷΈ • vnode(Cassandra ver1.2~)Λ࢖༻ ͍ͯ͠ΔͨΊɺख࡞ۀͰͷtokenͷ ܭࢉˍׂΓ౰͕ͯඞཁͳ͘୯७ͳ ىಈͰOK
  10. Monitoring • threshold • use sensu • how to check

    • Community&Original sensu plugin • how to notify • mail & hipchat
  11. Monitoring • trend • use OpsCenter • data size&latency •

    use sensu & influxdb & grafana • how to check • Community&Original sensu plugin
  12. Monitoring • check • Cassandra • read&write_qps,latency • thread pool

    • ReadStage • FlushWriter • Compaction • HintedHandoff...
  13. Operation • repair & cleanup • about 20h / weekly

    • backup & restore • snapshot & sstableloader • restore CI • ?? h / weekly
  14. Operation • repair & cleanup • ϨϓϦΧͷෆ੔߹Λ๷͙ͨΊʹఆظతͳrepairΛ࣮ߦ • σʔλͷ෮׆Λ๷͙ͨΊʹಉ࣌ʹcleanup΋࣮ߦ •

    ࣮ߦपظ͸ 7days ʻ gc grace seconds(default:10days)2 2 gc grace secondsɿTombstoneͷGarbageCollection࣮ߦ·Ͱͷ࣌ؒ
  15. Operation • backup • 2hຖʹ֤nodeͰsnapshotΛ࡞੒͠Swiftʹอଘɻ • restore • test-clusterʹͯఆظతʹrestore͕ग़དྷ͍ͯΔ͔֬ೝ •

    sstableloaderΛ࢖͍ۭClusterʹdataΛྲྀ͠ࠐ͜Ή3 3 ͨͩsstableloaderΊͬͪΌ͔͔࣌ؒΔ͔Βɺ࣮ࡍͷrestore࡞ۀ͸snapshot௚ஔ͖ͷ෮چʹͳΔ͔΋
  16. Կ͔͋ͬͨ࣌ʹΑ͘࢖͏nodetool • nodetool cfstats • cfຖʹ৘ใΛݟ͍ͨ4 • nodetool disablegossip,disablethrift,disablebinary,flush •

    disable* : ֤protocolແޮԽ • flushɿmemtable͔Βflushͤ͞Δ 4 CassandraશମͰSlowdownͯ͠Δͷ͔ɺಛఆcfͰ٧·ͬͯΔͷ͔֬ೝ͍ͨ͠ΑͶ
  17. ͓͖ͨ͜ͱɺରԠͨ͜͠ͱ • ॠஅ͕ଓ͖L2ϨϕϧͰͷ׬શͳΔஅʹͳΔ • Clusterతʹ͸શnode͕ಠཱͨ͠ঢ়ଶʹɻ • max hint window ms

    (default:3h)Λ௒͑ͨ(!!)ͷͰhint7ͷ৘ใ ͸શͯഁغ͞ΕϦηοτ͞ΕΔܗʹɻ • NW෮چޙʹશnodeͰrepairΛ͔͚ͯɺClusterͷ෮چʹɻ 7 hint:ଞnode͕μ΢ϯͨ͠ࡍʹॻ͖ࠐ·ΕΔ͸ͣͩͬͨσʔλΛଞͷϨϓϦΧ͕อ࣋͢Δ max_hint_window_ms:↑ ͷhintΛഁغ͢Δ·Ͱͷ࣌ؒ
  18. ͦͷଞઌਓͷ஌ݟ9 • slow queryΛݟΔ͜ͱ͕ग़དྷͳ͍ͷͰɺࠔΔલʹΞϓϦଆʹ slow logΛ࣮૷͢Δ • εΩʔϚઃܭେࣄ໰୊ • wide

    rowΛආ͚Δɻࣄલ෼ׂग़དྷΔͳΒͪΌΜͱ͠Α͏ • Cassandraʹݶͬͨ࿩Ͱͳ͍͚ΕͲ... 9 ઌਓͷي੻ http://www.slideshare.net/oranie/cassandra-summit-jpn-2014-100-node-cluster-admin-operation