Cassandra To Infinity And Beyond

How Teads scale with Apache Cassandra.
Internet scale means tons of data, read heavy workload, massive data ingestion and low latency.
The French AdTech company Teads uses Cassandra massively, a reliable and performant Open Source database.
Spawning Cassandra nodes in AWS is a piece of cake with Terraform and Chef.

Transcript

1. Cassandra To Infinity And Beyond Romain Hardouin Senior Cloud Infrastructure Engineer The Global Media Platform
2. Cassandra @ Teads About Teads Workload Infrastructure Ops Monitoring & Alerting Data model Why a fork?
3. About Teads
5. Ad Slot Available Ad Available Video Start User click Video Complete
6. French AdTech AWS / GCP Scala / JS / Go Machine learning Docker / CoreOS Terraform / Chef / Debian Cassandra / Kafka / MySQL / Redis Spark / Flink
7. Workload
8. Workload Different kinds of workloads but all of them are latency sensitive Internet scale Massive amount of data ingested from partners We also create lots of data by ourselves No more analytics • Tons of business critical counters • Time series • TTL, TTL, TTL
9. 3 years ago 1 million qps Write heavy Analytics stack
10. Now +2 millions qps Read heavy…
11. Now ...but also lots of streamed SSTables not counted as writes
12. Infrastructure
13. Gimme some figures! 250 nodes Mostly ephemeral data 28 TB 100 billions keys 3 regions Regional vs Worldwide 21 DCs
14. Does it scale? Requests vs Nodes over the past year (linreg)
15. Our Stack
16. AWS instances c5d.4xlarge 16 vCPU @ 3.00GHz 32 GB RAM 400 GB NVMe
17. Why not i3? Production P99 read latency i3.2xlarge
18. Why not i3? Production P99 write latency i3.2xlarge
19. Why not EBS? No more EBS Cheap storage, great for STCS Snapshots (S3 backup) No coupling between disks and CPU/RAM High latency, high I/O wait Throughput: 160 MB/s Unsteady performances
20. Ops
21. Ops Workflow More on https://medium.com/teads-engineering/easy-cassandra-scaling-with-terraform-chef-rundeck-9443e0375aa7 Scale out Scale in Any changes
22. Ops Workflow apply
23. Ops Workflow replace apply Unreachable instance Replace
24. Repair No incremental repair Scheduled with Reaper http://cassandra-reaper.io
25. Monitoring & Alerting
26. Monitoring Overview dashboards Advanced dashboards for troubleshooting Alerting dashboards Outliers Compare a node to average Compare 3 DCs (multi regions)
27. Monitoring Ratio cross DCs/Clusters to grasp workloads Examples: • R/W Spread: ( (max qps -min qps )/max qps )*100 • P99/95 jitter factor: ( P99 - P95 ) / P99 • Memory cached / disk ratio
28. Monitoring YAML Configuration - include: bean_regex: org.apache.cassandra.metrics:type=ReadRepair,name=.* attribute: - Count - include: bean: org.apache.cassandra.metrics:type=CommitLog,name=TotalCommitLogSize
29. Alerting Down node Exceptions Commitlog size High latency High pendings tasks Many hints Clock out of sync IO Wait Disk space ...
30. Data model
31. KISS principle No secondary index No materialized views No UDT/UDF No SASI*
32. Tables settings All* type of compactions are used: STCS, LCS, TWCS LZ4 @ 4KB almost everywhere
33. Back in 2016 Old analytics data model: How do you repair 45TB of data within gc grace period?
34. Why a fork?
35. Apache Cassandra version Fork started with C*2.1 Now in C*3.0 Maybe C*4.0 directly
36. Fork motivation 1. Need to add a patch ASAP High Blocked NTR CASSANDRA-11363 2. Why not backport interesting tickets? 3. Why not add small features/fixes? Expose tasks queue length via JMX CASSANDRA-12758
37. Backport example CASSANDRA-13418 Allow TWCS to ignore overlaps when dropping fully expired sstables
38. Custom feature: Securing a legacy cluster without any downtime Allow to create roles prior to use standard auth stack • AllowAnyCredentialsAuthenticator ◦ "User '{}' has been authenticated without password checking" • UnsecureAutoLoginCassandraRoleManager ◦ Allow any non-existing role to login: "Auto login of role '{}'" ◦ If a role exists standard checking will be done. • UnsecureCassandraAuthorizer ◦ Authorize DML on any resource for any authenticated user. ◦ Super users will be given all permissions i.e. DML, DDL and DCL. • No spam logs
40. One-liner of the year 2016 CASSANDRA-12580: Fix Merkle tree size calculation Paulo Motta: log2 vs ln
41. Fix Merkle tree size calculation “How do you repair 45TB of data within gc grace period?” For a subset of tables: • Before: 23 days • With CASSANDRA-12580: 16 hours
42. Can’t upgrade?
43. Can’t upgrade? Quick reaction High Blocked NTR ticket has been released in 2.1 but you need to wait after a release Stability, reliability Critical counters e.g. CASSANDRA-14958
44. Questions?
45. We hire ;-)
46. Thank you!
47. References cassandra.apache.org thelastpickle.com Buzz Lightyear is a character in the Toy Story franchise. © Disney/Pixar Back to the Future is a trademark and copyright of Universal Studios and U-Drive Joint Venture. Licensed by Universal Studios. All Rights Reserved. Crown icon designed by Good Ware from Flaticon www.chef.io www.datadoghq.com www.packer.io www.rundeck.com www.terraform.io