work with NoSQL databases ~ 5 years • Cassandra ~6 month • Besides working with data, I love to bike, which can be a challenge in San Francisco 2 Me Hello, Thank you for attending my talk this morning. My name is Susanne and I am a DBA at yelp. I work with 3 other dbas together to support ~300 developers in their datastore needs. And that’s me biking home from work. Not really, but almost. I want to mention that my slides will be on speaker deck in case you are interested in them.
would have said faster horses. (Henry Ford) I would like to start my talk with this quote, because given the fact, that we are at a MySQL conference, and I will mainly talk about us using Cassandra I thought it is rather fitting.
great local businesses • Yelp had an average of approximately 135 million monthly unique visitors in Q4 2014 • Yelpers have written over 71 million local reviews • In addition to reviews, you can use Yelp to find events, lists and to talk with other Yelpers • Yelp communities have taken root in major metros across the US, Canada, Europe, Asia Yelp is an online review portal. And what you can take away from these numbers is: - we have lots of data - we have lots of users and our users are distributed over big parts of the world Also you can see that we were founded in 2004 and hence
pattern, rather key-value style • Too much writes for one master => logical sharding • Move data out of MySQL 8 Why Do We Want To Move Away From MySQL For Certain Use Cases?
our requirements were multiple master writes we were in the NoSQL realm and because of the rather key-value style of data pattern, we were in the key-value/ column db area of the NoSQL dbs. As you know from my presentation title, we have eventually chosen Cassandra. Let me show you some of the highlights, to show why:
gracefully • Scales horizontally 10 Highlights Of Cassandra Cassandra has a peer-to-peer distribution model it. Therefore a cluster is also often called a ring, the individual host are called nodes in this ring.
are durable • Consistency tunable (CAP) 11 Highlights Of Cassandra, Cont’d Writes: Although our app is very read heavy in general, since as you can imagine way more people read reviews, than they write one, we do have services, that are very write heavy as you will see. Tunable consistency I will talk later more about.
(ALMOST as good as the MySQL one ;-) ) 12 Highlights Of Cassandra, Cont’d Since Cassandra is a wide column db, the schema is more flexible than a pure key-value db. Wide column stores, store data in records with an ability to hold very large numbers of dynamic columns. Since the column names as well as the record keys are not fixed, wide column stores can be seen as two-dimensional key-value stores. Cassandra has matured quite a bit over the years. Whereas in earlier years you had to manually assign the tokens, which mark the ownership of range keys, that was a pain! Now the default is to use so called virtual nodes, that Cassandra balances automatically. An aspect which was also important for us, was that Cassandra a very strong and supportive community.
with more powerful machine • One cluster per service • SOA based architecture • Why: can be tuned per use case, .. 13 Cluster Setup Choices First decision we had to make after deciding that we want to use Cassandra was, how do we wanted to deploy it. In AWS or in our own datacenters? And our first decision was that we wanted to have one cluster per service, fits into our SOA based architecture. It’s easier to target the cluster to an individual service,
• Instance type: c3.8xlarge • Durable writes = true 14 Our (Default) Production Setup Then we decided, to run solely in AWS. Datastax is for Cassandra what Percona is for MySQL. This is our default prod set up, we especially like the 10 Gigabit networking support, but we do vary on the instance type, depending on each individual service Cassandra provides write durability by appending writes to a commit log. Depending on the commitlog_sync option, the commit log is then synced to disk either periodically (every 10 seconds by default) or in the batch mode, Cassandra will wait with acknowledging writes until a commit log has been fully flushed (fsynced) to disk. This mechanism is called durable_writes.
different AWS regions, spread over AZs This is a presentation of one of our production Cassandra clusters a so called ring. On the right hand site you can see the two different amazon regions, data centers in two regions, the two different AZs, all nodes are up
per DC • Datacenter awareness through GossipingPropertyFileSnitch 16 Our (Default) Production Setup, Cont’d We use only SSD based ephemeral storage for our data partition How many copies of my data do I want to have? (replication factor) Cassandra has the concept of a so-called snitch. A snitch determines which data centers and racks nodes belong to and therefore inform Cassandra about the network topology so that requests are routed efficiently.
-l unlimited, ulimit -n 48000 • Synchronize clocks via NTP • Different compaction strategies, per use case • All our nodes are seeds 17 Our (Default) Production Setup We have implemented on our Cassandra nodes the recommended settings for production cluster. But we do use different compaction strategies per use case, I will talk about this later more.
18 Our Architecture Set-Up Next we decided on our architecture setup: As I said already, we are deploying Cassandra into AWS. We are using puppet at yelp as our configuration management tool. We needed to find some way to get an IP from the cluster into the application. We could have used a static file for this, but since we were using Zookeeper and Smartstack in-house already, we thought Zookeeper, would probably serve us better. Nerve is being used for seed discovery.
by Airbnb) • Ideal for SOA architecture • Haproxy + nerve + synapse + zookeeper = smartstack 19 Our Architecture Set-Up For those of you, who might not be so familiar with smartstack. SmartStack is an open source tool developed by Airbnb to manage SOA stacks. It consists of two different services Nerve and Synapse. We use Nerve of the smartstacks tool, it keeps a central Zookeeper updated with the status of its local Cassandra nodes.
install Cassandra on an AWS instance using Puppet. This example shows you that we have one puppet class per Cassandra cluster. In this case this is a cluster called demo. We also configure the Smartstack/nerve configuration. Then Puppet updates the nerve config running on that instance. Let’s now look more into our use cases
the CAP theorem • By trading away C, we’ve picked use cases with eventual consistency in mind When you think about a distributed system, you have to have the CAP theorem in mind. Basically, the CAP theorem says that of the 3 guarantees availability, partition tolerance and consistency, you can pick only 2. Cassandra focuses on availability and partition tolerance, that’s why your app can still fully write and read if some nodes went down. But on the other hand that means that you have to deal with looser consistency then in MySQL.
you 3 key use cases, all of which affect the user experience. This -for example-is a screenshot from our IOS mobile app, where we offer a “friend” feed and a “nearby” feed.
a read looks similar to select * from feed_table where user_id=123 and feed_type in (xx,yy) order by time_created; 23 First Yelp Use Case: Feed Data When we were using MySQL, when working to display a user feed, the application would directly fetch all the feed entries that the user followed from a single MySQL db, sort them by creation time. This is called a fan out on read approach. Since you are all MySQL experts, you will probably see right away that this query with the in clause and the order by is difficult to tune in a relational db.
Fan Out On Write vs Fan Out On Read • Cassandra is perfect for Fan Out on Write • The general nature of feed data is complicated: Jeremy Stoppelman use case 24 First Yelp Use Case: Feed Data, cont’d In building an aggregation timeline, the primary architecture decision is when the fan-out occurs. As I said on the previous slide when we were using MySQL, we were using a fan out on read approach. But when your timeline most resembles Twitter, a roughly permanent, append-only, per-user aggregation of a set of source feeds, fan out on write is better. Of course Fanout-On-Write approach is very well suited for fast reads, we have chosen that. Write costs are essentially equal to the number of followers each user has and is done for each post. To do this reliably for an edge case, who has a lot of friends and followers in our app, this process needed to be handled asynchronously and in the background. outlier jeremy! user like our very own CEO Jeremy Stoppelman,
families • user_personal_feed • user_friend_feed • user_following_feed 25 First Yelp Use Case: Feed Data, cont’d Column families = tables in Cassandra. So the key here is to denormalize the data. Although we are writing even more with every new event, the reads are fast because each CF is essentially a materialized view of every event in the end user has visibility into.
is Caching. When you use the yelp app on your mobile device, yelp is offering you to search within your surrounding, when you allow us to see your location.
• We want to have this data specifically close to the user => Multi-master writes • Lots of long primary keys being written 27 Second Yelp Use Case: Caching For this location based searches we wanted the data specifically close to the user and fast writes. You might know, the “rule of the thumb” for InnoDB primary key design that they should be as short as possible. Cassandra is ok with long primary keys.
DC 28 Second Yelp Use Case: Caching, cont’d * so instead of MySQL we were using Memcached for a while * Memcached was set up in single cluster per DC, then in case of a DC failover the caches where empty So with Cassandra this working now very wel for us.
the result of a search request and we are presenting ads here. The top two businesses are ads, The data with the little yellow square next to the business names show ad content. We are storing a lot of data for a short period of time. * which ad have we shown you already * have you clicked on it or not * then later billing our ad partners based on some of this data is done in Cassandra.
transient data • Lots of writes • Multi-master writes are favorable 30 Third Yelp Use Case: Time Series Data, Cont’d We store a ton of data about ads, when people click, when people don’t click. Again we wanted to have multi-master writes and it very transient data.
Time To Live mechanism • Cassandra cleans for us after 48 hours The data that we are storing here is expiring very fast. If you want to expire data in a database, you don’t have much choice: you need a periodic task that somehow finds expired data and removes it. With lots of data, keeping this efficient can be a challenge especially in a relational db. Taking advantage of TTL mechanism, time to live, it is cleaning up for us after 2 days the data will be deleted.
• More projects in the pipeline • Great addition parallel to MySQL These were 3 of our use cases where we are currently using Cassadra, we have a 2nd feed running, some analytic services, and some monitoring tools for example Zipkin. I will now share some of the lessons that we learned along the way. What we felt was important to know…
accuracy • Read/Write Consistency Levels: all, each_quorum, quorum, local_quorum, one etc. 33 Consistency Level As I was mentioning already, one of the perks of Cassandra is that you can tune Cassandra’s eventual consistency. For any given read or write operation, the client application decides how consistent the requested data must be.
for reads in some cases • A write must be written on a quorum of replica nodes (sum_of_replication_factors /2) + 1 • E.g. a two data center cluster where each data center has a RF = 3, a quorum is 4 nodes―the cluster can tolerate 2 replica nodes down 34 Consistency Level We have decided to use quorum, which provides strong consistency if you can tolerate some level of failure. What is quorum? A quorum is calculated, and then rounded down to a whole number As an example how this influences availability: when you use quorum in combination with RF 3, the cluster can tolerate 2 nodes down.
into the workflow of how Cassandra handles an update request: You send a request to update your table -> this request is immediately written into a CommitLog -> then into an in-memory table called a Memtable (like innodb buffer pool!) -> Once the Memtable hits a certain size, it flushes the entire Memtable to an on disc Sorted Strings Table, which can no longer be modified. * Writes are sequential append operations, that ’s why they are so fast. * Reads will be checked in the memtable first, before they go to disk.
write “tombstones” • Updates are written as new data 36 Updates And Deletes How do updates and deletes happen in this model? Because the Sorted string Tables are immutable and you need to keep track between nodes Tombstones mark a row or column as deleted, data still exists, but can not be read by API With this concepts updates and deletes are as fast as inserts. The downside to this approach is Of course, this data needs to be cleaned eventually
• Evicts tombstones • Consolidates SSTables • Creates a new index in the merged SSTable 37 What Is Compaction In Cassandra? This cleaning process is called Compaction. It re-organizes SSTables to optimize data structures on disk as well as reclaim unused space. Compaction is a regular part of working with Cassandra and the impact can be minimized if handled properly.
heavy, read-lights workloads • Leveled compaction (Use Case: Caching) • Is more aggressive about compacting • Thus recommended for workloads with a read-heavy, frequent updates pattern • Date-tiered compaction (Use Case: Timeseries Data) 38 What compaction strategies to use? Best performance out of reads: correct compaction strategy!! data tiered compaction is best for time series data
Grafana • (We do not use Datastax’s opscenter) 40 Monitoring If things go sideways, we need monitoring and alerting. Cassandra -being a big java process basically-it comes out of the box with jmxtrans, an open source JMX monitoring application. Via this protocol CAssandras statistics are available and it is possible to plug this into any graphing tool you want to use. We are using Graphite and Grafana. We do not use Datastax’s opscenter, because we want to have all the metrics in one place.
• Cassandra specific alerts e.g. replication_lag (Jolokia) 42 Alerting If you have attend my colleagues talk on Tuesday, you know already, the we are using sensu at yelp to monitor our machines and so do we with our Cassandra boxes. With Sense we have implemented checks on CPU… Additionally, we use a jmx plug in, called Jolokia, in which we implement specific Cassandra alerts, e.g, replication lag. Which gives you more flexibility to write your own alerts.
Encrypt and load backups to s3 43 Backup Big philosophical question, do you need backup in distributed system? We think yes, e.g. as in all databases, to help you to recover from logical error
Failover is a breeze • Cqlsh gives us a great sql-like interface 44 Experience So Far (Technical) Although Cassandra has a bit of the reputation of being a set-it-and-forget-it system, it still needs regular care. The masterless node architecture is a really nice, appreciated feature, when you come from MySQL and had to make master/slave fail-overs in the middle of the night. For example when you work with Mongo DB you better learn javascript to be able to talk with your db on the command line, but Cassandra offers a very SQL like shell.
Some statements do not translate easily from SQL to CQLSH In some cases the sql like shell doesn't help in the way you were hoping, e.g. if you want to see which threads are running.
basic rules are not true in Cassandra: • Minimize the Number of Writes • Minimize Data Duplication • Determine beforehand exactly what queries you need to support 47 Experience So Far (Technical) A rule of thumb in Cassandra is for every query you want to answer in Cassandra you should use your own CF. This demands a different way of thinking. You not only need to know your data, you need to know what you want to ask your data.
• Troubleshooting in a distributed system • Changes to a prod system can become very complicated 48 Experience So Far (Technical) When you run into problems, information is sometimes harder to find than as for a mature (very well documented) system like MySQL Troubleshooting in a distributed system can be more complicated than in a clear master-slave setting Schema changes are a breeze with Cassandra but to change some other settings, like Data types, number of DCs etc. can become more complicated than you were hoping for, so you need to plan in advance
lot so far • Ramp up the experience 49 Experience So Far (Cultural) we have now already 10 Services using Cassandra and more in planning try to stay ahead of the curve
“War games” 50 Future In the near future reporting tools for teams like log stash processes for deploying More “war games”, be better prepared for if things go sideways
siroemk
chobishiba
falcon8823
ryu1013
timakin
rshindo
oku053
panda728
hollycummins
palkan
syumai
kinopeee
aleyda
greggifford
sugarenia
codingconduct
portentint
samlambert
samtorres
rabernat
tammyeverts
konakalab
eileencodes
