Counting (Notes)

This talk is going to cover some techniques of counting using a computer. Counting problems come up very often in areas of databases, networking, and elsewhere. Counting is so simple that by itself it isn’t even worth talking about, but there are some techniques that have truly impressive gains.

Notes accompanying these slides: https://speakerdeck.com/roshmat/counting-using-computer

Transcript

1. FINAL Roshan Mathews Counting (using computer) (Slide 1) This talk is going to cover some techniques of counting using a computer. Counting problems come up very often in areas of databases, networking, and elsewhere. Counting is so simple that by itself it isn’t even worth talking about, but there are some techniques that have truly impressive gains. (Slide 2) We’ll discuss this topic in the context of two problems. These problems have simple direct solutions that are usually good enough for most purposes. But when dealing with large data sets, there are alternate solutions that have very surprising characteristics, and are better suited for reasons that we will see. The first problem is this – if given a collection of data we need to count the number of elements present. An example of this can be counting the number of requests that hit an HTTP server. The second problem is similar to the first, but instead of counting how many elements are present, we want to count how many unique elements are there. For detecting network attacks, one of the indicators is the number of source-destination pairs that are present in traffic. In databases, the query optimiser chooses which strategy to use by estimating sizes of sets it is joining. Again, I’d like to emphasise that with small sets of data, we don’t have a problem per se, but the techniques we’ll see soon become very useful for large data sets. (Slide 3) Let us consider the first problem, which is conceptually simple and sets the tone for the discussion of the second problem. There is a simple direct solution to this. PAUSE The direct solution to the first problem is to keep a counter. Go over the data, for each item you see, you increment the counter. The final value of the counter is your answer. This is a good solution, it gives you the correct answer. It is linear in time, since you only go over the data once. To improve on time you’ll need to find a way of counting without looking at all data. It is also logarithmic in space, since to store a number as large as n you need log2n bits of space. (Slide 4) Two bits allows you to count to 4, to be able to count up to 256, we need 8 bits. To be able to count up to a million elements, you need 20 bits. To count a billion you need 30bits, etc. So this looks pretty good, maybe there is no need to do any better. PAUSE (Slide 5) But memory wasn’t always cheap. In 1977, Robert Morris, a scientist and programmer at Bell Labs, encountered a situation where he needed to keep track of a large number of 8-bit counters. Moving to 16-bits was not an option given the memory he had. But 8 bits clearly wasn’t enough. (Slide 6) So the problem he faced was that with 8-bit counters he could count up to 256, but the events he was counting exceeded that. Could we count to more than 256 using 8 bits, the answer is no. That’s the minimum required to get to that number. So what could we do to count to say, double that number, that is 512? of 1 7
2. FINAL Roshan Mathews (Slide 7) Here’s some ideas. We could try to maybe “reuse” our counter, count to 256, then start over from zero? We keep a flag that tells us if we’re using, or reusing our code. Another idea is this: maybe we could count every alternate event, thus doubling our capacity? Flag to keep track of whether the event we see next should to be counted or skipped? What’s the problem with these two ideas? PAUSE (Slide 8) The problem is that a flag is one bit of state. It is a 9th bit in addition to the 8 we’re already using for the counter. We don’t have any bits to spare. We can’t get an accurate counter to count more than 256. Maybe we can get a not-so-accurate counter? Accuracy is overrated. It is useful to have Re.1 coins if you want to buy a Rs5 cup of coffee, not especially required if you want to buy a Ferrari. For many use cases it is okay to have a number that’s approximately right. (Slide 9) What we shall do – or rather, what Robert Morris did – was to toss a coin. A fair coin, that has equal chances of heads or tails. 50 - 50. If we get heads, we increase our counter. If tails, we stay at the same value This is a strange idea, we’re no longer deterministic. We’ve introduced a probabilistic element to our program, and now our answer might not be exactly right. We’re trading accuracy to be able to break the information theoretic bound to the number that we can count using just 8-bits. This is important, this is the main insight of this talk. That we can “trade accuracy for space”. PAUSE (Slide 10) Here’s a picture to illustrate how this new “counter” of ours works. A simple regular counter goes from 1 to 2, from 2 to 3, and so on. Here, if we are at 1, and we increment the counter, then we can go to 2 with 50% chance if we get heads, or we can stay at 1, with 50% chance if we get tails. PAUSE This might sound like a neat idea, but it might not be obvious if this is solving the problem at hand. The coin tossing counter can go anywhere. (Slide 11) It helps to think of the coins like this: on average we expect to see one head, for every two tosses. If we’ve tossed the coin 10 times, we expect to see 5 heads. We can see more, or maybe less. But the most likely outcome is 5. So in the case of our counter, we expect the value to increment once for every two tosses. PAUSE Let’s look at a probability chart. If we toss a fair coin 10 times, then the x-axis here is the number of heads we see, and the y-axis is the probability of that outcome. When we say that we expect to see 5 heads when we toss the fair coin 10 times, we mean that it is the most likely outcome. It is possible that we see no heads in 10 tosses, but that is very unlikely. (Slide 12) One thing is obvious directly, that the “counter” isn’t a counter anymore. It isn’t storing the count of the number of elements. Instead it is an “estimator”, we can use this to get an “estimate” of the number of elements. In this case, if the estimator is k, then our estimate is 2k. If we’ve seen n elements, then the estimator should be approximately n/2 So our estimator, which is 8 bits, can still go to 256. You can’t do more than that! But we can use that to estimate numbers up to 512. PAUSE of 2 7
3. FINAL Roshan Mathews (Slide 13) This is very easy to implement on a computer. We use a uniform random number generator that gives us values between 0 and 1, and increment if the value generated is less than 0.5. This can be done in four lines of Python. (Slide 14) So we’ve seen that we can count to 512, double of 256, by using a coin. To count to higher numbers, say to 3 times 256, 768, we use a biased coin. A biased coin isn’t fair, the probabilities of heads and tails are not equal. With a coin that has 1 in 3 chance of being heads, and 2/3 chance of tails. We now “expect” our counter to move forward once every three tosses. With a coin that has 1 in 1000 chance of being heads, our counter to move forward once every 1000 tosses, allowing us to count to much higher values. (Slide 15) Just for completeness, lets look at the code for this, we just introduce the bias of the coin as a parameter to our function. (Slide 16) One issue with our estimators so far, has been the error. For small counts the errors can be significant. Take the case of the fair coin, with a actual count of 1, our estimate will be either 0 or 2, that is off by 1 … which is a relative error of 100%. (Slide 17) For a biased coin, similarly, for an actual count of 1, the estimate will be 0 or 1000! The error is 1 or 999. (Slide 18) So the problem we’ve seen is that the error can be large for small counts. Another approach is to use a different estimator, one that tracks log of the count. If we have an estimator k, which represents the log of the actual count, then we can get the actual estimate by using 2k. The error is better, always 100%, but we’ll see why that’s better. How do we implement this estimator? With this log-estimator after n increments we expect the value of the estimator k to be approx log n … we want the estimator to go to k + 1 after another n increments. But we have to do this with only the value k at hand, we don’t know how many increments we have seen after the first n. We have a value, k, that k should not increase for another n increments! How can do we do this? PAUSE (Slide 19) We use the idea of coin tossing here again, with probability 1/2k we shall increment the value of the counter, and with probability 1 - 2-k, we shall stay the same. This might look weird, so lets look at a picture. (Slide 20) Here’s the same information in a picture. This will work out to the numbers we want on average. We go from 0 to 1 in 1 count, from 1 to 2 in 2 counts, from 2 to 3 in 4 counts, and so on. Instead of a fixed probability of counting up, we now have a counter that counts up slower and slower as the counter value increases. PAUSE (Slide 21) This estimator can be implemented just as easily as earlier. ** is exponentiation in Python. of 3 7
4. FINAL Roshan Mathews (Slide 22) The error, as mentioned in the passing earlier, is approximately one binary order of magnitude. The error is the same as the current value of the counter. This is not good, but we now have a constant relative error. 100%, as it is. If the estimator is 10 instead of being at 9, then the count will be 1024 instead of 512. So the error is 512! This is a constant relative error of 100%. Lets see how we can fix this. (Slide 23) We can reduce this error by changing the base of the estimator, instead of a base 2, we use a number smaller than 2, to get a relative error that is a smaller fraction. With the base of 2, the error was a factor of 2, we will use a smaller base. (Slide 24) With a value of a that is chosen as shown, we can count up to n digits using only log2log2n bits plus a fixed number of bits. For example, with a set to 1.044 we can count up to 65,000 using 8 bits, which is half the number of bits what is required by an exact counter. And much more than what you can count exactly with 8 bits. (Slide 25)
 This algorithm was given the name “approximate counting”, it has a small relative error, very small space requirements and as we’ve seen it can be implemented on a computer in a very simple and efficient manner. The interesting ideas introduced are that of “estimation”, “probabilistic counting”, and that of “trading accuracy for space” This concludes our treatment of the first of the two problems that we’re considering. The more interesting stuff is coming up, but this has laid a foundation to understand that. To tie this to our counting requests example from before, we probably don’t need to save 8-bits any more, but the ideas introduced are interesting. Before we go on are there any QUESTIONS (Slide 26) Now we come to the second problem of counting that we’re considering, and that is to count the number of distinct elements in a collection of data. This is called the cardinality of the data. This problem comes up often, for example if we have some HTTP logs and we want to count the different IPs that we have requests coming from. In databases, counting the number of uniques is used for query optimisation. Used for detecting attacks in a network. Let us first look at the straightforward solution. How would we solve this? PAUSE We can use a hash map, and then count the keys. You go over the data, putting each item into the hash and then count the number of keys. (Slide 27) This solution is good, it is accurate, is fast. It uses too much space though. Like earlier we’re going to try to trade accuracy for space. Unlike the previous case, we’re going to motivate how the fancy algorithms here work, instead of going into the details. (Slide 28) What we learned from Approximate Counting was that you can give up some accuracy, and use probabilistic techniques and get good enough solutions to problems. The key idea was using the notion of an estimator. What estimator can we use here? (Slide 29) One way to approach the problem can be to use sampling. We take a sample that has the same “structure” in some sense as the complete data set. Then we count – exactly – the number of unique elements in the sample, and scale that up to estimate the cardinality of the full set. of 4 7
5. FINAL Roshan Mathews The problem with this approach is that it can have a very high error rate depending on the replication in the data. Consider a multi set containing a million integers all between 1 and 10. If I draw a sample of 1000 numbers and count the uniques, I’ll get 10. If I scale that up by 1000, I’d estimate that the full data set has 10,000 unique elements when it actually has only 10. So this isn’t very good. We need a better solution. (Slide 30) Let us motivate a better solution, by thinking about a simpler problem. First let us try to distribute some numbers between 0 and 100, if we want 9 numbers evenly spaced, then what numbers would we choose? No matter how often they are repeated, we have to choose the numbers 10, 20, 30, 40, 50, 60, 70, 80, 90 … only those will be evenly spaced. What is the minimum value? It has to be 10! (Slide 31) Can we do the reverse of this puzzle. What if we our data consisted of perfectly evenly distributed numbers, with arbitrary repetitions, and given to us randomly shuffled. How can we estimate the number of unique elements in that. PAUSE (Slide 32) Knowing the range of the data, we just scan the data to find the minimum value and then use that to estimate the cardinality. So if we know that the data ranges from 0 to 100, and if we find the minimum to be 10, we know that there are 9 unique elements in the set! PAUSE This is a good solution, looks every item once, uses fixed additional storage, very easy to implement, and does not depend on how often the numbers are repeated. What is the problem with this? (Slide 33) It is a good solution to a problem that we don’t have. It has two problems, our data is not going to be numbers, and even if the data was numerical it is unlikely to be uniformly distributed. Data found in the wild might have any probability distribution. So what can we do about that? PAUSE (Slide 34) We can use hashing to convert our data into a form that we can use. A hash function maps the given data into 32-bit integers that are uniformly distributed over the range 0 – 232. This converts our data into a shape that is easy to work with. So we’ve simplified the problem to be solved to a certain extent. PAUSE And this brings us to the point where we look at some history and introduce the other major contributor to our understanding of this problem space. (Slide 35) Philippe Flajolet was a computer scientist at INRIA, a French research institute. Robert Morris had presented his problem and solution in a short 3 page note in 1977, it wasn’t analysed completely till 1983, when Flajolet published his analysis. He gave it the name Approximate Counting. Around the same time, Flajolet came up with an algorithm called Probabilistic Counting that solves the Cardinality Estimation problem we’re considering right now. In 2003, he introduced the LogLog algorithm to solve the same problem, and in 2007, he introduced HyperLogLog a near-optimal solution to this problem. We’ll cover these solutions to the cardinality estimation problem from a high level. of 5 7
6. FINAL Roshan Mathews (Slide 36) Min is just one estimator that can be used. Flajolet and Martin instead used the maximum count of the number of leading zeros in the hash values of the data set. So if you look at this sample 16-bit hash-value, then the estimator value is 3. There are 3 leading zeros. What sort of estimator is that? We’ve already seen two estimators: the fair coin tossing estimator, and the log-estimator by Morris. As it turns out this will be like the log-estimator we saw earlier. (Slide 37) If I toss a fair coin a bunch of times, I expect half of them to be heads. Similarly, the first bit of a bunch of hash values can be 1 or 0. If 32-bit numbers are uniformly distributed, the we’d expect about half of them to start with the number zero, and half with one. Of the strings that start with zero, half will have zero as the second digit and half will have one as the second digit, and so on. (Slide 38) Putting that idea into a table, we expect to find binary values starting with these patterns, with these probabilities. PAUSE (Slide 39) For a data item, x .. the function ρ(x) returns the number of leading zeros in the hash of x. We scan the data, calculating this value and keeping track of the maximum ρ(x) seen, call this S. S is a good indicator of log n … and is used as our estimator. Given that estimator, we can calculate our estimate as 2S. That is we expect to find 2S uniques in our dataset. This is approximately correct, there is a bias in this estimator which is corrected, but that is a detail that we won’t be covering in our talk. PAUSE (Slide 40) Like the logarithmic estimator we used in approximate counting, this estimator too has an error that’s approximately one binary order of magnitude. In that case we had changed the base of the logarithm to reduce the error. We’re going to take another approach in this case. We will take m different hash functions and calculate m different values of S. and then average them. As it turns out, doing so reduces the error by a factor of 1/√m (Slide 41) The problem with this approach is that it is hard to find m good hash functions, and there is a CPU cost involved with calculating all these values for each data item. (Slide 42) Flajolet and Martin came up with a innovative idea, instead of using m different hashing functions, they divided the data into m buckets using the first l bits of data with m = 2l … the remaining L - l bits can be used now to estimate the value of n/m … and then we average those and scale up by m. This technique is called Stochastic Averaging. (Slide 43) Here’s an example of what I just described. Assume we have a 12-bit hash function and we get this value for some data point. If we take l = 3, then we have m = 23 = 8 buckets that we can put our data into, what we’re left with is 12 - 3 = 9 bits, and we can use these 9-bits with the technique described earlier to estimate n / 8, since we’ve removed 3 bits, our estimate will be off by 1/8, which we can then scale up. We scale up and we get the estimate of the number of uniques! (Slide 44) Ignoring the m-values that we average (stochastically or otherwise), what we’ve done is kept one single value that somehow estimates the log of the unique count, and use that to find our answer. of 6 7
7. FINAL Roshan Mathews There are two insights in this approach, one was to use “bit-patterns” – the number of leading zeros – to estimate the log of the cardinality. The other was to use Stochastic Averaging to reduce the error. PAUSE (Slide 45) This approach described just now, describes a family of algorithms. All of them solve the cardinality estimation problem. All of them are linear in time, and use constant space. The first of these is called Probabilistic Counting, and uses a fixed amount of storage of m words. About 20 years after PCSA, Flajolet introduced another algorithm. The error of this algorithm was a little worse than PCSA, but this used less than 1/4 the amount of space because it only needed to store a number. That number, our estimator, is a logarithmic estimator, which means that the total number of bits needed to be able to estimate values up to N, is log log N. That’s how the algorithm gets it’s name. In 2007, another improvement was made to the LogLog solution. That algorithm was called HyperLogLog and it used the same observable as the LogLog algorithm, but instead of using arithmetic means to average out the numbers it used Harmonic means. Harmonic means tend to the effect of the large outliers, and tends closer to the smaller numbers. HyperLogLog is nearly optimal. HyperLogLog is what you’ll use if you want to solve this problem today. HyperLogLog is the point of this talk. Everything was motivation just to say that name. (Slide 46) Here is an interesting image from the LogLog paper, it shows the estimator used to count the number of unique words in the complete works of Shakespeare. These 128 bytes, spread across 256 counters of 4 bits each, in some sense encode all the unique words in Shakespeare. The estimate that is derived from this is off by about 9% from the actual count. PAUSE (Slide 47) Now to bring this all to a close, let us look at what we have here. These are very simple algorithms. They are not obvious, and the analysis involved is very intricate. But they processes themselves are very simple. They are very easy to implement. The Approximate Counting counter could be implemented in 4 lines of code. The time and space complexity of all of them are pretty good. The three cardinality estimation algorithms we saw can be paralleled perfectly. If the data is spread across a bunch of nodes, then each of them can independently calculate the estimator. Then a master node can easily merge these values. The only data transferred between nodes are these “summaries”. So even with a 1000 node cluster we can combine the values and produce a global estimate in no time. (Slide 48) Some of the ideas worth taking away are: the notion of estimation and probabilistic approaches to estimation. These have the feeling of back of the envelope calculations. We used multiple estimates and averaged them to reduce errors. Another idea that was very interesting was of using hashing to convert the problem domain, that idea in one shot simplified our problem greatly. Hyperloglog, the main algorithm for cardinality estimation is present in database systems like Postgresql, Cassandra, Redis. (Slide 49) This talk was mostly derived from six papers, I can send copies to those interested. Most of them are quite readable if you skip over all the math, which I did. LogLog and HyperLogLog have lots of other detail that I didn’t cover, and they are worth looking into. (Slide 50) Thank you of 7 7