Web Content Analytics at Scale with Parse.ly

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
February 17, 2016

Web Content Analytics at Scale with Parse.ly

Using Elasticsearch, Parse.ly wrote a time series backend for its real-time content analytics product. This talk will cover time-based indices, hot/warm/cold tiers, doc values, index aliases/versioning, and other techniques to run a multi-terabyte Elasticsearch cluster to perform time series at scale.

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

February 17, 2016
Tweet

Transcript

  1. ‹#› Andrew Montalenti! CTO, Parse.ly @amontalenti Web Content Analytics at

    Scale with Parse.ly
  2. content analytics fuels real-time decision-making for people who run the

    largest sites
  3. None
  4. None
  5. None
  6. None
  7. None
  8. None
  9. None
  10. None
  11. 175 TB of compressed customer data.! Growing 20 TB+ per

    month.
  12. before MongoDB, before Cassandra, before “NoSQL”, there was Lucene.

  13. 2013! "Can one use Solr as a Time Series Engine?"

  14. 2014! "Are Elasticsearch aggregations a dream query layer?"

  15. 2015! "Can we push ES to take 10K+ writes/sec and

    store 10TB+ of customer data?"
  16. (turns out, answer to each question is "yes", but lots

    of caveats!)
  17. fielddata: don't do it!! ! doc_values All The Things!

  18. _source: don't do it!! ! especially if your schema has

    high-cardinality multi-value fields.
  19. "logstash-style" raw records! are nice, but...! ! to operate with

    good query latency, you need rollups, and these are tricky.
  20. { "url": "http://arstechnica.com/12345", "ts": "2015-01-02T00:00:000Z",! "visitors": ["3f3f", "3f3g", ...millions],! !

    "metrics": { "$all/page_views": 6200000, "desktop/page_views": 4200000,! "mobile/page_views": 2000000,! "$all/engaged_secs": 27500000,! "new/engaged_secs": 250000000,! "returning/engaged_secs": 25000000, }, ! "metas": { "title": "Obama gives speech",! "authors": ["Mike Journo"],! "section": "Politics",! "pub_date": "2015-01-02T08:00:000Z", } } partition and time bucket high-cardinality metric numeric metrics metadata 1day rollup! (1 per day)
  21. { "url": "http://arstechnica.com/12345", "ts": "2015-01-02T08:05:000Z",! "visitors": ["3f3f", "3f3g", ...hundreds],! !

    "metrics": { "$all/page_views": 62, "desktop/page_views": 42,! "mobile/page_views": 20,! "$all/engaged_secs": 275,! "new/engaged_secs": 250,! "returning/engaged_secs": 25, }, ! "metas": { "title": "Obama gives speech",! "authors": ["Mike Journo"],! "section": "Politics",! "pub_date": "2015-01-02T08:00:000Z", } } partition and time bucket high-cardinality metric numeric metrics metadata 5min rollup! (288 per day)
  22. { "url": "http://arstechnica.com/12345", "ts": "2015-01-02T08:05:123Z",! "visitors": ["3f3f3"],! ! "metrics": {

    "$all/page_views": 1, "desktop/page_views": 1,! "mobile/page_views": 0,! "$all/engaged_secs": 0,! "new/engaged_secs": 0,! "returning/engaged_secs": 0, }, ! "metas": { "title": "Obama gives speech",! "authors": ["Mike Journo"],! "section": "Politics",! "pub_date": "2015-01-02T08:00:000Z", } } partition and time bucket high-cardinality metric numeric metrics metadata raw event! (millions per day)
  23. url-raw by hour url-5min by day url-1day by month document

    grouping! in time-based indices
  24. top_things(...)! thing_details(...)! site_timeline(...) initial data access layer

  25. historical analytics for spotting long-term opportunities, trends, and insights in

    heaps of data
  26. Parse.ly "Batch Layer" Topologies with Spark and Amazon S3 Parse.ly

    "Speed Layer" Topologies with Storm & Kafka Parse.ly Dashboards and APIs with Elasticsearch & Cassandra Parse.ly Raw Data Pipeline with Amazon Kinesis & S3 Access mage building blocks
  27. rebuild the world!

  28. None
  29. Mid-2015! "This sort of works, but seems that we need

    more hardware... and what's up with response times?"
  30. "You need to give big ! customers their own indices."

    - Otis "You need to use node-shard! allocation for hot/cold tiers." - Radu
  31. Time-based indices Index versioning Customer namespaces Node-shard allocation all together

    now!
  32. Time-Based • v1_shared-1day-2015.01 • v1_shared-1day-2015.02

  33. Versioning • v1_shared-1day-2015.01 • v2_shared-1day-2015.01

  34. Namespaces • v1_shared-1day-2015.01 • v1_condenast-1day-2015.01

  35. Node-Shard Allocation • v1_shared-1day-2015.01 => cold (mem, rust)! • v1_shared-5min-2015.02.01

    => warm (mem, ssd)! • v1_shared-5min-2015.03.15 => hot (mem, cpu)! • v1_shared-raw-2015.03.15T12 => raw (cpu)
  36. • Cluster: 40 nodes, 500+ indices, 7,000+ shards • Tiers:

    4 client, 3 master, 9 raw, 9 hot, 12 warm, 3 cold • Instances: 1TB+ of RAM, 500+ CPU cores • Disks: 12+ TB data, >50% in SSDs, rest in rust • Writes: 10K+ writes per second • Reads: 100's of aggregations per second
  37. None
  38. None
  39. Late-2015! "This is shipped! It works! ... but, some issues

    remain."
  40. OOMs = bugs timeouts = lies queries = hogs bugs,

    lies, and hogs
  41. In the worst case, a bad query takes longer than

    its timeout, hogs the cluster, and hits an OOM bug.
  42. better resiliency store compression task management aggregation paging query profiling

    excited about future
  43. Questions? Tweet to @amontalenti!

  44. links • Lucene: The Good Parts! • Mage: The Magical

    Time Series Backend! • Pythonic Analytics with Elasticsearch! • Visit us: http://parse.ly • Join us: http://parse.ly/jobs
  45. appendix

  46. building mage streaming time series engine for our next 1,000

    customers
  47. None
  48. None
  49. None
  50. None
  51. pykafka ingest raw event data at high speed

  52. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 consumer = ... # balanced while True: msg = consumer.consume() msg = json.loads(msg) urlparse(msg["url"]) Python State Code Python State Code Python State Code Python State Code Python State Code pykafka.producer Python State Code
  53. None
  54. scale-out functions over a stream of inputs in order to

    generate a stream of outputs
  55. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 Python State Code Python State Code Python State Code Python State Code pykafka.producer Python State Code multi-lang json protocol class UrlParser(Topology): url_spout = UrlSpout.spec(p=1) url_bolt = UrlBolt.spec(p=4, input=url_spout)
  56. None
  57. pyspark scale-out batch functions over static dataset to perform transformations

    and actions
  58. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 Python State Code Python State Code Python State Code Python State Code pyspark.SparkContext sc = SparkContext() file_rdd = sc.textFile(files) file_rdd.map(urlparse).take(1) cloudpickle py4j and binary pipes
  59. None
  60. lesson learned log-oriented "lambda architecture" works well, but it costs

    time and money!
  61. multi-process, not multi-thread multi-node, not multi-core message passing, not shared

    memory ! heaps of data and streams of data