Cloudphysics SNIA presentation

Transcript

1. Wisdom from Crowds of Machines Analytics and Big Data Summit September 19, 2013 Chetan Conikee Irfan Ahmad Tuesday, October 1, 13
2. CloudPhysics Copyright 2013. All Rights Reserved. About Us CloudPhysics' mission is to discover the underlying principles that govern systems behavior and use this knowledge to transform computing. Tuesday, October 1, 13
3. CloudPhysics Copyright 2013. All Rights Reserved. About Us CloudPhysics' mission is to discover the underlying principles that govern systems behavior and use this knowledge to transform computing. Chetan Conikee Chief Data Officer • Founder, SeekVence • Data Scientist - Intuit • Founding Engineer - Business Signatures (Entrust), CashEdge (Fiserv), Smartpipes (Sophos) Irfan Ahmad Chief Technology Officer • VMware tech lead - DRS (Distributed Resource Scheduling) Team • Transmeta (microprocessors) Tuesday, October 1, 13
4. Operations Management SaaS Tuesday, October 1, 13
5. Modeling Rules Engines Card Card Card Card Card Apps Parsers Analyzers Parsers HTTPS CloudPhysics, Inc. Copyright 2013. All rights reserved. HTTPS High-resolution performance, config, fault, change, event, task data Perf Stat Datastore Inventory Datastore Event, Task Datastore Secure multi-tenant architecture Schema Normalization Web/App Servers Predictive Analytics Benchmarks Scale Out Query Engine www.CloudPhysics.com Collector vApp Scalable Cache Ingest Collective Intelligence Tuesday, October 1, 13
6. CloudPhysics Copyright 2013. All Rights Reserved. We collect • 100 billion+ data points /day • Avg 1.3 million properties/datacenter • Continuous time-series data stream • VMs, servers, networks, storage • Configuration, state, tasks, events Tuesday, October 1, 13
7. CloudPhysics Copyright 2013. All Rights Reserved. Our Software ToolSet Apache Kafka Tuesday, October 1, 13
8. CloudPhysics Copyright 2013. All Rights Reserved. Our dialect is Productive Fast Functional Statically typed Actors - an elegant way to handle concurrency ... and inter-op Java with Scala Tuesday, October 1, 13
9. CloudPhysics Copyright 2013. All Rights Reserved. Data Pipeline Criteria • Secure Data Retention • NO data-loss guarantee • Scale out consumption process on demand • Query near-to-real-time • Query historical data on demand (replay) Tuesday, October 1, 13
10. Circa 2012 • Small team - Lean and Minimum Viable Product • Launched Beta offering - license to “move fast and break things” • Techcrunch moment at VMworld 2012 • Exponential customer growth Tuesday, October 1, 13
11. How do we scale? courtesy: http://www.capricemotorinn.com/Dam1_.html Tuesday, October 1, 13
12. Data Collection/Processing Pipeline Generation-1 Object Storage AWS S3-Object Store HTTP Play! Akka Cluster HTTP Play! Akka Cluster CloudPhysics Observer(s) HTTP Cluster (Message Router) HTTP Cluster (Message Processer) Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage In-Memory Cache with TTL Tuesday, October 1, 13
13. 1 Data Collection/Processing Pipeline Generation-1 Object Storage AWS S3-Object Store HTTP Play! Akka Cluster HTTP Play! Akka Cluster CloudPhysics Observer(s) HTTP Cluster (Message Router) HTTP Cluster (Message Processer) Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage In-Memory Cache with TTL Tuesday, October 1, 13
14. 2 1 Data Collection/Processing Pipeline Generation-1 Object Storage AWS S3-Object Store HTTP Play! Akka Cluster HTTP Play! Akka Cluster CloudPhysics Observer(s) HTTP Cluster (Message Router) HTTP Cluster (Message Processer) Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage In-Memory Cache with TTL Tuesday, October 1, 13
15. 2 3 1 Data Collection/Processing Pipeline Generation-1 Object Storage AWS S3-Object Store HTTP Play! Akka Cluster HTTP Play! Akka Cluster CloudPhysics Observer(s) HTTP Cluster (Message Router) HTTP Cluster (Message Processer) Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage In-Memory Cache with TTL Tuesday, October 1, 13
16. 2 3 4 1 Data Collection/Processing Pipeline Generation-1 Object Storage AWS S3-Object Store HTTP Play! Akka Cluster HTTP Play! Akka Cluster CloudPhysics Observer(s) HTTP Cluster (Message Router) HTTP Cluster (Message Processer) Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage In-Memory Cache with TTL Tuesday, October 1, 13
17. 2 3 4 1 5 5 5 Data Collection/Processing Pipeline Generation-1 Object Storage AWS S3-Object Store HTTP Play! Akka Cluster HTTP Play! Akka Cluster CloudPhysics Observer(s) HTTP Cluster (Message Router) HTTP Cluster (Message Processer) Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage In-Memory Cache with TTL Tuesday, October 1, 13
18. 2 3 4 1 5 5 5 Data Collection/Processing Pipeline Generation-1 Object Storage AWS S3-Object Store HTTP Play! Akka Cluster HTTP Play! Akka Cluster CloudPhysics Observer(s) HTTP Cluster (Message Router) HTTP Cluster (Message Processer) Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage In-Memory Cache with TTL 6 Tuesday, October 1, 13
19. Tuesday, October 1, 13
20. Deconstructing - Play! backed Akka service Tuesday, October 1, 13
21. Deconstructing - Play! backed Akka service HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) Tuesday, October 1, 13
22. Deconstructing - Play! backed Akka service 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) Tuesday, October 1, 13
23. Deconstructing - Play! backed Akka service 2 2 Dequeue 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) Tuesday, October 1, 13
24. Deconstructing - Play! backed Akka service 2 2 Dequeue 3 3 Dispatch 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) Tuesday, October 1, 13
25. Deconstructing - Play! backed Akka service 2 2 Dequeue 3 3 Dispatch 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) 4 Process 4 Tuesday, October 1, 13
26. Deconstructing - Play! backed Akka service 2 2 Dequeue 3 3 Dispatch 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) 5 Complete 5 4 Process 4 Tuesday, October 1, 13
27. Deconstructing - Play! backed Akka service 2 2 Dequeue 3 3 Dispatch 6 6 Next (Repeat 1) 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) 5 Complete 5 4 Process 4 Tuesday, October 1, 13
28. Deconstructing - Play! backed Akka service 2 2 Dequeue 3 3 Dispatch 6 6 Next (Repeat 1) 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) 5 Complete 5 • Lightweight Objects with Universal Concurrency primitives 4 Process 4 Tuesday, October 1, 13
29. Deconstructing - Play! backed Akka service 2 2 Dequeue 3 3 Dispatch 6 6 Next (Repeat 1) 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) 5 Complete 5 • Lightweight Objects with Universal Concurrency primitives • Message Passing Concurrency & no shared state 4 Process 4 Tuesday, October 1, 13
30. Deconstructing - Play! backed Akka service 2 2 Dequeue 3 3 Dispatch 6 6 Next (Repeat 1) 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) 5 Complete 5 • Lightweight Objects with Universal Concurrency primitives • Message Passing Concurrency & no shared state • Supervisory Strategy to handle and recover from failures 4 Process 4 Tuesday, October 1, 13
31. Deconstructing - Play! backed Akka service 2 2 Dequeue 3 3 Dispatch 6 6 Next (Repeat 1) 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) 5 Complete 5 • Lightweight Objects with Universal Concurrency primitives • Message Passing Concurrency & no shared state • Supervisory Strategy to handle and recover from failures • Scale UP and Scale OUT on demand 4 Process 4 Tuesday, October 1, 13
32. Deconstructing - Play! backed Akka service 2 2 Dequeue 3 3 Dispatch 6 6 Next (Repeat 1) 1 1 Enqueue HTTP Request HTTP Cluster Router Mailbox Dispatcher ThreadPool - Actor(s) 5 Complete 5 • Lightweight Objects with Universal Concurrency primitives • Message Passing Concurrency & no shared state • Supervisory Strategy to handle and recover from failures • Scale UP and Scale OUT on demand • You get a PERFECT FABRIC to construct a SCALE-OUT SaaS/PaaS CLOUD Service 4 Process 4 Tuesday, October 1, 13
33. Issues we faced • Consumption falls below the rate of production • Increased backlog of streamed messages into the Mailbox can overload an Actor’s memory footprint • Negatively Impact responsiveness leading to application crashes Tuesday, October 1, 13
34. Let’s discuss design patterns https://twitter.com/derekwyatt Akka Concurrency Derek Wyatt A must read for serious Akka hAkkers https://twitter.com/jamie_allen Effective Akka Jamie Allen Tuesday, October 1, 13
35. Refactoring Exercise • Using Akka Persistence (http://doc.akka.io/docs/akka/snapshot/scala/persistence.html) • Use Akka master-worker pattern to distribute workload based on worker’s processing queue depth (deployed zookeeper for cluster co-ordination and service discovery) • Akka Cluster was early in development (http://doc.akka.io/docs/akka/snapshot/common/cluster.html#cluster) Tuesday, October 1, 13
36. Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
37. Decoupling Messaging from Processing - Why? Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
38. Decoupling Messaging from Processing - Why? • Redundancy - via replication Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
39. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
40. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability • Resiliency - in consumption Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
41. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability • Resiliency - in consumption • Asynchronous Communication Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
42. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability • Resiliency - in consumption • Asynchronous Communication • Replay Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
43. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability • Resiliency - in consumption • Asynchronous Communication • Replay Choices Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
44. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability • Resiliency - in consumption • Asynchronous Communication • Replay Choices • JMS Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
45. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability • Resiliency - in consumption • Asynchronous Communication • Replay Choices • JMS • AWS SQS Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
46. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability • Resiliency - in consumption • Asynchronous Communication • Replay Choices • JMS • AWS SQS • ZeroMQ Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
47. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability • Resiliency - in consumption • Asynchronous Communication • Replay Choices • JMS • AWS SQS • ZeroMQ • RabbitMQ Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
48. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability • Resiliency - in consumption • Asynchronous Communication • Replay Choices • JMS • AWS SQS • ZeroMQ • RabbitMQ • Flume Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
49. Decoupling Messaging from Processing - Why? • Redundancy - via replication • Scalability • Resiliency - in consumption • Asynchronous Communication • Replay Choices • JMS • AWS SQS • ZeroMQ • RabbitMQ • Flume • Kafka Producer Topic Consumer(s) Push Push Pull Tuesday, October 1, 13
50. Producer-0 Producer-1 Producer-2 Broker-0 Broker-1 Broker-2 Consumer-0 Consumer-1 Consumer-2 Zookeeper-0 Zookeeper-1 Zookeeper-2 Zookeeper-Ensemble Producers Brokers Consumers Push Push Pull Pull Tuesday, October 1, 13
51. Kafka - LinkedIn SNA Team 2011 Producer-0 Producer-1 Producer-2 Broker-0 Broker-1 Broker-2 Consumer-0 Consumer-1 Consumer-2 Zookeeper-0 Zookeeper-1 Zookeeper-2 Zookeeper-Ensemble Producers Brokers Consumers Push Push Pull Pull Tuesday, October 1, 13
52. Kafka - LinkedIn SNA Team 2011 Producer-0 Producer-1 Producer-2 Broker-0 Broker-1 Broker-2 Consumer-0 Consumer-1 Consumer-2 Zookeeper-0 Zookeeper-1 Zookeeper-2 Zookeeper-Ensemble Producers Brokers Consumers Push Push Pull Pull Tuesday, October 1, 13
53. Kafka - Core Concepts • Distributed, partitioned, replicated publish subscribe system • Messages are grouped into topics • Producer (pushes to) Broker (pulls from) Consumer • Messages are retained for a configured interval of time • A topic can be further divided into partitions • Each partition is replicated across configurable number of brokers for fault tolerance (> 0.8x version) • Consumers are not contending with each other during message consumption Tuesday, October 1, 13
54. • Producers decoupled from Consumers • Messages are delivered to a partition on a topic on a broker either • uses round-robin fashion by default • (or) using a partition function • Send messages at or below the configured size • Choose between “fire-and-forget” delivery or request broker for acknowledgement on receiving message by setting “request.require.ack” Kafka Core Concepts - Producers Tuesday, October 1, 13
55. Compression(Batch of Messages)- GZIP and Snappy Data is written to a sets of segment files on a broker (topic + partition) Kafka Core Concepts - Brokers Zero-copy transfer from files to socket - All delivery done through sendfile calls on mmap’d files - Avoid double buffering and GC overhead On-disk format of message: message-length : 4 bytes (value: 1 + 4 + n) “magic” value “ : 1 byte crc32 checksum : 4 bytes payload : n bytes Topic + Partition Replication at topic - partition level Tuesday, October 1, 13
56. • Can form a logical group (with defined “groupId”) to jointly consume from a single topic • Each partition is consumed by exactly one consumer in a consumer group • Register with zookeeper, in order to track their offsets in each partition for each broker that they consume from • Consumers must correctly handle the same message multiple times • Possibility of out of order delivery and redelivery require idempotent, commutative consumers when dealing with systems of record Kafka Core Concepts - Consumers Tuesday, October 1, 13
57. Tuesday, October 1, 13
58. Data Collection/Processing Pipeline Generation-2 HTTP Cluster (Producers) Object Storage Kafka - Broker T1 T2 T3 HTTP Cluster (Consumers) AWS S3-Object Store HTTP Play! Akka Cluster Auto Scaling Group CloudPhysics Observer(s) HTTP Play! Akka Cluster Auto Scaling Group Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage Tuesday, October 1, 13
59. 1 Data Collection/Processing Pipeline Generation-2 HTTP Cluster (Producers) Object Storage Kafka - Broker T1 T2 T3 HTTP Cluster (Consumers) AWS S3-Object Store HTTP Play! Akka Cluster Auto Scaling Group CloudPhysics Observer(s) HTTP Play! Akka Cluster Auto Scaling Group Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage Tuesday, October 1, 13
60. 2 1 Data Collection/Processing Pipeline Generation-2 HTTP Cluster (Producers) Object Storage Kafka - Broker T1 T2 T3 HTTP Cluster (Consumers) AWS S3-Object Store HTTP Play! Akka Cluster Auto Scaling Group CloudPhysics Observer(s) HTTP Play! Akka Cluster Auto Scaling Group Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage Tuesday, October 1, 13
61. 2 3 1 Data Collection/Processing Pipeline Generation-2 HTTP Cluster (Producers) Object Storage Kafka - Broker T1 T2 T3 HTTP Cluster (Consumers) AWS S3-Object Store HTTP Play! Akka Cluster Auto Scaling Group CloudPhysics Observer(s) HTTP Play! Akka Cluster Auto Scaling Group Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage Tuesday, October 1, 13
62. 2 3 1 4 Pull Pull Data Collection/Processing Pipeline Generation-2 HTTP Cluster (Producers) Object Storage Kafka - Broker T1 T2 T3 HTTP Cluster (Consumers) AWS S3-Object Store HTTP Play! Akka Cluster Auto Scaling Group CloudPhysics Observer(s) HTTP Play! Akka Cluster Auto Scaling Group Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage Tuesday, October 1, 13
63. 2 3 5 1 4 Pull Pull Data Collection/Processing Pipeline Generation-2 HTTP Cluster (Producers) Object Storage Kafka - Broker T1 T2 T3 HTTP Cluster (Consumers) AWS S3-Object Store HTTP Play! Akka Cluster Auto Scaling Group CloudPhysics Observer(s) HTTP Play! Akka Cluster Auto Scaling Group Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage Tuesday, October 1, 13
64. 2 3 5 1 6 6 6 4 Pull Pull Data Collection/Processing Pipeline Generation-2 HTTP Cluster (Producers) Object Storage Kafka - Broker T1 T2 T3 HTTP Cluster (Consumers) AWS S3-Object Store HTTP Play! Akka Cluster Auto Scaling Group CloudPhysics Observer(s) HTTP Play! Akka Cluster Auto Scaling Group Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage Tuesday, October 1, 13
65. 2 3 5 1 6 6 6 4 Pull Pull Data Collection/Processing Pipeline Generation-2 HTTP Cluster (Producers) Object Storage Kafka - Broker T1 T2 T3 HTTP Cluster (Consumers) AWS S3-Object Store HTTP Play! Akka Cluster Auto Scaling Group CloudPhysics Observer(s) HTTP Play! Akka Cluster Auto Scaling Group Event Metrics Store Configuration Store Performance Metrics Store Time Series DB Cluster Sharded JSON Store Time Series DB Cluster Storage 4 Pull HDFS Spark Cluster Compute Collective Intelligence Kafka HDFS Consumer ALPHA Storage Tuesday, October 1, 13
66. Kafka Tuning Considerations • Advantages • Increases I/O parallelism for writes • Increases degree of parallelism for consumers • Tune memory of consumer based on (number of threads in consumer group) * (queuedchunks.max) * (fetch.size) • Increase Heap size • Reduce consumer thread count • Lower fetch size or max queue size • Overheads • More open file handles • More offsets to be check-pointed increasing load on zookeeper ensemble Increased Partition Configuration (for high volume topics) Tuesday, October 1, 13
67. Data Analysis Tuesday, October 1, 13
68. Tuesday, October 1, 13
69. Event Metrics Store Configuration Store Performance Metrics Store Query Engine Play/Akka based Web Application HTTP Cluster (Producers) Spray / Akka Cluster Time Series DB Cluster Sharded JSON Store ‘ENSO’ Data Analysis Pipeline Time Series DB Cluster Storage Authorization / Access Control Meta Data Time Machine Query Planner Aggregation Off-Heap Cache A L E R T S Tuesday, October 1, 13
70. 1 Event Metrics Store Configuration Store Performance Metrics Store Query Engine Play/Akka based Web Application HTTP Cluster (Producers) Spray / Akka Cluster Time Series DB Cluster Sharded JSON Store ‘ENSO’ Data Analysis Pipeline Time Series DB Cluster Storage Authorization / Access Control Meta Data Time Machine Query Planner Aggregation Off-Heap Cache A L E R T S Tuesday, October 1, 13
71. 1 2 Event Metrics Store Configuration Store Performance Metrics Store Query Engine Play/Akka based Web Application HTTP Cluster (Producers) Spray / Akka Cluster Time Series DB Cluster Sharded JSON Store ‘ENSO’ Data Analysis Pipeline Time Series DB Cluster Storage Authorization / Access Control Meta Data Time Machine Query Planner Aggregation Off-Heap Cache A L E R T S Tuesday, October 1, 13
72. 1 2 3 Event Metrics Store Configuration Store Performance Metrics Store Query Engine Play/Akka based Web Application HTTP Cluster (Producers) Spray / Akka Cluster Time Series DB Cluster Sharded JSON Store ‘ENSO’ Data Analysis Pipeline Time Series DB Cluster Storage Authorization / Access Control Meta Data Time Machine Query Planner Aggregation Off-Heap Cache A L E R T S Tuesday, October 1, 13
73. 1 2 3 4 Event Metrics Store Configuration Store Performance Metrics Store Query Engine Play/Akka based Web Application HTTP Cluster (Producers) Spray / Akka Cluster Time Series DB Cluster Sharded JSON Store ‘ENSO’ Data Analysis Pipeline Time Series DB Cluster Storage Authorization / Access Control Meta Data Time Machine Query Planner Aggregation Off-Heap Cache A L E R T S Tuesday, October 1, 13
74. 1 2 3 4 5 Event Metrics Store Configuration Store Performance Metrics Store Query Engine Play/Akka based Web Application HTTP Cluster (Producers) Spray / Akka Cluster Time Series DB Cluster Sharded JSON Store ‘ENSO’ Data Analysis Pipeline Time Series DB Cluster Storage Authorization / Access Control Meta Data Time Machine Query Planner Aggregation Off-Heap Cache A L E R T S Tuesday, October 1, 13
75. 1 2 3 4 5 6 6 Event Metrics Store Configuration Store Performance Metrics Store Query Engine Play/Akka based Web Application HTTP Cluster (Producers) Spray / Akka Cluster Time Series DB Cluster Sharded JSON Store ‘ENSO’ Data Analysis Pipeline Time Series DB Cluster Storage Authorization / Access Control Meta Data Time Machine Query Planner Aggregation Off-Heap Cache A L E R T S Tuesday, October 1, 13
76. 1 2 3 4 7 8 5 6 6 Event Metrics Store Configuration Store Performance Metrics Store Query Engine Play/Akka based Web Application HTTP Cluster (Producers) Spray / Akka Cluster Time Series DB Cluster Sharded JSON Store ‘ENSO’ Data Analysis Pipeline Time Series DB Cluster Storage Authorization / Access Control Meta Data Time Machine Query Planner Aggregation Off-Heap Cache A L E R T S Tuesday, October 1, 13
77. 1 2 3 4 7 8 5 6 6 Event Metrics Store Configuration Store Performance Metrics Store Query Engine Play/Akka based Web Application HTTP Cluster (Producers) Spray / Akka Cluster Time Series DB Cluster Sharded JSON Store ‘ENSO’ Data Analysis Pipeline Time Series DB Cluster Storage Authorization / Access Control Meta Data Time Machine Query Planner Aggregation Off-Heap Cache A L E R T S Push Notifications Tuesday, October 1, 13
78. Query Engine (Enso) Query Engine • Authored a Data Virtualization Layer • An type-safe Scala API/DSL library (domain specific language) that auto-generates MongoDB and Cassandra query operators • Allows querying complex deeply nested JSON streams in near real time • Supports grouping, aggregation and filtering • Supports on-demand querying of historical data • Supports resource-management query priorities Tuesday, October 1, 13
79. • The Dojo Cluster is a part of an auto scaling group • Compute nodes are added/deleted based on throughput characteristics - e.g. lag in consumption from Kafka brokers. • We are currently leveraging AnsibleWorks IT Orchestration Engine IT/DevOps Orchestration Tuesday, October 1, 13
80. We’re hiring Infrastructure Data Science DevOps Email : info@cloudphysics.com twitter: @cloudphysics Tuesday, October 1, 13