Big Data For Better Datacenters

Despite all the recent advancements in the operations management field, data center management today still largely remains as a black art. Administrators have limited visibility into their data center operations today and yet they have to make important operations management decisions every day. A typical data center generates about a Billion data points every day. A lot of insight could be gathered from this data but due to the large volume and scale, on-premise software solutions only collect limited subset of this data. This limits them to a very narrow view of the data center. We at CloudPhysics have taken a different approach to this problem. We created an analytics platform in the cloud, that provides the ability to query, slice and dice and mashup the data with multiple data-sources. This approach not only yields incredible insights but also solves many of the teething operational management issues that have not been solved before. In this talk we give an overview of the data center metadata and provide details on how CloudPhysics handles this data at scale using its platform.

Transcript

1. Big  Data  for  Better  Datacenters ! Balaji  Parimi     @vimAPIGuru   Krishna  Raj  Raja     @esxtopGuru   ! Strata  2014,  Santa  Clara
2. World  Today
3. The  Irony
4. Analytics
5. Networking  becomes   mainstream   ! No  effective  means  to   pool  storage  and   compute  capacity   ! 100s  of  silos  to  manage 1990s Application Network Application OS OS Compute Compute Storage Storage
6. Virtualization  becomes   mainstream   ! Improved  resource   pooling   ! Increased  utilization   2000s Storage Network VM Hypervisor Compute VM VM VM VM Hypervisor Compute VM VM VM
7. 2010s Cloud  has  become   mainstream   ! Centralized  management VM Storage Network VM Software  Defined  Datacenter Compute VM VM VM VM Compute VM VM 70%  of  all  the  x86  OS  instances  run     in  virtual  machines,  80%  by  2016     ! -­‐  Gartner  (2012)
8. Data  Centers  Have  Evolved  But   IT  Operations  Has  Not
9. IT  Operations  Today Retrofitted
10. IT  Operations  Today Tedious
11. IT  Operations  Today Paralyzing
12. IT  Operations  Today Lot  of  Guesswork!
13. IT  Operations  Should  Be  Data-­‐Driven,     “Intuition  and  Guesswork”   Should  be  Replaced  By
 “Analytics  and  Predictive  Intelligence”

14. Datacenter  Efficiency Statistically Significant Machine Metadata Available for Data Scientists
15. Half  Million   Private  Clouds Can  Private  Clouds  Match  The  Efficiency? Data  Scientists Machine     Metadata +
16. Operational  Metadata  Per  Day 500  Virtual  Machines                    50  Servers Private  Cloud 1  Billion  Data  Points 25GB
17. 60 million tweets 650  Million  Users 2.5 billion pieces of content 1  Billion  Users Quadrillion datapoints (1015) 500,000 Datacenters 50  Million  VMs Massive  Scale
18. Challenges VM                    Server                    Server                    Server                    Server VM VM VM                                              Data  Center VM VM VM VM VM VM VM VM                            Management  Server                                                                                Control  Plane   Management     Driven  by   Collective  Intelligence (Collect) (Predict/Alert)
19. ORG MGMT SERVER CLUSTER SERVER DISK STORE STORAGE! CLUSTER VDISK VM VNIC VSWITCH VPORTGROUP One to Many Data  Model
20. CONFIGURATION PERFORMANCE TASK & EVENTS EXTERNAL DATA SOURCES Types  of  Data  Streams
21. On-­‐Demand     Analytics  Engine Prediction  and   Simulation  Engine Pre-­‐Alerts   and  Notification Distributed  Datastore            Complex   Stream  Processing              Collector Data  Analyst   Administrator Partner Data  Stream
22. Extract  /  Transform Persist Model Stream  Mashup Prediction Alerting Job  Scheduling Resource  Management Enabling  Private  Clouds Collective  Intelligence
23. Use  Cases
24. Predicting  Out  of  Disk  Space  Event Datastore  Capacity 0% 25% 50% 75% 100% April May June July August Datastore  1 Datastore  2 Datastore  3 Disk  Space  Usage  is  Hard  to  Predict
25. Predicting  Out  of  Disk  Space Probability 0% 25% 50% 75% 100% <  1  Month 6  Months 1  Year 100%  Full 90%  Full 80%  Full Probability  Distribution  Using  Monte  Carlo  Simulation
26. Discovering  Causality 1. Virtual  Machine  18   2. Virtual  Machine  23   3. Virtual  Machine  108
27. Predicting  Potential  Outage
28. Predicting  SSD  Performance  Impact Latency  Reducjon 0% 25% 50% 75% 100% Solid  State  Drive  Cache  Per  VM 16  GB 32  GB 64  GB 128  GB VM  1  VM  2  VM  3 Target  Latency 62  GB 24  GB
29. Predicting  Configuration  Outliers
30. Predicting  Performance  Outliers CPU  Co-­‐efficient 0 22.5 45 67.5 90 Virtual  Machine  # 0 3 6 9 12 Organizaion  A Organizajon  B Organizajon  C
31. Some  Closing  Thoughts
32. Towards  a  New  World
33. Thank  You ! Balaji  Parimi     @vimAPIGuru   Krishna  Raj  Raja     @esxtopGuru