MicroStrategy World 2014: Scaling MicroStrategy at eBay

eBay has one of the largest data warehouses in the world! See how the BI Platform team at eBay had to rethink and rebuild their system architecture and processes in order to support the ever-growing data volume and scalability needs of their developers and users.

Transcript

1. HOW BIG IS BIG: SCALING MICROSTRATEGY AT EBAY Business Intelligence Architect Tim Case
2. TODAY’S TOPICS • eBay Overview • Challenges & Solutions • Plans for 2014 • Extending MicroStrategy SCALING MICROSTRATEGY AT EBAY 2
3. 128M Active users 7 out of 10 Items sold are new 7 out of 10 Items sold are fixed price EBAY MARKETPLACES
4. A new retail is emerging. It’s consumer driven and technology enabled.
5. THE PHONE IS BECOMING A MISSION CONTROL DEVICE
6. People want to get what they want, when they want it, wherever they are.
7. 2012 Mobile GMV $13B $22B in 2013 Countries 190 Application downloads 190M EBAY LEADS IN MOBILE
8. every 10 min. a car or truck is bought via mobile every 1 min. a tablet is bought via mobile every 15 sec. a ladies handbag is bought via mobile EBAY MOBILE VOLUME IS STAGGERING
9. We connect people to the things they need and love.
10. DATA STATISTICS SCALING MICROSTRATEGY AT EBAY 10
11. DATA PLATFORMS SCALING MICROSTRATEGY AT EBAY 11
12. 52N Co-existent 5300/5350/5380 (30 TB /50 TB Max) ‘99 ’01 ‘11 ’03 ‘05 ‘07 ‘09 Access DB & MS Excel Reports (3 MB) Informatica Oracle DW & Business Objects (500 GB) Campaign Mgmt DMs: FADE, RFM, Power Sellers, VCRU… (4 TB) Teradata 12N 5300 (8 TB) Dual System Primary: 64N 5400 Secondary: 60N 5380 Auto Gen ETL SOA Virtual Data Marts (VDM) Tactical Workload Member Insight ABC (RAM) Singularity Connected Commerce Cloud (Internal) Hadoop and NoSQL MicroStrategy SAS MAX Portal Ab Initio Tableau ANALYTICS EVOLUTION SCALING MICROSTRATEGY AT EBAY 12
13. MICROSTRATEGY BY THE NUMBERS 64 Projects 5,200 Users 440,000 Sessions 15,000,000 Jobs 70,000 Reports 240 Cubes SCALING MICROSTRATEGY AT EBAY 13
14. MICROSTRATEGY ARCHITECTURE SCALING MICROSTRATEGY AT EBAY 14
15. CHALLENGES SCALING MICROSTRATEGY AT EBAY 15
16. INTELLIGENT CUBE GROWTH SCALING MICROSTRATEGY AT EBAY 16 PROBLEM • No available memory (RAM) for additional Intelligent Cubes
17. INTELLIGENT CUBE GROWTH SCALING MICROSTRATEGY AT EBAY 17 256 1024 2048 2011 2012 2013 RAM (in GB) PROBLEM • No available memory (RAM) for additional Intelligent Cubes SOLUTION • Increased the RAM to physical limit of servers (2TB)
18. INTELLIGENT CUBE PROCESSING PROBLEM • Intelligent cube processing taking too long and missing SLAs SCALING MICROSTRATEGY AT EBAY 18
19. INTELLIGENT CUBE PROCESSING PROBLEM • Intelligent cube processing taking too long and missing SLAs SOLUTION • Upgraded to a 10Gbps connection to backend network • Added additional server to cluster • Balanced jobs across nodes Cube 05 Cube 04 Cube 03 Cube 02 Cube 01 Cube Processing Times (in seconds) Before After SCALING MICROSTRATEGY AT EBAY 19
20. PROBLEM • Moving servers between environments caused issues URL REWRITE SCALING MICROSTRATEGY AT EBAY 20
21. PROBLEM • Moving servers between environments caused issues SOULTION • URL Rewrite rules URL REWRITE SCALING MICROSTRATEGY AT EBAY 21       <rule  name="test"  stopProcessing="true">      <match  url="^asp/Main.aspx”  />      <conditions>          <add  input=“{QUERY_STRING}”  type=“Pattern”                  pattern=“(.*)server=[^&]+(.*)”      </conditions>      <action  type="Redirect"  url="bix/asp/Main.aspx?              {C:1}server=ISERVER01{C:2}"  />   </rule>  
22. PROBLEM • Users need to be able to gauge data freshness at a glance CUBE MONITOR SCALING MICROSTRATEGY AT EBAY 22
23. PROBLEM • Users need to be able to gauge data freshness at a glance SOULTION • Custom Cube Monitor application CUBE MONITOR SCALING MICROSTRATEGY AT EBAY 23
24. PROBLEM • No way to measure end-to-end user experience PAGE LOAD TIMES SCALING MICROSTRATEGY AT EBAY 24
25. PROBLEM • No way to measure end-to-end user experience SOULTION • Plugin to measure client-side performance   PAGE LOAD TIMES SCALING MICROSTRATEGY AT EBAY 25
26. PROBLEM • No way to measure end-to-end user experience SOULTION • Plugin to measure client-side performance • Email Alerts PAGE LOAD TIMES SCALING MICROSTRATEGY AT EBAY 26
27. PROBLEM • Need a low friction way to log anything MEASURE EVERYTHING SCALING MICROSTRATEGY AT EBAY 27
28. PROBLEM • Need a low friction way to log anything SOULTION • HTTP Log Service MEASURE EVERYTHING SCALING MICROSTRATEGY AT EBAY 28 http://api.bix.corp.ebay.com/LogService/ Log?app=TEST &function=TestFunction &user=tcase &executiontime=9999 &error=FALSE &note=This%20is%20a%20sample
29. PROBLEM • Difficult to monitor multiple applications • No single point-of-view   CROSS-PLATFORM INSIGHT SCALING MICROSTRATEGY AT EBAY 29
30. PROBLEM • Difficult to monitor multiple applications • No single point-of-view SOULTION   CROSS-PLATFORM INSIGHT SCALING MICROSTRATEGY AT EBAY 30
31. SPLUNK SCALING MICROSTRATEGY AT EBAY 31
32. SPLUNK SCALING MICROSTRATEGY AT EBAY 32
33. PLANNED IN 2014 SCALING MICROSTRATEGY AT EBAY 33
34. PROBLEM • Managing Windows and Linux servers increases complexity MIXED ENVIRONMENT SCALING MICROSTRATEGY AT EBAY 34
35. PROBLEM • Managing Windows and Linux servers increases complexity SOULTION • Standardize on Linux   MIXED ENVIRONMENT SCALING MICROSTRATEGY AT EBAY 35
36. PROBLEM • Shared storage is a single point of failure SHARED STORAGE SCALING MICROSTRATEGY AT EBAY 36
37. PROBLEM • Shared storage is a single point of failure SOULTION • Flash-based “storage bubble”   SHARED STORAGE SCALING MICROSTRATEGY AT EBAY 37
38. PROBLEM • Widespread impact during planned and unplanned downtime PLATFORM STABILITY SCALING MICROSTRATEGY AT EBAY 38
39. PROBLEM • Widespread impact during planned and unplanned downtime SOULTION • Sharded architecture – Physical servers – Linux cgroups   – Internal cloud (C3) PLATFORM STABILITY SCALING MICROSTRATEGY AT EBAY 39
40. PROBLEM • High memory requirements • Duplicated data • Scalability issues ANALYTIC DATA STORE SCALING MICROSTRATEGY AT EBAY 40
41. PROBLEM • High memory requirements • Duplicated data • Scalability issues SOULTION • Analytic data layer ANALYTIC DATA STORE SCALING MICROSTRATEGY AT EBAY 41
42. EXTENDING MICROSTRATEGY SCALING MICROSTRATEGY AT EBAY 42
43. METRICS EXPLORER • Integrated with internal DataHub • Leverages MicroStrategy Visual Insight • Access to Intelligent Cubes for quick analysis • Warehouse reports also available for deeper analysis • Can save and share results SCALING MICROSTRATEGY AT EBAY 43
44. METRICS EXPLORER SCALING MICROSTRATEGY AT EBAY 44
45. METRICS EXPLORER SCALING MICROSTRATEGY AT EBAY 45
46. METRICS EXPLORER SCALING MICROSTRATEGY AT EBAY 46
47. METRICS EXPLORER SCALING MICROSTRATEGY AT EBAY 47
48. ELEMENT CACHING SERVICE • Improve prompt performance – Critical for Metrics Explorer • Daily and weekly schedules • Developer self-service SCALING MICROSTRATEGY AT EBAY 48
49. ELEMENT CACHING SERVICE SCALING MICROSTRATEGY AT EBAY 49
50. ELEMENT CACHING SERVICE SCALING MICROSTRATEGY AT EBAY 50
51. ELEMENT CACHING SERVICE SCALING MICROSTRATEGY AT EBAY 51
52. ELEMENT CACHING SERVICE SCALING MICROSTRATEGY AT EBAY 52
53. WEB BASED COMMAND MANAGER (WBCM) • No need to install Command Manager • Ability to trigger events via REST service • Simple workflows supported • Simplified user management • Empowers developers SCALING MICROSTRATEGY AT EBAY 53
54. WEB BASED COMMAND MANAGER SCALING MICROSTRATEGY AT EBAY 54
55. WEB BASED COMMAND MANAGER SCALING MICROSTRATEGY AT EBAY 55
56. WEB BASED COMMAND MANAGER SCALING MICROSTRATEGY AT EBAY 56
57. WEB BASED COMMAND MANAGER SCALING MICROSTRATEGY AT EBAY 57
58. SELF SERVICE MIGRATION TOOL (SSMT) • Developer-focused tool for object management • Enables software development lifecycle (SDLC) • Built-in automated testing • Improved visibility into object migration • Leverages System Manager SCALING MICROSTRATEGY AT EBAY 58
59. SELF-SERVICE MIGRATION TOOL SCALING MICROSTRATEGY AT EBAY 59 1.  Get Object Manager access 2.  Generate package using Object Manager 3.  Upload package 4.  Import package o  Update Schema o  Purge Object Cache o  Purge Element Cache o  Email Notifications
60. OData • Industry-standard Open Data Protocol • Get data in XML or JSON format • Consume MicroStrategy data from your tool of choice: – Excel – Tableau – Visualization libraries (D3.js, Highcharts) SCALING MICROSTRATEGY AT EBAY 60
61. OData http://localhost:8080/owind.svc/Categories/Description   http://localhost:8080/owind.svc/Categories/$count   http://localhost:8080/owind.svc/Categories?$orderby=CategoryName  desc&$top=4   http://localhost:8080/owind.svc/Categories?$filter=CategoryID  eq  8   SCALING MICROSTRATEGY AT EBAY 61
62. WRAP UP • Challenges & Solutions • Plans for 2014 • Extending MicroStrategy SCALING MICROSTRATEGY AT EBAY 62
63. QUESTIONS? tcase@ebay.com @timrcase http://www.linkedin.com/in/timrcase/