Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Big Data, Big Deal - Daniel Miller, Splunk

TSWG Innovators 2013
March 07, 2013
540

Big Data, Big Deal - Daniel Miller, Splunk

Daniel Miller is the Australia and New Zealand Country Manager of Splunk. 9th Annual Innovators Conference -presentation day 1: March 7th @ the Palazzo Versace, Gold Coast, Australia

TSWG Innovators 2013

March 07, 2013
Tweet

More Decks by TSWG Innovators 2013

Transcript

  1. Safe Harbor Statement 2 During the course of this presentation,

    we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
  2. Company Overview 3 Company (NASDAQ: SPLK) Founded 2004, first software

    release in 2006 HQ: San Francisco / Region HQ: London, Hong Kong Over 750 employees, based in 12 countries FY2012 $120 million; +83% year-over-year Business Model / Products Free download to massive scale On-premise, in the cloud and SaaS 4,800+ Customers Customers in over 85 countries 54 of the Fortune 100 Largest license: 100 Terabytes per day
  3. Industry Recognition Splunk named one of the World’s most innovative

    companies Ranked #1 Big Data Innovator Ranked #4 Most Innovative Company 4 Fast Company's Most Innovative Companies Issue (March 2013)
  4. What is ‘big data’? The 3 V’s……. Velocity The data

    is being produced at a rate that is beyond the performance limits of traditional systems Volume The volume of data is too large for traditional database software tools to cope with Variety The data lacks the structure to make it suitable for storage and analysis in traditional databases and data warehouses – “Big data” describes the realization of greater business intelligence by storing, processing and analyzing data that was previously ignored due to the limitations of traditional data management technologies to handle its volume, velocity and/or variety.
  5. What is ‘big data’? The 3 I’s….. Immediate The data

    is being produced at a rate that is beyond the performance limits of traditional systems Intimidating The volume of data is too large for traditional database software tools to cope with Ill-defined The data lacks the structure to make it suitable for storage and analysis in traditional databases and data warehouses – “Big data” describes the realization of greater business intelligence by storing, processing and analyzing data that was previously ignored due to the limitations of traditional data management technologies to handle its volume, velocity and/or variety.
  6. What is ‘big data’? Simply… “data that exceeds the processing

    capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.” Ed Dumbill, O’Reilly
  7. The World of Business Analytics is Changing 8 “Feeding transactional

    data into a traditional data warehouse no longer represents the extent of capabilities necessary for BI.” “….require new information management capabilities to integrate information from disparate, external and unstructured information sources.” “The simple idea of building a traditional data warehouse to support a BI platform is no longer sufficient.” Source: Business Analytics Require New Information Management Capabilities, Nov, 2011.
  8. My most advanced Hadoop clients are also getting disillusioned …

    The only consistent success, reported by my clients, is with Splunk. Svetlana Sicular Gartner Research Director January 22, 2013 “ “ 9
  9. End User Demands Driving Shift 10 Sales Products Marketing Support

    How do we use mobile and geo location data to improve content mix for new mobile services? How do we get better visibility into customer interactions with online service in real time? How do we get real-time insights into purchases online and from new devices? How do we drive product innovation with insight into how customers use our products?
  10. Big Data Comes from Machines Volume | Velocity | Variety

    | Variability GPS, RFID, Hypervisor, Web Servers, Email, Messaging Clickstreams, Mobile, Telephony, IVR, Databases, Sensors, Telematics, Storage, Servers, Security Devices, Desktops Machine-generated data is one of the fastest growing, most complex and most valuable segments of big data 11
  11. Machine Data (from customer interaction) Product Information Geo location Data

    Customer interacts with service online or from any device Real-Time Business Insights from Machine Data Example: Business Visibility From Machine Data 66.57.19.112 ..[05/Dec/2011 07:05:22:152]”GET /card.do?action=addtocart&itemid=EST-17& product_id=K9-BD- 01&JSESSIONID.SD7SLSFF8ADFF8HTTP 1.1” 200 3923 AppleWebKit/535.2 (KHTML.like Gecko) Chrome/15.0.874.121 Safari535.2 Product Action User session User browser information Product_id=K9-BD-01 Product Name=2 TB Portable Drive Manufacturer=iomega Geo location data Correlated with product information from database Location data based on where the customer purchased / interacted with service – What products are popular in what region? – Which product are customers leaving in cart? – What are interaction paths by devices? – How can we improve customer experience? 12
  12. Managing Machine Data vs. Structured Data 13 Stored Digital Information

    (exabytes) Business transaction data Well understood & analyzed Slow growth Handled by traditional BI Unstructured data Tremendous source of business value Under leveraged by business Cannot be handed by BI Needs a new approach Structured Data Machine Data
  13. Typical Process for Structured Data 14 Reporting Analysis CRM ERP

    Apps ETL Data- warehouse Data Mart Data Mart Data Mart Data Mart Dashboards
  14. Typical Approaches Don’t Work for Machine Data 17 Reporting Analysis

    CRM Apps ETL Data- warehouse Data Mart Data Mart Data Mart Data Mart Dashboards Key Challenges  Requires pre-defined schema – limits flexibility  Difficult to handle data diversity in real time  Adding new and changing data sources is hard  Scaling for large volumes of data is difficult  Time consuming with long deployments
  15. Customer Facing Data Outside the Datacenter Applications Web logs Log4J,

    JMS, JMX .NET events Code and scripts Networking Configurations syslog SNMP netflow Databases Configurations Audit/query logs Tables Schemas Virtualization & Cloud Hypervisor Guest OS, Apps Cloud Linux/Unix Configuration s syslog File system ps, iostat, top Windows Registry Event logs File system sysinternals Logfiles Configs Messages Traps Alerts Metrics Scripts Tickets Changes Click-stream data Shopping cart data Online transaction data Manufacturing, logistics… CDRs & IPDRs Power consumption RFID data GPS data Splunk Collects and Indexes Any Machine Data 18
  16. Splunk Collects and Indexes Any Machine Data 19 Customer Facing

    Data Outside the Datacenter Applications Web logs Log4J, JMS, JMX .NET events Code and scripts Networking Configurations syslog SNMP netflow Databases Configurations Audit/query logs Tables Schemas Virtualization & Cloud Hypervisor Guest OS, Apps Cloud Linux/Unix Configuration s syslog File system ps, iostat, top Windows Registry Event logs File system sysinternals Logfiles Configs Messages Traps Alerts Metrics Scripts Tickets Changes Click-stream data Shopping cart data Online transaction data Manufacturing, logistics… CDRs & IPDRs Power consumption RFID data GPS data No custom connectors No RDBMS No need for ETL No upfront schema •Any amount, any location, any source.
  17. Splunk Turns Machine Data into Operational Intelligence Report and analyze

    Custom dashboards Monitor and alert Ad hoc search Real-time Collection and Indexing Developer Platform 20 Splunk storage Other Stores Optimized for real-time, low latency and interactivity
  18. Operational Intelligence Across the Business 21 Search and Investigate Proactive

    Monitoring Operational Visibility Real-time Business Insights IT & Ops Gain real-time insight from operational data to make better-informed business decisions Business
  19. Business Insights with Splunk 22 Representative Use Cases Across Customers

    Application Analytics (usage clickstream + feature descriptions + customer profile) Content & Search Analytics (mobile access + content downloads + search) Real-time Sales Analytics (device activation + billing plans + geo location) Service Cost Analytics (call detail records + tariffs database + VOIP peering) Online Monetization Analytics (customer clickstream + virtual goods pricing + billing) Marketing Analytics (web/mobile logs + ad pricing + click through)
  20. 23 Financial Services Firms Drive Results with Splunk Troubleshoot and

    monitor trading and settlement applications. Improve uptime and reduce MTTR. Monitor and manage online investment application and servers. Network security monitoring and rapid incident response to mitigate security risks. Ensures effective compliance while improving productivity of compliance team. End to end monitoring across trading applications – improving uptime and customer experience. Cross-tier visibility to improve dev ops coordination and accelerate MTTR. Index data across trading applications and FIX order processing to improve customer service.
  21. 24 Splunk in Financial Services Trade Processing Customer Support Settlement

    Systems FIX Order Management Payment Processing Core Banking Processes Infrastructure Management Branch / ATM Management Operational Intelligence Across Diverse Business Processes and Services Security Compliance (SOX/Basel) Online Banking Trading Credit Cards Loans Representative Business Processes Product & Services Insurance Business Services
  22. Financial Services Division One of largest providers of outsourced online

    financial management solutions Serving 1800+ financial institutions and 4 million+ end customers (from 2011) Applications include: – Consumer and business internet banking – Electronic bill payment and presentment – Personal online financial management – Website hosting and development for financial institutions “Fraud team’s goal is to provide fraud analysis on more of a proactive basis.” 26
  23. Using Big Data to Prevent Fraud Using ALL the data,

    historical analysis of past 30 / 90 / all time periods identifies new fraud patterns As patterns emerge build real-time alerts when evidence of similar patterns of known fraudsters emerge (SMS, email) Result: – Watching fraudster in real-time—seeing $5M, $7M, $8M wire attempts – Splunk exposed every element of the infrastructure that he touched – Next we could correlate activities based on time to understand his pattern of activity 27
  24. Truth from the Trenches: Geolocation 28 Identified fraud pattern and

    enabled location of attacker Noticed a similar fraud pattern across 15 banks Then we mapped them to see they were within 15 miles of one another Fraud was coming from one data processing vendor who they all shared Big Data enables Intuit to make better decisions using the data they already have
  25. Previously had customized parser Searches conducted in batch taking 3+

    hours Reports came in piecemeal across 5000 emails with different syntax Only sophisticated (aka highly- paid) users could track patterns Splunk provides single pane of glass with full visibility Role-based access provides secure views into data Customer service and banking customer teams can begin queries on their own—no waiting for access/ permission—no highly paid engineer required Identify fraudulent activity in seconds Went from reactive response to real-time monitoring Splunk Speeds Analysis and Action 29
  26. 30 Splunk in Financial Services Trade Processing Customer Support Settlement

    Systems FIX Order Management Payment Processing Core Banking Processes Infrastructure Management Branch / ATM Management Operational Intelligence Across Diverse Business Processes and Services Security Compliance (SOX/Basel) Online Banking Trading Credit Cards Loans Representative Business Processes Product & Services Insurance Business Services
  27. 31 Enabling High Performance Global Trade Infrastructure  Trading infrastructures

    processes thousands of transactions across 25+ applications  Indexes data across all applications and tiers including mission critical middle-tier trade service  Splunk provides in-depth visibility across tiers: – Improves system uptime with timely alerts – Understand user trends and improve user experience – Accelerates trade processing times – Troubleshoot difficult issues within seconds Leading European Financial Services Firm “Today, we run threshold-based alerts to address issues before they cause downtime” - Team Lead for Trading Infrastructure
  28. 32 Splunk in Financial Services Trade Processing Customer Support Settlement

    Systems FIX Order Management Payment Processing Core Banking Processes Infrastructure Management Branch / ATM Management Operational Intelligence Across Diverse Business Processes and Services Security Compliance (SOX/Basel) Online Banking Trading Credit Cards Loans Representative Business Processes Product & Services Insurance Business Services
  29. Gaining Visibility Across Order Routing Systems 33  Challenging to

    gain insight across financial industry exchange (FIX) logs to address customer requests  Indexes data across trading applications – includes FIX logs, syslog from hosts and applications  Splunk provides in-depth visibility across tiers: – Automated transaction record monitoring – Significantly reduced customer request response times – Reduced broker response times – Increased productivity of existing resources Improves Customer Service and Streamlines Operations “Splunk helps us improve customer service. We are closing queries as fast as we open them” - Matt Easley, Customer Support Manager
  30. 34 Splunk in Financial Services Trade Processing Customer Support Settlement

    Systems FIX Order Management Payment Processing Core Banking Software Infrastructure Management Branch / ATM Management Operational Intelligence Across Diverse Business Processes and Services Security Compliance (SOX/Basel) Online Banking Trading Credit Cards Loans Representative Business Processes Product & Services Insurance Business Services
  31. End to End Visibility Delivers $6 MM in Annual ROI

    35  Indexes millions of events/day. Machine data across: – Custom & 3rd party software, servers – Java code, databases, Operating systems  Splunk provides insights to improve service: – Decreased downtime through proactive monitoring – Reduced MTTR with faster troubleshooting – Better coordination across Development & Operations – Improved team efficiency and better resource utilization Financial Service Firm Improves Service and Customer Satisfaction “We paid for Splunk in the first month…..now, we are actually proactively solving problems”
  32. 36 Splunk in Financial Services Trade Processing Customer Support Settlement

    Systems FIX Order Management Payment Processing Core Banking Software Infrastructure Management Branch / ATM Management Operational Intelligence Across Diverse Business Processes and Services Security Compliance (SOX/Basel) Online Banking Trading Credit Cards Loans Representative Business Processes Product & Services Insurance Business Services
  33. Enabling Timely Transaction Settlement Processing 37  Peak trading volume

    exceeds 20 MM transactions per day – settlement process is critical  Indexes data across all applications and engines  Splunk provides insights to improve performance: – Alerting supports pro-active customer engagement – Engine dashboards highlight KPI to monitor performance – Developers see impact of changes in real-time – Rapidly perform forensic investigations to troubleshoot Leading Global Equities and Commodities Exchange “Tried months and several homegrown solutions to surface real time insight – with Splunk it was working in 1 week”
  34. 38 Splunk in Financial Services Trade Processing Customer Support Settlement

    Systems FIX Order Management Payment Processing Core Banking Software Infrastructure Management Branch / ATM Management Operational Intelligence Across Diverse Business Processes and Services Security Compliance (SOX/Basel) Online Banking Trading Credit Cards Loans Representative Business Processes Product & Services Insurance Business Services
  35. 39 Multi Scale Unit Risk Metrics Group Our high performance

    cloud computing offering has Splunk integrated from the ground up. It is the go-to solution for every type of question – it helps in decision making at every level right from dev/test/ops to account management, product management and pricing. “ ” Vincent Bumgarner Principal Systems Engineer Running Compute Instances Avg CPU Utilization CPU Percentile Splunk integrated into Risk Metrics’ high performance cloud computing effort from the ground up Used throughout the organization for delivering deep insights to vendor pricing, CFO, account management and sales, in addition to dev/test and support
  36. Insights Across Roles & Departments Product Managers Sales Operations Executive

    Management Customer Service & Support IT Management & Operations Marketing Managers 40
  37. Machine Data into Real-time Business Insights Complement traditional BI Tools

    Deliver rapid time to value Leverage new class of data 41 Real-time business insights
  38. Broad Adoption Across 5,200+ Customers in 85+ Countries 42 Over

    Half the Fortune 100 Cloud and Online Services Cloud and Online Services Education Cloud and Online Services Energy and Utilities Cloud and Online Services Financial Services & Insurance Cloud and Online Services Government Cloud and Online Services Manufacturing Cloud and Online Services Media & Entertainment Cloud and Online Services Cloud and Online Services Healthcare Travel and Leisure Cloud and Online Services Retail Cloud and Online Services Telecommunications Cloud and Online Services Technology Cloud and Online Services
  39. Easy to Get Started 44 Download and install in minutes

    3. Start Splunking 1. Download 2. Eat your Machine Data
  40. Some tips 45 3. Give lots people access 1. Have

    some requirements 2. Start small – different types, but manageable volumes