How Autodesk Leverages Splunk as an Assurance Platform on AWS

In this presentation I talked about how a couple divisions at Autodesk run Splunk on AWS and leverage Splunk as a platform to provide operational and security visibility and confidence for our AWS adoption

Transcript

1. November 13, 2014 | Las Vegas, NV Alan Williams, Principal Engineer at Autodesk Consumer Group Praveen Rangnath, Director of Cloud Product Marketing at Splunk
2. © 2014 Autodesk !  Engineer @ Autodesk !  Infrastructure Background !  AWS for ~4 years !  Splunk for ~1 year !  Motorcyclist !  Soft spot for pit bulls Who Am I?
3. © 2014 Autodesk !  Leader in 3D design, engineering and entertainment software !  Introduced AutoCAD in 1982 !  Empowering the Maker movement !  Help our customers imagine, design and create a better world Who is Autodesk? http://www.autodesk.com/products/personal-design-and-creativity
4. © 2014 Autodesk iOS Apps on App Store
5. © 2014 Autodesk Android Apps on Play Store
6. © 2014 Autodesk students.autodesk.com Get free access to the software used to make the games, movies, buildings, and products that inspire you.
7. © 2014 Autodesk Why AWS?
8. © 2014 Autodesk !  Take inventory of existing hardware !  Use the AWS Calculator !  http://calculator.s3.amazonaws.com/index.html !  Cost/compute analysis Let’s begin with cost analysis…
9. © 2014 Autodesk Cost Analysis – Account for Everything Support & Maintenance Power & Cooling Rack space Storage (FC + SATA) Servers Load Balancers Adds up quickly
10. © 2014 Autodesk What we noticed… Total cost of server hardware vs Total cost of AWS instances = 35% lower for AWS Total cost of all on- premise infra vs Total cost of all AWS infra = 50% lower for AWS
11. © 2014 Autodesk !  We can’t compete on price !  Economies of scale !  We can’t compete on speed !  Time to provision !  Time to innovate - deliver new features Outcome
12. © 2014 Autodesk !  Production workloads !  Customer facing !  Business critical !  Big data analytics !  Dev / Test environments !  Net new systems Many AWS Use Cases
13. © 2014 Autodesk Why Splunk?
14. © 2014 Autodesk !  Leverages existing investments !  Standard log aggregation platform !  Familiar technology !  Logging = Splunk !  Single view across all accounts !  Splunk App for AWS Why Splunk?
15. © 2014 Autodesk !  Operations Insights & Troubleshooting !  Analyzing data from thousands of endpoints globally !  Centralized visibility across all AWS accounts !  Splunk App for AWS !  Security Incidents Response !  Event auditing !  Splunk App for Enterprise Security !  Business Intelligence !  Product metrics dashboards – executive visibility using Splunk !  Product analytics Provide Assurance for AWS adoption
16. © 2014 Autodesk CloudTrail Dashboard
17. © 2014 Autodesk Business Intelligence
18. © 2014 Autodesk Incident Response
19. © 2014 Autodesk Has this compromised host made any API calls?
20. © 2014 Autodesk Where were these IAM keys used?
21. © 2014 Autodesk Where are sign-ins originating?
22. © 2014 Autodesk Running Splunk On AWS
23. © 2014 Autodesk Tenets to running Splunk on AWS !  Automation !  AWS CloudFormation template - http://goo.gl/Hn309p !  Ansible Playbook - http://goo.gl/fulJPc !  Scalability !  Easy and quick to add/remove nodes !  Auto-scale everything !  Splunk Search Head Pooling* !  Performance !  Search heads (CPU bound) – C3 instances !  Indexers (IO bound) – C3+EBS, I2, HS1 instances !  Maximize IOPS with RAID O
24. © 2014 Autodesk Splunk Architecture on AWS Auto scaling Group Availability Zone (1b) Virtual Private Cloud Availability Zone (1a) Auto scaling Group Application Subnet Application Subnet Presentation Subnet Internal LB Subnet Presentation Subnet Internal LB Subnet Auto scaling Group Auto scaling Group Auto scaling Group Auto scaling Group Search Head 1 Indexer Peer 1 Indexer Peer …8 Bastion (Ansible Master) Direct Connect splunk.mycompany.com IGW ELB Deploy Server A NFS 200 GB Inter- mediate Forwader Cluster Master Node NAT 1 Gbps 1 Gbps Internet LB Subnet Internet LB Subnet Arrow Legend Load Balancing Traffic Splunk inter-communication S3 shuttl archiving NFS Search Head 3 Auto scaling Group License Master Node
25. © 2014 Autodesk CloudTrail + Splunk SNS Topic SQS Queue CloudTrail S3 Bucket SNS Topic CloudTrail 1 1 2 2 3 3 4 4 5 Account A Account B Core Services Account !  Simple to configure !  Decoupled components !  Scalable to many accounts !  Central logging view across all accounts
26. © 2014 Autodesk !  AWS + Splunk = Happy Marriage !  Scalable to 100s of accounts !  Platform Use Cases for Operations, Security and Business Summary
27. © 2014 Autodesk !  CloudFormation Splunk Cluster Template !  https://github.com/alanwill/cfn-splunk !  Ansible Splunk Playbook !  https://github.com/alanwill/ansible-splunk Try it yourself… Pull Requests encouraged
28. Autodesk is a registered trademark of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document. © 2014 Autodesk. All rights reserved. @alanwill alanwill
29. 29   Splunk  Company  Overview   Company  (NASDAQ:  SPLK)     Founded  2004,  first  so=ware  release  in  2006     HQ:  San  Francisco  /  Regional  HQ:  London,  Hong  Kong     Over  1,200  employees,  based  in  12  countries     Annual  revenue:  $302.6M  (YoY  +52%)   Business  Model  /  Products     Free  download  /  online  sandbox  to  massive  scale     On-­‐premises,  in  the  cloud  and  SaaS     8,000+  Customers     Customers  in  100  countries     Over  2/3  of  the  Fortune  100     Largest  license:  100  Terabytes  per  day   Fast  Company  2013:  Named  Splunk  #4  Most  Innova]ve   Company  in  the  World  and  #1  Big  Data  Innovator     Leader:  Gartner  SIEM  Magic  Quadrant,  2014    
30. What  Is  Machine  Data?   Volume    |    Velocity    |    Variety    |    Variability   GPS,   RFID,   Hypervisor,   Web  Servers,   Email,  Messaging,   Clickstreams,  Mobile,     Telephony,  IVR,  Databases,   Sensors,  Servers,  Storage,   Security  Devices,  AWS  CloudTrail  and  AWS  Config   Machine  data  is  the  fastest  growing,  most   complex,  most  valuable  area  of  big  data   30  
31. 31   What  Does  Machine  Data  Look  Like?   Sources   TwiJer   Care  IVR   Middleware     Error   Order  Processing  
32. 32   Machine  Data  Contains  Cri]cal  Insights   Customer  ID   Order  ID   Customer’s  Tweet     Time  Wai]ng  On  Hold   Twiher  ID   Product  ID   Company’s  Twiher  ID   Sources   TwiJer   Care  IVR   Middleware     Error   Order  Processing   Customer  ID   Order  ID   Customer  ID  
33. 33   Machine  Data  Contains  Cri]cal  Insights   Order  ID   Customer’s  Tweet     Time  Wai]ng  On  Hold   Product  ID   Company’s  Twiher  ID   Sources   TwiJer   Care  IVR   Middleware     Error   Order  Processing   Order  ID   Customer  ID   Twiher  ID   Customer  ID   Customer  ID  
34. IT   Opera]ons   Security  and   Compliance   Digital   Intelligence   App  Dev  and   App  Mgmt.   Developer  Plaiorm  (REST  API,  SDKs)   Business   Analy]cs   Industrial  Data   and  Internet  of   Things   Small  Data.  Big  Data.  Huge  Data.   Use  Cases  for  Machine  Data  Analy]cs   34   Core  Use  Cases   Emerging  Use  Cases  
35. Industry-­‐Leading  Plaiorm  For  Machine  Data    Machine  Data:  Any  LocaPon,  Type,  Volume   Online   Services   Web   Services   Servers   Security   GPS   Loca]on   Storage   Desktops   Networks   Packaged   Applica]ons   Custom   Applica]ons   Messaging   Telecoms   Online   Shopping   Cart   Web   Clickstreams   Databases   Energy   Meters   Call  Detail   Records   Smartphones   and  Devices   RFID   On-­‐   Premises   Private     Cloud   Public     Cloud   PlaQorm  Support  (Apps  /  API  /  SDKs)   Enterprise  Scalability   Universal  Indexing   Answer  Any  QuesPon   Developer   PlaQorm   Report  and     analyze   Custom     dashboards   Monitor     and  alert   Ad  hoc     search   35  
36. Comprehensive  AWS  Solu]ons   So=ware-­‐as-­‐a-­‐Service   Self-­‐managed  so=ware   App  for  AWS   Integra]ons   Amazon  Machine  Images   Hunk  –  EMR  IntegraPon   CloudTrail CloudWatch S3 Config Billing ELB 36  
37. Full   Featured   Enterprise  Ready   Easy   37  
38. Splunk  Cloud  –  Full-­‐Featured  SaaS  Plaiorm   Full  Power   of  Splunk   Enterprise   Access  to   600+  Apps   Hybrid   Deployment   Architecture   38  
39. Hybrid  Search   Search  Head(s)   Indexer(s)   Search  Head(s)   Indexer(s)   On  Premises   Private  Cloud   Public  Cloud   On  Premises   Private  Cloud   Public  Cloud   Single  Pane  of  Glass  Visibility   39  
40. Splunk  Cloud  –  Enterprise-­‐Ready  Service   Industry-­‐leading   scalability  &   flexibility   Architected   for  upPme  &   performance     100%  UpPme  SLA   Robust   enterprise   security   5GB/day  -­‐  5TB/day  plans   No  data  comingling;   Security  aJestaPons   40  
41. High  availability   across  indexers   &  search  heads   MulPple  AWS     availability  zones   Dedicated  cloud   environments   Splunk  Cloud  fully  monitored  using  Splunk  Enterprise   Built  for  100%  Up]me   41  
42. Splunk  Cloud  –  Easy   ACCELERATED   TIME  TO   VALUE  &      FASTER  ROI   Starts  at  $675  per  month  /   33%  price  reducPon   Splunk  Online  Sandbox   Immediate  deployment   42  
43. Forward  data   Search   Monitor   Get  value  fast   What  You  Do   Hardware  setup   Storage   Scaling   Monitoring   What  We  Do   43  
44. Copyright  ©  2014  Splunk,  Inc.   Thank  You!  
45. ENT212