Hammerspace - IT Press Tour December 2018

Transcript

1. 1 IT Press Tour Data Beyond Storage December 6th, 2018
2. 2 § Introduction § Hammerspace Overview – Global Namespace – Use cases – Benefits § Product Deep-dive – Architecture – Metadata-as-a-Service with Demo – Performance – Multi Cloud with Demo § Wrap-up Agenda
3. 3 § Headquartered in Silicon Valley § 30 person team § Technology developed over 5 years Company Overview Mission: “Access to unstructured data anywhere, on-demand.”
4. 4 Data Agility – Moving from Questions to Answers IT Service Queue Data Visibility Multiple Silos Multiple Formats Copying Pre- Planning ? Time-to-Value Run Job
5. 5 Data Consumers Need Data Agility Movie Production Marketing Team Oil & Gas • Speed up production • Relate data to movie • Cost sensitive • Use new products from Cloud • Where is the data? • IT “free” • Analyze recent seismic data • Get results faster • Scale with increased data ”Mike” ”Tiffany” ”Frank”
6. 6 Data Silos and Copying Kill Agility § Data silos restrict access and discovery § Copying data creates sprawl File and Object Increased costs Increased infrastructure Increased risks NFS, SMB, S3 Data Center
7. 7 The Product – Data-as-a-Service Cloud Service Software Appliance Hybrid Cloud Data Management Metadata Management Data Catalogs Global Index and Search Data Governance Hybrid Multi Cloud Unstructured Data On-demand File Data Metadata Services Pay-as-you-Go Environments Data Services On-demand Object Data
8. 8 Data in Hammerspace
9. 9 Hybrid Cloud Data Management Metadata Management Data Catalogs Global Index and Search Data Governance Cloud Data Pipelines with On-Premises Data NFS, SMB, S3 Data Center Rekognition Data Lakes Elasticsearch Macie
10. 10 Metadata Harvesting using Cloud Analytics AWS Rekognition Cloud On-premises W I N D O W S L I N U X Applications NFS/SMB Data Metadata Existing Storage Other NFS Data Services Lambda Data Metadata
11. 11 Additional Use Cases with Examples Multi Cloud Data Management – Self-service for DevOps Cloud Burst – Cloud rendering for M&E App Data Mobility – Hybrid Cloud for Enterprise IT
12. 12 On-premises data center Bi-directional Metadata Replication Applications WAN-optimized data replication Native Object NFS SMB Data Metadata Data Services Data Services Existing Storage Metadata Management Metadata Analytics Global Index and Search Data Governance Applications NFS SMB S3 Windows Linux Geo-spanning Namespace Side-car model
13. 13 Broad Platform Support Cloud – Amazon AWS § S3 § Glacier – Azure § Blob Storage – Google Cloud Platform § Cloud Storage On-premises
14. 14 How does Hammerspace deliver these results? 100% global visibility across silos Discover and prepare data in minutes not hours / days Start data jobs immediately Reduce cloud data costs by 10x Easy to deploy, non- disruptive to IT operations
15. 15 Product Deep-dive
16. 16 Architecture
17. 17 § Automatic capacity load-balancing – No more shares stuck on storage islands § Extensive client and file performance telemetry – Performance load-balance shares across arrays – Automatically remediate hot spots § Data Recovery from backup/snapshots is simple and automated – Recover snapshots (data) from any storage to any storage – Policies can automatically recover data from Tier 3 to Tier 1 without administrator action – Users can self-service recovery by via .snapshot or Windows Previous Versions § Tier data with file-level granularity – Non-disruptive data movement between NAS volumes and Object Operational Simplicity with Hammerspace
18. 18 NFS v4.2 IO Path § Clients read/write directly to underlying storage § Metadata in Hammerspace Server § IO performance is maximized with direct access to data § Hammerspace Server can be deployed using Bare-metal or Virtual Metadata Data Path Metadata + Data Linux File
19. 22 IO Path Overview § Access to the same data across multiple protocols § Namespace and IO path separated from physical location of data § Live Mobility between any managed storage Metadata Data Path Metadata + Data Win, Mac, ESX Linux Portal Mover Store File Object Block
20. 23 Site A Site B § Metadata – Asynchronous – Optimistic – Collision resolution – Prior version § Data – Policy based – On-demand – WAN Optimized Geo-Spanning Win, Mac, ESX Linux File Object Block Linux Win, Mac, ESX File Object Block
21. 24 § Share-level snapshots with scheduling § Snapshot objectives allow snapshot data to tier into – Object § Content is automatically deduped and compressed – NAS § Integrated with the underlying storage (NetApp, Isilon, DSX) – Leverage vendor API to perform file clones – Vendor-specific storage optimizations are automatic § Snapshot lifecycle can easily be managed separately from live data Share Snapshots and Data Lifecycle NAS (original) NAS Tier 2 Object Glacier
22. 25 How Non-Disruptive Mobility Works NFS v4.2 example A P P L I C A T I O N N F S v 4 . 2 M E T A D A T A STORAGE A D A T A Data Services
23. 26 How Non-Disruptive Mobility Works NFS v4.2 example A P P L I C A T I O N N F S v 4 . 2 M E T A D A T A STORAGE A D A T A § Application opens a file, receives layout from Hammerspace § Application can now start I/O L A Y O U T Data Services I / O
24. 27 How Non-Disruptive Mobility Works NFS v4.2 example A P P L I C A T I O N N F S v 4 . 2 M E T A D A T A D A T A § Data mobility is requested to move file from A to B § Layout is recalled § File remains open (no data flush) L A Y O U T R E C A L L STORAGE A Data Services STORAGE B I / O
25. 28 How Non-Disruptive Mobility Works NFS v4.2 example A P P L I C A T I O N N F S v 4 . 2 M E T A D A T A D A T A STORAGE A Data Services § Client requests new layout (file remains open, no I/O impact) § New write layout points to data mover § New read layout points to original source § Mobility process starts to mirror data to new location Data Mover N E W L A Y O U T Writes Reads STORAGE B I / O
26. 29 How Non-Disruptive Mobility Works NFS v4.2 example A P P L I C A T I O N N F S v 4 . 2 M E T A D A T A D A T A STORAGE A Data Services L A Y O U T STORAGE B § When mobility is completed, the same process is followed in the reverse order – Recall layout pointing to data mover and original source – New layout pointing to new location I / O
27. 30 § Build multi cloud data strategy § Manage the cost of data in the cloud § Dark data discovery Hammerspace Cloud Data Profiler
28. 31 Metadata-as-a-Service
29. 32 § Enhanced metadata in addition to POSIX/SMB – Access history for load & performance § User-defined metadata – Attributes § Name value pairs, pre-declared schema – Tags § Name value pairs without pre-declared schema – Labels § Strings, pre-declared – Keywords § Strings, not pre-declared § Collections – Instantly find data through metadata – Navigate shares using filter-like views Powerful Metadata for Data Analytics and Control
30. 33 Metadata Demo Harvesting
31. 37 Performance
32. 38 § Understand when a file is meeting or not meeting desired performance § Open Files are evaluated every ~20 seconds => ~180 placement decisions per hour § Watch the file move between storage tiers in near real-time § Telemetry collected for all open files in the Global Namespace File Performance Visibility 1-Week Live
33. 39 AWS: Storage Volume Scalability in a single Namespace • Scaling 1…4 Storage Volumes, 128 KB Read • Scaling 10Gbps and 25Gbps networks 0 2,000,000 4,000,000 6,000,000 8,000,000 10,000,000 12,000,000 0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 90,000 100,000 1x1x4x8 2x2x4x8 4x4x4x8 8x8x4x8 16x16x4x8 32x32x4x8 64x64x4x8 128x128x4x8 IOPS KB/s Clients X Threads X QD DSX R/W=100/0 BS=131072 4KB Scalability, 1 and 4 10GbE SVs (AWS r4.8xlarge) and 4 25GbE SVs (AWS r4.16xlarge). Four SSDs added per DSX (very slow writes, so not charting). 1DSX 10GbE IOPS 4DSX 10GbE IOPS 4DSX 25GbE IOPS 1DSX 10GbE KB/s 4DSX 10GbE KB/s 4DSX 25GbE KB/s
34. 40 Multi Cloud
35. 41 Hammerspace with NFS v4.2 delivers maximum performance M E T A D A T A A P P L I C A T I O N L I N U X N F S v 4 D A T A N FS – N od e 1 G L O B A L N A M E S P A C E Proven to saturate 100 Gbit/s networking • 11.75 GB/s throughput • Single 100 Gbit/s from client into 11 underlying storage arrays • Data automatically load- balanced across all volumes • Started smaller, increased dynamically without interruption for max performance N FS – N od e 2 N FS – N od e 3 N FS – N od e 4 N FS – N od e 5 N FS – N od e 6 N FS – N od e 7 N FS – N od e 8 N FS – N od e 9 N FS – N od e 10 N FS – N od e 11 Single share 100 Gbit/s Hammerspace Server
36. 42 Demo setup: Geo-spanning, 4-site Site A Site B Site C Site D Shared S3 Other NFS Other NFS
37. 43 Multi Cloud Demo On-prem / AWS / Azure
38. 44 Data Consumers Need Data Agility Movie Production Marketing Team Oil & Gas • Faster cloud rendering workflow • Multi-cloud distribution for price advantage • Works with existing infrastructure • Faster time to start analytics jobs • Fast and complete discovery • Non-disruptive to IT • 5x faster time to start analytics jobs • Optimized Edge site data transfer
39. 45 Data Beyond Storage