Showcase Booth - Open and Intelligent Data Autonomy

Introduction to OpenSDS and Open Data Autonomy.
To be used at the showcase booth.

Transcript

1. Open and Intelligent Data Autonomy
2. 2 How to Efficiently Use Data Across Multiple Clouds？ • Data Provisioning • Data Operation • Data Protection • Data Migration Cinder 60+ Drivers Manila 10+ Drivers libStorage CoprHD Swift Ceph RexRay libStorage Ceph RexRay ViPR Cinder 60+ drivers Manila 10+ drivers OceanStor DJ Swift Others NAS Local storage SAN NVMeof Scale-out Cloud CLOUD ENTERPRISE IT integration Virtualization layer APP VM VM On-Premise Virtualization layer APP VM VM Virtualization layer VM VM VM VM Private Cloud Virtualization layer APP VM VM Virtualization layer VM VM APP Public Cloud Hot data Warm data Cold data Frozen data • How can use data management service effectively? • How can data be associated with applications on the cloud? • How can data be managed in a unified manner? Key Customer Requirements Key Challenges for Developers
3. OpenSDS is the first top-level open-source project of the Linux Foundation in the Data Domain. It was jointly built by Huawei and world-leading data vendors, as well as excellent enterprises and universities. OpenSDS is aimed at addressing the data challenges in the cloud-based transformation of enterprise IT infrastructure. 3 Linux Foundation's First Top-Level Project in the Data Domain
4. 4 OpenSDS: Open and Intelligent Data Autonomy Orchestrate data management service automatically and intelligently Data Provisioning Data Operation Data Migration Data Protection Intelligent Data Management
5. 5 OpenSDS Architecture and Projects v SUSHI The Northbound Plug-ins Project Common plug-ins to enable OpenSDS storage services for cloud and application frameworks v HOTPOT The Storage Controller Project Single control for block, file, and object services across storage on premise and in clouds v Gelato The Multi-Cloud Project Policy based multi-cloud data control to enable data mobility across clouds
6. 6 OpenSDS: Intelligent Data Provisioning Data Service workflows actions Simple Data Provisioning Workflow: Volume Creation Key Points • Supports Service Catalog • Supports Customized Workflows • Easy integration of new actions Automation and Orchestration provide a framework for intelligent data provisioning Seamless File Sharing API Server File Share API Controller Dock NFS Server Manila Other drivers Key Points • Supports file share in hotpot • Supports NFS Server, Manila and etc.
7. 7 Automation and Orchestration BUILD WORKFLOWS DEPLOY AUTOMATE • Define Actions. • Integrate with workflow Engine • Define generic services. • Limitless use cases • Customize the services Simplify orchestration and automation of complex workflows intelligently API Server Service Connector Instance task st2Connector DB Orchestration Manager Workflow Engine Create Volume Get Volume Attach to Host Notify Provisioning workflow Check Bucket Create Bucket Create Migration Notify Migration workflow Task1 Task2 Task3 Task5 Custom workflow Task4
8. 8 Illustration : Volume Provisioning Automation • User registers a service, e.g. volume provisioning service, by defining a template in YAML or JSON. • Service Catalog Manager Updated • Create Service Instance • Schedule the service instance • Orchestration Automatically ``yaml --- version: 1.0 name: “Provisioning service” description: “Provisioning a volume for a host” parameters: “volume_name”: type: “string” description: “Name of the volume to be provisioned” constraints": allowed_pattern: "[A-Z]+[a-zA-Z0-9]*" “profile”: type: “string” default: “default” description: “Profile ID or name” “volume_size”: type: “integer” “default”: 1 “host”: type: “string” description: “Name or IP address of the host” workflows: “provisioning-workflow”: description: “This is the workflow for provisioning a volume” input: [“volume_name”, “profile”, “volume_size”, “host”] output: <json> tasks: “create-volume”: action: “opensds.create-volume” on-success: - “mapping-volume-to-host” "mapping-volume-to-host": action: "opensds.attach-volume" on-success: - ”notify” “notify”: action: “opensds.notify” "input": "cmd: "printf 'volume <% $.volume_id %> was attached to host successfully'" ```
9. OpenSDS: Intelligent Data Operation Intelligent analysis of telemetry data and anomaly detection Kafka OpenSDS Metrics API OpenSDS Metrics Controller Collector LVM Metrics driver Ceph Metrics driver 3rd-party Metrics driver Dock Collector Exporter Converter 3rd-party ML Module POST GET Intelligent Anomaly Detection Intelligent Collection Anomaly Detection Framework Key Points • Scalable interfaces for different data processing • Scalable metrics driver to support heterogeneous storages dbscan algorithm Intelligent Telemetry
10. 10 Telemetry: Integrate with Prometheus and Grafana • General and Scalable API Sets • Exporter and Converter – Support Various Backends – Support Various Third-party Telemetry and Analytics Modules – Custom integration support • Easy Integration of Visualization Kafka OpenSDS Metrics API OpenSDS Metrics Controller Collector LVM Metrics driver Ceph Metrics driver 3rd-party Metrics driver Dock Collector Exporter Converter 3rd-party ML Module POST GET Framework Ready for Intelligent analytics
11. 11 Anomaly Detection • Periodic Generation of Data • Kafka streams of Data • Training service to get trained Models • Prediction Service with specific trained model • Seamless integration with Alert Manager • Intelligent Actions on trigger Support integrated default analytics framework
12. 12 OpenSDS: Intelligent Data Migration Data in the past, at present, and in the future is connected and end-to-end data lifecycle management cross multi-clouds. Key Points • Support object data migration across different cloud service providers. • Support of choosing the storage tier though the lifecycle management rules. • Support customized Tier definition.
13. Multi-Cloud Data Management Connector Manager API Handler Migration Data Mover Policy Manager Job Manager Plan Manager Other Data Mover … DB Scheduler Apache Kafka DataFlow Manager Keystone put jobs pull jobs run plan create/delete/update/get policy create/delete/update/get connector create/delete/update/get plan get jobs restart/cancel job get plan create jobs update plan auth update job S3 Service Backend Service Dataflow Service Database Backend Adapters GUI API Gateway Data Mover Identity Service Scalable Framework for Multi-Cloud Data Management
14. Framework for Object Lifecycle Management • Scalable and Generic API Life Cycle Configurations • Schedule and Orchestrate • Smart Storage Tier Management across Multiple Cloud. • Inter and Intra Cloud Lifecycle Management Smart Data Lifecycle Management across cloud across storage tiers
15. 15 15 2018H1 ARUBA • OpenStack • Replication Array-Based, Host-Based • Dashboard • Storage Profiles • Enumeration • Block Storage • Cinder Drivers • Ceph • LVM • Huawei: Dorado 2017H2 ZEALAND • Kubernetes FlexVolume • Vol CRUD • Standalone Cinder Integration • CSI Support • Ceph, LVM 2018H2 BALI • S3 Object • Multi-Cloud AWS, Azure, Ceph, Huawei • Multi-OpenStack • CSI v1.0 2019H1 CAPRI* • Object Lifecycle • Telemetry • Anomaly Detection • Orchestration & Automation • File Share • NVMeoF • Multi-Cloud: GCP, IBM • Storage • Fujitsu • HPE Nimble 2019H2 DAITO* • Data Protection • …more Roadmap v0.25
16. 16 The Open Autonomous Data Project To build an open autonomous data storage platform with self-driving protection, availability, security and optimization capabilities for real world uses; to provide a neutral platform for open source data storage projects collaboration, and to build a multi-vendor ecosystem of products, solutions and services SODA PROJECT REBRANDING
17. @opensds_io info@opensds.io / lfopensds@gmail.com https://www.opensds.io https://github.com/opensds opensds.slack.com Join & Know More…