Using OpenStack Orchestration for Big Data Workloads

Talk on OpenStack Orchestration (Project Heat) using the Enterprise working group's Big Data reference architecture. Demo deployed on Cisco Metacloud

Transcript

1. Hart Hoover | @hhoover Using OpenStack Orchestration for Big Data
2. Big Data
3. The Digital Economy is Disrupting Everything
4. 2.5 Quintillion bytes of data is generated every day, most of which is never captured, never collected, with no corresponding action taken. 2,500,000,000,000,000,000 Source: IBM/Cisco
5. By year-end 2018, 25% of durable good manufacturers will utilize data generated by smart machines in their customer-facing sales, billing, and service workflows. And by 2018, 6 billion “Things” will request support. 6 Billion Things Source: Gartner
6. Data & Analytics Use Cases MEDIA/ ENTERTAINMENT Viewers / advertising effectiveness COMMUNICATIONS Location-based advertising EDUCATION & RESEARCH Experiment sensor analysis CONSUMER PACKAGED GOODS Sentiment analysis of what’s hot, problems HEALTH CARE Patient sensors, monitoring, EHRs Quality of care LIFE SCIENCES Clinical trials Genomics HIGH TECHNOLOGY / INDUSTRIAL MFG. Mfg. quality Warranty analysis OIL & GAS Drilling exploration sensor analysis FINANCIAL SERVICES Risk & portfolio analysis New products AUTOMOTIVE Auto sensors reporting location, problems RETAIL Consumer sentiment Optimized marketing LAW ENFORCEMENT & DEFENSE Threat analysis - social media monitoring, photo analysis TRAVEL & TRANSPORTATION Sensor analysis for optimal traffic flows Customer sentiment UTILITIES Smart Meter analysis for network capacity, ON-LINE SERVICES / SOCIAL MEDIA People & career matching Web-site optimization
7. • An Open Source software platform (file system) that process vast amounts of data (MapReduce, HBase, HDFS) • Yahoo! and Google are its biggest contributors • Attributes: • Scalable: Store and process petabytes of data • Economical: Processes across clusters of commonly available servers • Efficient: Processes in parallel on the nodes where the data is located • Reliable: Maintains multiple copies of the data • Highly available: Automatically redeploys computing tasks on failures What Is Hadoop?
9. https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf
10. OpenStack Orchestration (Heat)
11. What is OpenStack Orchestration? OpenStack Dashboard Standard Hardware OpenStack Shared Services Your Applications Compute Networking Storage APIs
12. • python-heatclient || python-openstackclient • heat-api • heat-api-cfn • heat-engine • (MySQL & RabbitMQ) OpenStack Orchestration Components
13. OpenStack Orchestration Workflow: POST heat-engine heat-api RabbitMQ MySQL POST
14. OpenStack Orchestration Workflow: GET heat-engine heat-api RabbitMQ MySQL GET
15. HOT
16. • Sections • Parameter Constraints • Pseudo Parameters • Intrinsic Functions HOT Components
17. •Meta •Parameters •Resources •Outputs heat_template_version: 2015-10-15 description: Deploys a three-tier web application. HOT Sections
18. •Meta •Parameters •Resources •Outputs parameters: key_name: type: string label: Key Name description: Name of keypair to be used for compute instance HOT Sections
19. •Meta •Parameters •Resources •Outputs resources: my_instance: type: OS::Nova::Server properties: flavor: m1.small image: ubuntu-14.04 HOT Sections
20. •Meta •Parameters •Resources •Outputs outputs: instance_ip: description: IP address of the deployed compute instance value: { get_attr: [my_instance, first_address] } HOT Sections
21. length: (String) { min: <lower limit>, max: <upper limit> } range: (number) { min: <lower limit>, max: <upper limit> } allowed_values: [ <value>, <value>, ... ] Parameter Constraints allowed_pattern: <regular expression> custom_constraint
22. OS::stack_name OS::stack_id OS::project_id Pseudo Parameters
23. get_attr get_file get_resource list_join resource_facade Intrinsic Functions str_replace digest repeat str_split map_merge
24. Use the community as a reference, not for production
26. https://www.openstack.org/software/sample-configs/#big-data Big Data Deployment