Big data on the rise: Views of emerging trends and predictions from real-life end users

There are some key trends that are emerging in 2017 within the Hadoop and big data ecosystem, which center around the increase of the use of the cloud. These trends are the underpinning for a larger shift toward purpose-driven products positioned as the core of an organization’s data strategy. While these trends are clearest among early adopters with mature deployments, what about those more traditional end user organizations in the midst of a digital transformation? What are their views on these trends? How are their organizations reconciling the needs and desires to pave a path forward?

John Mertic and Cupid Chan present findings on big data trends, based on data from the ODPi End User Advisory Board (TAB). John and Cupid share real end-user perspectives from companies like GE on how they are using big data tools, the challenges they face, and where they are looking to focus investments—all from a vendor-neutral viewpoint.

Transcript

1. John Mertic, Director, ODPi @jmertic | @ODPIorg Cupid Chan, Managing Partner, 4C Decision @cupidckchan The open ecosystem of big data Big data on the rise: Views of emerging trends and predictions from real-life end users
2. @ODPiOrg
3. @ODPiOrg SIZE ISN’T THE REAL CHALLENGE
4. @ODPiOrg INFRASTRUCTURE NOT BOUND TO A DATACENTER
5. @ODPiOrg TOP 3 KEY CONSIDERATIONS FOR DATA PROJECTS What problem am I trying to solve? What data do I have and where is it coming from? What questions need answered?
6. Create Collect/Ingest/ Process/ETL Analysis Apply /Use Share/Publish/ Access Store/Archive Preserve Destroy/ Delete/Purge Quality Security Governance Management Architecture and Schema Compliance Metadata Strategy Platform
7. @ODPiOrg ENTERPRISE HADOOP DEPLOYMENT CONTINUUM
8. @ODPiOrg ENTERPRISE PRODUCTION HADOOP NEEDS
9. @ODPiOrg Disclaimer 9 This Strata session was prepared or accomplished by Cupid Chan in his personal capacity. The opinions expressed in this talk are the author's own and do not reflect the view of any United States Government Agency or Department.
10. AI in Movies
13. Give a Man a Fish, and You Feed Him for a Day… Teach a Man To Fish, and You Feed Him for a Lifetime – Chinese Proverb
14. Give a Computer a Code, and You Help It Solving for one Situation… Teach a Computer To Code, and You Help It Solving for a Collection of Situations – Cupid Proverb
16. CPU VS GPU
18. Who sings this song? Daddy's Car: (https://www.youtube.com/watch?v=LSHZ_b05W7o) Detail: http://www.flow-machines.com/
19. Georgian Character 1 2 3 4 A B D E C
20. FDA Demo
21. $60,000,000,000 23,400 Bad Addresses http://abcnews.go.com/Politics/medicare-funds-totaling-60-billion-improperly-paid-report/story?id=32604330 Let’s play a Number Game
22. Claims OK Fraud
23. @ODPiOrg Cupid’s Prediction on Emerging Trends for Big Data in the next 5 years 23 AI + BI Artificial Intelligence + Business Intelligence
25. Descriptive: What happened?
26. Diagnostic: Why did it happen?
27. Predictive: What is likely to happen?
28. Prescriptive: What is the best course of action
29. Create Collect/Ingest/ Process/ETL Analysis Apply /Use Share/Publish/ Access Store/Archive Preserve Destroy/ Delete/Purge Quality Security Governance Management Architecture and Schema Compliance Metadata Strategy Platform
30. @ODPiOrg Cupid’s Prediction on Emerging Trends for Big Data in the next 5 years 30 AI + BI Artificial Intelligence + Business Intelligence = CI (Cognitive Intelligence) Intelligence by combining the Speed of how a Machine Learns and Direction Intuited from Human Insight
31. @ODPiOrg STRENGTHENING THE ENGAGEMENT MODEL
32. @ODPiOrg WHY ADD A FEEDBACK LOOP Modern data tools and platforms are non-trivial ( by design ) Enterprise deployment pattern considerations don’t make it upstream Interoperability standards are inconsistent, even though platform vendors often derive from the same codebase
33. @ODPiOrg AN OPEN COMMUNITY STRUCTURE CATALYZES THE FEEDBACK LOOP
34. @ODPiOrg SUSTAINABLE DATA INVESTMENT Successful Projects depend on members, developers, infrastructure to develop products that the market will adopt. Downstream constituents (ISVs/IHVs, Solution Providers, End Users) as a part of this helps them be an active participant in their software supply chain.
35. @ODPiOrg ODPi PROGRAMS: COMPLIANCE ADDS TRUST
36. @ODPiOrg ODPi PROGRAMS: INVESTMENT IN UPSTREAM APACHE AND BIGTOP ODPi is a proud Gold sponsor of the ASF. In addition, ODPi is investing $50,000 to fund developer work with the world’s top Apache and big data developers and architects to expand Bigtop’s functionality and usability. Projects funded include: § Adding containerization support for big data projects built from BigTop § Strengthening the testing framework to improve interoperability These investments amongst others put ODPi investing nearly $100,000 annually back into upstream communities.
37. @ODPiOrg ODPi PROGRAMS: ADVANCING DATA GOVERNANCE ODPi is investing in enabling cross industry collaboration to solve the key data governance challenges facing enterprises today. Focuses include: • Shared R&D on metadata models to support current and upcoming regulations such as GDPR • Driving standardized APIs across metadata repositories ( both Hadoop and outside Hadoop )
38. @ODPiOrg HOW COMPANIES AND DEVELOPERS CAN GET INVOLVED WITH ODPI Have your organization support ODPi https://www.odpi.org/about/join Check out ODPi Data Governance on GitHub https://github.com/odpi/data-governance Join the ODPi User Advisory Board https://www.odpi.org/odpi-compliance-directory/odpi-end-users See how Apache Atlas is implementing this https://cwiki.apache.org/confluence/display/ATLAS/Open+Metadata+and+Governance
39. @ODPiOrg