Presentation at Open Innovation Symposium 2013

Why Big Data matters and how open source can help with the efforts in understanding and benefiting from big data analysis. Presentation jointly done with BATC of A-STAR.

This presentation was done after a speech by Minister Balakrishnan. The transcript of his speech is at: http://app.mewr.gov.sg/web/Contents/Contents.aspx?Yr=2013&ContId=1891

After him, the speaker was Red Hat CEO, Jim Whitehurst.

Transcript

1. Drama Centre, National Library Board October 1, 2013 GUEST-OF-HONOUR: Dr. Vivian Balakrishnan, Minister for the Environment and Water Resources
2. Big Data Questions: #ois2013
3. Big Data: ➔ Comes from computational sciences
4. Big Data: ➔ Comes from computational sciences ➔ Describes scenarios where the amount (volume) of data COMING IN vastly outstrips the software tools to store and even process it.
5. And the big deal is?
6. Credit: http://www.domo.com/blog/blog/2012/06/08/how-much-data-is-created-every-minute/
7. We now have the tech and tools to work with them
8. We now have the tech and tools to work with them - and in real time!
9. OK, so WIIFM?
10. Today, data is being gathered and stored in vast volumes
11. And data is coming in with high velocity
12. And in many varieties
13. VVV of Big Data Volume Velocity Variety - Credit: Doug Laney, Gartner
14. What is Business Analytics spreadsheets scorecards dashboards pivot tables data warehousing month-end reports Prepared Data, Known Problem
15. What is Business Analytics spreadsheets scorecards dashboards pivot tables data warehousing month-end reports Prepared Data, Known Problem text mining information visualisation social media analytics sentiment analysis unstructured data predictive analytics on-demand reports integrated data platform Raw Data, Unknown Problem
16. Let's look at “what ifs?”
17. What if: you can know how crowded the MRT train is and move to the platform next to the less crowded carriage?
18. What if: you can detect, manage and mitigate forest fires using data from the NEA?
19. We now have a good collection of tools and technologies
20. We now have a good collection of tools and technologies - but could use more
21. http://www.bigdata-startups.com/
22. Case study: Visualizing training …
23. <demo>
24. What does this mean?
25. What does this mean? • Value proposition – different for different stakeholders
26. What does this mean? • Value proposition – different for different stakeholders • End-users?
27. What does this mean? • Value proposition – different for different stakeholders • End-users? • Source for the best course(s) that fits my needs
28. What does this mean? • Value proposition – different for different stakeholders • End-users? • Source for the best course(s) that fits my needs • Personalised recommendations
29. What does this mean? • Value proposition – different for different stakeholders • End-users? • Source for the best course(s) that fits my needs • Personalised recommendations • Organisations?
30. What does this mean? • Value proposition – different for different stakeholders • End-users? • Source for the best course(s) that fits my needs • Personalised recommendations • Organisations? • Holistic training programme for my employees
31. What does this mean? • Value proposition – different for different stakeholders • End-users? • Source for the best course(s) that fits my needs • Personalised recommendations • Organisations? • Holistic training programme for my employees • Specific areas to build skills and capability
32. What does this mean? • Training providers
33. What does this mean? • Training providers • Does this give me insights into my competitors?
34. What does this mean? • Training providers • Does this give me insights into my competitors? • Where's my niche?
35. What does this mean? • Training providers • Does this give me insights into my competitors? • Where's my niche? • Where's my blue-ocean?
36. What does this mean? • Training providers • Does this give me insights into my competitors? • Where's my niche? • Where's my blue-ocean? • Policy-makers
37. What does this mean? • Training providers • Does this give me insights into my competitors? • Where's my niche? • Where's my blue-ocean? • Policy-makers • Which areas to invest to build capability?
38. What does this mean? • Training providers • Does this give me insights into my competitors? • Where's my niche? • Where's my blue-ocean? • Policy-makers • Which areas to invest to build capability? • Who are the key players to work with?
39. What does this mean? • New businesses?
40. What does this mean? • New businesses? • New services to offer? Offering insights as a service?
41. What does this mean? • New businesses? • New services to offer? Offering insights as a service? • New businesses to co-create
42. Are we missing something?
43. Are we missing something? • Using the tools and technology alone can’t address all your business needs
44. Are we missing something? • Using the tools and technology alone can’t address all your business needs • What are the business problems?
45. Are we missing something? • Using the tools and technology alone can’t address all your business needs • What are the business problems? • How real and painful are these problems?
46. Are we missing something? • Using the tools and technology alone can’t address all your business needs • What are the business problems? • How real and painful are these problems? • Which analytical techniques to apply? What are the pros and cons to each technique?
47. Are we missing something? • Using the tools and technology alone can’t address all your business needs • What are the business problems? • How real and painful are these problems? • Which analytical techniques to apply? What are the pros and cons to each technique? • How do you translate the business problems into viable solutions that can be deployed?
48. Are we missing something? • Using the tools and technology alone can’t address all your business needs • What are the business problems? • How real and painful are these problems? • Which analytical techniques to apply? What are the pros and cons to each technique? • How do you translate the business problems into viable solutions that can be deployed?
49. Bridging the Translation Gap Talent & Capability Building IHLs A*STAR
50. The big picture – Eco-system
51. What Can You Do Today?
52. Just consider this
53. The Large Hadron Collider (LHC) produces millions of collisions every second in each detector, generating approximately one petabyte of data per second. http://home.web.cern.ch/about/updates/2013/04/animation-shows-lhc-data- processing
54. But today’s computing systems aren't capable of recording at such rates ... http://home.web.cern.ch/about/updates/2013/04/animation-shows-lhc-data- processing
55. So sophisticated selection systems are used for a fast electronic pre- selection, only passing one out of every 10,000 events ... http://home.web.cern.ch/about/updates/2013/04/animation-shows-lhc-data- processing
56. Tens of thousands of processor cores then select 1% of the remaining events for analysis ... http://home.web.cern.ch/about/updates/2013/04/animation-shows-lhc-data- processing
57. That's it: 1% of 1 in every 10,000 from 1 petabyte Does science need to make that compromise? Can't we do better? http://home.web.cern.ch/about/updates/2013/04/animation-shows-lhc-data- processing
58. We can do even better when driven by Open Source Tools
59. And Here's The Call To Action
60. 1. Sign up on: – openshift.redhat.com – github.com 2. Install Hadoop (and R) 3. Get data from data.gov.sg 4. Engage with BATC to bring your ideas to life
61. Thank You Harish Pillay Wilson Lau Red Hat BATC Questions: #ois2013
62. One more thing
63. Keep in mind to observe the Bonferroni Principle (https://en.wikipedia.org/wiki/Bonferroni_correction) [homework for you]