IHPC/A*Star Seminar on Big Data

Transcript

1. Big Data – Why Do I care? Harish Pillay Head, Community Architecture and Leadership Red Hat hpillay@redhat.com
2. Big Data
3. Big Data: - Comes from computational sciences - describes scenarios where the amount (volume) of data coming IN vastly outstrips the software tools to store and even process it.
4. So, what's the big deal then?
6. We now have the tech and tools to work with them – and in real time!
7. OK, WIIFM?
8. What's in it for me? (for those who are acronym-challenged)
9. Today, data is being gathered and stored in vast volumes
10. And data is coming in with high velocity
11. And in many varieties
12. VVV of Big Data Volume Velocity Variety - Credit: Doug Laney, Gartner
13. So why should you care?
14. Let's look at the survey that you participated in
15. <limesurvey stuff>
16. This is very simple analysis
17. Let's look at “what if?”s
18. What if you can know how crowded the MRT train is and moved away from that part of the platform before boarding?
19. What if you can improve the chances of knowing the exam questions based on analysis of previous exams?
20. What if you can detect, manage and mitigate forest fires (and NEA will thank you for that)?
21. Internet of Things
22. With IoT, your Big Data will be a Big Tsunami
23. And we now have a lot of the tools and technologies
24. A small sampling: –Hadoop –Spark –R –SciDB –Solr
25. Credit: www.bigdata-startups.com
26. What Can You Do Today?
27. Just consider this
28. The Large Hadron Collider (LHC) produces millions of collisions every second in each detector, generating approximately one petabyte of data per second. None of today’s computing systems are capable of recording such rates, so sophisticated selection systems are used for a first fast electronic pre-selection, only passing one out of 10,000 events. Tens of thousands of processor cores then select 1% of the remaining events for analysis. • http://home.web.cern.ch/about/updates/2013/04/animation-shows-lhc-data-processing
29. And all of this is Open Source
30. 1. Sign up on: • openshift.redhat.com and • github.com 2. install R, Hadoop 3. check out: data.gov.sg for data
31. But do keep in mind to observe the Bonferroni Principle (https://en.wikipedia.org/wiki/Bonferroni_correction) [homework for you]
32. Now Go Change The World!
33. Thank You! @harishpillay hpillay@redhat.com