Introduction to Big Data

What is Big Data ? What trends in IT support it ?
Some examples and What tools are used ? The future ?

Transcript

1. Big Data • What is it ? • What trends in IT support it ? • Some examples • What tools are used ? • The future ?
2. Big Data – What is it ? • Very large data set volumes • Very long / unacceptable processing times • Very large data velocity ( inputs / outputs ) • Very large varieties of data • High level of complexity
3. Big Data – Supporting Trends • Moore's Law An observation that the number of transistors on integrated circuits doubles every two years.
4. Big Data – Supporting Trends • Kryder's Law The density of storage is increasing and the cost decreasing at a rate faster than Moore's Law.
5. Big Data – Supporting Trends • Butter's Law Relates to network capacity and states that the cost of sending data over an optical network halves every nine months.
6. Big Data – Supporting Trends • Parallel Processing Task parallelism, breaking the task down into its constituent parts and processing them simultaneously.
7. Big Data – Examples • NASA Climate Simulation 32 petabytes • The Large Hadron Collider 25 petabytes annually, 200 petabytes after replication • Wall mart 2.5 petabytes per hour
8. Big Data – Tools • Hadoop Hadoop is often used at the server level to organise the cluster along with a NoSQL database for data storage. • NoSQL Databases ( non sql ) that use looser consistency models than relational databases. Performance gains via simplification using key value stores. • MPP Massively parallel processing and analytics databases. Fast for data aggregation but slow for data loading.
9. Big Data – The Future • Data sets will continue to grow • Storage unit costs will continue to decrease • Processing costs will decrease • Network capacity will continue to grow • Data growth may exceed processing capacity
10. Contact Us • Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems