big data

big data keynote at the opencode quebec, introducing cassandra, hadoop and pig.

Transcript

1. big data
2. cassandra - Facebook - 2007. - Apache - 2008. - Netflix, Digg, Twitter, Rackspace...
3. cassandra - non-relationnal - schema-less - open-source - horizontally scalable - easy replication - large datasets
4. cassandra - datacenters - «no single point of failure».
5. cassandra data model - no joins (maybe joints, we don’t know as of version 1.0.9..) - denormalization
6. cassandra data model - keyspace - column family - row key - super column - column / value
7. cassandra data model application = { users = { ‘ngarneau’: { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } }
8. cassandra data model application = { users = { ‘ngarneau’: { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace
9. cassandra data model application = { users = { ‘ngarneau’: { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family
10. cassandra data model application = { users = { ‘ngarneau’: { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key
11. cassandra data model application = { users = { ‘ngarneau’: { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key column
12. cassandra data model application = { users = { ‘ngarneau’: { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key column value
13. cassandra keep in mind memory disk memtable commit log
14. cassandra keep in mind memory disk memtable commit log
15. cassandra keep in mind memory disk memtable commit log
16. cassandra keep in mind memory disk memtable commit log memtable
17. cassandra keep in mind memory disk memtable commit log memtable memtable
18. cassandra keep in mind memory disk memtable commit log memtable memtable memtables
19. cassandra keep in mind memory disk memtable commit log memtable memtable memtables
20. cassandra keep in mind memory disk memtable commit log memtable memtable memtables SSTables
21. cassandra keep in mind memory disk memtable commit log memtable memtable memtables SSTables SSTables
22. cassandra keep in mind memory disk memtable commit log memtable memtable memtables SSTables SSTables SSTables
23. cassandra keep in mind memory disk memtable commit log memtable memtable memtables SSTables SSTables SSTables
24. cassandra keep in mind memory disk memtable commit log memtable memtable memtables SSTables SSTables SSTables SSTables
25. cassandra keep in mind memory disk memtable commit log memtable memtable memtables SSTables SSTables SSTables SSTables SSTables
26. cassandra keep in mind memory disk memtable commit log memtable memtable memtables SSTables SSTables SSTables SSTables SSTables SSTables
27. hadoop - Yahoo! - 2006. - Apache - 2008.
28. hadoop - mapreduce - hadoop distributed filesystem
29. hadoop mapreduce - map - reduce
30. hadoop mapreduce
31. hadoop mapreduce
32. hadoop mapreduce
33. hadoop HDFS
34. hadoop HDFS data
35. hadoop HDFS hadoop data
36. hadoop HDFS hadoop data
37. hadoop HDFS hadoop data hadoop
38. hadoop HDFS hadoop data hadoop hadoop
39. hadoop HDFS hadoop data hadoop hadoop data
40. hadoop HDFS hadoop data hadoop hadoop data data
41. hadoop HDFS hadoop data hadoop hadoop data data data
42. hadoop HDFS hadoop data hadoop hadoop data data data
43. hadoop HDFS hadoop data hadoop hadoop data data data
44. hadoop HDFS hadoop data hadoop hadoop data data data cassandra
45. hadoop HDFS hadoop data hadoop hadoop data data data cassandra cassandra
46. hadoop HDFS hadoop data hadoop hadoop data data data cassandra cassandra cassandra
47. hadoop keep in mind - business intelligence - machine learning - collective intelligence
48. pig - Yahoo! - 2007. - Apache - 2008.
49. pig - pigs eat anything. - pigs live anywhere. - pigs are domestic. - pigs fly.
50. pig keep in mind
51. let’s play! https://ngarneau@github.com/ngarneau/opencode.git
52. let’s play! dataset Salons = { ’1’: { ‘id’: 1, ‘attendants’: 47, ‘name’: ‘Salon Laval’, ‘year’: 2010 } } Commandes = { ’1’: { ‘amount’: 799, ‘salon’: 1 } }
53. let’s play! we want to know what is the correlation between the number of attendants and the total revenues by salon.