Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

big data

Avatar for ngarneau ngarneau
March 09, 2012
Programming
400
5

big data

big data keynote at the opencode quebec, introducing cassandra, hadoop and pig.

Avatar for ngarneau

ngarneau

March 09, 2012

More Decks by ngarneau

See All by ngarneau
Introduction au machine learning avec Scitkit-learn
Avatar for ngarneau ngarneau
0
49
Mocks, stubs & seams
Avatar for ngarneau ngarneau
0
110

Other Decks in Programming

See All in Programming
モジュールの視点からSwiftを読み解く #iosdc
Avatar for Shigure Shimotori s_shimotori
0
100
Family mrubyの進捗
Avatar for Katsuhiko Kageyama kishima
1
110
Heart of Swift Concurrency
Avatar for Yuta Koshizawa koher
0
320
GKE で Pod の見方を変えたら、スケールアウト時の挙動を真に捉えられた話
Avatar for Sota Takaki stkk
0
110
型解析で実現する Go の言語内 DSL / Conference に Go! タイムテーブルの歩き方 for Gophers
Avatar for mazrean mazrean
0
140
What We Talk About When We Talk About XP
Avatar for seki at druby.org m_seki
2
570
信頼性の目標を誰も求めてない
Avatar for Shuma shubox
0
500
AIを上手に使っていこうとしたら越境せざるを得なくなった話 〜実践1年で見えた境界を越えなければならない理由と進め方〜 / Crossing borders with AI
Avatar for TomoyaKitaura tomoyakitaura
4
1.1k
XHTMLが残したもの
Avatar for Yosuke Furukawa yosuke_furukawa
PRO
2
710
Streamlitで実現する自然言語データアプリ開発
Avatar for Ayumu Yamaguchi ayumu_yamaguchi
0
210
GKE アップグレード前に知っておきたい Blue/Green と PDB の関係
Avatar for Sota Takaki stkk
0
160
思考垂れ流し開発　～音声入力 × AIエージェント × 開発ハーネスによる試行錯誤～
Avatar for NPOS npostring
0
1.1k

Featured

See All Featured
How GitHub (no longer) Works
Avatar for Zach Holman holman
316
150k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
19k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
55
8.3k
Beyond borders and beyond the search box: How to win the global "messy middle" with AI-driven SEO
Avatar for David Carrasco davidcarrasco
3
240
Fantastic passwords and where to find them - at NoRuKo
Avatar for Phil Nash philnash
52
3.8k
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
4k
Believing is Seeing
Avatar for Spiro Bolos oripsolob
1
220
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
The Mindset for Success: Future Career Progression
Avatar for Greg Gifford greggifford
PRO
0
490
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
Avatar for Rebecca Miller-Webster rmw
35
3.8k
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
Avatar for r-kagaya rkaga
63
46k
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
249
1.3M

Transcript

  1. big data

  2. cassandra - Facebook - 2007. - Apache - 2008. -

    Netflix, Digg, Twitter, Rackspace...

  3. cassandra - non-relationnal - schema-less - open-source - horizontally scalable

    - easy replication - large datasets

  4. cassandra - datacenters - «no single point of failure».

  5. cassandra data model - no joins (maybe joints, we don’t

    know as of version 1.0.9..) - denormalization

  6. cassandra data model - keyspace - column family - row

    key - super column - column / value

  7. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } }

  8. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace

  9. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family

  10. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key

  11. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key column

  12. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key column value

  13. cassandra keep in mind memory disk memtable commit log

  14. cassandra keep in mind memory disk memtable commit log

  15. cassandra keep in mind memory disk memtable commit log

  16. cassandra keep in mind memory disk memtable commit log memtable

  17. cassandra keep in mind memory disk memtable commit log memtable

    memtable

  18. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables

  19. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables

  20. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables

  21. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables

  22. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables

  23. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables

  24. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables SSTables

  25. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables SSTables SSTables

  26. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables SSTables SSTables SSTables

  27. hadoop - Yahoo! - 2006. - Apache - 2008.

  28. hadoop - mapreduce - hadoop distributed filesystem

  29. hadoop mapreduce - map - reduce

  30. hadoop mapreduce

  31. hadoop mapreduce

  32. hadoop mapreduce

  33. hadoop HDFS

  34. hadoop HDFS data

  35. hadoop HDFS hadoop data

  36. hadoop HDFS hadoop data

  37. hadoop HDFS hadoop data hadoop

  38. hadoop HDFS hadoop data hadoop hadoop

  39. hadoop HDFS hadoop data hadoop hadoop data

  40. hadoop HDFS hadoop data hadoop hadoop data data

  41. hadoop HDFS hadoop data hadoop hadoop data data data

  42. hadoop HDFS hadoop data hadoop hadoop data data data

  43. hadoop HDFS hadoop data hadoop hadoop data data data

  44. hadoop HDFS hadoop data hadoop hadoop data data data cassandra

  45. hadoop HDFS hadoop data hadoop hadoop data data data cassandra

    cassandra

  46. hadoop HDFS hadoop data hadoop hadoop data data data cassandra

    cassandra cassandra

  47. hadoop keep in mind - business intelligence - machine learning

    - collective intelligence

  48. pig - Yahoo! - 2007. - Apache - 2008.

  49. pig - pigs eat anything. - pigs live anywhere. -

    pigs are domestic. - pigs fly.

  50. pig keep in mind

  51. let’s play! https://[email protected]/ngarneau/opencode.git

  52. let’s play! dataset Salons = { ’1’: { ‘id’: 1,

    ‘attendants’: 47, ‘name’: ‘Salon Laval’, ‘year’: 2010 } } Commandes = { ’1’: { ‘amount’: 799, ‘salon’: 1 } }

  53. let’s play! we want to know what is the correlation

    between the number of attendants and the total revenues by salon.