Upgrade to Pro — share decks privately, control downloads, hide ads and more …

big data

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for ngarneau ngarneau
March 09, 2012
Programming
400
5

big data

big data keynote at the opencode quebec, introducing cassandra, hadoop and pig.

Avatar for ngarneau

ngarneau

March 09, 2012

More Decks by ngarneau

See All by ngarneau
Introduction au machine learning avec Scitkit-learn
Avatar for ngarneau ngarneau
0
49
Mocks, stubs & seams
Avatar for ngarneau ngarneau
0
110

Other Decks in Programming

See All in Programming
Japan Community Day at Kubecon + CloudNativeCon Japan 2026: Learning Container Privilege Control by Building My Own Low-Level Container Runtime
Avatar for ternbusty ternbusty
1
170
Pythonの実行はどこまで賢くなったのか？ CPythonとPyPyから見る最適化のしくみ
Avatar for curekoshimizu curekoshimizu
3
1.9k
Gmail/Google DriveをトリガーにAIエージェントを動かそう！ / Run AI agents with Gmail/Google Drive as triggers!
Avatar for Har1101 har1101
2
440
リアルな遅延を測る仕様
Avatar for kota-yata kota_yata
1
130
in-process GraphQL のすすめ #ginzajs
Avatar for izumin5210 izumin5210
4
1.5k
170k Jobs a Day on GKE: Scaling Mercari's CI Platform - and What's Next for AI-Native Development
Avatar for Junya Okabe junyaokabe
0
120
Detecting Compromised CI with eBPF and Cilium Tetragon
Avatar for Liz Rice lizrice
0
250
AIと壁打ちしながら進めるコスト管理
Avatar for Ryoma Fujiwara fufuhu
2
1.7k
AI時代に学ぶ 好きなルール 嫌いなルール Linter編
Avatar for Shota Nemoto shorty5121
0
140
引き算の組織 ― アウトカムとAIに全振りするために辞めたこと ― / Organization by Subtraction
Avatar for ヤマモトヒロキ hirokiyamamoto14
PRO
0
300
ドリフトを絶対に許さない（？）CDK運用 / CDK Ops with Zero Tolerance for Drifts (?)
Avatar for アキキー | Akihisa Ikeda akihisaikeda
1
230
ALB ログから Trace を気合で繋げる技術
Avatar for Hayato Kawai fohte
3
300

Featured

See All Featured
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
133
19k
Practical Orchestrator
Avatar for Shlomi Noach shlominoach
191
12k
Designing Powerful Visuals for Engaging Learning
Avatar for Mike Taylor tmiket
1
510
Introduction to Domain-Driven Design and Collaborative software design
Avatar for Kenny Baas-Schwegler baasie
1
950
The untapped power of vector embeddings
Avatar for Frank van Dijk frankvandijk
2
1.8k
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
2.1k
How GitHub (no longer) Works
Avatar for Zach Holman holman
316
150k
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
Avatar for Ines Montani inesmontani
PRO
0
660
AI: The stuff that nobody shows you
Avatar for John Nunemaker jnunemaker
PRO
9
940
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
55
12k
Navigating the Design Leadership Dip - Product Design Week Design Leaders+ Conference 2024
Avatar for Andy Polaine apolaine
2
400
Building Adaptive Systems
Avatar for Chris Keathley keathley
44
3.2k

Transcript

  1. big data

  2. cassandra - Facebook - 2007. - Apache - 2008. -

    Netflix, Digg, Twitter, Rackspace...

  3. cassandra - non-relationnal - schema-less - open-source - horizontally scalable

    - easy replication - large datasets

  4. cassandra - datacenters - «no single point of failure».

  5. cassandra data model - no joins (maybe joints, we don’t

    know as of version 1.0.9..) - denormalization

  6. cassandra data model - keyspace - column family - row

    key - super column - column / value

  7. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } }

  8. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace

  9. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family

  10. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key

  11. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key column

  12. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key column value

  13. cassandra keep in mind memory disk memtable commit log

  14. cassandra keep in mind memory disk memtable commit log

  15. cassandra keep in mind memory disk memtable commit log

  16. cassandra keep in mind memory disk memtable commit log memtable

  17. cassandra keep in mind memory disk memtable commit log memtable

    memtable

  18. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables

  19. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables

  20. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables

  21. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables

  22. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables

  23. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables

  24. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables SSTables

  25. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables SSTables SSTables

  26. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables SSTables SSTables SSTables

  27. hadoop - Yahoo! - 2006. - Apache - 2008.

  28. hadoop - mapreduce - hadoop distributed filesystem

  29. hadoop mapreduce - map - reduce

  30. hadoop mapreduce

  31. hadoop mapreduce

  32. hadoop mapreduce

  33. hadoop HDFS

  34. hadoop HDFS data

  35. hadoop HDFS hadoop data

  36. hadoop HDFS hadoop data

  37. hadoop HDFS hadoop data hadoop

  38. hadoop HDFS hadoop data hadoop hadoop

  39. hadoop HDFS hadoop data hadoop hadoop data

  40. hadoop HDFS hadoop data hadoop hadoop data data

  41. hadoop HDFS hadoop data hadoop hadoop data data data

  42. hadoop HDFS hadoop data hadoop hadoop data data data

  43. hadoop HDFS hadoop data hadoop hadoop data data data

  44. hadoop HDFS hadoop data hadoop hadoop data data data cassandra

  45. hadoop HDFS hadoop data hadoop hadoop data data data cassandra

    cassandra

  46. hadoop HDFS hadoop data hadoop hadoop data data data cassandra

    cassandra cassandra

  47. hadoop keep in mind - business intelligence - machine learning

    - collective intelligence

  48. pig - Yahoo! - 2007. - Apache - 2008.

  49. pig - pigs eat anything. - pigs live anywhere. -

    pigs are domestic. - pigs fly.

  50. pig keep in mind

  51. let’s play! https://[email protected]/ngarneau/opencode.git

  52. let’s play! dataset Salons = { ’1’: { ‘id’: 1,

    ‘attendants’: 47, ‘name’: ‘Salon Laval’, ‘year’: 2010 } } Commandes = { ’1’: { ‘amount’: 799, ‘salon’: 1 } }

  53. let’s play! we want to know what is the correlation

    between the number of attendants and the total revenues by salon.