Upgrade to Pro — share decks privately, control downloads, hide ads and more …

big data

Avatar for ngarneau ngarneau
March 09, 2012
Programming
400
5

big data

big data keynote at the opencode quebec, introducing cassandra, hadoop and pig.

Avatar for ngarneau

ngarneau

March 09, 2012

More Decks by ngarneau

See All by ngarneau
Introduction au machine learning avec Scitkit-learn
Avatar for ngarneau ngarneau
0
49
Mocks, stubs & seams
Avatar for ngarneau ngarneau
0
110

Other Decks in Programming

See All in Programming
ECSアプリログをFireLensでコスト削減しようとしたけど諦めた話 in Fargate×Node.js
Avatar for アキキー | Akihisa Ikeda akihisaikeda
2
4.2k
ユニットテストの先へ：テスト技法で要求・仕様を整理するJava開発実践 / Beyond_Unit_Testing_Practical_Java_Development_Techniques_for_Organizing_Requirements_and_Specifications
Avatar for SHIMANE, Yoshikazu shimashima35
0
400
Semantic Version 単位で戦略を柔軟に変えて、パッケージアップデートを自動化する
Avatar for daitasu daitasu
1
240
[2026年度第1回ORセミナー] 計画最適化ベンチャーと競技プログラミング人材
Avatar for terry-u16 terryu16
0
260
LLMによるContent Moderationの本番運用の裏側と品質担保への挑戦
Avatar for suikabar suikabar
3
680
PHPで使える日時の表現と、その知り方 #frontend_phpcon_do
Avatar for hideki kinjyo o0h
PRO
0
240
メソッドのジェネリクスでGoの夢は広がるか？ / Kyoto.go #65
Avatar for utagawa kiki utgwkk
3
780
Skillsは効率化、Agentsは"自分の拡張"——Builder時代のエージェント編成（CC Night 2026）
Avatar for Hiroki Uemura wemra
1
130
その問い、本当に正しいですか？AI時代のエンジニアに必要な哲学と認知科学 / ai-philosophy-cognitive-science
Avatar for MinoDriven minodriven
9
5.4k
依存関係から依存物へ―Dependencyという言葉の歴史をひも解く
Avatar for j_lee_inst j_lee
0
120
Observability in Practice：Grafana 與 Edge Device SRE 的那些事
Avatar for Blueswen blueswen
0
160
コンテキストの使い捨てをやめる — ビジネスルール駆動開発と miko —
Avatar for ioki ioki
0
200

Featured

See All Featured
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
128
56k
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
Avatar for UX Y'all uxyall
1
1.7k
How GitHub (no longer) Works
Avatar for Zach Holman holman
316
150k
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
3.7k
The State of eCommerce SEO: How to Win in Today's Products SERPs - #SEOweek
Avatar for Aleyda Solis aleyda
2
11k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
21
1.5k
Abbi's Birthday
Avatar for Violet Bock coloredviolet
2
8.1k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
408
66k
SEO for Brand Visibility & Recognition
Avatar for Aleyda Solis aleyda
0
4.6k
Noah Learner - AI + Me: how we built a GSC Bulk Export data pipeline
Avatar for Tech SEO Connect techseoconnect
PRO
0
200
Beyond borders and beyond the search box: How to win the global "messy middle" with AI-driven SEO
Avatar for David Carrasco davidcarrasco
3
160
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
Avatar for Kenny Baas-Schwegler baasie
0
410

Transcript

  1. big data

  2. cassandra - Facebook - 2007. - Apache - 2008. -

    Netflix, Digg, Twitter, Rackspace...

  3. cassandra - non-relationnal - schema-less - open-source - horizontally scalable

    - easy replication - large datasets

  4. cassandra - datacenters - «no single point of failure».

  5. cassandra data model - no joins (maybe joints, we don’t

    know as of version 1.0.9..) - denormalization

  6. cassandra data model - keyspace - column family - row

    key - super column - column / value

  7. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } }

  8. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace

  9. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family

  10. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key

  11. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key column

  12. cassandra data model application = { users = { ‘ngarneau’:

    { ‘first_name’: ‘nicolas’, ‘last_name’: ‘garneau’ } } } keyspace column family row key column value

  13. cassandra keep in mind memory disk memtable commit log

  14. cassandra keep in mind memory disk memtable commit log

  15. cassandra keep in mind memory disk memtable commit log

  16. cassandra keep in mind memory disk memtable commit log memtable

  17. cassandra keep in mind memory disk memtable commit log memtable

    memtable

  18. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables

  19. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables

  20. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables

  21. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables

  22. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables

  23. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables

  24. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables SSTables

  25. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables SSTables SSTables

  26. cassandra keep in mind memory disk memtable commit log memtable

    memtable memtables SSTables SSTables SSTables SSTables SSTables SSTables

  27. hadoop - Yahoo! - 2006. - Apache - 2008.

  28. hadoop - mapreduce - hadoop distributed filesystem

  29. hadoop mapreduce - map - reduce

  30. hadoop mapreduce

  31. hadoop mapreduce

  32. hadoop mapreduce

  33. hadoop HDFS

  34. hadoop HDFS data

  35. hadoop HDFS hadoop data

  36. hadoop HDFS hadoop data

  37. hadoop HDFS hadoop data hadoop

  38. hadoop HDFS hadoop data hadoop hadoop

  39. hadoop HDFS hadoop data hadoop hadoop data

  40. hadoop HDFS hadoop data hadoop hadoop data data

  41. hadoop HDFS hadoop data hadoop hadoop data data data

  42. hadoop HDFS hadoop data hadoop hadoop data data data

  43. hadoop HDFS hadoop data hadoop hadoop data data data

  44. hadoop HDFS hadoop data hadoop hadoop data data data cassandra

  45. hadoop HDFS hadoop data hadoop hadoop data data data cassandra

    cassandra

  46. hadoop HDFS hadoop data hadoop hadoop data data data cassandra

    cassandra cassandra

  47. hadoop keep in mind - business intelligence - machine learning

    - collective intelligence

  48. pig - Yahoo! - 2007. - Apache - 2008.

  49. pig - pigs eat anything. - pigs live anywhere. -

    pigs are domestic. - pigs fly.

  50. pig keep in mind

  51. let’s play! https://[email protected]/ngarneau/opencode.git

  52. let’s play! dataset Salons = { ’1’: { ‘id’: 1,

    ‘attendants’: 47, ‘name’: ‘Salon Laval’, ‘year’: 2010 } } Commandes = { ’1’: { ‘amount’: 799, ‘salon’: 1 } }

  53. let’s play! we want to know what is the correlation

    between the number of attendants and the total revenues by salon.