Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Making sense of big data

Making sense of big data

The outline of the presentation that was given for 2012, February 7th's webmardi:

Life without MySQL: Manipulating data sets that break the single machine barrier
Understanding the role of Hadoop & Map/Reduce
Handling large amounts of storage and computing instances
Applications: Logs, Open Data, Streams

2fcc875f98607b3007909fe4be99160d?s=128

Pierre-Yves Ritschard

February 11, 2012
Tweet

Transcript

  1. Making Sense of Big- Data Saturday, February 11, 12

  2. Big Data ? • what do we mean by big

    ? • why is it becoming an issue now ? • a shift in internet usage: from consumption to sharing • cost of storage Saturday, February 11, 12
  3. Interesting Numbers • 400% growth of generated data per year,

    5% of global IT spending growth • 4 billion pieces of content shared everyday • 1 billion tweet sent out every 3 days • 1 petabyte processed by google every 72 minutes Saturday, February 11, 12
  4. Challenges • Extracting meaningful results from such large amounts of

    data • Presenting relevant data to users • It’s not a new problem • Remember life before google analytics ? (and what it would mean to do the same on your own) Saturday, February 11, 12
  5. Traditionnal Approaches • Huge Systems • Oracle BI • Still

    bound to a single processing unit • Huge upfront costs: the value of the result must be much higher than the cost of producing it Saturday, February 11, 12
  6. Who needs a new approach • Web applications • Logs

    • Event Processing • Open Data Applications • Everywhere cost of infrastructure matters Saturday, February 11, 12
  7. The Google Way • Search is the canonical big data

    use case • Needed to use off-the-shelf hardware • A tool suite: • Storage: GFS • Database: Big Table • Processing: Map/Reduce Saturday, February 11, 12
  8. The Open Source Ecosystem • Not settled yet • Hadoop:

    Backed by Yahoo • A lot of other players Saturday, February 11, 12
  9. Category Theory what we do with data • Simple Operations

    • counting • Relational Algebra • Projection: “SELECT a,b,c” • Selection: “WHERE a > 1” • Joins: “LEFT JOIN on” Saturday, February 11, 12
  10. A focus on map/reduce • One of the more established

    big data tools • Primarily for batch jobs, analytics • Addresses category theory on huge amounts of data • Doesn’t solve every problem Saturday, February 11, 12
  11. Map/Reduce principles • Breaks category theory into map and reduce

    operations • Map phase: operations that can run in parallel: projection, selection • Operations in the reduce phase: joins, counts • Just like cooking! Saturday, February 11, 12
  12. Pratical Overview (I) • http://xkcd.com/1007 • we can feed off

    of google’s ngram data • simple projection and selection • let’s see some code Saturday, February 11, 12
  13. Pratical Overview (II) Saturday, February 11, 12

  14. Quick Poll: Next Time ? • Infrastructure automation • Event

    Streaming with Apache Kafka • The CAP Theorem and Cassandra • Cluster ordering with zookeeper • Hadoop workshop Saturday, February 11, 12