How to survive the Data Deluge: Cloud computing for large scale data analysis

How to survive the Data Deluge: Cloud computing for large scale data analysis

Presentation of my Ph.D. proposal at Yahoo! Research

Transcript

  1. How to survive the Data Deluge: Cloud computing for large

    scale data analysis Gianmarco De Francisci Morales IMT Institute for Advanced Studies Lucca CSE PhD XXIV Cycle 8 Mar 2010 lunedì 8 marzo 2010
  2. Outline Part 1: Introduction What, Why and History Part 2:

    State of the art Current technologies and research Part 3: Proposal Ideas for future improvements lunedì 8 marzo 2010
  3. Part 1 Introduction lunedì 8 marzo 2010

  4. How would you sort... lunedì 8 marzo 2010

  5. How would you sort... ... 1GB of data? lunedì 8

    marzo 2010
  6. How would you sort... ... 1GB of data? ... 100GB

    of data? lunedì 8 marzo 2010
  7. How would you sort... ... 1GB of data? ... 100GB

    of data? ... 10TB of data? lunedì 8 marzo 2010
  8. How would you sort... ... 1GB of data? ... 100GB

    of data? ... 10TB of data? Scale matters! Because More Isn't Just More, More Is Different lunedì 8 marzo 2010
  9. The Petabyte Age lunedì 8 marzo 2010

  10. The Data Deluge The world is drowning in data Web

    2.0 (user generated content) Scientific experiments Physics (particle accelerators) Astronomy (satellite images) Biology (genomic maps) Sensors (GPS, RFID) lunedì 8 marzo 2010
  11. “Data is not information, information is not knowledge, knowledge is

    not wisdom.” Clifford Stoll lunedì 8 marzo 2010
  12. The “Big Data” problem “Data whose size forces us to

    look beyond the tried- and-true methods that are prevalent at that time” Jacobs, CACM 2009 Requirements for a large scale data analysis system Scalability (scale free) Cost effectiveness (autonomic) Fault tolerance (highly available) lunedì 8 marzo 2010
  13. Methodology evolution DBMS are the most common tool for data

    analysis ‘60s CODASYL ‘70s Relational DBMS ‘80s & ‘90s Parallel DBMS Not much has happened since the ‘70s The fundamental model and code are still the same lunedì 8 marzo 2010
  14. Relational DBMS evolution Yesterday: Relational model & OLTP & SQL

    Today: Different markets (OLTP, OLAP, Stream, etc..) Stored Procedures & User Defined Functions High performance requirements lunedì 8 marzo 2010
  15. Parallel DBMS A solution for performance problems Scale-out on shared

    nothing architectures using dataflow operators and horizontal partitioning Problems: Not enough flexibility and ease of use Limited fault-tolerance and scalability lunedì 8 marzo 2010
  16. Is parallel wrong? Parallel computing is dead Amdahl’s law: SpUp(N)

    = 1 / ((1-Pa)+Pa/N) Long live parallel computing Gustafson’s law: SpUp(N) = PG*N + (1-PG) Physical limits Manycore Money lunedì 8 marzo 2010
  17. Cloud Computing Convergence of parallel computing, virtualization and service oriented

    architectures Focus on being scale-free, fault tolerant, cost effective and easy to use Buzzword! Distributed system Scalability Location, replication and failure transparency lunedì 8 marzo 2010
  18. Data Intensive Cloud Computing I/O bound problems Move computing near

    data Simple, scale-agnostic programming interface Shared nothing architecture Commodity hardware lunedì 8 marzo 2010
  19. Part 2 State of the art lunedì 8 marzo 2010

  20. Who is involved? lunedì 8 marzo 2010

  21. Distributed Data Computation Coordination Data Abstraction High Level Languages Architecture

    lunedì 8 marzo 2010
  22. Google Yahoo Microsoft Others High Level Languages Computation Data Abstraction

    Distributed Data Coordination Sawzall Pig/Latin DryadLINQ SCOPE Hive Cascading MapReduce Hadoop Dryad BigTable HBase PNUTS Cassandra Voldemort GFS HDFS Cosmos Dynamo Chubby Zookeeper Software stacks lunedì 8 marzo 2010
  23. Comparison with PDBMS CAP Theorem BASE vs ACID Computing on

    large data vs Handling large data OLAP vs OLTP User Defined Functions vs Select-Project-Join Nested vs Flat data model lunedì 8 marzo 2010
  24. Comparison with PDBMS MapReduce, a major step backwards DeWitt, Stonebraker

    "If the only tool you have is a hammer, you tend to see every problem as a nail" Abraham Maslow SQL and Relational Model are not the answer lunedì 8 marzo 2010
  25. Computational Models I/O cost, Computability, Functional Programming Paradigm Enrichments MapReduceMerge,

    HadoopDB Online Analytics Templates, MapReduce Online Research directions lunedì 8 marzo 2010
  26. A new computation model for rack-based computing Goal: I/O cost

    characterization Issues: only theoretical analysis no existing reference system Future: best algorithms for the model, model adaptation to real systems F. Afrati and J. Ullman. Unpublished lunedì 8 marzo 2010
  27. A model of computation for MapReduce Goal: theoretical computability characterization

    of MapReduce algorithms Result: algorithmic design technique for MapReduce Future: develop algorithms in this class, find relationships with other classes H. Karloff, S. Suri, and S. Vassilvitskii. In SODA, 2010 lunedì 8 marzo 2010
  28. Google’s MapReduce programming model - revisited Goal: functional style reverse

    engineering of MapReduce How: top down functional analysis Result: simplified and rationalized MapReduce model with runnable functional specification R. Lammel. In Science of Computer Programming, 2007 lunedì 8 marzo 2010
  29. Map-Reduce-Merge Simplified relational data processing on large clusters Goal: implement

    relational operators efficiently How: new final phase that merges 2 key-value lists Issues: very low level and hard to use, needs integration into a high level language H. Yang, A. Dasdan, R. Hsiao, and D. Parker. In SIGMOD 2007 lunedì 8 marzo 2010
  30. HadoopDB An architectural hybrid of MapReduce and DBMS technologies for

    analytical workloads. Goal: advantages of both DB and MapReduce How: integrate a DBMS (PostgreSQL) in Hadoop, Hive as interface Issues: better reuse principles than artifacts A. Abouzeid, K. Bajda-Pawlikowski, D. Abadi, A. Silberschatz, A. Rasin. In VLDB, 2009 lunedì 8 marzo 2010
  31. Interactive analysis of web-scale data Goal: speed up general queries

    for big data How: pre-computed templates to fill at run-time Future: which templates are useful for interactive? help the user to formulate templates (sampling?) C. Olston, E. Bortnikov, K. Elmeleegy, F. Junqueira, B. Reed. In CIDR, 2009 lunedì 8 marzo 2010
  32. MapReduce Online Goal: speed up turnaround of MapReduce jobs How:

    operator pipelining, online aggregation Issues: limited inter-job pipelining (data only) inter-job aggregation problematic (scratch data) T. Condie, N. Conway, P. Alvaro, J. Hellerstein, K. Elmeleegy, and R. Sears. Technical report, University of California, Berkeley, 2009 lunedì 8 marzo 2010
  33. Part 3 Proposal lunedì 8 marzo 2010

  34. Research Problems Cloud Computing is batch oriented High level languages

    push for efficient relational operators Not clear which algorithms and problems are best for these systems Research efforts are “erratic”, no common research agenda yet Early stage of development, more effort needed lunedì 8 marzo 2010
  35. Research Questions How to design novel algorithms for large scale

    data analysis? How to support these algorithms on cloud computing systems? Is it possible to carry out online data analysis on such systems? lunedì 8 marzo 2010
  36. Methodology Top down: start by studying existing algorithms, extract a

    representative workload Identify weaknesses in existing systems Use principles of database research to fill the gaps Evaluate contributions from both theoretical and experimental point of view lunedì 8 marzo 2010
  37. Some Ideas Sampling and result estimation A good enough result

    is often acceptable Semantic clues Leverage properties of M/R functions (distributivity, associativity, commutativity) Properties of the input may speed up the computation lunedì 8 marzo 2010
  38. Thesis Goals Build and evaluate a toolbox of algorithms for

    large scale data analysis on cloud computing systems Design extensions to existing programming paradigms in order to support these algorithms Develop methods to speed up these algorithms to support online processing lunedì 8 marzo 2010
  39. Questions? lunedì 8 marzo 2010