Big ‘Queries’, not (only) Big Data

Transcript

1. Big ‘Queries’, not (only) Big Data Daniela Florescu Oracle
2. Lots of data. Little information. • Data is the new oil. • ‘Data’ in itself has no value. It’s just (lots of) bits. • Value could be extracted from data. • This can only be done through complex data processing -- the ‘Big Queries’.
3. Then.... • Several Years Ago... a VC told me.... • “There will never be another Oracle.”
4. ....and Now • “Did I hear the word ‘database’ ? • “Is it a Big ‘database’ ?” • General hysteria among investors, analysts and customers. • Entrepreneurs just satisfy the demand... • We had the Internet Bubble, the Social Bubble, now it’s the Database Bubble. • We had it all. Geeks are winning.
5. General Cacophony
6. “Big Data” Problem • Turning Data Into Actionable Information • It’s not a database. It’s a process. • What “big” means ? • It’s the value • The three Vs of Big Data: • Volume • Velocity • Variety
7. Big Data vs. Hadoop (huh?) • It’s like equating cancer with chemotherapy • General problem vs. one particular (partial) solution • Confusion helps Hadoop • This confusion kills every other solution • NOTE: Please do not parallelize by hand anymore !
8. Big Data vs. NoSQL (huh?) • NoSQL is the revenge of the application developers • NoSQL are databases, or at least data stores • NoSQL : • Live data behind an application • Transactional: queries and updates • Complex structures • Flexible, schema-less data • Small volume....
9. Big data success so far... • Turning the Internet into a car salesman play ground. • ... and get harassed by Facebook.
10. Lots of data. No information. • We live in a world full of data, but starved of information • Don’t compare to what we had 30 years ago • Compare to what we could have.
11. Example: LaunchPad • I manage a team of open source developers. • We store our code, and bugs in LaunchPad. • I would like to know what’s the average of bugs solved per week per developer. • How hard can that be !?
12. Example: my email • I have 100.000 emails in my inbox. • I am looking for a guy who worked in Neo4J, who wrote to me an email around the time of the NOSQL conference. • I don’t remember his name. • How hard can that be !?
13. Example: Google • I would like to have all the investment announcements in Big Data and NoSQL made in the last 24 hours. • How hard can that be !?
14. Example: Healthcare • I am a Stanford patient. • The doctors treat about 100.000 with the same problem as me. About 10K have the exact precise symptoms. • They have all electronic records. • I ask: “how is this particular treatment working ?” • Answer: “we don’t know, we have no statistics”. • How hard that can be !?
15. Example: Ebay • I like rare watches: Patek Phillippe, Breguet, Jaeger- LeCoultre, etc • I know that some of them appreciate better then others. • Ebay has all the sales data of all products in the world. • How can tell which one of the brands above appreciates better ?
16. Example: LinkedIn • I have a new query language for JSON (JSONiq) • This can be used by all MongoDB or Couch users. • How can I get from LinkedIn all potential customers for my technology?
17. Example: Genentech • We are doing tons of testing for tons of new potential medicines. • I have hundreds of thousands of spreasheets describing results of such testing. • I just realized that one of my testing methodology is flawed. • How can I see which results are wrong, potentially leading to serious problems with my medicine ?
18. Example: Salesforce • I have thousands of sales people, talking to tens of thousands of prospective customers about a new product. • The conversations are stored in Salesforce. • How can I tell if my new product launch will be successful, or if there are serious problems with it ?
19. Example: Facebook • I have all the consumer data in the world. • A new movie is released. • How can I tell how the public reacts to it, by age group ?
20. Example: presidential campaign • How can I know the characteristics of the undecided voters in the swing states ?
21. Example: DHS • I have all the emails of everybody (oh, well..they do) • How can I tell if there is a threat ? • How could a tragedy like the one in Boston be avoided ?
22. No information, and no hope either. • Are any of those products helping with my questions ? • What’s wrong with everybody ? Why did we loose hope ?
23. Integrated Information Processing Platform Lots of data. Extract the value.
24. We are missing this. A data refinery.
25. I am having fun • We are building such a platform. • Purely pragmatic. • We are concerned with the global architecture, not each component in particular. • Integrate best of breed from open source • We are concerned with scaling • data size, query complexity • I can write in a matter of hours programs to solve my questions above.
26. It’s Information, not Data • ‘Data’ sounds too much like records and tables. • I speak in sentences; I read sentences. Information is often in natural language. • CSV, XML, Text, PDF, 11073, HTML, JSON,.... • Whole spectrum: • records to markup to natural language • schema-flexible • no schema, multiple schemas, schema later, open schemas, complex schemas, schemas as vocabularies XML
27. It’s Processing, not Querying • Information massaging: • index • search • query • transform • analytics • entity extraction • entity resolution • information classification • machine learning • graph algorithms • statistics • pattern extraction • NLP • visualize • communicate XQuery
28. It has to be Complete • A Big Data Platform needs all the above components • Traditional ‘queries’ are just a small part • Integrate best of breed from open source • scale them independently • scale the general architecture • Has also to include components for human interaction
29. It has to be Integrated • A platform needs to allow • smooth integration of components • easy data communication between components • How many people in this room stitched together: • Lucene with a database • System R with a database • Spreadsheets with a database • an XML parser
30. Our focus 0-GB GB- TB TB-PB Queries analytics, transforms ETL NLP machine learning MySQL SQLServer DB2 Oracle Phyton More Phyton ???? YES YES YES
31. Cloud computations • Need to say more ? • Bigger single machines vs. lots of smaller machines • One word: parallelization • NOTE: please observe that automatic parallelization of information processing is NOT done today ! • With map-reduce we went backwards, not forward....
32. In memory computing • During information processing, avoid touching the disk • No more external sorts, please ! • Data network communication penalty less then the penalty of touching the local disk • Transactions: writing in main memory 3 times is safer then writing to disk • Exploit the aggregated main memory of a cloud of machines
33. How to make it scale ? • Traditional 3 tier architecture scaling: Middle tier Java, etc state-less small datasets Database SQL transactional large data sets Scale unlimited Scale: painful replication RAC
34. How to make it scale ?(2) • What changes in the “big data” scenario ? Middle tier state-less large datasets Data Stores NOT transactional large data sets problem less of a problem
35. Three Tier Architecture 2.0 Data Storing Layer Limited queries (*) Data Processing Layer queries NLP, etc stateless both unlimited scale scale independently No more Java, etc ! DataFlow languages (*)
36. Data Storing Layer Data Storing Layer Limited queries (*) unlimited scale Any query that cannot be executed in constant memory should not be allowed in this tier. DO: selections, projections, simple aggregates, simple transforms DON’T: joins MongoDB
37. Data Processing Layer Data Processing Layer queries NLP, etc stateless unlimited scale No more Java, etc ! Automatic parallelization • Query Languages: SQL, XQuery • Rules Languages: Prolog • Streams, and Queues • Data Workflows • Functional Programming: ML, Haskell • Set Oriented: APL XQuery
38. I am having fun • We are building such a platform. • Purely pragmatic. • We are concerned with the global architecture, not each component in particular. • Integrate best of breed from open source • We are concerned with scaling • data size, query complexity • I can write in a matter of hours programs to solve my questions above.
39. Real research (and industry...) problem • Holy grail: 1 billion $ question • Given my platform, I can write a program to do the information processing that I need. • I am a nerd....and I wrote the XQuery language.... • How can my admin do the same ?
40. Conclusion • Big Data problem: extracting value out of data • Big Data is not a database • The 4 Vs of Big Data: Value, Volume, Variety, Velocity • Size doesn’t always not matter • Look beyond Hadoop • There is more to programming then Phyton • Don’t accept Google as the answer • Look around you for questions that you need answered !
41. Please .. • Teach statistics, NLP, query processing, data flow architectures, machine learning, knowledge representation in “database” classes. • Teach functional programming, dataflow architectures • Please do not forget about query processing and automatic parallelization...
42. Thank you.