Data Context 
In the Field and in the Lab

Data Context 
In the Field and in the Lab

Keynote talk, 1st Workshop on Context in Analytics, IEEE International Conference on Data Engineering (ICDE) 2018.

Fb47910b51938c597b6ed6291206cb6e?s=128

Joe Hellerstein

April 16, 2018
Tweet

Transcript

  1. Data Context In the Field and in the Lab JOE

    HELLERSTEIN, UC BERKELEY 1 1ST WORKSHOP ON CONTEXT IN ANALYTICS, PARIS, APRIL 2018
  2. 2 Perspectives on Data Context Wrangling Context Services Six years

    with Trifacta and Google Cloud Dataprep The Common Ground model, and Ground system Managing the rise of Empirical AI flor ML Lifecycle
  3. 3 An Overview for Academics April 24, 2017

  4. 4 Data Preparation: A key source of data context. Shifting

    Market WHO WHA T WHER E Problem Relevance 80% Data Prep Market Use Cases Platform Challenges O n-Pre m Da ta A g e n t s A D L S D AT A S E CU RIT Y & ACCE S S CON T ROLS T RAN S P ARE N T D AT A LIN E AG E D AT A CAT ALOG IN T E G RAT ION ANALYTICS DATA SCIENCE AUTOMATION Perspectives on Data Context Wrangling Context Services Six years with Trifacta and Google Cloud Dataprep The Common Ground model, and Ground system Managing the rise of Empirical AI flor ML Lifecycle
  5. Data Preparation: A key source of data context. Shifting Market

    WHO WHAT WHERE Problem Relevance 80% Data Prep Market Use Cases Platform Challenges O n-Pre m Da ta A ge nts A D LS D AT A S E CU RIT Y & ACCE S S CON T ROLS T RAN S P ARE N T D AT A LIN E AG E D AT A CAT ALOG IN T E G RAT ION ANALYTICS DATA SCIENCE AUTOMATION
  6. 1990’s: IT Governance In the Data Warehouse “There is no

    point in bringing data … into the data warehouse environment without integrating it.” — Bill Inmon, Building the Data Warehouse, 2005
  7. 2018: Business Value From A Quantified World “Get into the

    mindset to collect and measure everything you can.” — DJ Patil, Building Data Science Teams, 2011
  8. “Get into the mindset to collect and measure everything you

    can.” — DJ Patil, Building Data Science Teams, 2011 “There is no point in bringing data … into the data warehouse environment without integrating it.” — Bill Inmon, Building the Data Warehouse, 2005 2018: Business Value From A Quantified World
  9. The World is Changing 7 End-User Self-Service WHO Power Shift

    Big Data Analytics Empirical AI WHAT Data Shift Open Source Cloud WHERE Platform Shift
  10. Whitespace: Data Wrangling DATA PLATFORMS ANALYSIS & CONSUMPTION 80% “It’s

    impossible to overstress this: 80% of the work in any data project is in cleaning the data.” — DJ Patil, Data Jujitsu, O’Reilly Media 2012
  11. Research Roots: Open Source Data Wrangler, 2011 + Predictive Interaction

    + Immediate feedback on multiple choices + Data quality indicators “Wrangler: Interactive Visual Specification of Data Transformation Scripts.” S. Kandel, A.Paepcke, J.M. Hellerstein, J. Heer. CHI 2011. “Proactive Wrangling: Mixed-Initiative End-User Programming of Data Transformation Scripts”, P.J. Guo, S. Kandel, J. M. Hellerstein, J. Heer. UIST 2011. “Enterprise Data Analysis and Visualization: An Interview Study.” S. Kandel, A. Paepcke, J.M. Hellerstein, and J. Heer IEEE VAST 2012 “Profiler: Integrated Statistical Analysis and Visualization for Data Quality Assessment”, S. Kandel, et al. AVI 2012. “Predictive Interaction for Data Transformation” J. Heer, J.M. Hellerstein and S. Kandel CIDR 2015
  12. The State of the Data Prep Market, 2018 Market category

    well established Forrester did its first “Wave” ranking in 2017 Gartner now estimates > $1 Billion market for Data Prep by 2021 12 “Trifacta delivers a strong balance for self-service by analysts and business users. Customer references gave high marks to Trifacta’s ease of use. Trifacta leverages machine learning algorithms to automate and simplify the interaction with data.”
  13. Data Wrangling Standard Across Industry Leaders Proprietary & Confidential 13

    Financial Services Insurance Healthcare and Pharmaceuticals Retail and Consumer Goods Government Agencies
  14. A DATA WRANGLING PLATFORM 14 On-Prem Data Agents ADLS DATA

    SECURITY & ACCESS CONTROLS TRANSPARENT DATA LINEAGE DATA CATALOG INTEGRATION
  15. A DATA WRANGLING PLATFORM 15 On-Prem Data Agents ADLS DATA

    SECURITY & ACCESS CONTROLS TRANSPARENT DATA LINEAGE DATA CATALOG INTEGRATION Millions of lines of recipes Multi-terabyte flows 24x7 Elasticity & Scalability
  16. Use Case

  17. CDC AIDS Intervention in Indiana

  18. The CDC reduced data preparation from 3 months to 3

    days with Trifacta • Refuted an assumption in analysis that would not have been possible without enriching datasets • Expects to scale this model to other, similar outbreaks, such as Zika or Ebola Benefits In future, we need to combine “a variety of sources to identify jurisdictions that, like this county in Indiana, may be at risk of an IDU- related HIV outbreak. These data include drug arrest records, overdose deaths, opioid sales and prescriptions, availability of insurance, emergency medical services, and social and demographic data.” - CDC “The Anatomy of an HIV Outbreak Response in a Rural Community” E. M. Campbell, H. Jia, A. Shankar, et al. “Detailed Transmission Network Analysis of a Large Opiate-Driven Outbreak of HIV Infection in the United States”. Journal of Infectious Diseases, 216(9), 27 November 2017, 1053–1062. https://academic.oup.com/jid/article/216/9/1053/4347235 https://blogs.cdc.gov/publichealthmatters/2015/06/the-anatomy-of-an-hiv outbreak-response-in-a-rural-community/
  19. A DATA WRANGLING PLATFORM 19 On-Prem Data Agents ADLS DATA

    SECURITY & ACCESS CONTROLS TRANSPARENT DATA LINEAGE DATA CATALOG INTEGRATION
  20. A DATA WRANGLING PLATFORM 20 On-Prem Data Agents ADLS DATA

    SECURITY & ACCESS CONTROLS TRANSPARENT DATA LINEAGE DATA CATALOG INTEGRATION ANALYTICS DATA SCIENCE AUTOMATION
  21. Common ground? • Burgeoning SW market • n2 connections? •

    Common formats must emerge • Need a shared place to Write it down, Link it up • Critical to market health!
  22. 22 ground A DATA CONTEXT SERVICE Joseph M. Hellerstein Sean

    Lobo Nipun Ramakrishnan Avinash Arjavalingam Vikram Sreekanti
  23. 23 Beyond Metadata Architecture Model Serving Model Debugging Parsing &

    Featurization Catalog & Discovery Wrangling Analytics & Vis Data Quality Reproducibility Scavenging and Ingestion Search & Query Versioned Storage ID & Auth ABOVEGROUND API TO APPLICATIONS UNDERGROUND API TO SERVICES METAMODEL COMMON GROUND Common Ground C: Version Graphs A: Model Graphs B: Usage Graphs The ABCs of Context Application Context: Views, models, code Behavioral Context: Data lineage & usage Change over time: Version history Awareness Community Health Metadata & Data Management Data Analysis Data Wrangling Perspectives on Data Context Wrangling Context Services Six years with Trifacta and Google Cloud Dataprep The Common Ground model, and Ground system Managing the rise of Empirical AI flor ML Lifecycle
  24. Vendor-Neutral, Unopinionated Data Context Services Beyond Metadata Architecture Model Serving

    Model Debugging Parsing & Featurization Catalog & Discovery Wrangling Analytics & Vis Data Quality Reproducibility Scavenging and Ingestion Search & Query Versioned Storage ID & Auth ABOVEGROUND API TO APPLICATIONS UNDERGROUND API TO SERVICES METAMODEL COMMON GROUND Common Ground C: Version Graphs A: Model Graphs B: Usage Graphs The ABCs of Context Application Context: Views, models, code Behavioral Context: Data lineage & usage Change over time: Version history Awareness Community Health Metadata & Data Management Data Analysis Data Wrangling
  25. 25 A Recurring Conversation with Big Data Community Metadata: 


    The last thing anybody 
 wants to work on Isn’t this just
 metadata? Community Health Metadata & Data Management Data Analysis Data Wrangling Time to Go Meta (on Use) Strata New York 2015 Grounding Big Data Strata San Jose 2016 Data Relativism Strata London Keynote 2016 https://speakerdeck.com/jhellerstein
  26. What is Metadata?

  27. Data about data This used to be so simple! But

    … schema on use One of many changes What is Metadata?
  28. Lay the groundwork for rich data context. Opportunity: A Bigger

    Context Don’t just fill a metadata- sized hole in the big data stack.
  29. What is Data Context? All the information surrounding the use

    of data.
  30. Emerging Data Context Space

  31. is Unopinionated Be conservative in what you do, be liberal

    in what you accept from others Postel’s Law
  32. The ABCs of Data Context Generated by—and useful to—many applications

    and components. Application Context: Views, models, code Behavioral Context: Data lineage & usage Change over time: Version history
  33. C: Version Graphs Common Ground: A Metamodel A: Model Graphs

    B: Usage Graphs
  34. 34 ABOVEGROUND API TO APPLICATIONS UNDERGROUND API TO SERVICES METAMODEL

    COMMON GROUND GROUND ARCHITECTURE
  35. GROUND ARCHITECTURE Model Serving Model Debugging Parsing & Featurization Catalog

    & Discovery Wrangling Analytics & Vis Data Quality Reproducibility Scavenging and Ingestion Search & Query Versioned Storage ID & Auth ABOVEGROUND API TO APPLICATIONS UNDERGROUND API TO SERVICES METAMODEL COMMON GROUND
  36. GROUND ARCHITECTURE Model Serving Model Debugging Parsing & Featurization Catalog

    & Discovery Wrangling Analytics & Vis Data Quality Reproducibility Scavenging and Ingestion Search & Query Versioned Storage ID & Auth ABOVEGROUND API TO APPLICATIONS UNDERGROUND API TO SERVICES METAMODEL COMMON GROUND
  37. Current Status Ground Server: Release v0.1.2 Java Play + PostgreSQL

    Grit: Ground over Git Bedrock Elastic coordination-free cloud storage “Anna” prototype ICDE 2018 www.ground-context.org grit
  38. 38 flor GROUNDING THE NEXT FLOWERING OF AI Rolando Garcia

    Vikram Sreekanti Dan Crankshaw Neeraja Yadwadkar Joseph Gonzalez Joseph Hellerstein Malhar Patel Sona Jeswani Eric Liu
  39. 39 ML Lifecycle Management: A Context-Rich Application Empirical AI ML

    Lifecycle Flor: Lifecycle Mgmt flor Demo ML Lifecycle Management: A Context-Rich Application Big Data Context Beyond Metadata The ABCs of Context flor Common Ground Architecture Model Serving Model Debugging Parsing & Featurization Catalog & Discovery Wrangling Analytics & Vis Data Quality Reproducibility Scavenging and Ingestion Search & Query Versioned Storage ID & Auth ABOVEGROUND API TO APPLICATIONS UNDERGROUND API TO SERVICES METAMODEL COMMON GROUND Perspectives on Data Context Wrangling Context Services Six years with Trifacta and Google Cloud Dataprep The Common Ground model, and Ground system Managing the rise of Empirical AI flor ML Lifecycle
  40. ML Lifecycle Management: A Context-Rich Application Empirical AI ML Lifecycle

    Flor: Lifecycle Mgmt flor Demo
  41. 41 A Conversation-Starter Look, AI today is more like Experimental

    Science than Engineering! Or math!
  42. 42 A Conversation-Starter OMG! Have you never read Herbert Simon?

    That is so 1995!
  43. 43 AI is more Experimental Science than Engineering Not a

    new observation
  44. 44 AI is more Experimental Science than Engineering Not a

    new observation But increasingly timely Overseen by tweakers
  45. 45 The Fourth Paradigm of Science That was for “plain

    old” science! ML & AI generate combinatorially more experiments and data A Transformed Scientific Method E HAVE TO DO BETTER AT PRODUCING TOOLS to support the whole re- search cycle—from data capture and data curation to data analysis and data visualization. Today, the tools for capturing data both at the mega-scale and at the milli-scale are just dreadful. After you have captured the data, you need to curate it before you can start doing any kind of data analysis, and we lack good tools for both data curation and data analysis. Then comes the publication of the results of your research, and the published literature is just the tip of the data iceberg. By this I mean that people collect a lot of data and then reduce this down to some number of column inches in Science or Nature—or 10 pages if it is a computer science person writing. So what I mean by data iceberg is that there is a lot of data that is collected but not curated or published in any systematic way. There are some exceptions, and I think that these cases are a good place for us to look for best practices. I will talk about how the whole process of peer review has got to change and the way in which I think it is changing and what CSTB can do to help all of us get access to our research. W 1 National Research Council, http://sites.nationalacademies.org/NRC/index.htm; Computer Science and Telecom- munications Board, http://sites.nationalacademies.org/cstb/index.htm. 2 This presentation is, poignantly, the last one posted to Jim’s Web page at Microsoft Research before he went missing at sea on January 28, 2007—http://research.microsoft.com/en-us/um/people/gray/talks/NRC-CSTB_eScience.ppt. EDITED BY TONY HEY, STEWART TANSLEY, AND KRISTIN TOLLE | Microsoft Research Based on the transcript of a talk given by Jim Gray to the NRC-CSTB1 in Mountain View, CA, on January 11, 20072 Jim Gray, 2007
  46. 46 We have to do better at producing tools to

    support the whole research cycle!
  47. 47 ML applications are tightly tied to the ML lifecycle

  48. 4 8 ML applications are tightly tied to the ML

    lifecycle
  49. 4 9 ML applications are tightly tied to the ML

    lifecycle
  50. 5 0 ML applications are tightly tied to the ML

    lifecycle
  51. 51 PROBLEM The ML lifecycle is poorly tooled

  52. 52 • How we develop pipelines • Undo/Redo in model

    design is lacking • Failure to detect poor methods • Version skew: easy for versions to diverge • How we use pipelines: their resources and products • Difficult to predict how models will affect system behavior • Changes to data may not be tracked or recorded • No record of who uses which resources and why • Disorganized models are easy to lose and hard to find The ML lifecycle is poorly tooled
  53. 53 Version Skew: easy for versions to diverge

  54. 5 4 Disorganized models are easy to lose and hard

    to find Models will likely be organized by an individual’s standard, but not by an organization’s standards. https://xkcd.com/1459/ http://dilbert.com/strip/2011-04-23
  55. 55 Failure to detect poor methods • Data dredging or

    P-hacking • Weak isolation of test data • Training on attributes that are unknown during testing time Nature, 25 May 2016 https://xkcd.com/882/
  56. 56 Goals for flor 1. Enable safe and agile exploration

    of alternative model designs 2. Passively track and sync the history and versions of a pipeline and its executions across multiple machines 3. Answer questions about the history and provenance, and procure artifacts from the versions • Approach: • Build a system to leverage widely used tools in a principled manner.
  57. 57 Flor lives Above Ground • Unlike Ground, Flor is

    “opinionated”. • Three basic subclasses of Node Artifact Literal Action
  58. 58 Flor lives Above Ground • Unlike Ground, Flor is

    “opinionated”. • Three basic subclasses of Node • And Edges to capture workflow • Artifacts to Actions, Literals to Actions, Actions to Artifacts Artifact Literal Action Artifact
  59. 59 Action Artifact Literal Artifact Flor lives Above Ground •

    Unlike Ground, Flor is “opinionated”. • Three basic subclasses of Node • And Edges to capture workflow • Artifacts to Actions, Literals to Actions, Actions to Artifacts • Versions of Artifacts/Literals/Edges • git hashes Artifact Literal Action Artifact #86a1c71bc #cde3e1c51 #cde3e1c5 #86a1c71b #86a1c71bc #cde3e1c51 #86a1c71bc #cde3e1c51
  60. 6 0 Action Artifact Literal Artifact Flor lives Above Ground

    • Unlike Ground, Flor is “opinionated”. • Three basic subclasses of Node • And Edges to capture workflow • Artifacts to Actions, Literals to Actions, Actions to Artifacts • Versions of Artifacts/Literals/Edges • git hashes • Lineage Edges: • Track ArtifactVersions generated in workflows Artifact Literal Action Artifact #86a1c71bc #cde3e1c51 #cde3e1c5 #86a1c71b #86a1c71bc #cde3e1c51 #86a1c71bc #cde3e1c51
  61. 61 USE CASE: “I have an existing pipeline, how can

    Flor help me?”
  62. 62

  63. 6 3 Taxi.ipynb run_existing_pipeline train.csv model.pkl score.txt rmse.txt

  64. 6 4 When the pipeline executes... • Data Versioning: All

    artifacts are versioned in Git, and associated with their respective experiments • New run = New commit • Metadata versioning: git history reflected in ground. ArtifactVersions autogenerated to track git commits. • Provenance: The provenance relationships between objects (artifacts or otherwise) are recorded in Ground
  65. 65 Because we version and record data context... • Materialize

    any artifact, in context • Know which artifact to materialize • Replay all previous experiments, with new data • [Opportunity] Sync local and remote versions of the pipeline, run the pipeline anywhere
  66. 6 6 USE CASE: “Ok, how can Flor help me

    refine my pipeline?”
  67. 67

  68. 6 8 3x Taxi.ipynb run_existing_pipeline train.csv num_estimators model.pkl score.txt rmse.txt

  69. 6 9 When the pipeline executes... • Data Versioning: All

    artifacts are versioned in Git, and associated with their respective experiments • Metadata versioning: git history reflected in ground. ArtifactVersions autogenerated to track git commits. • Provenance: The provenance relationships between objects (artifacts or otherwise) are recorded in Ground • Parallel multi-trial experiments • Our example (3x): num_est=15, num_est=20, num_est=30.
  70. 7 0 Because we declare and track a literal... •

    Materialize any artifact, in richer context • Know which artifact to materialize • Replay all previous experiments, with new data • [Opportunity] Sync local and remote versions of the pipeline, run the pipeline anywhere • [Opportunity] Scripting, set literal from the command line or externally
  71. 71 USE CASE: “I’ll build my next pipeline with Flor

    from the start.”
  72. 72

  73. 73

  74. 74 3x Taxi.ipynb test Taxi.ipynb train Taxi.ipynb split Taxi.ipynb preproc

    Taxi.ipynb calculate_distance Taxi.ipynb dataframize train.csv num_estimators xTrain.pkl xTest.pkl yTrain.pkl yTest.pkl train_ready.pkl train_dist_df.pkl train_df.pkl score.txt rmse.txt model.pkl
  75. 75 When the pipeline executes... • Versioning: All artifacts are

    versioned in Git, and associated with their respective experiments • New run = New commit • Provenance: The relationships between objects, artifacts or otherwise, are recorded in Ground • Parallel multi-trial experiments • Trial invariant artifacts don’t have to be recomputed
  76. 76 Because we built a pipeline with Flor... • Materialize

    any artifact, in richer context • Know which artifact to materialize • Replay all previous experiments, with new data • Share resources, with the corresponding changes • Swap components • Maintain the pipeline • [Opportunity] Inter-version Parallelism • [Opportunity] Undo/Redo
  77. We automatically track all the metadata, context, and lineage with

    • Timestamps • Which resources your experiment used • How many trials your experiment ran • What the configuration was per trial • The evolution of your experiment over time (versions) • The lineage that derived any artifact in the workflow • The metadata you need to retrieve a physical copy of any artifact in the workflow, ever • The current state of your experiment in the file system, in context • Whether you’ve forked any experiment resources, and which ones • When you executed an experiment, whether you executed it to completion, or only partially • Whether you’ve peeked at intermediary results during interactive pipeline development, and what you did in Flor after you learned this information • Whether you peek at the same result multiple times, or each time peek at a different trial and see a different result • The location of the peeked artifacts so they may be re-used in future computations without repeating work • Whether two specifications belonging to the same experiment used the same or different resources, and whether they derived the same artifacts. • Whether any resource or artifact was renamed • ….
  78. 78 CONCLUSION

  79. 79 Perspectives on Data Context Six years with Trifacta and

    Google Cloud Dataprep The Common Ground model, and Ground system Managing the rise of Empirical AI Wrangling Context Services flor ML Lifecycle
  80. METAMODEL Wrangling Context Services flor ML Lifecycle Opportunity At All

    Levels
  81. Opportunity At All Levels Application-Specific Context Generation and Mining Data

    Context Modeling Systems Infrastructure for Data Context Management
  82. 82 We have to do better at producing tools to

    support the whole research cycle! One of the most high-impact (and fun!) topics in CS today.
  83. http://ground-context.org flor http://github.com/ucbrise/jarvis Joe Hellerstein UC Berkeley / Trifacta hellerstein@berkeley.edu

    8 3 Context at Berkeley