data management and processing systems that are reliable, powerful and agile. ▪ We use open frameworks & patterns rather than a single vendor's product. Our approach is tailored to your needs, allowing you to build on what you have, improve incrementally, and take ownership of the solution. ▪ Our methods enable automated audit and facilitate rapid, agile development of data pipelines and analytics using metadata-based data governance.
experience ranging from G7 governments to leading tech companies in the Silicon Valley ▪ We’ve watched the growing drive for data-driven decision making and advanced analytics … and often seen it flounder in the “data swamps” – manual or undocumented processes, data integration challenges, poor data modelling …
for Two Years”* – Sep 2016, Wallstreet Journal Big ad buyers and marketers are upset with Facebook Inc. after learning the tech giant vastly overestimated average viewing time for video ads on its platform for two years … the earlier counting method likely overestimated average time spent watching videos by between 60% and 80% … [this] is an embarrassment for Facebook. … Publicis [$77bn in ad buys a year] said, “... Two years of reporting inflated performance numbers is unacceptable.” * https://www.wsj.com/articles/facebook-overestimated-key-video-metric-for-two-years-1474586951
are supposed to be tech leaders and they can’t get this right ▪ Diminished trust If this is wrong what else is wrong? ▪ Internal costs Embarrassment, scandal management, fixing the problem (and what about their own internal KPIs) https://www.bloomberg.com/news/articles/2016-09-23/ facebook-says-it-gave-advertisers-inflated-video-view-metrics
(time spent watching videos) is likely the result of tens or hundreds of processing steps that convert raw platform usage data into a final metric ▪ When not properly managed, complex data pipelines like this offer multiple opportunities for error: Misunderstanding between business and tech teams as to what should be measured. For example, for FB the problem was including only video views longer than 3s in a metric advertisers assumed included all views. Errors in manually coded computation steps – whilst the whole process is automated, each step will involve manually produced code subject to human error. ▪ Organizations face huge pressure to move quickly and adapt fast. New metrics and KPIs are added quickly for both internal and external use. Often speed comes at the cost of reliability & quality. “Move fast and break things” Facebook is a leading tech organization … how did this happen?
in Excel or data warehouse hell? Need to integrate data from inside and outside the enterprise whilst maintaining quality and accountability? Are your data assets underutilized? Is there redundancy? Do you want to mix and match technologies or avoid lock-in to a single vendor or the latest fad?
number you think you are looking at? Where did it come from and who “owns” its definition? Is it being computed correctly? Budget v s. Actua l ( x1000) $0.00 $125.00 $250.00 $375.00 $500.00 A B C D E F G H I Budget Actual ? → → → →
lake Standardized External Data (dimensions) external data streams internal data streams mirrors/scrapes/snapshots normalized and standardized projection of data warehouse business user automated decision making ad-hoc data exploration metadata definition, repeated refinement application quality and health measurement integration – requires proper modeling standards and conformance providers/ market place 1 2 3 4 5 6 7 8 might be materialized/ persisted X X X X X X quality indicators
data lakes – we don’t think you should either! Data lakes quickly become data swamps. A data lake is a symptom of an underlying business challenge (rapid arrival of unorganized data) rather than a desired outcome. In our experience, to become useful a data lake has to be supplemented with a proper data warehouse/factory – where its data can be purified and distilled into reliable, transparent and discoverable data.
data integration ▪ Data model assessment and development ▪ Data conformance assessment and development ▪ Metrics audit / assessment We are experts in the design and implementation of data governance and management systems.
analytical data Produce metrics and insights faster in automated way using metadata. Assure consistency. ▪ Data governance strategy assessment and development ▪ Master data repository design and development ▪ Data maturity assessment and enhancement
make your data manageable again ▪ Metrics and dimensions assessment and catalog development for quality assurance, provenance and data discoverability ▪ Data redundancy and consistency assessment ▪ Robust data quality indicators
of the metrics? How is record-level security assured? ▪ If the metrics are not accessible through reports, what are the other ways one can get into metrics that they are not supposed to see? What are the possible untrusted access points? ▪ How is data modification (write) access controlled? Who is allowed to create or drop tables? Who is allowed to load data? We have a strong focus on security. Our approach addresses common security issues in data processing such as:
Focus on business opportunities and challenges – not tech Focus on scaling against business complexity Optimization through relevancy, traceability and reducing duplication ▪ Holistic approach – considering whole ecosystem and its business context ▪ Non-invasive – we don't replace your data warehouse ▪ Experts in integrating external data including open data
rufuspollock
kentarofujii
teamlab
carta_engineering
riyaamemiya
shogo4131
gionn
puhitaku
yahonda
pfn
satomasahiro2005
jozono
ishikawa_satoru
.jpg){kind=avatar}
cargoodrich
mojombo
mclloyd
tamaranovitovic
cassininazir
aleyda
samanthasiow
marketingsoph
ryanjones
yeseniaperezcruz
letsgokoyo
samtorres
![Cleaning up data lakes [swamps] Reliable, transparent, discoverable data data](https://files.speakerdeck.com/presentations/02b8eb4b06004fc7ae702abb8d3a90d4/slide_30.jpg){kind=link}
