How to scale Data Ops

Data Engineering roles have increased 10x in the past 3 years.
Their role is to provide all the teams in a company with fresh, reliable, accurate data.

Despite that, very few fundamental changes have emerged in the ways we scale Data Ops. Companies today are faced with either dealing with fragmented data, or hiring more data engineers. But it doesn't have to be that way.

This deck talks about why you need data engineering, what should be your strategy and focus, the newer ways to integrate your different data silos, where to start your data consolidation efforts, how to scale processes data engineering teams, and what it will bring to your business in the end.

Transcript

1. How to scale #Data Ops
2. - Co-founder, CEO @ Hull.io - Product guy - Advisor, Techstars Mentor - Always tried to unify all the things
 github.com/unity
 First company name... Unity ¯\_(ツ)_/¯ I’m Romain Dardour
3. ❤ Unifying Things ❤Automating Stuff
4. What is “Data Ops”
5. “The data you need
 to work in one tool, Is usually in another tool” — Romain Dardour
6. Data Ops is Infrastructure More Teams than ever More Tools than ever More Data than ever
9. Data Engineers
 10X increase
 since 2015
10. Why scale Data Ops
11. 1900-1970 Factory Era
 The one who produces the most wins 1970-2000 Mass Marketing Era
 The one who shouts the loudest wins 2000-2015 Online Marketing Era
 The one with the biggest online presence wins 2015+ Personalized Marketing Era
 The one with the best usage of data wins
12. ✅ Collect ✅ Understand Act
13. How many have you seen ?
14. ￫ Attribution ￫ Team Alignment ￫ Reporting ￫ Personalization ￫ Lead Qualification ￫ Customer success ￫ Product Qualified Leads ￫ Growth Experiments ￫ GDPR (Compliance)
15. Who scales Data Ops?
17. Data engineer ≠ Data Scientist Data Scientist Cleans, massages, organizes (big) data, performs statistics and analysis to develop insights, build models, find patterns, tell stories to stakeholders Data Engineer Develops, constructs, Tests, maintains architectures and large-scale processing systems - Acquire Data - Marry systems together - Recommend ways to Improve Data reliability
18. Growth Engineers, BizOps Teams, Demand Gen Teams - Transversal, Horizontal, Growth mindset - All purpose enablers - “Getting things done” focus Skills
 SQL & NoSQL, APIs, Python, Ruby, Javascript, Database Systems,
 Data Modeling & ETL Tools, MapReduce, Data warehousing.
19. When to Scale Data Ops
20. Manual customer data integration?
21. Automated customer data integration?
22. Engineered customer data integration?
23. Right Answer:
 When it hurts
24. CRM == Brain Intimately knowing your customers &
 everything about them. Wasn’t that fun?
25. Fits Brain & Workday ? Don’t Scale Scale Yep Nope
26. How to Scale Data Ops
27. Simpler data Complex data ?
28. What matters is how customer data is used What experiences can you create from your tools, teams & data?
29. Explore orchestration by lifecycle stage & experience OPPORTUNITY LEAD CUSTOMER
30. 5 steps to maturity Best practices from working with marketing teams
31. Warning: Politics
32. Developer Ads Analytics Sales Support Email CEO
33. Step 0. Get in a room together. No dial-ins. Everyone around a whiteboard. Morning or whole day
34. Start Small Define one Initial Goal (You’ll build more later. Walk before you run)
35. How could I send an email like this from Intercom? Hi Romain, I noticed you hadn’t got your ticket for Inbound.org for this year. Are you thinking of coming? We’ve a focus this year on in-house speakers, particularly from SaaS. Have you met Hana Abaza from Shopify? You know the deal. 3-days. Resort. Justin’s karaoke. And the weather in London’s really not great in September, so.. Let me know! Darmesh
36. Payments Industry Location Hi Romain, I noticed you hadn’t got your ticket for Inbound.org for this year. Are you thinking of coming? We’ve a focus this year on in-house speakers, particularly from SaaS. Have you met Hana Abaza from Shopify? You know the deal. 3-days. Resort. Justin’s karaoke. And the weather in London’s really not great in September, so.. Let me know! Darmesh Friends
37. #demo_requests Hana Abaza Head of Marketing Toronto $ Fit: Very Good Signals: Data Warehouse, Salesforce, $SHOP Matching tech: 8 Blog Reader for 2 months 
 Interested in data warehousing Viewed pricing 3 times this week 1 How could I send a notification like this from Slack?
38. #demo_requests Hana Abaza Head of Marketing Toronto $ Fit: Very Good Signals: Data Warehouse, Salesforce, $SHOP Matching tech: 8 Blog Reader for 2 months 
 Interested in data warehousing Viewed pricing 3 times this week 1 Tech stack Website Scoring
39. Unify 
 Customer Data
40. This is the hardest part
41. Incomplete Sources Inaccurate Sources Limited APIs Duplicate Items Legacy Data “Impedence Mismatch” between objects
 (i.e Accounts, Companies, Organizations…)
42. You must do it right. No Shortcuts. This is what gives you an edge
43. Start by unifying all profiles (Identity Resolution) Subscribed to newsletter + Chat conversation
 “Can I see a demo?”
44. You don't “integrate tools”
45. You integrate profiles between tools HubSpot Contact
 
 
 HubSpot Company Intercom User Intercom Company
46. You integrate profile fields between tools HubSpot Contact Ed Fry ed@hull.io Subscribed Head of Growth London, UK Intercom User Ed Fry ed@hull.io
47. Map all key profiles & profile data Contact Name Email Lead status Job title Location ...
48. Enrich profiles to identify ICPs + Job title + Location + 85 data points
49. Build
 Custom Logic
50. “Qualified lead” Job role = “Marketing” Employees > 30 Segment profiles
52. Which drink
 got you drunk?
53. “Standard Attribution models suck. Let’s build our own”
54. For the original source fields, we'd like the highest ranking source & details on the oldest date that any of these events occurred. If two events of the same rank occur, we want the earliest in the day. For the latest source fields, we'd like the highest ranking source & details on the most recent date that any of these events occurred. If two events of the same rank occur, we want the last in the day. Everything has the same logic on the account level, but we of course want the oldest source across all users on that account for the original source and the latest across all for the latest.
56. They built it in 1 week
57. Synchronize between tools quickly, predictably, reliably
60. Which tools to use?
61. What’s important isn’t what your data stack
 looks like. It’s how connected it is,
 and how easy it is to iterate on it. Everything is a loop around the customer profile.
62. But... Use Salesforce. Seriously
 You'll end up there eventually anyways.
63. In 2018 Customer data platforms
64. 1. Unified customer profile
65. 2.Transform raw customer data into new traits
66. 3.Query, build & update segments of customers
67. 4. Map & sync profile data to all tools (up to real-time)
68. Use different CDPs for different data models Enterprise E-Commerce Mobile B2B
69. Pick the integration method that works for you Simpler data Complex data
70. Prioritize
 Fast & Secure
 Iteration
71. Thanks romain@hull.io