Improving feature engineering in the lab and production with Ivory

Talk given by Ben Lever at Strata+Hadoop World in London, 6 May 2015:

Feature engineering is a critical and time-consuming activity in the development and deployment of any modeling pipeline. It is also exacerbated as data science teams seek to incorporate new data sources into their pipelines that are at a scale far larger than previously employed. Furthermore, the transition to production environments is littered with complexity as these pipelines are exposed to the dynamic, and fragile, world of ongoing data feeds, data corrections, and evolving data models.

In this talk we will introduce Ivory, a new open-source, Hadoop-based data store that seeks to address these challenges. Ivory is a scalable and extensible data store for storing facts and extracting features. It is optimised specifically for the feature engineering stages of modelling pipelines, simultaneously simplifying and adding rigour to them.

This session will walk through an example of how Ivory can be used in the typical data scientist’s workflow, and then how that extends to migrating pipelines into production. It will impart all of the basic concepts of Ivory such as repositories, the dictionary, its fact-based data model, and virtual features. It will also demonstrate the benefits of Ivory being an immutable data store and the unique opportunities that creates.

Transcript

1. IVORY © Ambiata 2015
2. Ben Lever CTO & co-founder at Ambiata © Ambiata 2015 @bmlever
3. IMPROVING FEATURE ENGINEERING © Ambiata 2015
4. BUILDING & DEPLOYING MODELS © Ambiata 2015
5. © Ambiata 2015
6. © Ambiata 2015
7. © Ambiata 2015 Feature preparation
8. © Ambiata 2015 Feature preparation Model training
9. © Ambiata 2015 Feature preparation Model training
10. © Ambiata 2015 Model training Features Feature preparation
11. © Ambiata 2015 0.00 3 “3001” 451.20 634.83 16 “4670” 128.22 15.12 2 - 15.45 33.56 2 - 17.12 98.34 12 “3303” 328.34 523.81 23 “2046” 63.98 1086.05 17 - 71.59 224.81 9 - 1042.43 78.21 2 “2134” 27.65 126.48 4 - 135.20 true false false false true true false false false false “M” - “F” “M” “F” - “F” “F” “M” - gender balance purchases zipcode max.spend is.new 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236
12. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “M” - “F” “M” “F” - “F” “F” “M” - “3001” “4670” - - “3303” “2046” - - “2134” -
13. 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 “M” - “F” “M” “F” - “F” “F” “M” - “3001” “4670” - - “3303” “2046” - - “2134” - INSTANCE
14. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “M” - “F” “M” “F” - “F” “F” “M” - “3001” “4670” - - “3303” “2046” - - “2134” - FEATURE VECTOR
15. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 “M” - “F” “M” “F” - “F” “F” “M” - gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “3001” “4670” - - “3303” “2046” - - “2134” -
16. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new FEATURE “3001” “4670” - - “3303” “2046” - - “2134” - “M” - “F” “M” “F” - “F” “F” “M” -
17. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “M” - “F” “M” “F” - “F” “F” “M” - “3001” “4670” - - “3303” “2046” - - “2134” -
18. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “M” - “F” “M” “F” - “F” “F” “M” - “3001” “4670” - - “3303” “2046” - - “2134” -
19. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “3001” “4670” - - “3303” “2046” - - “2134” - “M” - “F” “M” “F” - “F” “F” “M” -
20. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “M” - “F” “M” “F” - “F” “F” “M” - “3001” “4670” - - “3303” “2046” - - “2134” -
21. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “M” - “F” “M” “F” - “F” “F” “M” - “3001” “4670” - - “3303” “2046” - - “2134” -
22. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 M - F M F - F F M - gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 “M” - “F” “M” “F” - “F” “F” “M” - 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “3001” “4670” - - “3303” “2046” - - “2134” -
23. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 “M” - “F” “M” “F” - “F” “F” “M” - 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “3001” “4670” - - “3303” “2046” - - “2134” -
24. © Ambiata 2015 Model training Feature preparation Features
25. © Ambiata 2015 Feature preparation
26. © Ambiata 2015
27. © Ambiata 2015 Feature preparation
28. © Ambiata 2015 Feature preparation
29. © Ambiata 2015 Feature preparation
30. © Ambiata 2015 Feature preparation
31. © Ambiata 2015 Feature preparation
32. © Ambiata 2015 Feature preparation
33. © Ambiata 2015 Feature preparation Model training
34. © Ambiata 2015 Model training Feature preparation Features
35. © Ambiata 2015 Model training Data sources Feature preparation Features
36. © Ambiata 2015
37. © Ambiata 2015
38. © Ambiata 2015
39. © Ambiata 2015
40. © Ambiata 2015 DIFFERENT FORMATS
41. © Ambiata 2015 DIFFERENT FORMATS DIFFERENT SCALES
42. © Ambiata 2015 DIFFERENT FORMATS DIFFERENT SCALES DIFFERENT STRUCTURES
43. © Ambiata 2015 Model training Feature preparation Data sources Features
44. © Ambiata 2015 Model training
45. © Ambiata 2015 Model training
46. © Ambiata 2015 Model training
47. © Ambiata 2015 Model training
48. © Ambiata 2015 Model training
49. © Ambiata 2015
50. © Ambiata 2015 Subscription Transactions Support queries
51. © Ambiata 2015 Subscription Transactions Support queries
52. © Ambiata 2015 Subscription Transactions Support queries
53. © Ambiata 2015 Subscription Transactions Support queries
54. © Ambiata 2015 Subscription Transactions Support queries
55. © Ambiata 2015 Subscription Transactions Support queries
56. © Ambiata 2015 Trial Paid Cancelled Subscription Transactions Support queries
57. © Ambiata 2015 Subscription Trial Paid Cancelled Transactions Support queries
58. © Ambiata 2015 Subscription Trial Paid Cancelled Transactions Support queries
59. © Ambiata 2015 Subscription Trial Paid Cancelled Transactions Support queries
60. © Ambiata 2015 Subscription Trial Paid Cancelled Transactions Support queries 2 w
61. © Ambiata 2015 Subscription Trial Paid Cancelled Transactions Support queries 2014-01-23
62. © Ambiata 2015 Subscription Trial Paid Cancelled Support queries 2014-01-23 Transactions
63. © Ambiata 2015 Subscription Trial Paid Cancelled Support queries 2013-11-09 Transactions
64. © Ambiata 2015 Subscription Trial Paid Cancelled Support queries 2014-05-17 Transactions
65. © Ambiata 2015 Model training
66. © Ambiata 2015 Model training Model scoring
67. © Ambiata 2015 Subscription Support queries Transactions
68. © Ambiata 2015 Subscription Trial Support queries Transactions
69. © Ambiata 2015 Subscription Trial Support queries Transactions
70. © Ambiata 2015 Subscription Trial Support queries Transactions
71. © Ambiata 2015 Subscription Trial Support queries Transactions
72. © Ambiata 2015 Subscription Trial Support queries Transactions
73. © Ambiata 2015 Subscription Trial Paid Support queries Transactions
74. © Ambiata 2015 Subscription Trial Paid Support queries Transactions
75. © Ambiata 2015 Subscription Trial Paid Support queries Transactions
76. © Ambiata 2015 Subscription Trial Paid Cancelled Support queries Transactions
77. © Ambiata 2015 Subscription Trial Paid Cancelled Support queries Transactions
78. © Ambiata 2015 Subscription Trial Paid Cancelled Transactions Support queries
79. © Ambiata 2015 Subscription Trial Paid Cancelled Transactions Support queries CONTINUAL DATA GROWTH
80. © Ambiata 2015 Subscription Trial Paid Cancelled Transactions Support queries CONTINUAL DATA GROWTH CONSTANT TIME FEATURE PREPARATION
81. © Ambiata 2015 Model training Model scoring Feature preparation
82. © Ambiata 2015 Model training LAB Feature preparation
83. © Ambiata 2015 Model training FACTORY Feature preparation
84. © Ambiata 2015 Lab Factory
85. © Ambiata 2015 Lab Once-off data set Factory Ongoing data feeds
86. © Ambiata 2015 Lab Once-off data set If it’s wrong, start again Factory Ongoing data feeds Fix broken data on the fly
87. © Ambiata 2015 Lab Once-off data set If it’s wrong, start again Variable time Factory Ongoing data feeds Fix broken data on the fly Constant time
88. © Ambiata 2015 Feature preparation Model training
89. © Ambiata 2015 Feature preparation Model training Feature preparation Model training
90. © Ambiata 2015 Feature preparation Model training Feature preparation Model training Feature preparation Model training
91. Feature preparation Modeling 85% 15% © Ambiata 2015 … optimistically
92. © Ambiata 2015
93. © Ambiata 2015 Model training Model scoring Feature preparation
94. © Ambiata 2015
95. © Ambiata 2015 Receive data every day
96. © Ambiata 2015 Receive data every day Prepare features every day
97. © Ambiata 2015 Receive data every day Batch score models every day Prepare features every day
98. © Ambiata 2015 Receive data every day Batch score models every day Prepare features every day x N
99. © Ambiata 2015 Feature preparation Model training
100. © Ambiata 2015 Model training
101. © Ambiata 2015 IVORY Model training
102. IVORY © Ambiata 2015
103. IVORY © Ambiata 2015 A Data Warehouse for Data Science
104. IVORY © Ambiata 2015 Apache V2 Licence
105. © Ambiata 2015
106. A scalable and extensible data store for storing facts and extracting features © Ambiata 2015
107. A scalable and extensible data store for storing facts and extracting features © Ambiata 2015
108. Facts: immutable, typed values keyed along 3 dimensions © Ambiata 2015
109. © Ambiata 2015 v Facts: immutable, typed values keyed along 3 dimensions
110. © Ambiata 2015 v Values are structured: primitives structs list of values Facts: immutable, typed values keyed along 3 dimensions
111. © Ambiata 2015 entity v Like the primary key of a DB row: customer ID account ID user ID Facts: immutable, typed values keyed along 3 dimensions
112. © Ambiata 2015 entity attribute v Like the name of a DB column: gender purchases zipcode balance Facts: immutable, typed values keyed along 3 dimensions
113. © Ambiata 2015 entity attribute time v When the value is valid from Facts: immutable, typed values keyed along 3 dimensions
114. © Ambiata 2015 entity attribute time v Facts: immutable, typed values keyed along 3 dimensions
115. © Ambiata 2015 entity attribute time v E-A-V-T Dimensions are unbounded Unordered Facts: immutable, typed values keyed along 3 dimensions
116. © Ambiata 2015 E A V T
117. © Ambiata 2015 48149407 gender 2007-04-01 “F” E A V T
118. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” E A V T
119. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T
120. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T
121. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F”
122. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M”
123. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M”
124. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M”
125. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M” 48149407 zipcode 2015-03-19 “5192”
126. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M” 48149407 zipcode 2015-03-19 “5192”
127. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M” 48149407 zipcode 2015-03-19 “5192” 62948721 balance 2014-02-01 3478.23
128. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M” 48149407 zipcode 2015-03-19 “5192” 62948721 balance 2014-02-01 3478.23 62948721 balance 2013-02-01 12099.21
129. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M” 48149407 zipcode 2015-03-19 “5192” 62948721 balance 2014-02-01 3478.23 62948721 balance 2013-02-01 12099.21
130. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M” 48149407 zipcode 2015-03-19 “5192” 62948721 balance 2014-02-01 3478.23 62948721 balance 2013-02-01 12099.21 48149407 has.children 2015-02-17 true
131. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 balance 2014-02-04 46.54 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M” 48149407 zipcode 2015-03-19 “5192” 62948721 balance 2014-02-01 3478.23 62948721 balance 2013-02-01 12099.21 48149407 has.children 2015-02-17 true
132. A scalable and extensible data store for storing facts and extracting features © Ambiata 2015
133. A scalable and extensible data store for storing facts and extracting features © Ambiata 2015
134. © Ambiata 2015 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 purchases 2014-02-04 4 E A V T 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M” 48149407 zipcode 2015-03-19 “5192” 62948721 balance 2014-02-01 3478.23 62948721 balance 2013-02-01 12099.21 48149407 has.children 2015-02-17 true
135. © Ambiata 2015 gender balance purchases zipcode max.spend is.new 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 “M” - “F” “M” “F” - “F” “F” “M” - 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false “3001” “4670” - - “3303” “2046” - - “2134” -
136. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 “M” - “F” “M” “F” - “F” “F” “M” - 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false “3001” “4670” - - “3303” “2046” - - “2134” - max.spend is.new
137. © Ambiata 2015 0.00 3 3001 1.00 634.83 16 4670 0.6875 15.12 2 - 0.50 33.56 2 - 1.00 98.34 12 3303 0.8333 523.81 23 2046 0.4782 1086.05 17 - 1.00 224.81 9 - 0.2222 78.21 2 2134 0.50 126.48 4 - 0.0 1 3 1 1 4 1 2 1 1 1 M - F M F - F F M - gender balance purchases zipcode max.spend is.new 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 “M” - “F” “M” “F” - “F” “F” “M” - 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false “3001” “4670” - - “3303” “2046” - - “2134” -
138. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 “M” - “F” “M” “F” - “F” “F” “M” - 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false “3001” “4670” - - “3303” “2046” - - “2134” - max.spend is.new
139. © Ambiata 2015
140. © Ambiata 2015 gender balance purchases zipcode entity max.spend is.new 0.00 3 3001 1.00 634.83 16 4670 0.6875 15.12 2 - 0.50 33.56 2 - 1.00 98.34 12 3303 0.8333 523.81 23 2046 0.4782 1086.05 17 - 1.00 224.81 9 - 0.2222 78.21 2 2134 0.50 126.48 4 - 0.0 1 3 1 1 4 1 2 1 1 1 M - F M F - F F M - 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 “M” - “F” “M” “F” - “F” “F” “M” - 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false “3001” “4670” - - “3303” “2046” - - “2134” -
141. © Ambiata 2015 gender balance purchases zipcode max.spend is.new entity @ time 0.00 3 3001 1.00 634.83 16 4670 0.6875 15.12 2 - 0.50 33.56 2 - 1.00 98.34 12 3303 0.8333 523.81 23 2046 0.4782 1086.05 17 - 1.00 224.81 9 - 0.2222 78.21 2 2134 0.50 126.48 4 - 0.0 1 3 1 1 4 1 2 1 1 1 M - F M F - F F M - 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 “M” - “F” “M” “F” - “F” “F” “M” - 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false “3001” “4670” - - “3303” “2046” - - “2134” -
142. © Ambiata 2015 Subscription Trial Paid Cancelled Transactions Support queries
143. © Ambiata 2015 Subscription Trial Paid Cancelled Support queries Transactions
144. © Ambiata 2015 Subscription Trial Paid Cancelled Support queries 2015-04-03 Transactions
145. © Ambiata 2015 gender balance purchases zipcode max.spend is.new 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236
146. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 @ @ @ @ @ @ @ @ @ @ max.spend is.new
147. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 2015-04-03 @ @ @ @ @ @ @ @ @ @ Same times max.spend is.new
148. © Ambiata 2015 Snapshot
149. © Ambiata 2015 Snapshot All entities
150. © Ambiata 2015 Snapshot All entities Same time for all entities
151. © Ambiata 2015 Snapshot All entities Same time for all entities Features for model scoring
152. © Ambiata 2015
153. © Ambiata 2015 Subscription Trial Paid Cancelled 2014-05-17
154. © Ambiata 2015 Trial Paid Cancelled 89340218 2014-05-17
155. © Ambiata 2015 Trial Paid Cancelled 89340218 48149407 Trial Paid 2014-05-17
156. © Ambiata 2015 Trial Paid Cancelled 89340218 48149407 18452274 Trial Paid Paid 2014-05-17
157. © Ambiata 2015 Trial Paid Cancelled 89340218 48149407 18452274 Trial Paid Paid 2014-05-17 2014-01-23 2014-09-24
158. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 max.spend is.new
159. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new
160. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ Different times max.spend is.new
161. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ Same entity, different times max.spend is.new
162. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ Same entity, different times max.spend is.new
163. © Ambiata 2015 Snapshot Chord All entities Same time for all entities Features for model scoring
164. © Ambiata 2015 Snapshot Chord All entities Specific entities Same time for all entities Features for model scoring
165. © Ambiata 2015 Snapshot Chord All entities Specific entities Same time for all entities Different times for different entities Features for model scoring
166. © Ambiata 2015 Snapshot Chord All entities Specific entities Same time for all entities Different times for different entities Features for model scoring Features for model training
167. © Ambiata 2015 Snapshot Chord All entities Specific entities Same time for all entities Different times for different entities Features for model scoring Features for model training
168. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new
169. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new
170. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new
171. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new
172. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new
173. 48149407 gender 2007-04-01 “F” 48149407 zipcode 2011-04-01 “2134” 48149407 purchases 2014-02-04 4 62948721 gender 2013-03-14 “F” 93754723 gender 2013-09-27 “M” 48149407 zipcode 2015-03-19 “5192” 62948721 balance 2014-02-01 3478.23 62948721 balance 2013-02-01 12099.21 48149407 has.children 2015-02-17 true © Ambiata 2015 E A V T
174. © Ambiata 2015 E A T V
175. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new
176. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new
177. © Ambiata 2015 zipcode “3001” “2046” “2045”
178. © Ambiata 2015 zipcode “3001” “2046” “2045” 2013-04-03
179. © Ambiata 2015 zipcode “3001” “2046” “2045” 2013-04-03
180. © Ambiata 2015 “3001” gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new
181. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new “3001”
182. © Ambiata 2015 “3001” gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ 0.00 “M” max.spend is.new
183. © Ambiata 2015 0.00 “3001” 634.83 “4670” 15.12 33.56 98.34 “3303” 523.81 “2046” 1086.05 224.81 78.21 “2134” 126.48 “M” “F” “M” “F” “F” “F” “M” gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new
184. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new 0.00 “3001” 634.83 “4670” 15.12 - 33.56 - 98.34 “3303” 523.81 “2046” 1086.05 - 224.81 - 78.21 “2134” 126.48 - “M” - “F” “M” “F” - “F” “F” “M” -
185. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new 0.00 “3001” 634.83 “4670” 15.12 - 33.56 - 98.34 “3303” 523.81 “2046” 1086.05 - 224.81 - 78.21 “2134” 126.48 - “M” - “F” “M” “F” - “F” “F” “M” -
186. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ Base attributes max.spend is.new 0.00 “3001” 634.83 “4670” 15.12 - 33.56 - 98.34 “3303” 523.81 “2046” 1086.05 - 224.81 - 78.21 “2134” 126.48 - “M” - “F” “M” “F” - “F” “F” “M” -
187. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ Base features max.spend is.new 0.00 “3001” 634.83 “4670” 15.12 - 33.56 - 98.34 “3303” 523.81 “2046” 1086.05 - 224.81 - 78.21 “2134” 126.48 - “M” - “F” “M” “F” - “F” “F” “M” -
188. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new 0.00 “3001” 634.83 “4670” 15.12 - 33.56 - 98.34 “3303” 523.81 “2046” 1086.05 - 224.81 - 78.21 “2134” 126.48 - “M” - “F” “M” “F” - “F” “F” “M” -
189. © Ambiata 2015 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ max.spend is.new 0.00 “3001” 634.83 “4670” 15.12 - 33.56 - 98.34 “3303” 523.81 “2046” 1086.05 - 224.81 - 78.21 “2134” 126.48 - “M” - “F” “M” “F” - “F” “F” “M” -
190. © Ambiata 2015 zipcode “3001” “2046” “2045”
191. © Ambiata 2015 zipcode “3001” “2046” “2045” transaction
192. © Ambiata 2015 zipcode “3001” “2046” “2045” transaction 2013-04-03
193. © Ambiata 2015 zipcode “3001” “2046” “2045” transaction 2013-04-03 count 2w
194. © Ambiata 2015 zipcode “3001” “2046” “2045” transaction 2013-04-03 count 4w
195. © Ambiata 2015 zipcode “3001” “2046” “2045” transaction 2013-04-03 max 4w
196. © Ambiata 2015 gender balance purchases zipcode max.spend is.new 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ 3 0.00 “3001” 634.83 “4670” 15.12 - 33.56 - 98.34 “3303” 523.81 “2046” 1086.05 - 224.81 - 78.21 “2134” 126.48 - “M” - “F” “M” “F” - “F” “F” “M” -
197. © Ambiata 2015 gender balance purchases zipcode max.spend is.new 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ 3 0.00 “3001” 634.83 “4670” 15.12 - 33.56 - 98.34 “3303” 523.81 “2046” 1086.05 - 224.81 - 78.21 “2134” 126.48 - “M” - “F” “M” “F” - “F” “F” “M” - 451.20 true
198. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “3001” “4670” - - “3303” “2046” - - “2134” - “M” - “F” “M” “F” - “F” “F” “M” -
199. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false max.spend is.new “3001” “4670” - - “3303” “2046” - - “2134” - “M” - “F” “M” “F” - “F” “F” “M” -
200. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode max.spend is.new 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ Virtual attributes 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false “3001” “4670” - - “3303” “2046” - - “2134” - “M” - “F” “M” “F” - “F” “F” “M” -
201. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode max.spend is.new 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ Virtual features 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false “3001” “4670” - - “3303” “2046” - - “2134” - “M” - “F” “M” “F” - “F” “F” “M” -
202. © Ambiata 2015 0.00 3 634.83 16 15.12 2 33.56 2 98.34 12 523.81 23 1086.05 17 224.81 9 78.21 2 126.48 4 gender balance purchases zipcode max.spend is.new 89340218 48149407 18452274 07499337 62948721 48149407 00272446 13374497 62948721 46474236 2015-04-03 2014-02-01 2013-04-03 2015-01-13 2013-10-26 2014-12-14 2014-11-22 2013-09-17 2014-10-30 2015-08-27 @ @ @ @ @ @ @ @ @ @ 451.20 128.22 15.45 17.12 328.34 63.98 71.59 1042.43 27.65 135.20 true false false false true true false false false false “3001” “4670” - - “3303” “2046” - - “2134” - “M” - “F” “M” “F” - “F” “F” “M” -
203. © Ambiata 2015 gender balance purchases zipcode max.spend is.new
204. © Ambiata 2015 gender balance purchases zipcode max.spend is.new Attribute Dictionary
205. © Ambiata 2015 gender balance zipcode purchases max.spend is.new
206. © Ambiata 2015 gender balance zipcode purchases max.spend is.new name
207. © Ambiata 2015 gender balance zipcode purchases max.spend is.new name string double string encoding
208. © Ambiata 2015 gender balance zipcode purchases max.spend is.new name string double string encoding transaction {item:string, amnt:double}
209. © Ambiata 2015 gender balance zipcode purchases max.spend is.new name string double string encoding transaction {item:string, amnt:double} transaction 4w count source window expression
210. © Ambiata 2015 gender balance zipcode purchases max.spend is.new name string double string encoding transaction {item:string, amnt:double} transaction 4w count transaction 4w max(amnt) source window expression
211. © Ambiata 2015 gender balance zipcode purchases max.spend is.new name string double string encoding transaction {item:string, amnt:double} transaction 4w count transaction 4w max(amnt) source window expression support_query {type:string, rating:int}
212. © Ambiata 2015 gender balance zipcode purchases max.spend is.new name string double string encoding transaction {item:string, amnt:double} transaction 4w count transaction 4w max(amnt) source window expression support_query {type:string, rating:int} query 12w count > 0
213. Ivory data store © Ambiata 2015
214. Ivory data store © Ambiata 2015 Import dictionary
215. Ivory data store © Ambiata 2015 Ingest facts Import dictionary
216. Ivory data store Ingest facts © Ambiata 2015 Import dictionary Extract features
217. Ivory data store Ingest facts © Ambiata 2015 Import dictionary Snapshot
218. Ivory data store Ingest facts © Ambiata 2015 Import dictionary Chord
219. Ivory repository Ingest facts © Ambiata 2015 Import dictionary Chord
220. Ivory repository © Ambiata 2015
221. © Ambiata 2015 Ivory repository
222. Ivory repository © Ambiata 2015 Hadoop
223. /my-repo © Ambiata 2015 Hadoop
224. /my-repo © Ambiata 2015 HDFS / S3
225. /my-repo © Ambiata 2015 MapReduce
226. /my-repo © Ambiata 2015 Hadoop
227. /my-repo © Ambiata 2015 Ivory CLI Hadoop “Client”
228. /my-repo © Ambiata 2015 Ivory CLI Hadoop “Client” > ivory --help
229. © Ambiata 2015
230. © Ambiata 2015 > ivory create-repository /my-repo
231. © Ambiata 2015 > ivory create-repository /my-repo > ivory import-dictionary ...
232. © Ambiata 2015 > ivory create-repository /my-repo > ivory import-dictionary ... > ivory ingest ...
233. © Ambiata 2015 > ivory create-repository /my-repo > ivory import-dictionary ... > ivory ingest ... > ivory snapshot ...
234. © Ambiata 2015 > ivory create-repository /my-repo > ivory import-dictionary ... > ivory ingest ... > ivory snapshot ... > ivory chord ...
235. © Ambiata 2015
236. © Ambiata 2015
237. create © Ambiata 2015
238. © Ambiata 2015 import dictionary
239. © Ambiata 2015 ingest
240. © Ambiata 2015 ingest chord
241. © Ambiata 2015 ingest chord train model
242. © Ambiata 2015
243. © Ambiata 2015 ingest
244. © Ambiata 2015 snapshot ingest
245. © Ambiata 2015 snapshot ingest score model
246. © Ambiata 2015
247. © Ambiata 2015 ingest
248. © Ambiata 2015 ingest snapshot
249. © Ambiata 2015
250. © Ambiata 2015 import dictionary
251. © Ambiata 2015 ingest
252. © Ambiata 2015 ingest snapshot
253. © Ambiata 2015
254. © Ambiata 2015 ingest
255. © Ambiata 2015
256. © Ambiata 2015 1 2 3 0 4 5 6 7 8 9 10 11 12 13 14 15
257. © Ambiata 2015 1 2 3 0 4 5 6 7 8 9 10 11 12 13 14 15 snapshot
258. © Ambiata 2015 1 2 3 0 4 5 6 7 8 9 10 11 12 13 14 15 chord
259. © Ambiata 2015 1 2 3 0 4 5 6 7 8 9 10 11 12 13 14 15
260. © Ambiata 2015
261. PROJECT STATUS © Ambiata 2015
262. © Ambiata 2015 Ivory in production for ~18 months
263. © Ambiata 2015 Ivory in production for ~18 months Ambiata hosted repositories (AWS)
264. © Ambiata 2015 Ivory in production for ~18 months Ambiata hosted repositories (AWS) Self-hosted repositories (on-prem)
265. Apache V2 Licence github.com/ambiata/ivory © Ambiata 2015
266. © Ambiata 2015
267. © Ambiata 2015 Documentation
268. © Ambiata 2015 Feature expression language Ivory “SQL” (Icicle) Documentation
269. Apache V2 Licence github.com/ambiata/icicle © Ambiata 2015
270. THANK YOU © Ambiata 2015 @bmlever github.com/ambiata/ivory