Why do 87% of data science projects never make it into production? • Lack of leadership support • Siloed organizations don’t foster collaboration • Ownership • Need to educate business leaders “AI is not going to replace managers, but managers who use AI are going to replace those who don’t.”
be market focused, and should ask the question “What is our business and what should it be?” True innovation is rarely an extension of an already existing business practice. - Peter Drucker
• Plan outcome-based interviews with various departments around their goals • Use these to build collaborative projects, understand problem domain better • Build buy-in on different levels ▪ As team leaders, middle managers are at the intersection of the vertical and horizontal flows of information in the company. They serve as a bridge between the visionary ideals of the top and the often chaotic market reality of those on the front line of the business. ▪ There may be a strong desire to bring about change, but reality on the ground might make it difficult to happen. Prioritize effectively, make sure you can have allies on the ground.
process, get involved in theirs • Deliver prototypes, demos • Build interest, excitement ◦ Who cares? ◦ Who objects? ◦ Who is affected? ◦ How do decisions get made? • Find common goals • Help people connect dots, don’t confuse them with ML jargon
fragmentation • Vendor proliferation (difficulty in consistency of insights) • Change perceived as buying new products or moving to the cloud • Lacking standardization of data, KPI definitions • Lacking platform approach to data, multiple non-standard microservices • Attempts at broad sweeping transformations rather than business value centric
All complex systems that work evolved from simpler systems that worked. If you want to build a complex system that works, build a simpler system first, and then improve it over time. Axiom 2: It takes a village to raise a child [model]. --- Heuristic: Start a model family off solving new problems whenever possible. A model needs a village that wants to raise it, enough time & resources to play till it’s mature enough to handle complex problems. This requires more time and effort for well established problems rather than for new ones.
◦ Features reinforce biases • Emergence • Long term effects - deviation and drift • ML Debt ◦ Entanglement ◦ Dependencies ◦ Feature erosion • Measurements may not generalize
big engineering costs. • Validity of proxy metrics needs to be established • Heterogeneity in treatment effects • Market effects: ensure similarity in market conditions between variants • Interaction effects • Learning about a model’s ability to learn quickly, not long term convergence
can be safely funded. Requires good culture to be successful. Recommendations to an existing business solution. Where bulk of research is invested. Marry with continuous delivery. Consistency of delivery has higher reward than incremental innovation. Insights Free-form Research Leverage existing ML Products ML Research 04 02 01 03 High Urgency Low Urgency High Importance Low Importance