Driving Innovation In Retail with Machine Learning Nishan Subedi Head Of Algorithms | VP Technology Overstock, Inc.

Agenda: 1. Retail landscape 2. What is Innovation? 3. Collaboration critical to foster innovation 4. Challenges to ML adoption 5. ML practices 6. Decision matrix

Overstock.com

Retail is extremely competitive E-commerce trends 2020 | SearchNode

Retail Business Practices: Search, browse, personalization Storefront, discovery Emails, Acquisition, Marketing SEO, Paid keyword, Listing Ads Manufacturing, operations, customer service, shipping, delivery Supply Chain, Manufacturing Pricing, Promotions Curation, quality, navigation, Assortment

Lend themselves to well studied ML solutions ● Synthesize user engagement to aid better discovery ● Supply chain efficiencies ● Personalization ● Information Retrieval ● Marketplace dynamics ● Revenue, profit maximization

Yet we still struggle to create value from AI efforts Why do 87% of data science projects never make it into production? ● Lack of leadership support ● Siloed organizations don’t foster collaboration ● Ownership ● Need to educate business leaders “AI is not going to replace managers, but managers who use AI are going to replace those who don’t.”

Nature of Innovation Innovation in a business enterprise must always be market focused, and should ask the question “What is our business and what should it be?” True innovation is rarely an extension of an already existing business practice. - Peter Drucker

Collaboration / Buy In ● Embrace company goals & initiatives ● Plan outcome-based interviews with various departments around their goals ● Use these to build collaborative projects, understand problem domain better ● Build buy-in on different levels ■ As team leaders, middle managers are at the intersection of the vertical and horizontal flows of information in the company. They serve as a bridge between the visionary ideals of the top and the often chaotic market reality of those on the front line of the business. ■ There may be a strong desire to bring about change, but reality on the ground might make it difficult to happen. Prioritize effectively, make sure you can have allies on the ground.

Collaboration / Buy In ● Involve business partners in planning process, get involved in theirs ● Deliver prototypes, demos ● Build interest, excitement ○ Who cares? ○ Who objects? ○ Who is affected? ○ How do decisions get made? ● Find common goals ● Help people connect dots, don’t confuse them with ML jargon

System level challenges / data challenges ● Data silos & fragmentation ● Vendor proliferation (difficulty in consistency of insights) ● Change perceived as buying new products or moving to the cloud ● Lacking standardization of data, KPI definitions ● Lacking platform approach to data, multiple non-standard microservices ● Attempts at broad sweeping transformations rather than business value centric

Cultural Challenges ● Develop ML center of excellence ● Setting expectations ○ Delivery KPIs ○ ML myth level-setting ● Culture of experimentation ○ Multiple iterations on same product ○ Tests for learning ● Accountability setting ○ Product ownership ○ Manage data, production

ML Product Lifecycle

ML Product = Data Science + Engineering [Creation + Delivery] of Value

Machine Learning Productionization ● Production standards ● Training, experimentation platform ● Workflow management ● Visualization ○ Online metrics ○ Offline metrics ○ Operational metrics ○ Business metrics

Nurturing an infant model to maturity Axiom 1, Gall’s Law: All complex systems that work evolved from simpler systems that worked. If you want to build a complex system that works, build a simpler system first, and then improve it over time. Axiom 2: It takes a village to raise a child [model]. --- Heuristic: Start a model family off solving new problems whenever possible. A model needs a village that wants to raise it, enough time & resources to play till it’s mature enough to handle complex problems. This requires more time and effort for well established problems rather than for new ones.

Complexity: ● Feedback & bias ○ Positional & presentation bias ○ Features reinforce biases ● Emergence ● Long term effects - deviation and drift ● ML Debt ○ Entanglement ○ Dependencies ○ Feature erosion ● Measurements may not generalize

Experimentation and measurement challenges: ● Long-term experiments and holdouts have big engineering costs. ● Validity of proxy metrics needs to be established ● Heterogeneity in treatment effects ● Market effects: ensure similarity in market conditions between variants ● Interaction effects ● Learning about a model’s ability to learn quickly, not long term convergence

ML Decision Matrix Very high cost, exponential payoff possible if can be safely funded. Requires good culture to be successful. Recommendations to an existing business solution. Where bulk of research is invested. Marry with continuous delivery. Consistency of delivery has higher reward than incremental innovation. Insights Free-form Research Leverage existing ML Products ML Research 04 02 01 03 High Urgency Low Urgency High Importance Low Importance

Thank you! https://www.overstock.com/careers