Is Your AI Feature Actually Helping?

Transcript

1. Jill Cates · Senior Data Engineer, Shopify AI Council SF 2026 Is Your AI Feature
 Actually Helping? Building an attribution system for AI
2. Merchant Onboarding • Examples of milestones: • First product created • First theme edit • Store unlock, etc. • We can't tell whether the merchant did it or AI did it for them
3. Metrics Ladder Retention Attribution Acceptance caused lift Usage survived missing layer clicked yes opened it Incrementally
4. Not All Tasks Are the Same The metric only makes sense in context Deterministic Creative Output is binary Output is subjective • Toggle a setting • Fill a required field • Enable a feature • Write a product description • Design a theme • Choose a color scheme • AI either caused it or it didn't. • Partial attribution can be just as valuable.
5. The Blank Page Problem Cognitive load: There are too many decisions and no starting point. Slows the user down.
6. The Blank Page Problem Scaffolding effect: A teacher provides just enough structure to get a student past the point where they're stuck.
7. Attribution Taxonomy Attributed AI proposed a change; merchant kept it as-is Assisted AI proposed a change; merchant edited before saving Abandoned AI proposed a change; merchant threw it away Manual AI was not used at all
8. One Session, Three Outcomes Prompt: “Help me create a product” Title Description Image Kept ✅ Edited " Deleted # Attributed Assisted Abandoned attributed assisted abandoned
9. How Similar Is Similar Enough? Handcrafted Ceramic Mug Ocean Blue, 12oz. Ocean Blue Ceramic Mug. Levenshtein distance Token overlap Embedding similarity 64% rewrite 4 out of 7 tokens survived Semantic meaning is nearly identical Manual / Abandoned Assisted Attributed AI proposes: Merchant saves:
10. Frontend Backend
11. The Instrumentation Reality Scoring window Deduplication Credit assignment Nice clean table Grain
12. The Field Level Breakdown Field Attribution pattern SEO title Attributed SEO (meta) description Attributed Product category Attributed Product title Assisted Produce description Assisted Price Manual Vendor Manual
13. • AI cost curve is convex: each marginal improvement costs more than the last. • Partial automation isn't a transitional phase. It's the long-run equilibrium. Partial Automation Isn’t a Compromise
14. Guardrail Metrics • Edit rate • Revert rate • Retention at N days xkcd Goodhart’s Law
15. Attribution ≠ Experimentation A/B Experiments Attribution Did AI cause lift? What did AI touch? Impact measurement Credit assignment Attribution tells you where to look; experimentation tells you if it's real.
16. Key Takeaways 1. Assisted is (sometimes) the success state 2. Pair attribution with guardrails 3. Instrument lineage early 4. Attribution surfaces the pattern. Experimentation proves the lift. 5. Don't stop at usage and acceptance. Measure what survived.
17. Thank you!