brightonSEO US 2024: A Measured Approach: Testing UX with LLMs

Transcript

1. A Measured Approach: Testing UX with LLMs JR Oakes Locomotive Agency Speakerdeck.com/jroakes
2. JR Oakes JR Oakes is the VP of Strategy for LOCOMOTIVE Agency. He has been an SEO since 2011 and was formerly an architectural glass artist. His focus areas are in SEO, machine learning, language, and user experience. jroakes locomotive.agency
3. My Interests Have Broadened a Bit Generative AI Personalization Engines Causal ML Technical SEO
4. We OPENSOURCE Link Link Link
5. If you used an AI tool or technology for web design related tasks, what did you use it for… https://www.mindinventory.com/blog/ai-in-ui-ux-design/
6. 71 of consumers expect companies to deliver personalized interactions… 76 percent get frustrated when this doesn’t happen. - McKinsey & Company %
7. A Measured Approach: Types: In-Browser, Proprietary, Inference APIs Examples User State Implementation Testing UX with LLMs
8. CONFIDENTIAL Section 1a In Browser LLMs
9. AI in Chrome
10. AI in Chrome
11. Hacking the AI Example Code
12. Web LLM: High-Performance In-Browser LLM Inference Engine
20. Privacy Preserving Increasing Size / Performance ratio Pros Can be tuned to a particular task Reduces need for network requests
21. Cons Takes a long time to load Temporarily bricks your laptop Much less accurate than full-size models
22. Try it out now! Try it out now! Everything Calculator On Desktop
23. CONFIDENTIAL Section 1b Proprietary Systems
24. Vertex AI Search Search
25. Search Search Ask a new question
26. Vertex AI Agents Agents
27. Agents: Data Stores Agents
28. Agents: Tuning Agents
29. Agents: Integration Agents
30. Recommender Vertex AI Recommender
31. Recommender Recommender: Objectives
32. Recommender Recommender: Documents Google Analytics 4 View
33. Recommender Recommender: Events Google Analytics 4 View
34. Recommender Recommender: Results
35. CONFIDENTIAL Section 1c Inference APIs
36. LLM Inference is getting FAST and CHEAP 0
37. LLMs are becoming very dependable
38. Inference in ~1 second Based on the collected information, here’s a JSON object representing the core high-level objectives of the user: Groq API
39. Very fast Increasing Size / Performance ratio Pros Can be tuned to a particular task Cost/1M tokens dropping drastically
40. Cons Price prohibitive for very high-volume sites Dependant on potentially bubble companies Data shared with third-parties
41. CONFIDENTIAL Section 2 User State as Context
42. Think of User State as Context
43. Think of User State as Context
44. This is Great Context
45. What else can we add?
46. What else can we add?
47. Code for Collecting User State Example Code
48. CONFIDENTIAL Section 3 Example Workflows
49. Framework
50. Example 1: Reducing Cognitive Load
51. Example 1: Reducing Cognitive Load We can infer some preference from the context
52. Example 1: Reducing Cognitive Load Filter to relevant pages or sets
53. Example 2: Reduce Friction by Providing Answers Addition of RAG
54. Example 2: Reduce Friction by Providing Answers
55. Example 2: Reduce Friction by Providing Answers Can we find that important buried answer?
56. Example 2: Reduce Friction by Providing Answers To unlock/enable the sale?
57. Example 3: Data Labeling/Cleaning
58. Example 3: Data Labeling/Cleaning ➔ Infer location Other Examples ➔ Assign to user cohorts ➔ Assign client vs prospect behaviour ➔ Infer product/service interest category ➔ Include summary of browsing behaviour in contact forms
59. Example 4: Guiding the User Code
60. Example 5: Highlight Relevant Features
61. Example 6: Dynamic Testimonial Insertion
62. Example 7: Dynamic content reordering 1 1 2 2 3 3 4 4 5 5 6 6
63. Example 8: Dynamic Help Context shows frustrating activity
64. CONFIDENTIAL Section 4 Implementation
65. Code Size and Complexity is an Issue Source
66. Use JSON
67. Creepy Factor To minimize, focus on prioritization and curation Creepy Sensitivity Index Low High Medium
68. Cost
69. Controlling Costs ➔ Use heuristics to know when it is helpful to utilize models; ➔ Control usage to particular site sections; ➔ Select models carefully. Many times a smaller size model is adequate to the task; ➔ Minimize output (e.g. Select the number rather than print the name)
70. Libraries Source
71. CONFIDENTIAL Want to build stuff? Talk to me