Why You Need A Relevance Engineer Driving The Car - SEO Week 2026

This session will look at how specific content signals behave differently across different types of search queries and how those differences inform our experimentation at iPullRank. We will deep dive into designing the AI Search metrics that reflect how systems interpret your content, interpret metric behavior by query type and intent, translate the insights into controlled experiments, and apply our findings to content templates and editorial strategy.

Key Takeaways:
- See how iPullRank approaches experimenting with the metrics that matter for AI Search
- Learn how to analyze metric outputs to identify actionable patterns
- See how to take those patterns and apply them to optimizing your content strategy.

Transcript

1. WHY YOU NEED A RELEVANCE ENGINEER Driving The Car Zach Chahalis Senior Director of SEO and Data Analytics iPullRank
2. I’M ZACH CHAHALIS SR. DIRECTOR OF SEO AND DATA ANALYTICS iPullRank My context: 6+ Months at iPullRank (But 2.5 yrs in total) 15 Years in SEO, Marketing Analytics, and CRO Two Weeks Into Married Life (Almost) 11 Yr Old Goldendoodle
3. STORY TIME 3
4. MY LOVE FOR CARS STARTED AT A YOUNG AGE
5. IT WAS A DREAM COME TRUE TO GET MY LICENSE AND MY FIRST CAR
6. I’VE ALREADY OWNED OVER 20 DIFFERENT VEHICLES (THIS IS JUST A SAMPLE)
7. BUT MODERN CARS BORE ME
8. I’VE FOUND MYSELF LOOKING FOR CARS WITH MORE SOUL
9. SO I BOUGHT THIS
10. What Does This Have To Do With Relevance Engineering or AI Search? — Probably 95% of you, SEO Week Audience
11. My boredom with modern “appliance” cars matches my boredom with SEO before AI Search came along — Me, Sr. Director of SEO, iPullRank
12. I’VE FOUND MYSELF LOOKING FOR CARS WITH MORE SOUL BUT, CARS WITH SOUL REQUIRE A DRIVER BEHIND THE WHEEL AND SOMEONE WHO CAN WORK ON THEM
13. SO I’VE LEARNED HOW TO WORK ON AND MODIFY CARS
14. JUST LIKE I’VE EVOLVED MY SEO SKILLS INTO RELEVANCE ENGINEERING SKILLS
15. The fun part of my job is getting to experiment and reverse engineer with my amazing team. — Me, Sr. Director of SEO, iPullRank
16. LET’S ESTABLISH HOW MEASUREMENT HAS CHANGED 16
17. THREE BUCKETS FOR MEASUREMENT MEASUREMENT MUST EVOLVE BEYOND PERFORMANCE METRICS Passage Relevance Entity Salience Bot Activity Synthetic Query Rankings Share of Voice Citation Rate Citation Quality Citation Sentiment Traffic Events Conversions Engagement Depth Measures how visible, cited, and represented your brand is across search and AI surfaces. Outcomes that prove whether visibility and citations translate into business impact. Signals that show whether your content and site are aligned with how AI and search engines understand and retrieve information.
18. AI Overviews are Causing Traffic Losses Wikipedia, one of the most Organic Search reliant websites saw its traffic grow with the introduction of ChatGPT, but has seen its traffic decline with the advent of AI Overviews. AI Overviews are Changing User Behavior
19. BUT ORGANIC SEARCH STILL MATTERS • Search remains the primary discovery channel • AI systems still retrieve from search • Ranking is still table stakes https://www.hubspot.com/marketing-statistics
20. BUT ORGANIC SEARCH STILL MATTERS • Search remains the primary discovery channel • AI systems still retrieve from search • Ranking is still table stakes https://datos.live/report/state-of-search-q4-2025/
21. BEST PRACTICES ARE NOT ENOUGH
22. YOU NEED TO EXPERIMENT CONSTANTLY • No universal playbook • Experimentation is required • Relevance must be engineered
23. THE METRICS 23 WHAT CAN BE MEASURED?
24. HOW DO YOU COME UP WITH NEW AI SEARCH METRICS? • Started by identifying what modern search and AI systems reward (accuracy, depth, structure, answerability) • Translated those qualities into measurable NLP and retrieval signals • Designed metrics to be diagnostic optimization levers, not content scores • Tested metrics against performance outcomes • Iterated to keep metrics that explain performance changes consistently Learn More: https://ipullrank.com/resources/webinars/ai-search-metrics-case-study These metrics come from translating the qualities that search and AI systems reward into measurable signals we can use to diagnose and improve content performance.
25. CONTENT- KEYWORD COSINE SIMILARITY Measures topic coherence and focus; penalizes semantic drift.
26. COMPREHENSIVE COVERAGE INDEX Balances length with informational value and completeness. Composite of multiple quality signals.
27. STRATEGIC ENTITY RICHNESS SCORE Weights entity count by quality factors; penalizes irrelevant padding.
28. EXPLANATORY EFFICIENCY INDEX Measures fact density vs. narrative overhead. Indicates content bloat.
29. VERIFIABLE CLAIMS Measures how much of a document’s claim-making content can be substantiated against trusted external knowledge sources.
30. OPTIMAL CHUNKABILITY SCORE Measures how well a document's existing structure (paragraphs, headings) aligns with underlying semantic structure.
31. CONCEPTUAL DEPTH SCORE Measures specificity and hierarchical depth of concepts discussed. Rewards detailed, expert-level information.
32. INFORMATION GAIN SCORE Measures how much new or novel information a passage provides. Rewards unique insights, penalizes redundant information.
33. ENTITY DENSITY Ratio of entities to words, ensuring appropriate information density without crowding.
34. SO LET’S DISCUSS OUR EXPERIMENT 34
35. EXPERIMENT DETAILS • We started by finding and crawling three websites with obvious AI- generated content. We intentionally chose websites from different industries • Crawled a sample of each site (enough URLs to reach significance) • Translated those qualities into measurable NLP and retrieval signals • Ran each of the websites against our diagnostic optimization levers and evaluated the websites against one another We’ve previously run experiments on these metrics and presented them Now, we’ve identified three AI-generated content websites and compared them against the metrics
36. CONTESTANT 1 - A WELL-KNOWN PROJECT MANAGEMENT SAAS This is a major project management SaaS platform who heavily invested into an AI pipeline but has lost its way. Page 1 rankings have dropped 90% from their peak Data Source: SEMRush
37. CONTESTANT 2 - A FINANCE INFORMATIONAL WEBSITE This is a finance help/tools/guide website with content focused on educating and guiding users on topics ranging from mortgage and banking information to calculators and finance news. Page 1 rankings have essentially been flat over the past year. Data Source: SEMRush
38. CONTESTANT 3 - A TECH HOW-TO WEBSITE This is a tech/gadget-oriented website focused on how-to style content with various content pieces around topics such as using/customizing Android phones. Page 1 rankings have increased over the past year but are trending down since January. Data Source: SEMRush
39. SO HOW HAS AIO VISIBILITY FARED? Data Source: SEMRush
40. WAIT, LET’S NARROW IN ON THAT BLUE LINE Data Source: SEMRush
41. WELL, WHAT ABOUT TOTAL AI CITATIONS? Data Source: SEMRush
42. HMM Data Source: SEMRush
44. OK SO WHAT DID WE FIND? 44 Ranking Does Not Equal Extraction
45. “OFF-BRAND” CONTENT HURTS YOUR PERFORMANCE
46. A LOT OF WORDS
47. BUT NOTHING VALUABLE TO SAY
48. AND LIMITED VERIFIABLE INFORMATION This means the two performing sites are much more likely to present information that can be independently substantiated, reinforcing stronger factual grounding and trust signals.
49. ALSO THESE METRICS ARE NOT A ONE-SIZE FITS ALL 49 THIS IS WHY YOU NEED A RELEVANCE ENGINEER Ranking Does Not Equal Extraction
50. LET’S ZOOM IN ON THE TECH HOW-TO SITE
51. THE PRODUCTIVITY SAAS SITE IS MEASURABLY MORE TEMPLATIZED Optimal Chunkability measures how easily content can be broken into interchangeable sections — a signal of rigid, template-driven production.
52. AND THE FINANCE AND TECH HOW-TO SITES WIN IN DIFFERENT WAYS 52 BECAUSE IT’S NOT A ONE SIZE FITS ALL
53. THE FINANCE SITE EXCELS AT EXPLANATORY EFFICIENCY Finance needs fewer than half the words Productivity SaaS uses to deliver each unit of explanatory value. This is the clearest measure of content bloat.
54. TECH HOW-TO EXCELS AT ENTITY DENSITY It is effective at packing more concrete, named, technical entities (device names, OS versions, settings, apps) into each paragraph
55. SO WHAT DOES THIS ALL HAVE TO DO WITH YOUR CAR ANALOGY??
56. AN SEO/GEO STRATEGY IS ONLY AS GOOD AS THE RELEVANCE ENGINEER DRIVING THE CAR YOU CAN’T JUST ACTIVATE TESLA AUTOPILOT AND SIT BACK
57. PATRICK SCHOFIELD LEAD RELEVANCE ENGINEER iPullRank His context: Continues to move our relevance engineering practice forward Developed proprietary NLP metrics for AI citation optimization Produced 75k URL case study on Relevance Engineering metrics Actual car guy. Modifies them. Actually drives. AND THIS IS WHO’S BEEN DRIVING THE CAR
58. NEW ROLES ARE EMERGING The functionality of these new channels has evolved so new skills at the intersection of AI and content strategy (content engineering) and deeper understanding are required to navigate them.
59. NEW ROLES ARE EMERGING The functionality of these new channels has evolved so new skills at the intersection of AI and content strategy (content engineering) and deeper understanding are required to navigate them.
60. PROOF IS IN THE CASE STUDIES 60 CLOSING ARGUMENTS
61. 61 THE RESULTS ▪ 121.4% cumulative increase in Sign-ups (6 Months) ▪ 52.6% cumulative increase in Organic Traffic ▪ 17x higher conversion uplift on pages with improved Relevance Scores ▪ 24% increase in total AIOs (1 Month), 40% cumulative (3 Months) ▪ 72% increase in average AIOs per page (1 Month), 168% cumulative (3 Months) THE BACKGROUND A global financial services business wanted to move beyond vanity traffic metrics. They sought to validate our hypothesis: that optimizing content for Semantic Relevance—a proprietary metric measuring a piece of contents semantic alignment—would directly correlate with high-intent user acquisition and revenue growth. WHAT WE DID We executed a Content Optimization Program focusing on semantic optimization rather than just keyword volume. Semantic Overhaul: We updated content to improve their Relevance Scores, ensuring deeper alignment with specific user intents. Validated Forecasting: We built a CMGR model that accurately predicted the August breakout performance, securing confidence for a scaled investment. GLOBAL FINANCIAL SERVICES LEADER INCREASES SIGN-UPS 121% — PROVING THE ROI OF SEMANTIC OPTIMIZATION Case Studies OUR GOALS ▪ Drive high-intent, converting traffic (not just clicks). ▪ Prove the correlation between Semantic Relevance and Revenue. ▪ Validate a scalable content growth model. SERVICES USED ▪ Content Audit ▪ Content Plan ▪ Competitive Analysis ▪ Content Optimizations ▪ Implementation Audits
62. 62 THE RESULTS ▪ 278% increase in AI Overview Visibility ▪ 34% increase in AI Search Visibility ▪ 32% increase in AI Referred Sessions ▪ 21% increase in AI Referred Revenue THE BACKGROUND A national self-storage brand joined our AI Search Strategy Program to prove short-term impact. In 3 months, they wanted measurable gains in AI visibility, traffic, and revenue while building toward their goal of becoming the most searched self-storage brand in the U.S. WHAT WE DID We ran a focused AI search pilot built to drive early gains in 3 months. First, we addressed technical issues that were limiting discoverability by fixing 499 errors, correcting global navigation headings, and updating robots.txt. Then we delivered the strategic and measurement work needed to support ongoing performance: a Keyword Portfolio, Omnimedia Content Audit, Omnimedia Content Plan, AI Search Measurement Plan, AI Search Audit, and Strategic Roadmap. Operational Impact We helped the client make AI search a more structured business function, with clearer KPIs, tighter prioritization, and a more defined link between technical fixes, content decisions, and performance measurement. SELF-STORAGE BRAND INCREASES AI OVERVIEW VISIBILITY 278% AND DRIVES 21% MORE AI-REFERRED REVENUE IN 3 MONTHS Case Studies OUR GOALS ▪ Become the most searched self- storage brand in the U.S. ▪ Prove impact in 3 months ▪ Increase AI visibility and AI Overview inclusion ▪ Grow AI-referred traffic and revenue SERVICES USED ▪ Keyword Portfolio ▪ Omnimedia Content Audit ▪ Omnimedia Content Plan ▪ AI Search Measurement Plan ▪ AI Search Audit ▪ Strategic Roadmap Start Date Implementation
63. WRAPPING UP 63 CLOSING ARGUMENTS
64. 64 WARNINGS: BUYER BEWARE Outcomes Are Not Guaranteed Results are probabilistic. Impact varies by query and page. Define Your Risk Tolerance Test scope determines risk. Start narrow before scaling. Industry Context Matters Vertical behavior differs. One playbook does not fit all. Measure Before You Modify Baseline first. Change second. Watch for Second- Order Effects Changes cascade. Monitor beyond citations.
65. ALWAYS BE EXPERIMENTING My next experiment is already kicking off creating AI content at scale but aligned with the metrics that matter for my identified website audience.
66. 66 THANK YOU Zach Chahalis Senior Director of SEO and Data Analytics iPullRank X: @ZachChahalis LI: /zacharychahalis Tap in with us: ipullrank.com