TW-BERT: Good for Users, Good for SEOs Dave Cousin Oban International @davetheseo https://bit.ly/tw-bert https://www.linkedin.com/in/davecousin/

obaninternational.com What we’re going to cover Why BERT needs improving? What’s Term Weighting? What is BERT? What’s TW-BERT? Where this fits in Google’s System? How to start optimising obaninternational.com

TW-BERT: the basics

obaninternational.com obaninternational.com Humans understand context intuitively Nike Running Shoes Nike Running Shoes Running Shoes obaninternational.com obaninternational.com

obaninternational.com obaninternational.com What TW-BERT is basically •A model •Developed by Google Researchers •Explained in a paper called ‘End to End Query Term Weighting’* * https://dl.acm.org/doi/pdf/10.1145/3580305.3599815

obaninternational.com obaninternational.com What TW-BERT is basically •TW-BERT helps(/will help) Google think like a human • Deciding what results to show • What order to show them •Massively improves search intent understanding * https://dl.acm.org/doi/pdf/10.1145/3580305.3599815 It understand the importance of words and sequences of words Whereas BERT currently deals with wordpieces Because…

The problem to be solved

obaninternational.com obaninternational.com The problem for users •BERT was meant to understand intent better than ever •It helps Google… “grasp the subtle nuances of language that computers don’t quite understand the way humans do”

obaninternational.com obaninternational.com The problem for users • Users have over the past couple of years been maybe more critical of Google’s results than ever:

obaninternational.com obaninternational.com The problem for SEOs •We’ve all had SERPs where we used to be top and we see who took over and think WTF

obaninternational.com obaninternational.com Real or imagined? • There’s even a 2024 research paper saying Google’s results are worse: https://downloads.webis.de/publications/papers/bevendorff_2024a.pdf • Ok it’s not due to BERT • But BERT hasn’t fixed it • There IS more Spam

The Solution

obaninternational.com obaninternational.com What is Term Weighting? •Which sequences of words are important? •Non Weighted: • Match MORE terms Where can I get great Korean BBQ in Brighton

obaninternational.com obaninternational.com What is Term Weighting? •Which sequences of words are important? Weighted: • Match IMPORTANT terms Where can I get great Korean BBQ in Brighton

obaninternational.com obaninternational.com Words, n-grams and wordpieces Wordpieces (19): I|want|##ed|a|new|##er|day|##ly|##s|##paper|from|the|##agent|to| ##day|get|##est|one|##morrow I wanted a newer daily newspaper from the newsagent today, I want to get the newest one tomorrow Unigrams (16): |I|wanted|a|newer|daily|newspaper|from|the|newsagent|today|want| to|get|newest|one|tomorrow| Bigrams (17): |I wanted|wanted a|a newer|newer daily|daily newspaper|newspaper from|from the|the newsagent|newsagent today|today, i|i want|want to|to get|get the|the newest|newest one|one tomorrow|

obaninternational.com obaninternational.com Importance of n-grams 30,000 Wordpieces 35,000 Words

obaninternational.com obaninternational.com Importance of n-grams •Humans, understand other human’s without consciously thinking about it 30,000 Wordpieces 10 million+ N-grams

obaninternational.com obaninternational.com Importance of n-grams •We identify phrases or terms we recognise 30,000 Wordpieces 10 million+ N-grams

obaninternational.com obaninternational.com Importance of n-grams •Challenge to replicate in NLP 30,000 Wordpieces 10 million+ N-grams

obaninternational.com obaninternational.com Importance of n-grams •Needs massive high dimensional neural networks and training corpora 30,000 Wordpieces 10 million+ N-grams

obaninternational.com obaninternational.com What is BERT? Bidirectional Encoder Representations from Transformers

obaninternational.com obaninternational.com What is BERT? •Deep Learning based Lexical Retriever for NLP Bidirectional Encoder Representations from Transformers

obaninternational.com obaninternational.com What is BERT? •Based on a (relatively) huge corpus or training material Bidirectional Encoder Representations from Transformers

obaninternational.com obaninternational.com What is BERT? •This allows the creation of a neural network (right) Bidirectional Encoder Representations from Transformers

obaninternational.com obaninternational.com What is BERT? •Each wordpiece is mapped to others with direction & magnitude Bidirectional Encoder Representations from Transformers

obaninternational.com obaninternational.com •Weights all wordpieces •Dynamic and considers: •Context •Meaning •Semantics •Sentiment What BERT does to search queries? •Expands queries: not just synonyms also semantic & contextual terms e.g. +‘restaurant’ to ‘Korean bbq’ bose noise cancel ##ing head with micro and blue tooth ##phone

obaninternational.com obaninternational.com What BERT does to search queries? bose noise cancel ##ing head with micro and blue tooth ##phone

obaninternational.com obaninternational.com What BERT does to search queries? bose noise cancel ##ing head with micro and blue tooth ##phone

obaninternational.com 29 The basics of how search engines work Crawl (+Render) Index Ranking Algorithm Search / SERP Send Fetch Results Query

obaninternational.com 30 IR and where BERT fits in Index Match Score Ranking Algorithm BERT Search Candidate Docs Fetch Results Query Weighted expanded 3 : 2 Scored Pairs

obaninternational.com obaninternational.com TW-BERT’s bridging solution BERT’s Wordpiece Island TW-BERT Statistical Bridge Opened 2023 Fast Efficient Can Misunderstand Deeper Understanding of Language Slow Expensive Isle of Deep Learning

obaninternational.com obaninternational.com What TW-BERT does to search queries? bose noise cancel ##ing head with micro and blue tooth ##phone

obaninternational.com obaninternational.com What TW-BERT does to search queries? bose noise cancel ##ing head with micro and blue tooth ##phone

obaninternational.com obaninternational.com What TW-BERT does to search queries? bose noise cancel ##ing head with micro and blue tooth ##phone

obaninternational.com 35 And where TW-BERT fits Index Match Score Ranking Algorithm TW- BERT BERT Search Candidate Docs Fetch Results Query Weighted, expanded 3 : 2 Scored Pairs Feedback Loop

obaninternational.com obaninternational.com The paper: End to End Query Term Weighting •Findings Vs BERT & Deep learning:

obaninternational.com obaninternational.com The paper: End to End Query Term Weighting •Findings Vs BERT & Deep learning: Better overall precision Particularly good for top 10, top 100

obaninternational.com obaninternational.com The paper: End to End Query Term Weighting •Findings Vs BERT & Deep learning: Performs very well for 1-10 word terms

obaninternational.com obaninternational.com Is Google using TW-BERT? •Probably….. a bit •Why wouldn’t they? •It helps fix a major flaw •Some put July/August fluctuations down to TW- BERT: Source: Algoroo

obaninternational.com obaninternational.com If Google are, then is this it? •Can tweak this and significance •More feedback loop data = more certainty and accuracy •Easy to copy for other search engines

obaninternational.com obaninternational.com Are N-Grams the new keywords? •No KWs aren’t dead •Just a bit limp – another step in their diminishing importance

obaninternational.com obaninternational.com Significance of n-grams in SEO •We have to optimise for intent first •KW stuffing will do nothing if the content doesn’t deliver

obaninternational.com obaninternational.com Significance of improved query expansion •More competition from relevant content without the same KWs •Users will use more long tail & natural language

obaninternational.com obaninternational.com Are you ready for TW-BERT? •Less competition from irrelevant results •But what if you have the irrelevant result? •Put yourself in the shoes of searchers?

obaninternational.com obaninternational.com How to start optimising now: Match intent •Understand intent •What is ranking? •Who has gained? •What are they doing? •Look at SGE too for intent

obaninternational.com obaninternational.com How to start optimising now: n-gram optimisation •Identify important n- grams in: • Related hero search terms • Questions or 5+ words 1. How to make a beef goulash? 2. How to cook a beef goulash? 3. Cook a beef goulash 4. Beef goulash recipe 5. Step by step beef goulash 6. Beef goulash how to

obaninternational.com obaninternational.com How to stay optimised •Check every 3-6 months for key terms: • How’s language changing? • Is knowledge changing? • Consider intent in different countries • What is it? • Is it a scam? • How To Invest? • Focus on BitCoin • News • Prices • Latest scams 2012 2017 2024 ‘cryptocurrency’ intent over time

obaninternational.com obaninternational.com So Good for Users, Google and SEOs? • Improved intent matching helps Users find what they want faster • And see less answers full of brand mentions Users • More likely to keep using Google as a search engine, • Brand can’t cheat their way to the top, some will buy ads instead Google • SEOs get more relevant, qualified traffic • Don’t have to go after irrelevant terms SEOs @davetheseo https://bit.ly/tw-bert linkedin.com/in/davecousin