Slide 1

Slide 1 text

TW-BERT: Good for Users, Good for SEOs Dave Cousin Oban International @davetheseo https://bit.ly/tw-bert https://www.linkedin.com/in/davecousin/

Slide 2

Slide 2 text

obaninternational.com What we’re going to cover Why BERT needs improving? What’s Term Weighting? What is BERT? What’s TW-BERT? Where this fits in Google’s System? How to start optimising obaninternational.com

Slide 3

Slide 3 text

TW-BERT: the basics

Slide 4

Slide 4 text

obaninternational.com obaninternational.com Humans understand context intuitively Nike Running Shoes Nike Running Shoes Running Shoes obaninternational.com obaninternational.com

Slide 5

Slide 5 text

obaninternational.com obaninternational.com What TW-BERT is basically •A model •Developed by Google Researchers •Explained in a paper called ‘End to End Query Term Weighting’* * https://dl.acm.org/doi/pdf/10.1145/3580305.3599815

Slide 6

Slide 6 text

obaninternational.com obaninternational.com What TW-BERT is basically •TW-BERT helps(/will help) Google think like a human • Deciding what results to show • What order to show them •Massively improves search intent understanding * https://dl.acm.org/doi/pdf/10.1145/3580305.3599815 It understand the importance of words and sequences of words Whereas BERT currently deals with wordpieces Because…

Slide 7

Slide 7 text

The problem to be solved

Slide 8

Slide 8 text

obaninternational.com obaninternational.com The problem for users •BERT was meant to understand intent better than ever •It helps Google… “grasp the subtle nuances of language that computers don’t quite understand the way humans do”

Slide 9

Slide 9 text

obaninternational.com obaninternational.com The problem for users • Users have over the past couple of years been maybe more critical of Google’s results than ever:

Slide 10

Slide 10 text

obaninternational.com obaninternational.com The problem for SEOs •We’ve all had SERPs where we used to be top and we see who took over and think WTF

Slide 11

Slide 11 text

obaninternational.com obaninternational.com Real or imagined? • There’s even a 2024 research paper saying Google’s results are worse: https://downloads.webis.de/publications/papers/bevendorff_2024a.pdf • Ok it’s not due to BERT • But BERT hasn’t fixed it • There IS more Spam

Slide 12

Slide 12 text

The Solution

Slide 13

Slide 13 text

obaninternational.com obaninternational.com What is Term Weighting? •Which sequences of words are important? •Non Weighted: • Match MORE terms Where can I get great Korean BBQ in Brighton

Slide 14

Slide 14 text

obaninternational.com obaninternational.com What is Term Weighting? •Which sequences of words are important? Weighted: • Match IMPORTANT terms Where can I get great Korean BBQ in Brighton

Slide 15

Slide 15 text

obaninternational.com obaninternational.com Words, n-grams and wordpieces Wordpieces (19): I|want|##ed|a|new|##er|day|##ly|##s|##paper|from|the|##agent|to| ##day|get|##est|one|##morrow I wanted a newer daily newspaper from the newsagent today, I want to get the newest one tomorrow Unigrams (16): |I|wanted|a|newer|daily|newspaper|from|the|newsagent|today|want| to|get|newest|one|tomorrow| Bigrams (17): |I wanted|wanted a|a newer|newer daily|daily newspaper|newspaper from|from the|the newsagent|newsagent today|today, i|i want|want to|to get|get the|the newest|newest one|one tomorrow|

Slide 16

Slide 16 text

obaninternational.com obaninternational.com Importance of n-grams 30,000 Wordpieces 35,000 Words

Slide 17

Slide 17 text

obaninternational.com obaninternational.com Importance of n-grams •Humans, understand other human’s without consciously thinking about it 30,000 Wordpieces 10 million+ N-grams

Slide 18

Slide 18 text

obaninternational.com obaninternational.com Importance of n-grams •We identify phrases or terms we recognise 30,000 Wordpieces 10 million+ N-grams

Slide 19

Slide 19 text

obaninternational.com obaninternational.com Importance of n-grams •Challenge to replicate in NLP 30,000 Wordpieces 10 million+ N-grams

Slide 20

Slide 20 text

obaninternational.com obaninternational.com Importance of n-grams •Needs massive high dimensional neural networks and training corpora 30,000 Wordpieces 10 million+ N-grams

Slide 21

Slide 21 text

obaninternational.com obaninternational.com What is BERT? Bidirectional Encoder Representations from Transformers

Slide 22

Slide 22 text

obaninternational.com obaninternational.com What is BERT? •Deep Learning based Lexical Retriever for NLP Bidirectional Encoder Representations from Transformers

Slide 23

Slide 23 text

obaninternational.com obaninternational.com What is BERT? •Based on a (relatively) huge corpus or training material Bidirectional Encoder Representations from Transformers

Slide 24

Slide 24 text

obaninternational.com obaninternational.com What is BERT? •This allows the creation of a neural network (right) Bidirectional Encoder Representations from Transformers

Slide 25

Slide 25 text

obaninternational.com obaninternational.com What is BERT? •Each wordpiece is mapped to others with direction & magnitude Bidirectional Encoder Representations from Transformers

Slide 26

Slide 26 text

obaninternational.com obaninternational.com •Weights all wordpieces •Dynamic and considers: •Context •Meaning •Semantics •Sentiment What BERT does to search queries? •Expands queries: not just synonyms also semantic & contextual terms e.g. +‘restaurant’ to ‘Korean bbq’ bose noise cancel ##ing head with micro and blue tooth ##phone

Slide 27

Slide 27 text

obaninternational.com obaninternational.com What BERT does to search queries? bose noise cancel ##ing head with micro and blue tooth ##phone

Slide 28

Slide 28 text

obaninternational.com obaninternational.com What BERT does to search queries? bose noise cancel ##ing head with micro and blue tooth ##phone

Slide 29

Slide 29 text

obaninternational.com 29 The basics of how search engines work Crawl (+Render) Index Ranking Algorithm Search / SERP Send Fetch Results Query

Slide 30

Slide 30 text

obaninternational.com 30 IR and where BERT fits in Index Match Score Ranking Algorithm BERT Search Candidate Docs Fetch Results Query Weighted expanded 3 : 2 Scored Pairs

Slide 31

Slide 31 text

obaninternational.com obaninternational.com TW-BERT’s bridging solution BERT’s Wordpiece Island TW-BERT Statistical Bridge Opened 2023 Fast Efficient Can Misunderstand Deeper Understanding of Language Slow Expensive Isle of Deep Learning

Slide 32

Slide 32 text

obaninternational.com obaninternational.com What TW-BERT does to search queries? bose noise cancel ##ing head with micro and blue tooth ##phone

Slide 33

Slide 33 text

obaninternational.com obaninternational.com What TW-BERT does to search queries? bose noise cancel ##ing head with micro and blue tooth ##phone

Slide 34

Slide 34 text

obaninternational.com obaninternational.com What TW-BERT does to search queries? bose noise cancel ##ing head with micro and blue tooth ##phone

Slide 35

Slide 35 text

obaninternational.com 35 And where TW-BERT fits Index Match Score Ranking Algorithm TW- BERT BERT Search Candidate Docs Fetch Results Query Weighted, expanded 3 : 2 Scored Pairs Feedback Loop

Slide 36

Slide 36 text

obaninternational.com obaninternational.com The paper: End to End Query Term Weighting •Findings Vs BERT & Deep learning:

Slide 37

Slide 37 text

obaninternational.com obaninternational.com The paper: End to End Query Term Weighting •Findings Vs BERT & Deep learning: Better overall precision Particularly good for top 10, top 100

Slide 38

Slide 38 text

obaninternational.com obaninternational.com The paper: End to End Query Term Weighting •Findings Vs BERT & Deep learning: Performs very well for 1-10 word terms

Slide 39

Slide 39 text

obaninternational.com obaninternational.com Is Google using TW-BERT? •Probably….. a bit •Why wouldn’t they? •It helps fix a major flaw •Some put July/August fluctuations down to TW- BERT: Source: Algoroo

Slide 40

Slide 40 text

obaninternational.com obaninternational.com If Google are, then is this it? •Can tweak this and significance •More feedback loop data = more certainty and accuracy •Easy to copy for other search engines

Slide 41

Slide 41 text

obaninternational.com obaninternational.com Are N-Grams the new keywords? •No KWs aren’t dead •Just a bit limp – another step in their diminishing importance

Slide 42

Slide 42 text

obaninternational.com obaninternational.com Significance of n-grams in SEO •We have to optimise for intent first •KW stuffing will do nothing if the content doesn’t deliver

Slide 43

Slide 43 text

obaninternational.com obaninternational.com Significance of improved query expansion •More competition from relevant content without the same KWs •Users will use more long tail & natural language

Slide 44

Slide 44 text

obaninternational.com obaninternational.com Are you ready for TW-BERT? •Less competition from irrelevant results •But what if you have the irrelevant result? •Put yourself in the shoes of searchers?

Slide 45

Slide 45 text

obaninternational.com obaninternational.com How to start optimising now: Match intent •Understand intent •What is ranking? •Who has gained? •What are they doing? •Look at SGE too for intent

Slide 46

Slide 46 text

obaninternational.com obaninternational.com How to start optimising now: n-gram optimisation •Identify important n- grams in: • Related hero search terms • Questions or 5+ words 1. How to make a beef goulash? 2. How to cook a beef goulash? 3. Cook a beef goulash 4. Beef goulash recipe 5. Step by step beef goulash 6. Beef goulash how to

Slide 47

Slide 47 text

obaninternational.com obaninternational.com How to stay optimised •Check every 3-6 months for key terms: • How’s language changing? • Is knowledge changing? • Consider intent in different countries • What is it? • Is it a scam? • How To Invest? • Focus on BitCoin • News • Prices • Latest scams 2012 2017 2024 ‘cryptocurrency’ intent over time

Slide 48

Slide 48 text

obaninternational.com obaninternational.com So Good for Users, Google and SEOs? • Improved intent matching helps Users find what they want faster • And see less answers full of brand mentions Users • More likely to keep using Google as a search engine, • Brand can’t cheat their way to the top, some will buy ads instead Google • SEOs get more relevant, qualified traffic • Don’t have to go after irrelevant terms SEOs @davetheseo https://bit.ly/tw-bert linkedin.com/in/davecousin