News SEO is the cutting edge of all SEO

Slides from my talk at YoastCon 2023 where I showed some of the lessons I've learned about Google's inner workings from years of working with news publishers.

Transcript

1. #YoastCon #YoastCon News is at the cutting edge of SEO Barry Adams May 2023
2. #YoastCon Advance Warning “The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts.” - Bertrand Russell
3. #YoastCon #YoastCon
4. #YoastCon #YoastCon How does Google work?
5. #YoastCon #YoastCon Web Search Engines Crawling Indexing Ranking
6. #YoastCon
7. #YoastCon Google’s model Crawl Queue Crawling Processing Index Render Queue Rendering Index Index
8. #YoastCon #YoastCon 1. Crawling Ranking Crawling Indexing
9. #YoastCon
10. #YoastCon Three ‘layers’ of Googlebot? Crawling Processing Render Queue Rendering Crawling Crawling Index Crawl Queue Crawl Queue Crawl Queue
11. #YoastCon Three ‘layers’ of Googlebot 1. Realtime crawler 2. Regular crawler 3. Legacy content crawler
12. #YoastCon Realtime Crawler • Crawls VIPs ➢ Very Important Pages; Webpages that have a high change frequency and/or are seen as highly authoritative News website homepages & key section pages • Main purpose = discovery of valuable new content ➢ i.e., news articles • Rarely re-crawls newly discovered URLs ➢ Unless they’re new VIPs
13. #YoastCon Regular Crawler • Google’s main crawler; ➢ Does most of the hard work ➢ Probably the crawler that fetches page resources
14. #YoastCon Legacy Content Crawler • Crawls VUPs ➢ Very Unimportant Pages; URLs that have very little link value and/or are very rarely updated ➢ Re-crawls URLs that serve 4XX errors
15. #YoastCon Robots.txt = Crawl Management … or is it? User-agent: Googlebot-News Disallow: /
16. #YoastCon User-agent: Googlebot-News Disallow: / Robots.txt = Crawl Management … or is it? Robots.txt disallow for index management?!
17. #YoastCon What can non-news sites learn from this? 1. Turn key pages into VIPs; Make them more valuable by; - Improving link value - Increasing change frequency 2. Use robots.txt disallow rules to manage indexing & ranking; For example, block Googlebot-Image to prevent product images from showing in Image search
18. #YoastCon #YoastCon 2. Indexing Crawling Indexing Ranking
19. #YoastCon Indexing and Rendering Crawl Queue Crawling Processing Index Render Queue Rendering Index Index
20. #YoastCon Indexing and Rendering Render Queue Rendering Crawl Queue Crawling Processing Index Index Index
21. #YoastCon Indexing and Rendering Rendering takes time, and news doesn’t have time. Indexing is initially with raw HTML only. Crawl Queue Crawling Processing Index Render Queue Rendering Index Index
22. #YoastCon Rendering isn’t the only shortcut… Google wants publishers to noindex syndicated content. Because Google sucks at identifying duplicate content. At least, it can’t de-duplicate quickly.
23. #YoastCon Indexing is a multi-layered set of processes Render Queue Rendering Crawl Queue Crawling Processing Index Processing Processing Processing
24. #YoastCon What about the Index itself? Render Queue Rendering Crawl Queue Crawling Processing Index Index Index
25. #YoastCon Three Crawlers… Three Indices? Realtime crawler Regular crawler Legacy content crawler RAM storage SSD storage HDD storage
26. #YoastCon Three Layers of Index Storage 1. RAM storage ➢ Pages that need to be served quickly and frequently Includes news articles but also popular content 2. SSD storage ➢ Pages that are regularly served in SERPs but aren’t super popular 3. HDD storage ➢ Pages that are rarely (if ever) served in SERPs
27. #YoastCon It’s probably more complicated Realtime crawler Regular crawler Legacy content crawler RAM storage SSD storage HDD storage
28. #YoastCon What can non-news sites learn from this? 1. Make indexing easy for Googlebot; Put all your critical content in the HTML source Don’t rely on rendering to load valuable content 2. There’s no such thing as a duplicate content penalty; However, duplicate content on a single site means the site is competing with itself… and that’s stupid.
29. #YoastCon #YoastCon 3. Ranking Crawling Indexing Ranking
30. #YoastCon Search Intent – first BERT, then MUM
31. #YoastCon
32. #YoastCon However…
33. #YoastCon Most tools are out of date
34. #YoastCon Google Trends is better, but lacks numbers
35. #YoastCon Very few tools are (near) real-time
36. #YoastCon And even fewer accurately report on SERP features
37. #YoastCon
38. #YoastCon SERP Features • Many SERP features are volatile; ➢ None more than Top Stories & other news boxes • Top Stories are triggered when two conditions are met; ➢ Sudden increase in search volume ➢ Sudden increase in publishing volume
39. #YoastCon SERP Features and CTR
40. #YoastCon Who gets the Top Stories Top Spot? • Topical Authority • Authorship • E-E-A-T Expressions of the Knowledge Graph
41. #YoastCon Knowledge Graph Arnold Schwarzenegger Bodybuilding Predator (1987) Governor of California Maria Shriver Ronnie Coleman JFK
42. #YoastCon Knowledge Graph Arnold Schwarzenegger Bodybuilding Independent.co.uk … … … Predator (1987)
43. #YoastCon Internal Linking to Topic Hubs
44. #YoastCon Knowledge Graph Arnold Schwarzenegger Predator (1987) … … Independent.co.uk … Bodybuilding
45. #YoastCon
46. #YoastCon What can non-news sites learn from this? 1. Understand the intent behind the keywords you’re targeting; Don’t try to rank content that doesn’t match the intent If there are SERP features, try to get into those 2. Improve your Knowledge Graph presence; Category pages = topic hubs Use internal linking to your advantage Schema.org markup helps Google connect the dots
47. #YoastCon #YoastCon So why is news at the cutting edge of SEO?
48. #YoastCon News websites… … are crawled the most … are crawled the fastest … are indexed the quickest … are ranked according to the latest signals … are ranked based on the best interpretation of intent
49. #YoastCon From News SEO you can learn… … how Google crawls websites … how Google indexes content … how Google evaluates quality and authority … how SERP features impact on ranking and traffic … and much, much more.
50. #YoastCon www.SEOforGoogleNews.com
51. #YoastCon #YoastCon Thank You barry@polemicdigital.com @badams /in/barryadams/