Crawl Capacity Management - Melb SEO Meetup - Nov 2022

A brief overview of a few big things we've done on Envato Elements to improve crawl budget, or as I'm calling it here: Crawl Capacity Management.

This slide deck is presented at the Melbourne SEO meetup on Tuesday the 8th of November

Transcript

1. MELB SEO Meetup Crawl capacity management on Envato Elements Gastón Riera - @gastonriera
2. That's how I used to look, all well dressed and all. Gastón Riera - @gastonriera Gastón Riera
3. Everything you need to get your creative projects done. Gastón Riera - @gastonriera The big names: Other very cool products:
4. Gastón Riera - @gastonriera When should we care about crawl [capacity|budget]? Then there is a shit ton of pages ingested by Google. Here some examples of websites I've worked on
5. Let's get into SEO 🔎 Gastón Riera - @gastonriera
6. The problem: A big part of the site was not being indexed 👎 Gastón Riera - @gastonriera
7. ☹ Gastón Riera - @gastonriera
8. Gastón Riera - @gastonriera It looked like Google was not recrawling a large amount of pages!
9. Gastón Riera - @gastonriera And taking very long to recrawl some pages
10. Was Google actually crawling the site?🤔 Gastón Riera - @gastonriera
11. Gastón Riera - @gastonriera Yes, it was👍
12. But, it wasn't getting to crawl the entire site 😕 Gastón Riera - @gastonriera
13. And, why was that?󰤈 Gastón Riera - @gastonriera
14. No idea Gastón Riera - @gastonriera
15. Just kidding 😆 Gastón Riera - @gastonriera
16. We came up with two theories to work on 😎 Gastón Riera - @gastonriera
17. - Content quality - Internal linking Gastón Riera - @gastonriera We needed to work on
18. Gastón Riera - @gastonriera We needed to work on I'll get to them in a bit. - Content quality - Internal linking
19. What did we do? 🤔 Gastón Riera - @gastonriera
20. The basics! Gastón Riera - @gastonriera
21. The basics! - Noindex Gastón Riera - @gastonriera
22. The basics! - Noindex - Redirects Gastón Riera - @gastonriera
23. The basics! - Noindex - Redirects - Nofollow Gastón Riera - @gastonriera
24. The basics! - Noindex - Redirects - Nofollow - Crawl paths (more/less) Gastón Riera - @gastonriera
25. Gastón, that's nothing new! 😡 Gastón Riera - @gastonriera
26. I know! 😉 Gastón Riera - @gastonriera
27. How are we using them? Let's get to it Gastón Riera - @gastonriera
28. Battle_1: Content quality Gastón Riera - @gastonriera
29. Content is not just text on the page, but everything on it. Every page is content. Gastón Riera - @gastonriera A key takeaway
30. Battle_1: Content quality Gastón Riera - @gastonriera Two options: 1. Add content focussing on quality over quantity. 2. Remove content from Google's index. We already had +9M items!
31. Battle_1: Content quality Gastón Riera - @gastonriera Two options: 1. Add content focussing on quality over quantity. ❌ 2. Remove content from Google's index. We already had +9M items!
32. Battle_1: Content quality Gastón Riera - @gastonriera Two options: 1. Add content focussing on quality over quantity. ❌ 2. Remove content from Google's index. ✅ We already had +9M items!
33. Do you know what reduces the content quality of any site? Gastón Riera - @gastonriera
34. Do you know what reduces the content quality of any site? DUPLICATE CONTENT! Gastón Riera - @gastonriera
35. Noindex and remove duplicates, RUTHLESSLY Gastón Riera - @gastonriera Noindex a good part of the items library. -> Several million less discoverable pages! Why we decided to noindex instead of a fancier solution? Ask me later 😉
36. A few tips on how to get what to noindex? - Use google's crawled not indexed as a proxy - Check duplicate titles/urls/content description - Just a different image doesn't make it a different page to the eyes of Google! Gastón Riera - @gastonriera Battle_1: Content quality
37. Noindex and remove duplicates, RUTHLESSLY Gastón Riera - @gastonriera Why the redirected path had 15% of site's traffic and 20x the destination. Ask me later 😉 Merged two translations that ended up being way more similar that intended -> A few millions pages removed from Google.
38. Other big things we did • Turned Tag pages into Search pages • Search pages are noindex by default The overall result? Decreased the index size to a half without impacting organic traffic. Gastón Riera - @gastonriera 50% Battle_1: Content quality
39. Battle_2 Internal linking Gastón Riera - @gastonriera Reference
40. Battle_2: Internal linking Gastón Riera - @gastonriera Out of many tactics: 1. Reduce the number of crawl paths 2. Nofollow on links to low-value pages
41. Basically, Be intentional and smart about crawl paths. Gastón Riera - @gastonriera
42. Gastón Riera - @gastonriera Link to only valuable pages Added links between related search pages 10% Organic traffic! If it's a useful search page, it will not have a noindex. Note that
43. Gastón Riera - @gastonriera Link to only valuable pages Remove hreflang when you're uncertain of the quality on other languages 15% size of index! hreflang tags are bidirectional, remove them on every language. Remember 😉
44. What are valuable pages? Gastón Riera - @gastonriera In short, pages we want Google to index.
45. Gastón Riera - @gastonriera Link to only valuable pages As per nofollow: • Nofollow on links to noindex pages • Filters and facets, all nofollow The overall result? Google re-crawled more pages. 60%
46. So, why crawl capacity management? Gastón Riera - @gastonriera
47. Crawl budget stayed the same. Gastón Riera - @gastonriera *On average, over the last 2yrs.
48. BONUS TRACK and unpopular opinion. Gastón Riera - @gastonriera Sitemaps didn't help indexing AT ALL 󰤃
49. But actually, it is Gastón Riera - @gastonriera
50. An unpopular Gastón Riera - @gastonriera
51. BONUS TRACK and unpopular opinion. We learnt that • Sitemaps didn't help indexing AT ALL 󰤃 • Helpful only for debugging 🤓 Gastón Riera - @gastonriera
52. Gastón Riera - @gastonriera Gracias! - Thank You!