Crawl like an expert: Bolster your SEO strategy with the right data

Crawling a website is a good way to collect information about the website and its pages at scale. However, if you’ve ever tried to crawl a website with a specific goal in mind, you've likely discovered that getting the right information for the right pages is not so straightforward. What are the different reasons to crawl a website? What modifications should you make to get the exact information you need?

In this webinar, Rebecca Berbel, Frédéric Gérard and Mickael Serantes will explore how to guide your crawler to do exactly what you want. We’ll examine different scenarios from website monitoring to crawling key pages to accessing protected sites with a crawler (without breaking any rules).

You’ll walk away confident that your next crawls will include the right pages to provide actionable solutions for your technical SEO needs.

Transcript

1. Crawl like an expert: Bolster your SEO strategy with the right data
2. • Welcome! • There will be a replay — you'll receive it in a few days. • Feel free to ask questions in the questions tab at the bottom of your screen. We'll answer them at the end. Crawl like an expert: Bolster your SEO strategy with the right data
3. Our panel today Rebecca Berbel Frédéric Gérard Mickaël Serantes Product Marketing Manager Senior SEO Strategist Head of Product
4. Rich data and cross-analysis Excellent scalability Powerful segmentation Permanent data availability and history Support your competitive digital strategy and ensure website and brand visibility on search engines. Industry-leading Technical SEO Data for Competitive Websites www.oncrawl.com
5. Crawl like an expert: Bolster your SEO strategy with the right data • Why crawling is not one-size-fits-all • Crawling a sitemap • Crawling only part of a site • Different types of crawl when migrating a site • Efficient alerting • Q&A
6. Crawling isn't one-size-fits-all
7. How to ensure a good crawl strategy with Oncrawl?
8. The 3 pillars of a consistent crawl strategy Goal of the crawl Monitor changes Crawl regularly Global crawl strategy
9. What is a crawl for? You can analyze how pages are linked together and whether the structure is logical and efficient It can uncover navigational issues, broken links, slow-loading pages, and other barriers that might affect the user experience Crawling your website helps in understanding how search engines view your site Identifying potential security vulnerabilities and ensuring compliance with various standards and regulations User Experience Improvement Search Engine Optimization (SEO): Site Structure Analysis: Security and Compliance Checks
10. Start with a new crawl
11. Start with the crawl profile configuration
12. Start with the crawl profile configuration
13. Start with the crawl profile configuration
14. Crawling the URLs in a sitemap
15. Why crawl using a sitemap? • Ensuring Search Engine Accessibility • You can allow Oncrawl to discover sitemaps from a directory, subdomain, or URL; or you can provide the URLs of one or more sitemaps that you want to use.
16. Why crawl using a sitemap? • Ensuring Search Engine Accessibility • You can allow Oncrawl to discover sitemaps from a directory, subdomain, or URL; or you can provide the URLs of one or more sitemaps that you want to use.
17. Crawling only part of a site
18. Why crawl part of a website • Prioritizing your key pages ◦ Focus your analysis on what you’re optimizing right now ◦ Monitor basic but vital information ◦ Let the crawler explore your key pages (PLPs, PDPs, articles, etc…) ◦ Extract specific informations from them (Scraping)
19. Why crawl part of a website • Prepare your migration ◦ Before migration overview ◦ Follow your migration / optimizations with precision
20. Virtual robots.txt URL Filtering
21. Virtual robots.txt • Override your live robots.txt • Crawl blocked pages • Crawl only some subdomains • Crawl faster than the speed set in the crawl delay • Test different rules on your Preprod / Prod environment • Easy to implement / correct Pros URL Filtering
22. Virtual robots.txt • Override your live robots.txt • Crawl blocked pages • Crawl only some subdomains • Crawl faster than the speed set in the crawl delay • Test your rules on your Preprod / Prod environment • Easy to implement / correct • May not crawl all your pages Cons Pros URL Filtering
23. Virtual robots.txt • Override your live robots.txt • Crawl blocked pages • Crawl only some subdomains • Crawl faster than the speed set in the crawl delay • Test your rules on your Preprod / Prod environment • Easy to implement / correct • May not crawl all your pages Cons Pros • Extremely powerful with Regex • Include AND Exclude • Take into account all pages • Respect your live robots.txt Pros URL Filtering
24. Virtual robots.txt • Override your live robots.txt • Crawl blocked pages • Crawl only some subdomains • Crawl faster than the speed set in the crawl delay • Test your rules on your Preprod / Prod environment • Easy to implement / correct • May not crawl all your pages Cons Pros • Extremely powerful with Regex • Include AND Exclude • Take into account all pages • Respect your live robots.txt • Be familiar with Regex Cons Pros URL Filtering
25. 1. Your staging is Disallowed
26. 2. Crawl your PDPs with a virtual robots.txt
27. Homepage 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2
28. Homepage 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Allowed Allowed & crawled Allowed & crawled
29. Homepage 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4 Allowed Allowed & crawled Allowed & crawled Landing 1 /category/cooking
30. Homepage 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4 Allowed Allowed & crawled Allowed & crawled Disallowed Landing 1 /category/cooking
31. Homepage 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4 Allowed Allowed & crawled Allowed & crawled Disallowed Allowed but not crawled Allowed but not crawled Landing 1 /category/cooking
32. Homepage Landing 1 /category/cooking 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4 Allowed Allowed & crawled Allowed & crawled Disallowed Allowed but not crawled Allowed but not crawled Your crawl won’t be accurate!
33. C0 - Public Crawl your product pages What’s the solution?
34. C0 - Public What’s the solution? Use URL filtering instead Crawl your product pages
35. 3. Crawl your PDPs with URL Filtering
36. 3. Crawl your PDPs with URL Filtering Homepage Landing 1 /category/cooking Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4
37. 3. Crawl your PDPs with URL Filtering Homepage Landing 1 /category/cooking Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4 explored not fetched explored not fetched explored & fetched explored & fetched explored & fetched explored & fetched
38. Crawling during a migration
39. 1. Before your migration • Full overview crawl or goal-oriented crawl • Different crawl profile for each goal • Crawl your staging environment • Scrape the data you need • List, extract & export useful informations (pages to redirect, pages to optimize, links to change, pages without content, off stock product pages, empty PLPs, etc…) • Step-by-step migration & crawl
40. 2. During your migration (Preprod) • Crawl your staging environment (even non-accessible to search engines)
41. 2. During your migration (Preprod) • Using specific credentials
42. 2. During your migration (Preprod) • Using specific credentials
43. 2. During your migration (Preprod) • Using specific credentials
44. 2. During your migration (Preprod) • To crawl your sites hosted on a different server, like a pre-production server
45. What’s new about Oncrawl's crawl configuration feature?
46. User Agent: Crawl with another user agent • Test results with JS pages ◦ SSR (Server side rendering) ◦ CSR (Client side rendering) • Follow all the parameters of the txt robot to verify that it is properly configured for Google. • Internally: security and crawlability management on an IP/Bot pair for certain sites (hidden preprod)
47. User Agent: Crawl with another user agent
48. 2. During your migration (Preprod) • Crawl your staging environment (even non-accessible to search engines) • Compare it to your live site & spot differences using crawl comparison: ◦ Tech SEO
49. 2. During your migration (Preprod) • Crawl your staging environment (even non-accessible to search engines) • Compare it to your live site & spot differences using crawl comparison: ◦ Tech SEO ◦ Internal structure ◦ Internal linking ◦ Internal popularity
50. 2. During your migration (Preprod) • Crawl your staging environment (even non-accessible to search engines) • Compare it to your live site & spot differences using crawl comparison: ◦ Tech SEO ◦ Internal structure ◦ Internal linking ◦ Internal popularity ◦ Content & duplicate content
51. 2. During your migration (Preprod) • Crawl your staging environment (even non-accessible to search engines) • Compare it to your live site & spot differences using crawl comparison: ◦ Tech SEO ◦ Internal structure ◦ Internal linking ◦ Internal popularity ◦ Content & duplicate content ◦ Webperf & Core Web Vitals
52. 3. After your migration • Run a full crawl or focused on part of the site • Check your redirections by crawling your old pages (now redirected in 301) ◦ Use the URL List crawl mode
53. 3. After your migration • Run a full crawl or focused on part of the site • Check your redirections by crawling your old pages (now redirected in 301) ◦ Use the URL List crawl mode • Compare your data before and after the migration to spot differences or issues (Crawl over Crawl)
54. TIPS • Create a Sitemap for your redirected pages
55. TIPS • Create a Sitemap for your 301 pages • Submit it to Google through GSC ◦ Faster detection and processing of your redirects
56. TIPS • Create a Sitemap for your 301 pages • Submit it to Google through GSC ◦ Faster detection and processing of your redirects • Use Sitemap crawl mode to ensure all your pages are redirected to valid pages.
57. 5. Schedule your crawls • Schedule a daily/weekly/monthly crawl to automatically collect fresh data
58. 5. Schedule your crawls • Schedule a daily/weekly/monthly crawl to automatically collect fresh data • Create custom alerts to detect any issue
59. Crawling to drive monitoring
60. Efficient alerting • Website monitoring, • Quality assurance, • Business cases with custom fields
61. Alerts on specific topics • Pages returning a specific status code • Pages with a duplicate or missing title tag • Pages forbidden by robots.txt You can also create some about business cases using your own custom fields • Scrape your home page • Mandatory elements in your page description • Stock verification…
62. To summarize: have a global crawl strategy • Understand the context of your crawl and create a dedicated crawl profile • Simplify your daily life by creating schedule crawls for the different crawl profiles you use. • Stay informed about changes or issues related to your site by Creating alerts
63. Crawling: Top takeaways
64. Goal of the crawl Monitor changes Crawl regularly Global crawl strategy Crawling: Top takeaways
65. Crawling: Top takeaways You don't always have to crawl your full site! • Crawl the pages in your sitemaps • Monitor basic but vital information • Explore your key pages only (PLPs, PDPs, articles, etc…)
66. Crawling: Top takeaways You don't always have to crawl your full site! • Crawl the pages in your sitemaps • Monitor basic but vital information • Explore your key pages only (PLPs, PDPs, articles, etc…) Use different settings to target different site sections • Robots.txt • URL filtering
67. Crawling: Top takeaways You don't always have to crawl your full site! • Crawl the pages in your sitemaps • Monitor basic but vital information • Explore your key pages only (PLPs, PDPs, articles, etc…) Use different settings to target different site sections • Robots.txt • URL filtering Use different settings and scopes depending on the context • Authentication • User-Agents • List mode (example: lets you check lists of redirects)
68. Crawling: Top takeaways You don't always have to crawl your full site! • Crawl the pages in your sitemaps • Monitor basic but vital information • Explore your key pages only (PLPs, PDPs, articles, etc…) Use different settings to target different site sections • Robots.txt • URL filtering Use different settings and scopes depending on the context • Authentication • User-Agents • List mode (example: lets you check lists of redirects) Always monitor changes • Regular crawls • Different teams can be alerted
69. Crawling: Next steps Are you: Looking at the right parts of your website? Capturing the right information at the right time? Taking "snapshots" frequently enough? Showing the right changes to the right people?
70. Any questions? (ask them in the questions tab)
71. Thank for your attention Book your demo www.oncrawl.com