Tweet
Tweet

Slide 1

Slide 1 text

Crawl like an expert: Bolster your SEO strategy with the right data

Slide 2

Slide 2 text

● Welcome! ● There will be a replay — you'll receive it in a few days. ● Feel free to ask questions in the questions tab at the bottom of your screen. We'll answer them at the end. Crawl like an expert: Bolster your SEO strategy with the right data

Slide 3

Slide 3 text

Our panel today Rebecca Berbel Frédéric Gérard Mickaël Serantes Product Marketing Manager Senior SEO Strategist Head of Product

Slide 4

Slide 4 text

Rich data and cross-analysis Excellent scalability Powerful segmentation Permanent data availability and history Support your competitive digital strategy and ensure website and brand visibility on search engines. Industry-leading Technical SEO Data for Competitive Websites www.oncrawl.com

Slide 5

Slide 5 text

Crawl like an expert: Bolster your SEO strategy with the right data ● Why crawling is not one-size-fits-all ● Crawling a sitemap ● Crawling only part of a site ● Different types of crawl when migrating a site ● Efficient alerting ● Q&A

Slide 6

Slide 6 text

Crawling isn't one-size-fits-all

Slide 7

Slide 7 text

How to ensure a good crawl strategy with Oncrawl?

Slide 8

Slide 8 text

The 3 pillars of a consistent crawl strategy Goal of the crawl Monitor changes Crawl regularly Global crawl strategy

Slide 9

Slide 9 text

What is a crawl for? You can analyze how pages are linked together and whether the structure is logical and efficient It can uncover navigational issues, broken links, slow-loading pages, and other barriers that might affect the user experience Crawling your website helps in understanding how search engines view your site Identifying potential security vulnerabilities and ensuring compliance with various standards and regulations User Experience Improvement Search Engine Optimization (SEO): Site Structure Analysis: Security and Compliance Checks

Slide 10

Slide 10 text

Start with a new crawl

Slide 11

Slide 11 text

Start with the crawl profile configuration

Slide 12

Slide 12 text

Start with the crawl profile configuration

Slide 13

Slide 13 text

Start with the crawl profile configuration

Slide 14

Slide 14 text

Crawling the URLs in a sitemap

Slide 15

Slide 15 text

Why crawl using a sitemap? ● Ensuring Search Engine Accessibility ● You can allow Oncrawl to discover sitemaps from a directory, subdomain, or URL; or you can provide the URLs of one or more sitemaps that you want to use.

Slide 16

Slide 16 text

Why crawl using a sitemap? ● Ensuring Search Engine Accessibility ● You can allow Oncrawl to discover sitemaps from a directory, subdomain, or URL; or you can provide the URLs of one or more sitemaps that you want to use.

Slide 17

Slide 17 text

Crawling only part of a site

Slide 18

Slide 18 text

Why crawl part of a website ● Prioritizing your key pages ○ Focus your analysis on what you’re optimizing right now ○ Monitor basic but vital information ○ Let the crawler explore your key pages (PLPs, PDPs, articles, etc…) ○ Extract specific informations from them (Scraping)

Slide 19

Slide 19 text

Why crawl part of a website ● Prepare your migration ○ Before migration overview ○ Follow your migration / optimizations with precision

Slide 20

Slide 20 text

Virtual robots.txt URL Filtering

Slide 21

Slide 21 text

Virtual robots.txt ● Override your live robots.txt ● Crawl blocked pages ● Crawl only some subdomains ● Crawl faster than the speed set in the crawl delay ● Test different rules on your Preprod / Prod environment ● Easy to implement / correct Pros URL Filtering

Slide 22

Slide 22 text

Virtual robots.txt ● Override your live robots.txt ● Crawl blocked pages ● Crawl only some subdomains ● Crawl faster than the speed set in the crawl delay ● Test your rules on your Preprod / Prod environment ● Easy to implement / correct ● May not crawl all your pages Cons Pros URL Filtering

Slide 23

Slide 23 text

Virtual robots.txt ● Override your live robots.txt ● Crawl blocked pages ● Crawl only some subdomains ● Crawl faster than the speed set in the crawl delay ● Test your rules on your Preprod / Prod environment ● Easy to implement / correct ● May not crawl all your pages Cons Pros ● Extremely powerful with Regex ● Include AND Exclude ● Take into account all pages ● Respect your live robots.txt Pros URL Filtering

Slide 24

Slide 24 text

Virtual robots.txt ● Override your live robots.txt ● Crawl blocked pages ● Crawl only some subdomains ● Crawl faster than the speed set in the crawl delay ● Test your rules on your Preprod / Prod environment ● Easy to implement / correct ● May not crawl all your pages Cons Pros ● Extremely powerful with Regex ● Include AND Exclude ● Take into account all pages ● Respect your live robots.txt ● Be familiar with Regex Cons Pros URL Filtering

Slide 25

Slide 25 text

1. Your staging is Disallowed

Slide 26

Slide 26 text

2. Crawl your PDPs with a virtual robots.txt

Slide 27

Slide 27 text

Homepage 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2

Slide 28

Slide 28 text

Homepage 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Allowed Allowed & crawled Allowed & crawled

Slide 29

Slide 29 text

Homepage 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4 Allowed Allowed & crawled Allowed & crawled Landing 1 /category/cooking

Slide 30

Slide 30 text

Homepage 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4 Allowed Allowed & crawled Allowed & crawled Disallowed Landing 1 /category/cooking

Slide 31

Slide 31 text

Homepage 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4 Allowed Allowed & crawled Allowed & crawled Disallowed Allowed but not crawled Allowed but not crawled Landing 1 /category/cooking

Slide 32

Slide 32 text

Homepage Landing 1 /category/cooking 2. Crawl your PDPs with a virtual robots.txt Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4 Allowed Allowed & crawled Allowed & crawled Disallowed Allowed but not crawled Allowed but not crawled Your crawl won’t be accurate!

Slide 33

Slide 33 text

C0 - Public Crawl your product pages What’s the solution?

Slide 34

Slide 34 text

C0 - Public What’s the solution? Use URL filtering instead Crawl your product pages

Slide 35

Slide 35 text

3. Crawl your PDPs with URL Filtering

Slide 36

Slide 36 text

3. Crawl your PDPs with URL Filtering Homepage Landing 1 /category/cooking Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4

Slide 37

Slide 37 text

3. Crawl your PDPs with URL Filtering Homepage Landing 1 /category/cooking Promoted Product 1 /products/my_product1 Promoted Product 2 /products/my_product2 Product 3 /products/my_product3 Product 4 /products/my_product4 explored not fetched explored not fetched explored & fetched explored & fetched explored & fetched explored & fetched

Slide 38

Slide 38 text

Crawling during a migration

Slide 39

Slide 39 text

1. Before your migration ● Full overview crawl or goal-oriented crawl ● Different crawl profile for each goal ● Crawl your staging environment ● Scrape the data you need ● List, extract & export useful informations (pages to redirect, pages to optimize, links to change, pages without content, off stock product pages, empty PLPs, etc…) ● Step-by-step migration & crawl

Slide 40

Slide 40 text

2. During your migration (Preprod) ● Crawl your staging environment (even non-accessible to search engines)

Slide 41

Slide 41 text

2. During your migration (Preprod) ● Using specific credentials

Slide 42

Slide 42 text

2. During your migration (Preprod) ● Using specific credentials

Slide 43

Slide 43 text

2. During your migration (Preprod) ● Using specific credentials

Slide 44

Slide 44 text

2. During your migration (Preprod) ● To crawl your sites hosted on a different server, like a pre-production server

Slide 45

Slide 45 text

What’s new about Oncrawl's crawl configuration feature?

Slide 46

Slide 46 text

User Agent: Crawl with another user agent ● Test results with JS pages ○ SSR (Server side rendering) ○ CSR (Client side rendering) ● Follow all the parameters of the txt robot to verify that it is properly configured for Google. ● Internally: security and crawlability management on an IP/Bot pair for certain sites (hidden preprod)

Slide 47

Slide 47 text

User Agent: Crawl with another user agent

Slide 48

Slide 48 text

2. During your migration (Preprod) ● Crawl your staging environment (even non-accessible to search engines) ● Compare it to your live site & spot differences using crawl comparison: ○ Tech SEO

Slide 49

Slide 49 text

2. During your migration (Preprod) ● Crawl your staging environment (even non-accessible to search engines) ● Compare it to your live site & spot differences using crawl comparison: ○ Tech SEO ○ Internal structure ○ Internal linking ○ Internal popularity

Slide 50

Slide 50 text

2. During your migration (Preprod) ● Crawl your staging environment (even non-accessible to search engines) ● Compare it to your live site & spot differences using crawl comparison: ○ Tech SEO ○ Internal structure ○ Internal linking ○ Internal popularity ○ Content & duplicate content

Slide 51

Slide 51 text

2. During your migration (Preprod) ● Crawl your staging environment (even non-accessible to search engines) ● Compare it to your live site & spot differences using crawl comparison: ○ Tech SEO ○ Internal structure ○ Internal linking ○ Internal popularity ○ Content & duplicate content ○ Webperf & Core Web Vitals

Slide 52

Slide 52 text

3. After your migration ● Run a full crawl or focused on part of the site ● Check your redirections by crawling your old pages (now redirected in 301) ○ Use the URL List crawl mode

Slide 53

Slide 53 text

3. After your migration ● Run a full crawl or focused on part of the site ● Check your redirections by crawling your old pages (now redirected in 301) ○ Use the URL List crawl mode ● Compare your data before and after the migration to spot differences or issues (Crawl over Crawl)

Slide 54

Slide 54 text

TIPS ● Create a Sitemap for your redirected pages

Slide 55

Slide 55 text

TIPS ● Create a Sitemap for your 301 pages ● Submit it to Google through GSC ○ Faster detection and processing of your redirects

Slide 56

Slide 56 text

TIPS ● Create a Sitemap for your 301 pages ● Submit it to Google through GSC ○ Faster detection and processing of your redirects ● Use Sitemap crawl mode to ensure all your pages are redirected to valid pages.

Slide 57

Slide 57 text

5. Schedule your crawls ● Schedule a daily/weekly/monthly crawl to automatically collect fresh data

Slide 58

Slide 58 text

5. Schedule your crawls ● Schedule a daily/weekly/monthly crawl to automatically collect fresh data ● Create custom alerts to detect any issue

Slide 59

Slide 59 text

Crawling to drive monitoring

Slide 60

Slide 60 text

Efficient alerting ● Website monitoring, ● Quality assurance, ● Business cases with custom fields

Slide 61

Slide 61 text

Alerts on specific topics ● Pages returning a specific status code ● Pages with a duplicate or missing title tag ● Pages forbidden by robots.txt You can also create some about business cases using your own custom fields ● Scrape your home page ● Mandatory elements in your page description ● Stock verification…

Slide 62

Slide 62 text

To summarize: have a global crawl strategy ● Understand the context of your crawl and create a dedicated crawl profile ● Simplify your daily life by creating schedule crawls for the different crawl profiles you use. ● Stay informed about changes or issues related to your site by Creating alerts

Slide 63

Slide 63 text

Crawling: Top takeaways

Slide 64

Slide 64 text

Goal of the crawl Monitor changes Crawl regularly Global crawl strategy Crawling: Top takeaways

Slide 65

Slide 65 text

Crawling: Top takeaways You don't always have to crawl your full site! ● Crawl the pages in your sitemaps ● Monitor basic but vital information ● Explore your key pages only (PLPs, PDPs, articles, etc…)

Slide 66

Slide 66 text

Crawling: Top takeaways You don't always have to crawl your full site! ● Crawl the pages in your sitemaps ● Monitor basic but vital information ● Explore your key pages only (PLPs, PDPs, articles, etc…) Use different settings to target different site sections ● Robots.txt ● URL filtering

Slide 67

Slide 67 text

Crawling: Top takeaways You don't always have to crawl your full site! ● Crawl the pages in your sitemaps ● Monitor basic but vital information ● Explore your key pages only (PLPs, PDPs, articles, etc…) Use different settings to target different site sections ● Robots.txt ● URL filtering Use different settings and scopes depending on the context ● Authentication ● User-Agents ● List mode (example: lets you check lists of redirects)

Slide 68

Slide 68 text

Crawling: Top takeaways You don't always have to crawl your full site! ● Crawl the pages in your sitemaps ● Monitor basic but vital information ● Explore your key pages only (PLPs, PDPs, articles, etc…) Use different settings to target different site sections ● Robots.txt ● URL filtering Use different settings and scopes depending on the context ● Authentication ● User-Agents ● List mode (example: lets you check lists of redirects) Always monitor changes ● Regular crawls ● Different teams can be alerted

Slide 69

Slide 69 text

Crawling: Next steps Are you: Looking at the right parts of your website? Capturing the right information at the right time? Taking "snapshots" frequently enough? Showing the right changes to the right people?

Slide 70

Slide 70 text

Any questions? (ask them in the questions tab)

Slide 71

Slide 71 text

Thank for your attention Book your demo www.oncrawl.com