Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serverless SEO / Edge SEO - SEOkomm 2021

Serverless SEO / Edge SEO - SEOkomm 2021

A hands-on guide on how to use "Edge SEO" to drive your SEO program forward, including a detailed explanation on how to setup, run and create Cloudflare Workers for SEO tasks. Presented at SEOkomm 2021 in Salzburg, Austria.

More Decks by Bastian Grimm

Other Decks in Marketing & SEO


  1. Bastian Grimm, Peak Ace AG | @basgr Serverless SEO a.k.a.

    “Edge SEO” - ein Hands-on-Guide.
  2. Cloudflare Worker

  3. In 2022 braucht ihr für SEO nicht mehr unbedingt einen

    Serverzugang! Access not needed!
  4. Daher: Folien auf Englisch … Ich bin nur Ersatz!

  5. Before we talk about Workers, we need to talk HTTP

    requests – and CDNs: Establishing a common ground
  6. pa.ag @peakaceag 6 A (very) simplified request lifecycle Your computer

    Your browser Database server (in most cases) DNS server e.g. to translate domain<>IP Web server aka “origin server”
  7. If you’re not familiar with the term CDN: “A content

    delivery network (CDN) is a globally distributed network of servers deployed in multiple data centers around the globe.” Let's introduce a CDN to the mix
  8. pa.ag @peakaceag 8 Using a CDN, all requests will pass

    through “edge servers“ When we ignore DNS, databases etc for a minute, this is what it would look like: First request, ever. peakace.js is not cached on edge server yet Origin server Request: peakace.js Request: peakace.js peakace.js delivered from origin server Response: peakace.js peakace.js gets cached on edge server
  9. pa.ag @peakaceag 9 Using a CDN, all requests will pass

    through “edge servers“ When we ignore DNS, databases etc for a minute, this is what it would look like: Origin server Request: peakace.js peakace.js delivered from edge server peakace.js is cached on edge server Second request (independent of user)
  10. pa.ag @peakaceag 10 CDNs at a glance Some of the

    most popular CDN providers out there
  11. pa.ag @peakaceag 11 Back in Sep. 2017, Cloudflare introduced their

    “Workers“ Which ultimately became publicly available in March 2018: Source: https://blog.cloudflare.com/introducing-cloudflare-workers/
  12. Workers use the V8 JavaScript engine built by Google and

    run globally on Cloudflare's edge servers. A typical Worker script executes in <1ms – that’s fast! So… what‘s a Worker?
  13. … using the latest standard language features You can execute

    any JavaScript…
  14. …directly from your Worker, or forward them elsewhere Respond to

  15. Also, you can do multiple requests, in series or parallel

    and combine the results Send requests to 3rd-party servers
  16. Intercept and modify HTTP request and response URLs, status, headers,

    and body content. Seriously though, this is WILD!
  17. Some of the potential use-cases could be to:

  18. 301s, 302s – or even geo-specific ones, if needed Implement

  19. Adding/changing X-robots or even hreflang annotations Modify HTTP headers

  20. Overwrite the full file, add or remove single directives Modify

  21. Inject, change or even remove robots meta-annotations Modify meta directives

  22. Create unique page titles or meta descriptions when needed Update

    page titles & descriptions
  23. … or schema.org mark-up Implement hreflang

  24. Essentially, you can do almost anything – because you have

    full access to the request and response objects! Inject/remove (body) content
  25. pa.ag @peakaceag 25 However, does this only work with Cloudflare?

    Similar implementations are also available with other CDN providers: [email protected] [email protected] Edge Workers Cloudflare Workers
  26. pa.ag @peakaceag 26 But today it‘s all about Cloudflare, because:

    The top 3 providers (CF, AWS, Akamai) have 89% of all customers; Cloudflare alone is used by 81% of all sites that rely on a CDN (according to W3Techs): Source: https://pa.ag/2U9kvAh
  27. A practical and hands-on guide to setting up and running

    Cloudflare Workers for your SEO Excited? Let‘s go!
  28. pa.ag @peakaceag 28 Go create your own (free) account over

    at cloudflare.com Once your account is activated, you can add your first site/domain: Add your domain name - it can be registered anywhere (as long as you can change the DNS at your current provider)
  29. pa.ag @peakaceag 29 To play with it, the free account

    + $0 plan is sufficient: This is good enough for testing things out…!
  30. pa.ag @peakaceag 30 Next, you‘ll get to see the current

    DNS configuration Yours should look a little like this: at least two records, one for the root-domain, one for the www sub-domain, both pointing to the IP address of your hosting provider: On to the next screen!
  31. pa.ag @peakaceag 31 Now, CF will show you which nameservers

    to use instead: Nameservers with the current provider, in my case nsX.inwx.de My new nameservers with Cloudflare to be used instead
  32. pa.ag @peakaceag 32 Switching existing nameservers over to Cloudflare At

    my hosting provider, it looks like this: My new nameservers Cloudflare told me to use instead (see prev. screen)
  33. pa.ag @peakaceag 33 Switch back to tell them you’re all

    set: Nameservers with the current provider, in my case nsX.inwx.de My new nameservers with Cloudflare, to be used instead
  34. pa.ag @peakaceag 34 Cloudflare is going to email you when

    things are ready: Beware, this can take up to 24hrs depending on the registrars and nameservers: Your CF dashboard should look like this after the successful NS change.
  35. So far, so good – let‘s talk Workers now.

  36. pa.ag @peakaceag 36 A Worker, in its simplest form: This

    function defines triggers for a Worker script to execute. In this case, we intercept the request and send a (custom) response. Our custom response is defined here, for now we simply: (6) log the request object (7) fetch the requested URL from the origin server (8) log the response object (10) send the (unmodified) response back to the client
  37. pa.ag @peakaceag 37 Cloudflare Workers Playground: cloudflareworkers.com Test-drive Cloudflare Workers;

    create/edit and see the results live:
  38. pa.ag @peakaceag 38 Cloudflare Workers Playground: cloudflareworkers.com Test-drive Cloudflare Workers;

    create/edit and see the results live:
  39. pa.ag @peakaceag 39 Let's build our own first Worker /

    custom handleRequest:
  40. pa.ag @peakaceag 40 Let‘s test-drive this on the Workers Playground:

  41. Let's live-deploy your first Worker! Enough testing…

  42. pa.ag @peakaceag 42 Let‘s live-deploy the Worker to Cloudflare's edge

    servers Select your domain > Workers > Manage Workers
  43. pa.ag @peakaceag 43 Here’s how to add a Worker You‘ll

    be redirected from the “all Workers“ overview to the following mask: Give your Worker a unique name Copy & paste the Workers code you just tested on the Playground
  44. pa.ag @peakaceag 44 Confirm deployment and assign routing Go back

    to > Workers > Add route 1 2
  45. pa.ag @peakaceag 45 Comparison: left (original), right (Worker-enabled) Double-check live!

    Also, don‘t fall victim to caching, use “Disable Cache“ (see: Network tab) in Chrome Dev Tools to be sure you‘re seeing the latest version: vs
  46. Let’s have some SEO fun then? So much for theory…

  47. Please understand that all scripts / source codes are meant

    as examples only. Ensure you know what you’re doing when using them in a production environment! Warning: (maybe) not production-ready!
  48. 301 / 302 / bulk redirects & proxy passthrough 1.

  49. pa.ag @peakaceag 49 Redirects on the edge using the Response

    API To execute any type of HTTP redirect, we need to use the Response Runtime API which – conveniently – also provides a static method called “redirect()”: Source: https://pa.ag/3gvXYoL let response = new Response(body, options) return Response.redirect(destination, status) or just:
  50. pa.ag @peakaceag 50 The Cloudflare Workers Docs is a solid

    starting point: More: https://pa.ag/3gNd8Gn
  51. pa.ag @peakaceag 51 Different types of implementations at a glance

    (#18): 302 redirect, (#22): 301 redirect, (#26): a reverse proxy call and (#31-36): multiple redirects, selecting a single destination from a map based on a URL parameter:
  52. pa.ag @peakaceag 52 A quick overview to see how things

    are working… Source: https://httpstatus.io Correct, in fact this is not a redirect ID is not configured in redirectMap
  53. pa.ag @peakaceag 53 To “reverse proxy” a request, you can

    use the Fetch API It provides an interface for (asynchronously) fetching resources via HTTP requests inside of a Worker script: Source: https://pa.ag/3wpS3YT const response = await fetch(URL, options) Asynchronous tasks, such as fetch, are not executed at the top level in a Worker script and must be executed within a FetchEvent handler.
  54. pa.ag @peakaceag 54 return await fetch(“https://example.com ”) Easily “migrate” a

    blog hosted on a sub-domain to a sub-folder on your main domain – without actually moving it. Great tutorial: https://pa.ag/2Tw7LD8 Content shown from example.com Request sent from bastiangrimm.dev
  55. pa.ag @peakaceag 55 Verifying that this all happens “on the

    edge“: Zoom into any of the response headers for an originally requested URL such as bastiangrimm.dev/redirects/302:
  56. Safeguards, monitoring, serving a default file, etc. 2. robots.txt

  57. pa.ag @peakaceag 57 Which version would you like to wake

    up to? Preventing “SEO heart attacks“ using a Worker to monitor and safeguard your robots.txt file is one of many use-cases that are super easy to do: This is how I uploaded the robots.txt file to my test server This is what the Worker running in the background changed the output to vs
  58. pa.ag @peakaceag 58 Preventing a global “Disallow: /“ in robots.txt

    (#5-6): define defaults, (#15-16): if robots.txt returns 200, read its content, (#19-24): replace if global disallow exists, (#27-29): return default “allow all” if file doesn’t exist
  59. So, let’s do some “dynamic serving“ shall we? Static files

    become dynamic
  60. pa.ag @peakaceag 60 For demonstration purposes only: UA-based delivery (#10):

    get User-Agent, (#16-17): add dynamic Sitemap-link if UA contains “googlebot”
  61. pa.ag @peakaceag 61 vs Live-test & compare robots.txt using technicalseo.com

    Left screen shows bastiangrimm.dev/robots.txt being requested using a Googlebot User-Agent string, right screen is the default output: Free testing tool: https://technicalseo.com/tools/robots-txt/ Or use…
  62. Some systems cause endless headaches for SEOs – routing them

    through Cloudflare and using a Worker works very well! Easily overwrite files which are “not meant” to be changed?
  63. Modifying robots HTML directives on the fly 3. Robots meta

  64. pa.ag @peakaceag 64 Say hello to the HTMLRewriter class! The

    HTMLRewriter allows you to build comprehensive and expressive HTML parsers inside of a Cloudflare Workers application: Source: https://pa.ag/2RTpqEt new HTMLRewriter() .on("*", new ElementHandler()) .onDocument(new DocumentHandler())
  65. pa.ag @peakaceag 65 Let's give it a try and work

    with <head> and <meta> first (#24-25): pass tags to ElementHandler, (#9-11): if it’s <meta name=“robots”>, set it to “index,nofollow”, (#14-16): if it’s <head>, add another directive for bingbot:
  66. pa.ag @peakaceag 66 I mean, this should be clear –

    but just in case: Verifying presence of Worker-modified robots meta directives via GSC
  67. pa.ag @peakaceag 67 If you want to work with/on every

    HTML element… This selector would pass every HTML element to your ElementHandler. By using element.tagName, you could then identify which element has been passed along: return new HTMLRewriter() .on("*", new ElementHandler()) .transform(response)
  68. Of course, updating, changing or entirely replacing both elements is

    also possible! 4. Title and meta description
  69. pa.ag @peakaceag 69 Using element selectors in HTMLRewriter Often, you

    want only to process very specific elements, e.g. <meta> tags – but not all of them. Maybe it’s just the meta description you care about? new HTMLRewriter() .on('meta[name="description"]', new ElementHandler()) .transform(response) More on selectors: https://pa.ag/35xw073
  70. pa.ag @peakaceag 70 Updating or replacing titles and descriptions is

    easy… (#10): forced <title> overwrite, (#14-22): conditional changes to the meta description Element selectors are super powerful yet easy to use:
  71. Maybe you missed some during your last migration? 5. Rewriting

  72. pa.ag @peakaceag 72 HTMLRewriter listening to <a> and <img> tags

    (#29-30): Passing href/src attributes to a class which (#20): replaces oldURL with newUrl and (#16-18): ensures https-availability Based on: https://pa.ag/35llTSo
  73. Tell Google about localised versions of your page 6. Deploying

  74. pa.ag @peakaceag 74 HTTP hreflang annotations on the edge We‘ve

    just had plenty of HTML, so let‘s use HTTP headers instead – of course both ways work just fine though:
  75. pa.ag @peakaceag 75 Verify, e.g. using Chrome Developer Console: Network

    > %URL (disable cache) > Headers > Response Headers
  76. pa.ag @peakaceag 76 Before you ask: X-Robots directives are also

    possible… … and the same is true for X-Robots Rel-Canonical annotations:
  77. You can serve “proper“ under maintenance pages directly from your

    Cloudflare Worker 7. Serving HTTP 503s
  78. pa.ag @peakaceag 78 Combining HTTP 503 error with a Retry-After

    header Retry-After indicates how long the UA should wait before making a follow-up request: The server is currently unable to handle the request due to a temporary overloading or maintenance of the server […]. If known, the length of the delay MAY be indicated in a Retry-After header.
  79. Just in case I’ve somehow not made my point yet

    – you can do REALLY cool stuff and have control over the full HTML response – so adding content is easy. 8. Injecting content
  80. pa.ag @peakaceag 80 Replacing, prepending, appending … whatever you like?

  81. pa.ag @peakaceag 81 You could also (dynamically) read from an

    external feed Feeding in content from other sources is simple; below shows reading a JSON feed, parsing the input and inject to the <h1> of the target page:
  82. Yeah… actually this is how it all started; and still

    it‘s (one of) the most powerful tools to use for it! 9. Web performance
  83. pa.ag @peakaceag 83 Add native lazy loading for images to

    your HMTL mark-up Keep in mind: you don‘t want to lazy load all of your images (e.g. not the hero image); also, if you‘re using iframes, you might want to pass “iframe“ to the HTMLRewriter:
  84. pa.ag @peakaceag 84 Cleaning up HTML code for performance reasons

    E.g. by removing unwanted pre*-stages, or by adding async/defer to JS calls: More clean-up Worker scripts: https://gist.github.com/Nooshu
  85. pa.ag @peakaceag 85 A detailed guide on how to cache

    HTML with CF Workers More: https://pa.ag/3xk8rdt
  86. You can use Workers to fix broken tracking, allow for

    better accessibility, and much more. And tons of other things…
  87. Downsides, risks – and more… We need to talk responsibility

  88. pa.ag @peakaceag 88 [This] dates back to the time of

    the French Revolution At least, if you believe Wikipedia that is… Source: https://pa.ag/35nQSx6 With great power comes great responsibility.
  89. You could essentially change everything you wanted.

  90. pa.ag @peakaceag 90 Great summary over at ContentKing, well worth

    a read: What are the downsides […] What risks are involved? Source: https://pa.ag/3xhYUUk
  91. 10 million requests are included, every 1 million currently costs

    $0.50 extra - not crazy expensive, but in larger-scale setups certainly means additional costs. Risk of costs
  92. This might interfere with current processes, or at the very

    least, ensure Workers become part of a standardised process (e.g. deployment). PCI compliance
  93. The underlying codebase might do/require something that could accidentally be

    overwritten on the edge Potential conflict in code
  94. Additional modifications on the edge could result in massive debugging.

    Again: proper documentation and processes are crucial! Potential to introduce frontend bugs
  95. Always synchronise your activities with relevant stakeholders!

  96. … there is also evil Where there is good…

  97. pa.ag @peakaceag 97 Yep, you can do evil things with

    Workers for sure: Source: https://pa.ag/3cFq0Nq
  98. pa.ag @peakaceag 98 Dynamically creating links to “Baccarat Sites” “[…]

    at the CF Workers management area, there was a suspicious Worker listed called hang. It had been set to run on any URL route requests to the website:” Source: https://pa.ag/3cFq0Nq After further investigation [by sucuri], it was found that the website was actually loading SEO spam content through Cloudflare’s Workers service. This service allows someone to load external third-party JavaScript that’s not on their website’s hosting server.
  99. pa.ag @peakaceag 99 The suspicious “hang” Worker injection in detail:

    Source: https://pa.ag/3cFq0Nq ▪ The JavaScript Worker first checks for the HTTP request’s user-agent and identifies whether it contains Google/Googlebot or naver within the string text. ▪ If the user-agent string contains either of these keywords, then the JavaScript makes a request to the malicious domain naverbot[.]live to generate the SEO spam links to be injected into the victim’s website. ▪ After this step, the Worker then injects the retrieved SEO spam link data right before the final </body> tag on the infected website’s HTML source. ▪ The malicious JavaScript can also be triggered if the user-agent matches a crawler that is entirely separate from Googlebot: naver.
  100. pa.ag @peakaceag 100 If you‘re now wondering how to distribute

    Workers…? Source: https://pa.ag/3zq0Mwd
  101. Use at least two-factor authentication with Cloudflare While you‘re at

  102. Care for the slides? www.pa.ag twitter.com/peakaceag facebook.com/peakaceag Take your career

    to the next level: jobs.pa.ag [email protected] Bastian Grimm [email protected]