My talk from SMX Advanced 2021 outlining how to use Cloudflare Workers to overcome challenges and limitations with popular CMS and ecommerce platforms.
pa.ag @peakaceag 13 A (very) simplified request lifecycle Your computer Your browser Database server (in most cases) DNS server e.g. to translate domain<>IP Web server aka “origin server”
If you’re not familiar with the term CDN: “A content delivery network (CDN) is a globally distributed network of servers deployed in multiple data centers around the globe.” Let's introduce a CDN to the mix
pa.ag @peakaceag 15 Using a CDN, all requests will pass through “edge servers“ When we ignore DNS, databases etc for a minute, this is what it would look like: First request, ever. peakace.js is not cached on edge server yet Origin server Request: peakace.js Request: peakace.js peakace.js delivered from origin server Response: peakace.js peakace.js gets cached on edge server
pa.ag @peakaceag 16 Using a CDN, all requests will pass through “edge servers“ When we ignore DNS, databases etc for a minute, this is what it would look like: Origin server Request: peakace.js peakace.js delivered from edge server peakace.js is cached on edge server Second request (independent of user)
pa.ag @peakaceag 17 Especially for global businesses, CDNs can be a great help Use CDNPerf.com to find the one that suits you best, depending on where you are and which regions/countries you serve most. This will positively impact TTFB! Give it a try: https://www.cdnperf.com/ vs
pa.ag @peakaceag 19 Back in Sep. 2017, Cloudflare introduced their “Workers“ Which ultimately became publicly available in March 2018: Source: https://blog.cloudflare.com/introducing-cloudflare-workers/
Workers use the V8 JavaScript engine built by Google and run globally on Cloudflare's edge servers. A typical Worker script executes in <1ms – that’s fast! So… what‘s a Worker?
pa.ag @peakaceag 33 However, does this only work with Cloudflare? Similar implementations are also available with other CDN providers: [email protected][email protected] Edge Workers Cloudflare Workers
pa.ag @peakaceag 34 But today it‘s all about Cloudflare, because: The top 3 providers (CF, AWS, Akamai) have 89% of all customers; Cloudflare alone is used by 81% of all sites that rely on a CDN (according to W3Techs): Source: https://pa.ag/2U9kvAh
pa.ag @peakaceag 36 Go create your own (free) account over at cloudflare.com Once your account is activated, you can add your first site/domain: Add your domain name - it can be registered anywhere (as long as you can change the DNS at your current provider)
pa.ag @peakaceag 38 Next, you‘ll get to see the current DNS configuration Yours should look a little like this: at least two records, one for the root-domain, one for the www sub-domain, both pointing to the IP address of your hosting provider: On to the next screen!
pa.ag @peakaceag 39 Now, CF will show you which nameservers to use instead: Nameservers with the current provider, in my case nsX.inwx.de My new nameservers with Cloudflare to be used instead
pa.ag @peakaceag 40 Switching existing nameservers over to Cloudflare At my hosting provider, it looks like this: My new nameservers Cloudflare told me to use instead (see prev. screen)
pa.ag @peakaceag 41 Switch back to tell them you’re all set: Nameservers with the current provider, in my case nsX.inwx.de My new nameservers with Cloudflare, to be used instead
pa.ag @peakaceag 42 Cloudflare is going to email you when things are ready: Beware, this can take up to 24hrs depending on the registrars and nameservers: Your CF dashboard should look like this after the successful NS change.
pa.ag @peakaceag 43 Speaking of nameservers – are you already using 1.1.1.1 ? Cloudflare runs the fastest DNS resolver available. Why wouldn‘t you use it? More: https://pa.ag/3zueHRX
pa.ag @peakaceag 47 A Worker, in its simplest form: This function defines triggers for a Worker script to execute. In this case, we intercept the request and send a (custom) response. Our custom response is defined here, for now we simply: (6) log the request object (7) fetch the requested URL from the origin server (8) log the response object (10) send the (unmodified) response back to the client
pa.ag @peakaceag 54 Here’s how to add a Worker You‘ll be redirected from the “all Workers“ overview to the following mask: Give your Worker a unique name Copy & paste the Workers code you just tested on the Playground
pa.ag @peakaceag 56 Comparison: left (original), right (Worker-enabled) Double-check live! Also, don‘t fall victim to caching, use “Disable Cache“ (see: Network tab) in Chrome Dev Tools to be sure you‘re seeing the latest version: vs
Please understand that all scripts / source codes are meant as examples only. Ensure you know what you’re doing when using them in a production environment! Warning: (maybe) not production-ready!
pa.ag @peakaceag 60 Redirects on the edge using the Response API To execute any type of HTTP redirect, we need to use the Response Runtime API which – conveniently – also provides a static method called “redirect()”: Source: https://pa.ag/3gvXYoL let response = new Response(body, options) return Response.redirect(destination, status) or just:
pa.ag @peakaceag 62 Different types of implementations at a glance (#18): 302 redirect, (#22): 301 redirect, (#26): a reverse proxy call and (#31-36): multiple redirects, selecting a single destination from a map based on a URL parameter:
pa.ag @peakaceag 63 A quick overview to see how things are working… Source: https://httpstatus.io Correct, in fact this is not a redirect ID is not configured in redirectMap
pa.ag @peakaceag 64 To “reverse proxy” a request, you can use the Fetch API It provides an interface for (asynchronously) fetching resources via HTTP requests inside of a Worker script: Source: https://pa.ag/3wpS3YT const response = await fetch(URL, options) Asynchronous tasks, such as fetch, are not executed at the top level in a Worker script and must be executed within a FetchEvent handler.
pa.ag @peakaceag 65 return await fetch(“https://example.com ”) Easily “migrate” a blog hosted on a sub-domain to a sub-folder on your main domain – without actually moving it. Great tutorial: https://pa.ag/2Tw7LD8 Content shown from example.com Request sent from bastiangrimm.dev
pa.ag @peakaceag 66 Verifying that this all happens “on the edge“: Zoom into any of the response headers for an originally requested URL such as bastiangrimm.dev/redirects/302:
pa.ag @peakaceag 68 Which version would you like to wake up to? Preventing “SEO heart attacks“ using a Worker to monitor and safeguard your robots.txt file is one of many use-cases that are super easy to do: This is how I uploaded the robots.txt file to my test server This is what the Worker running in the background changed the output to vs
pa.ag @peakaceag 69 Preventing a global “Disallow: /“ in robots.txt (#5-6): define defaults, (#15-16): if robots.txt returns 200, read its content, (#19-24): replace if global disallow exists, (#27-29): return default “allow all” if file doesn’t exist
pa.ag @peakaceag 71 For demonstration purposes only: UA-based delivery (#10): get User-Agent, (#16-17): add dynamic Sitemap-link if UA contains “googlebot”
pa.ag @peakaceag 72 vs Live-test & compare robots.txt using technicalseo.com Left screen shows bastiangrimm.dev/robots.txt being requested using a Googlebot User-Agent string, right screen is the default output: Free testing tool: https://technicalseo.com/tools/robots-txt/ Or use…
Some systems cause endless headaches for SEOs – routing them through Cloudflare and using a Worker works very well! Easily overwrite files which are “not meant” to be changed?
pa.ag @peakaceag 75 Say hello to the HTMLRewriter class! The HTMLRewriter allows you to build comprehensive and expressive HTML parsers inside of a Cloudflare Workers application: Source: https://pa.ag/2RTpqEt new HTMLRewriter() .on("*", new ElementHandler()) .onDocument(new DocumentHandler())
pa.ag @peakaceag 76 Let's give it a try and work with and first (#24-25): pass tags to ElementHandler, (#9-11): if it’s , set it to “index,nofollow”, (#14-16): if it’s , add another directive for bingbot:
pa.ag @peakaceag 78 If you want to work with/on every HTML element… This selector would pass every HTML element to your ElementHandler. By using element.tagName, you could then identify which element has been passed along: return new HTMLRewriter() .on("*", new ElementHandler()) .transform(response)
pa.ag @peakaceag 80 Using element selectors in HTMLRewriter Often, you want only to process very specific elements, e.g. tags – but not all of them. Maybe it’s just the meta description you care about? new HTMLRewriter() .on('meta[name="description"]', new ElementHandler()) .transform(response) More on selectors: https://pa.ag/35xw073
pa.ag @peakaceag 81 Updating or replacing titles and descriptions is easy… (#10): forced overwrite, (#14-22): conditional changes to the meta description Element selectors are super powerful yet easy to use:
pa.ag @peakaceag 83 Maybe you should have listened to me in the first place!? Check out my presentation over at SlideShare: Slides: http://pa.ag/migration_search_y
pa.ag @peakaceag 84 HTMLRewriter listening to and tags (#29-30): Passing href/src attributes to a class which (#20): replaces oldURL with newUrl and (#16-18): ensures https-availability Based on: https://pa.ag/35llTSo
pa.ag @peakaceag 86 HTTP hreflang annotations on the edge We‘ve just had plenty of HTML, so let‘s use HTTP headers instead – of course both ways work just fine though:
pa.ag @peakaceag 90 Combining HTTP 503 error with a Retry-After header Retry-After indicates how long the UA should wait before making a follow-up request: The server is currently unable to handle the request due to a temporary overloading or maintenance of the server […]. If known, the length of the delay MAY be indicated in a Retry-After header.
Just in case I’ve somehow not made my point yet – you can do REALLY cool stuff and have control over the full HTML response – so adding content is easy. 8. Injecting content
pa.ag @peakaceag 93 You could also (dynamically) read from an external feed Feeding in content from other sources is simple; below shows reading a JSON feed, parsing the input and inject to the of the target page:
One of the key challenges when using CDNs: logfiles are literally everywhere – and a lot of requests don‘t even make it to the origin server… 9. Collecting logfiles
pa.ag @peakaceag 95 Cloudflare provides extensive possibilities for logfiles What I really love about this: direct integration with Google Cloud products! Note: you need the Enterprise plan for this. More: https://pa.ag/3gnj8GF
pa.ag @peakaceag 96 Peak Ace log file auditing stack. Interested? > [email protected] Log files are stored in Google Cloud Storage, processed in Dataprep, exported to BigQuery and visualised in Data Studio via BigQuery Connector. 8 Google Data Studio Data transmission Display data Import / API Google Dataprep 6 7 Google BigQuery 1 Log files GSC API v3 GA API v4 GA GSC 2 3 6 5 Google Apps Script API 4
pa.ag @peakaceag 99 Add native lazy loading for images to your HMTL mark-up Keep in mind: you don‘t want to lazy load all of your images (e.g. not the hero image); also, if you‘re using iframes, you might want to pass “iframe“ to the HTMLRewriter:
pa.ag @peakaceag 100 Cleaning up HTML code for performance reasons E.g. by removing unwanted pre*-stages, or by adding async/defer to JS calls: More clean-up Worker scripts: https://gist.github.com/Nooshu
pa.ag @peakaceag 104 Sloth: an advanced CF Worker Code Generator & CMS A very handy (and free) UI to manage Workers for changing robots.txt, titles & descriptions, redirects, hreflang, and much more: Check it out: https://sloth.cloud
pa.ag @peakaceag 105 Tool recommendation: Lil Redirector “Lil Redirector works by persisting and querying redirects inside of Workers KV, and includes an administrator UI for creating, modifying, and deleting redirects.” More: https://pa.ag/3q3EZGx
pa.ag @peakaceag 106 Workers KV – wait, what? Source: https://pa.ag/3vmTiXB Workers KV is a global, low-latency, key- value data store. It supports exceptionally high read volumes […] Workers KV is generally good for use-cases where you need to write relatively infrequently, but read quickly and frequently. It is optimised for these high-read applications.
pa.ag @peakaceag 107 Web Scraper based on Cloudflare Workers “Web Scraper makes it effortless to scrape websites. You provide a URL & CSS selector, and it will return you JSON containing the text contents of the matching elements.” More: https://pa.ag/3woCv7T
pa.ag @peakaceag 110 [This] dates back to the time of the French Revolution At least, if you believe Wikipedia that is… Source: https://pa.ag/35nQSx6 With great power comes great responsibility.
pa.ag @peakaceag 112 Great summary over at ContentKing, well worth a read: What are the downsides […] What risks are involved? Source: https://pa.ag/3xhYUUk
10 million requests are included, every 1 million currently costs $0.50 extra - not crazy expensive, but in larger-scale setups certainly means additional costs. Risk of costs
This might interfere with current processes, or at the very least, ensure Workers become part of a standardised process (e.g. deployment). PCI compliance
Additional modifications on the edge could result in massive debugging. Again: proper documentation and processes are crucial! Potential to introduce frontend bugs
pa.ag @peakaceag 120 Dynamically creating links to “Baccarat Sites” “[…] at the CF Workers management area, there was a suspicious Worker listed called hang. It had been set to run on any URL route requests to the website:” Source: https://pa.ag/3cFq0Nq After further investigation [by sucuri], it was found that the website was actually loading SEO spam content through Cloudflare’s Workers service. This service allows someone to load external third-party JavaScript that’s not on their website’s hosting server.
pa.ag @peakaceag 121 The suspicious “hang” Worker injection in detail: Source: https://pa.ag/3cFq0Nq ▪ The JavaScript Worker first checks for the HTTP request’s user-agent and identifies whether it contains Google/Googlebot or naver within the string text. ▪ If the user-agent string contains either of these keywords, then the JavaScript makes a request to the malicious domain naverbot[.]live to generate the SEO spam links to be injected into the victim’s website. ▪ After this step, the Worker then injects the retrieved SEO spam link data right before the final
Care for the slides? www.pa.ag twitter.com/peakaceag facebook.com/peakaceag Take your career to the next level: jobs.pa.ag [email protected] Bastian Grimm [email protected]
bastiangrimm
ikuyamada
carta_engineering
tarao
line_developers_tw
niba1122
sbtechnight
colinfay
line_developers
se_o_chan
kentosuzuki
you5805
notwaldorf
wjessup
swwweet
smashingmag
aarron
speakerdeck
rocio
shpigford
denniskardys
malarkey
eitanlees
destraynor