Automating Technical SEO with Screaming Frog CLI and n8n

This session shows how Screaming Frog CLI and n8n can automate areas of technical SEO that are still widely handled manually, unlocking automation that would otherwise just be a tool expense.

Transcript

2. CHRIS LEVER Head of Technical SEO @ Glass Atlas • Professional SEO since 2011 • Hobbyist dev since mid 2000’s • FULL TIME GEEK • Co-founder of TechSEO North
3. AUTOMATING TECH SEO With Screaming Frog CLI and N8N
4. What could you build? Screaming Frog CLI + n8n +
5. SCREAMING FROG CLI Quick Introduction
7. GETTING TO KNOW THE BASICS WITH USING SCREAMING FROM CLI Navigating to the Screaming Frog folder cd "C:\Program Files (x86)\Screaming Frog SEO Spider"
8. CALL FOR HELP The help flag Lists every available command and flag • Documents the syntax and arguments each flag requires • Reflects the exact capabilities of your installed version • Covers crawl, export, report, and configuration options • The reference point for any automation or scheduled task • The first command to run before building any workflow - - help
9. BULK EXPORTS Two commands that expose the data you need for automation --help export-tabs • Lists every tab and filter available as exports • Internal, Response Codes, Page Titles, Canonicals, Hreflang, Validation --help bulk-export • Lists every option from the Bulk Export menu • Inlinks, Outlinks, Images, Anchor Text, Canonicals, Directives --help export-tabs --help bulk-export
10. THE ANATOMY OF A SF CLI COMMAND Executable Hygiene Action Mode Destination Save .\ScreamingFrogSEOSpiderCli.exe --crawl https://glassatlas.com --headless --save-crawl --output-folder "C:\crawls" --timestamped-output
11. YOUR FIRST CLI CRAWL The first headless crawl • --crawl sets the target URL • --headless runs without the GUI • --save-crawl produces a reopenable file • --output-folder defines where the data lands • --timestamped-output keeps every crawl separate Command .\ScreamingFrogSEOSpiderCli.exe --crawl https://example.com --headless --save-crawl --output-folder "C:\crawls" --timestamped-output
12. DON’T FORGET THE CONFIG FILE The CLI runs the crawl, but the .seospiderconfig file controls how that crawl behaves. • Save your crawl setup config from the Screaming Frog GUI • You can reuse the same config in CLI workflows • Keeps settings consistent across scheduled crawls • Controls extraction, rendering, limits, user-agent, robots handling and more • Reduces mistakes when moving from manual crawls to automation Command .\ScreamingFrogSEOSpiderCli.exe --crawl https://example.com --headless --config "C:\configs\seo-audit.seospiderconfig" --save-crawl --output-folder "C:\crawls" --timestamped-output
13. Screaming Frog CLI - Crawling
14. Screaming Frog CLI - Processes
15. N8N AUTOMATION PLATFORM Quick Introduction
17. N8N WORKFLOWS & CREDENTIALS
18. N8N SCHEDULERS & EMAILERS
19. EXAMPLE N8N SEO WORKFLOW Use email to trigger an n8n workflow and return a summary of the crawl insights automatically. More about this later!
20. MOST EXAMPLES IN THIS PRESENTATION ARE USING:
21. YOUR FIRST WORKFLOW With Screaming Frog CLI and N8N
22. SIMPLE WORKFLOW TO Run the Crawl Verifies Output
23. RAW CRAWL OUTPUT This is the output internal_all.csv before we preprocess the data
24. CRAWL DATA PROCESSING Runs the crawl Checks Output Preps Output
25. SF CRAWL PREPROCESSING Structured crawl data pipeline • Cleans raw Screaming Frog exports • Generates time stamped processed outputs • Organises crawl history automatically • Moves completed exports into archives • Produces cleaner downstream AI inputs (saves on tokens) Use Python to create smaller, structured datasets are easier to process downstream.
26. PROCESSED OUTPUT
27. EMAIL TRIGGERED CRAWLS With Screaming Frog CLI and N8N
28. Email Triggered Crawls Full workflow example: Trigger > Check Mail > Crawl > Process > Notify
29. Email Triggered Crawls
30. Email Triggered Crawls Scheduler set to 5 mins Listens for Email Trigger Trigger Email Present? Yes/No Duration Tracker Extract URL from Email Marks Email as Read
31. Email Triggered Crawls Trigger SF-CLI Confirms Crawl Data Preparation Builds email summary Sends email summary
32. Email Triggered When the workflow has finished, an email is sent to the email inbox set. • Send an email to n8n to start the crawl workflow • The subject line can define the crawl target or workflow type • The workflow waits for the crawl to finish, then builds the summary • Results are sent back to the inbox as a simple crawl report • Ship the crawl data and crawl report downstream if you like
33. XML SITEMAP MONITOR With Screaming Frog CLI and N8N
34. XML Sitemap Monitor Crawls a target XML sitemap every morning and emails a summary of what changed.
35. XML Sitemap Monitor The workflow finishes by sending a clear email summary of what changed. • Shows today’s sitemap URL count against yesterday • Flags newly added or removed URLs • Highlights status code or indexability changes • Surfaces title and H1 changes at a glance
36. PRICE MONITOR With Screaming Frog CLI and N8N
37. Price Monitor Crawls a list of URLs, extracts JSON-LD and emails a summary of price + stock changes .
38. XML Sitemap Monitor - Spider.config
39. Price Monitor Python script to preprocess the product schemas, log differences and builds the summary email.
40. Price Monitor This workflow checks product pages and turns price or stock changes into a clear email summary. • Crawls a saved list of product URLs • Extracts price and stock data from JSON-LD • Compares today’s data against the last run • Flags price changes, stock changes and missing schema • Sends a simple summary email for review
41. TECHNICAL SEO TRIAGE Always on website monitoring.
42. Technical SEO Triage This workflow crawls a site, classifies the findings, and sends them to Google Sheets, Google Chat and Gmail.
43. Technical SEO Triage A Python script turns the crawl output into a quick technical SEO triage view. • Pulls out the issues that need attention first • Highlights changes across status codes, indexability and key page elements • Helps you decide what needs investigating before opening the full crawl • A crawl workflow designed to find the bits worth checking first.
44. Technical SEO Triage
45. Technical SEO Triage The workflow finishes by sending a clear email summary of what changed. • Shows today’s sitemap count against yesterday • Flags newly added or removed URLs • Highlights status code or indexability changes • Surfaces title and H1 changes at a glance
46. TRIAL AND ERROR Takes a few tries to get it right.
47. Stop the workflow execution is something feels wrong For example: taking longer than crawl speed vs URL count
48. Get-CimInstance Win32_Process -Filter "Name='java.exe'" | Where-Object { $_.CommandLine -like "*ScreamingFrog*" -or $_.ExecutablePath -like "*Screaming Frog*" } | ForEach-Object { taskkill /F /T /PID $_.ProcessId } Kill Screaming Frog CLI - Powershell This kills the crawler processes
51. BUILDING TOOLS With Screaming Frog CLI (no n8n needed)
52. Build your own tools with SF-CL I Because Screaming Frog CLI is executable and headless , you can build on top of it . Your tools decide when it runs, what config it uses, where the exports go, and how the results are shown.
53. Build tools & dashboards with SF-CLI
54. Build tools & dashboards with SF-CLI
56. POWERING AUTOMATIONS Without the heavy tech stack.
57. AUTOMATION WITHOUT THE HEAVY TECH Screaming Frog Automations Without CLI or n8n
58. Screaming Frog allows you to schedule crawls (including headless mode ) from inside the main SF GUI .
59. Export to Google Sheets and use App Scripts to power your automations.
60. TAKEAWAYS AND TIPS Best practice learnings.
61. USE LOCAL HARDWARE or CLOUD VMs Use spare hardware or cloud VMs to run heavier crawls, schedule workflows, and keep automation running in the background.
62. START ON SMALL WEBSITES FIRST When your start building workflows, you need to start small. Scale up after experimenting and testing.
63. PREPROCESS BEFORE YOU PROMPT Before sending crawl data into an AI workflow, strip out the columns, rows and noise you do not need. Cleaner input means fewer wasted tokens, clearer summaries and better answers.
64. EXPERIMENT BEYOND SEO Once you can automate crawls, extract data and route outputs, the same thinking can support other teams too. The real value is not the tool. It is the workflow you build around it.
66. DOWNLOAD WORKFLOWS Download the example workflows shown in this presentation.
67. DOWNLOAD AT SPEAKERDECK