Getting Under the Hood of Screaming Frog CLI for Automating Technical SEO | brightonSEO 2026

Presented at brightonSEO, October 2026

Most SEOs know Screaming Frog as a desktop crawler, but there's a lot more you can do with it once you start exploring the command line.

In this session, I get under the hood of Screaming Frog CLI covering the commands needed to run headless crawls, automate exports and move repetitive technical SEO tasks away from your laptop.

I also explore how Screaming Frog can work alongside n8n and Python to build automated workflows, with practical examples including XML sitemap change monitoring, technical SEO triaging and competitor price tracking.

The aim is simple: spend less time running the same crawls and pulling reports, and more time investigating the issues that actually matter.

Transcript

2. HOW WE USE SCREAMING FROG TODAY
3. SCREAMING FROG GUI A VIEW MOST SEASONED SEO’S ARE FAMILIAR WITH Most SEOs know Screaming Frog as a desktop crawler. It’s where most of us learnt to use Screaming Frog.
4. THE DESKTOP CRAWL WORKFLOW Open Screaming Frog Configure Set Your Target Run The Crawl Open the desktop app, Select your crawl mode Load your settings APIs, rendering, extraction Enter a domain or Upload a URL list Start crawling Wait for it to finish It works. But you are the automation. Do It Again Next week. Next month. Next client. Export Analyse Review the data Find the opportunities Export the tabs and reports you need
5. YOUR LAPTOP IS LETTING YOU DOWN MEMORY Large crawls eat RAM CPU Rendering and processing adds load UPTIME Your machine has to stay running
6. WHAT IF THE CRAWL DIDN’T NEED YOUR LAPTOP?
8. SCREAMING FROG CLI
10. FINDING SOME --HELP --help is the quickest way to see what the CLI can do. It returns the available arguments, crawl modes, input options and output controls. Think of it as the CLI’s built-in cheat sheet. Command: .\ScreamingFrogSEOSpiderCli.exe --help
11. LIST EXPORT OPTIONS --help export-tabs shows every tab and filter the CLI can export. It gives you the exact names to use when building automated exports. This is where the crawl becomes usable data. Command: .\ScreamingFrogSEOSpiderCli.exe --help export-tabs
12. LEARNING THE CLI COMMANDS
14. START COMMANDS FOR WINDOWS AND Mac OS USERS POWERSHELL TERMINAL
15. WINDOWS NAVIGATE TO THE FOLDER FIRST Folder: cd "C:\Program Files (x86)\Screaming Frog SEO Spider" Executable .\ScreamingFrogSEOSpiderCli.exe
16. MacOS NAVIGATE TO THE FOLDER FIRST Folder: cd "/Applications/Screaming Frog SEO Spider.app" Executable ./Contents/MacOS/ScreamingFrogSEOSpiderLauncher
17. Executable .\ScreamingFrogSEOSpiderCli.exe Executable ./Contents/MacOS/ScreamingFrogSEOSpiderLauncher Action --crawl Mode --headless Save --save-crawl Export --export-tabs Destination --output-folder Organisation --timestamped-output
18. UNDER THE HOOD
19. TELL IT WHERE TO START --crawl gives Screaming Frog its starting point. Executable --crawl https://example.com
20. CRAWL EXACTLY WHAT YOU GIVE IT --crawl-list gives Screaming Frog a defined set of URLs. List Mode Executable --crawl-list "C:\SEO\urls.txt"
21. HEADLESS CRAWL MODE Executable --crawl https://example.com --headless
22. SAVE THE WHOLE CRAWL Executable --crawl https://example.com --headless --save-crawl
23. HOW --export-tabs WORKS SINGLE EXPORT --export-tabs "Internal:All" MULTIPLE EXPORTS --export-tabs "Internal:All,Response Codes:Client Error (4xx),Page Titles:Duplicate" Executable --crawl https://example.com --headless --save-crawl --export-tabs "Internal:All"
24. HOW --bulk-export WORKS SINGLE BULK EXPORT --bulk-export "Links:All Inlinks" MULTIPLE BULK EXPORTS --bulk-export "Links:All Inlinks,Links:All Outlinks" Executable --crawl https://example.com --headless --save-crawl --bulk-export "Links:All Inlinks,Links:All Outlinks"
25. POINT THE OUTPUT SOMEWHERE Executable --crawl https://example.com --headless --save-crawl --bulk-export "Links:All Inlinks,Links:All Outlinks" -output-folder "C:\crawls" THE FOLDER MUST ALREADY EXIST
26. WHAT HAPPENS ON THE NEXT RUN? Executable --crawl https://example.com --headless --save-crawl --bulk-export "Links:All Inlinks,Links:All Outlinks" -output-folder "C:\crawls" --timestamped-output
27. GIVE THE RUN A NAME Executable --crawl https://example.com --headless --save-crawl --bulk-export "Links:All Inlinks,Links:All Outlinks" -output-folder "C:\crawls" --timestamped-output --project-name "Glass Atlas" --task-name "Weekly Crawl"
28. YOUR FIRST CLI CRAWL The first headless crawl Full Command • --crawl sets the target URL • --headless runs without the GUI .\ScreamingFrogSEOSpiderCli.exe • --save-crawl produces a --crawl https://example.com --headless --save-crawl --export-tabs "Internal:All" --output-folder "C:\crawls" --timestamped-output reopenable file • --export-tabs selects the data to export • --output-folder defines where the data lands • --timestamped-output keeps every crawl separate
29. DON’T FORGET THE CONFIG FILE The CLI runs the crawl, but the .seospiderconfig file controls how that crawl behaves. • Save your crawl setup config from the Screaming Frog GUI • You can reuse the same config in CLI workflows • Keeps settings consistent across scheduled crawls • Controls extraction, rendering, limits, user-agent, robots handling and more • Reduces mistakes when moving from manual crawls to automation Command Executable --crawl https://example.com --headless --config "C:\configs\seo-audit.seospiderconfig" --save-crawl --export-tabs "Internal:All" --output-folder "C:\crawls" --timestamped-output
30. YOUR FIRST CLI CRAWL
32. TAKE IT OFF YOUR LAPTOP
33. THE OFFICE CRAWL BOX USE A DEDICATED MACHINE TO RUN THE AUTOMATIONS
34. N8N THE AUTOMATION POWERHOUSE
35. DEMO TIME
36. N8N AUTOMATION WORKFLOWS
37. N8N CREDS & CONNECTIONS
38. SCHEDULING N8N WORKFLOWS
40. SITEMAP DIFF MONITOR
41. THE RESULTS SITEMAP DIFF MONITOR Send the price monitoring summary to the team. • Shows today’s sitemap URL count against yesterday • Flags newly added or removed URLs • Highlights status code or indexability changes • Surfaces title and H1 changes at a glance
42. THE WORKFLOW SITEMAP DIFF MONITOR + + TO
43. AUTOMATING TECHNICAL SEO TRIAGING
44. 1, THE JOB TECHNICAL SEO TRIAGE
45. 2, THE WORKFLOW TECHNICAL SEO TRIAGE + + = ❤
46. THE WORKFLOW EXPLAINED Start / decide when to crawl Manual trigger or scheduled daily run Set up the crawl Load the settings for this run Run Screaming Frog Launch the CLI crawl Build reports Triage findings Process exports Turn prioritised findings into the report Prioritise issues and apply technical hygiene checks Ingest, clean and prune the data Send/share results Wait for crawl Wait for the crawl process to complete Review crawl data Analyse the completed crawl Then, automate it. Weekly or Monthly. Whatever the cadence.
47. 3, UNDER THE HOOD TECHNICAL SEO TRIAGE A set of Python scripts turns the crawl output into a quick technical SEO triage view. • Pulls out the issues that need attention first • Highlights changes across status codes, indexability and key page elements • Helps you decide what needs investigating before opening the full crawl • A crawl workflow designed to find the bits worth checking first
48. 4, THE RESULTS TECHNICAL SEO TRIAGE PYTHON: SCORING + DRIFT HISTORY
49. THE RESULTS TECHNICAL SEO TRIAGE Send the triage summary to the team. • Sends a concise Google Chat summary after each run • Groups findings into Critical, High and Medium priority • Pulls out the worst finding, example URLs and why it matters • Flags anything not assessed and keeps routine metadata in the background
50. PRICE MONITOR
51. THE RESULTS PRICE MONITORING Send the price monitoring summary to the team. • Highlights price and stock changes since the previous run • Shows the old price, new price and percentage change • Flags failed scrapes so missing data is visible, not silently ignored
52. THE WORKFLOW EXPLAINED Schedule the run Prepare URLs Run Screaming Frog/ Scrape prices Run automatically every day at 8am Start the timer and prepare the URL list Crawl the prepared URLs via CLI Build summary Python extracts the prices and builds the summary Send/share results Email the finished price report
54. BUILDING YOUR OWN
55. FROM SIGNAL TO ACTION PICK THE SIGNALS. DEFINE THE OUTPUT. Start with the question. Then build the workflow.
57. TURN THE QUESTION INTO A WORKFLOW TRIGGER IT. CRAWL IT. PROCESS IT. ACT ON IT. THAT'S YOUR AUTOMATION
58. OUT OF THE BOX AUTOMATIONS
59. AUTOMATION WITHOUT THE HEAVY TECH Screaming Frog Automations Without CLI or n8n
62. SCHEDULING THE EASIEST WAY Create automations straight from the GUI. • Schedule crawls to run automatically at set times • Reuse your saved configuration, APIs and crawl settings • Automatically save and export the reports you need • Send outputs onwards without manually running the crawl • Less flexible downstream than CLI, but a great place to start.
63. RUN HISTORY + IMPORT / EXPORT SCHEDULES ACROSS MACHINES ❤
64. WRAPPING IT UP
66. BUILD ON TOP OF THE CRAWLER
67. BUILD YOUR OWN APP USING SCREAMING FROG CLI.
68. COULD BE USEFUL IF YOU ARE BUILDING AGENTIC AUTOMATIONS
69. THANK YOU