A deep dive into Screaming Frog’s JavaScript snippets for SEO

This presentation explores Screaming Frog’s new JavaScript snippet feature and how SEOs can use it to automate tasks, extract data, and integrate AI tools like ChatGPT.

Transcript

1. A deep dive into Screaming Frog’s JavaScript snippets for SEO Ziggy Shtrosberg JUNE 2025 SENIOR TECHNICAL SEO ANALYST
2. Before we start… 2 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO
3. What if I don’t know JavaScript? • If you can write using natural language. You can also code. • Use your favourite LLM to generate the custom JavaScript: 3 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO
4. JavaScript snippets 4 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO
5. 5 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO Screaming Frog released their JavaScript snippet functionality in May 2024 as part of their v20.0 update
6. 6 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO What are JavaScript snippets? They are snippets of JavaScript code that you can run on processed URLs during a crawl. IMPORTANT • Only 200 OK URLs • Only HTML (no PDFs)
7. SUBTITLE HERE 7 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO You can do things like: • Extract custom data from a webpage • Communicate with APIs such as ChatGPT or other libraries. • Modify or clean up data Screaming Frog system snippets
8. 8 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO My personal experience of using JS snippets May 2024 Now Disappointed Excited Excitement level Timeline Phase 1 Phase 2 Phase 3
9. 9 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO JavaScript snippets
10. 10 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO JavaScript snippets
11. Limitations 11 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO
12. 12 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO Scrapy Playwright Selenium Puppeteer Cheerio
13. 13 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO But that’s okay Screaming frog is designed to be an accessible SEO crawler first. If you have more advanced scraping requirements, use more suitable tools.
14. The basic set up & UI 14 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO
15. 15 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO Set up your JS snippet functionality 1 2
16. 16 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO Set up your JS snippet functionality 3 4
17. 17 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO Set up your JS snippet functionality 5
18. 18 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO Set up your JS snippet functionality 6 Code editor JS Tester
19. Use cases 19 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO
20. USE CASE #1 Multi-prompting Use AI to create better and more engaging product descriptions. But with a slight twist… 20 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO
21. 21 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO 1. Scrape the relevant product information from the product detail page.
22. 22 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO Gemini’s Response Claude’s Response 2. Make simultaneous API calls to Gemini and Claude with the product data and ask the LLMs to produce an improved version. Here are the results ->
23. 23 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO You know, not bad but I'm surprised. Gemini crushed it and was a 9/10. I thought you were the better LLM? What's going on. For reference, here is the product description that you came up with: claudeDesc And here is what Gemini wrote as an FYI: geminiDesc See if you can do better. 3. Follow it up with another API call to Claude with a “jealousy” prompt Prompt concept by Greg Isenberg
24. 24 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO 4. The final improved product description ->
25. USE CASE #2 Internationalization Detect webpages containing the wrong language for a client’s international website that trades in 50+ markets 25 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO
26. 26 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO 1. Extract the lang attribute from the opening HTML tag
27. 27 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO 2. Display the language and country based on the path segments from the URL https://www.example.com/ja-jp/japan/ Japanese Japan
28. 28 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO 3. Extract the first 150 words of the content, send to Gemini and ask the LLM to detect the language
29. 29 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO The final output It’s extremely cost efficient with language detection for 5,384 URLs costing only 6 cents using the Gemini 1.5 Flash model!
30. USE CASE #3 Internal linking Keyword detection for internal linking… but on steroids 30 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO
31. 31 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO Add the keywords that you want to target for your internal linking
32. 32 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO I slice up the content into chunks
33. 33 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO Assign scores to headings and content chunks based on semantic importance and distance from top of the page
34. 34 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO I use a "treewalker" to scrape the page, which refers to the TreeWalker API in JavaScript. • It provides methods to navigate the Document Object Model (DOM ) tree in various directions (forward, backward, up, down). • It's a powerful tool within the DOM that allows you to travel across and filter nodes within a webpage's HTML structure.
35. 35 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO The final output
36. 36 A Deep Dive into Screaming Frog’s JavaScript Snippets for SEO What else can we do with the script? - Make the chunking more dynamic. - Identify if a page already contains an internal link to our destination URL. - Send the filtered output to an LLM for anchor text or sentence replacement suggestions. - Use the same methodology to chunk and convert to embeddings. Potentially even push to a vector database with an API.
37. Thank you linkedin.com/in/ziggyshtrosberg/ shtros.com