Measuring the whole SERP at any scale Ray Grieselhuber DemandSphere Speakerdeck.com/raygrieselhuber @raygrieselhuber

The era of traditional rank tracking is over Speakerdeck.com/raygrieselhuber @raygrieselhuber Up Down 23 12

But the shape of the SERP is changing faster than ever Speakerdeck.com/raygrieselhuber @raygrieselhuber

Impacts: ● Fold visibility ● CTR ● Clicks / Sessions ● Conversions ● Revenue 600+ distinct elements across the SERPs Paid Content ● Pixel depth: 120px ● Pixel height: 320px ● Visual position: 1 ● Nested position: 2 Organic Result ● Pixel depth: 440px ● Pixel height: 330px ● Visual position: 3 ● Title changed

Daily layout changes on Google SERPs Paid Content ● Position (order of items) ● Product Name ● Price ● URL ● Pixel Data Organic Results ● Organic Position ● Visual Position ● Title ● Meta Description ● Product URL ● Pixel Data Popular Products ● Position (order of items) ● Product Name ● Price ● URL ● Pixel Data

No content

Speakerdeck.com/raygrieselhuber @raygrieselhuber Scale driver #1: AI + content marketing is driving explosive growth in content on the web

Speakerdeck.com/raygrieselhuber @raygrieselhuber Scale driver #2: The cost of money is affecting investment in ads Money supply SPIKED but now is decreasing Money is not as cheap anymore

Speakerdeck.com/raygrieselhuber @raygrieselhuber Typical clients: 75K-300K+ SERPs Larger clients: 1M+ SERPs Let’s define scale (volume) for our purposes

What is the right volume to monitor? Speakerdeck.com/raygrieselhuber @raygrieselhuber 1%-10% of your indexed pages is a good place to start

The bigger question is: Speakerdeck.com/raygrieselhuber @raygrieselhuber Who uses this data and for what?

Old answer: Speakerdeck.com/raygrieselhuber @raygrieselhuber SEOs

Better answer: Speakerdeck.com/raygrieselhuber @raygrieselhuber Good SEO is good product management, and vice-versa.

Even better answer: exploding use cases Speakerdeck.com/raygrieselhuber @raygrieselhuber Complexity increases Developer ● APIs ● Data lakes SEO Analyst ● SERP data ● Crawl data ● GSC ● GA ● Exploration Content Team ● Content & topics ● Content performance ● Visibility trends Product ● Technical ● SERP rankings ● Indexation ● PAA + Knowledge Graph Executive / Reporting ● Permissions ● Dashboards ● Speed of answers ● Forecasting ● Predictive Performance Team ● Ad data ● Hotel Ads ● Shopping ● Google for Jobs ● etc.

Speakerdeck.com/raygrieselhuber @raygrieselhuber How to meet the needs of all of these groups?

Speakerdeck.com/raygrieselhuber @raygrieselhuber Step 1: understand the data pipeline

Typical data pipelines Speakerdeck.com/raygrieselhuber @raygrieselhuber Fetching (HTML) ETL (JSON & DB) Analytics (DB) Dashboards, etc. SERPs (HTML) GSC GA4 Build? Buy? Build? Buy? Keywords

Where do development bottlenecks occur? Speakerdeck.com/raygrieselhuber @raygrieselhuber Fetching (HTML) ETL (JSON & DB) Analytics (DB) Dashboards, etc. SERPs (HTML) GSC GA4 Build? Buy? Build? Buy? Keywords

Speakerdeck.com/raygrieselhuber @raygrieselhuber Step 2: examine parallels from other problem domains

Similar use cases Speakerdeck.com/raygrieselhuber @raygrieselhuber Data Operations Observability (log files + search index / database) Problem domains Asset / Technology Parallels to SEO ● Log files: SERPs ● Parsed log files: Parsed SERPS (JSON, etc.) ● Index: ??? Data Science BI Datasets ML Models Dashboards & Visualizations ● SERPs ● GSC ● GA4 ● Keyword sets, etc. ● Looker Studio ● Superset ● DOMO ● etc.

Speakerdeck.com/raygrieselhuber @raygrieselhuber Step 3: use tools with a track record of acceleration

Tools that can help accelerate and support many use cases Speakerdeck.com/raygrieselhuber @raygrieselhuber Data Lakes BI / Dashboards / Data apps Cloud Data Warehouse ● S3-compatible object storage ● BigQuery ● Snowflake ● Redshift ● Looker Studio ● Superset ● Jupyter & other notebooks ● Python data apps (Dash, Streamlit)

SQL is a decent lingua franca for turning SERP data into BI Speakerdeck.com/raygrieselhuber @raygrieselhuber select search_domain, url, title, desc, position, people_also_asked_snippets from results.organic_results where fetch_date = “2023-11-08” limit 100000

We mapped every single element on the SERP and put it into a data warehouse Speakerdeck.com/raygrieselhuber @raygrieselhuber

As data science matures, we’re seeing many Python-based tools become available Speakerdeck.com/raygrieselhuber @raygrieselhuber ● Pandas / Dask / Pola.rs ● Data Science notebooks ● Numpy ● PyTorch ● Scrapy ● Advertools ● Plotly ● spaCy ● NLTK

Simplified access to SERP data Speakerdeck.com/raygrieselhuber @raygrieselhuber Developers SEO Team Content Team Product Team Executive / Reporting Performance Team APIs Data Warehouse Data Lakes Data Warehouse Data Warehouse SEO & Content Tools BI

Let’s talk! Speakerdeck.com/raygrieselhuber @raygrieselhuber ● Scan the QR code to interact with these slides on Slido ● Follow us at x.com/demandsphere ● Connect with me at linkedin.com/in/raygrieselhuber