Jamie Indigo - The Philosophy Minor’s Guide to Log Files

1. The Philosophy Minor’s Guide to Log Files Jamie Indigo 100% Human Technical SEO Tech SEO Connect - November 2026
2. 429 too many requests
3. • • • • • Limited Search documentation Black box index No 1st party reporting hub Documented user-agents Declared IP ranges • • • • Limited Search documentation No 1st party reporting hub Documented user-agents Declared IP ranges • • • • Limited Search documentation No 1st party reporting hub Documented user-agent No published IP range list (uses Google Cloud and workspace IPs) • • • • No Search • documentation No 1st party • reporting hub No • documented user-agents • No declared IP ranges Limited Search documentation Has 1st party reporting hub Documented user-agents Declared IP ranges • • • • Limited Search documentation No 1st party reporting hub No documented user-agents No declared IP ranges
4. surface model login status mode enabled tools location
8. How can I know what's True?
9. I think, therefore I am René descartes Rationalism - Reason first Descartes doubts everything until he finds the one thing he can’t doubt: that something is doing the doubting. Existence is proven by activity.
10. Hello world I request, therefore I am.
11. Logs as Evidence GET /kant/categorical-imperative HTTP/2 Host: www.example.com User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.0.0 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8 Accept-Encoding: gzip, deflate, br, zstd Accept-Language: en-US,en;q=0.9 Upgrade-Insecure-Requests: 1 Sec-CH-UA: "Chromium";v="153" Sec-CH-UA-Mobile: ?0 Sec-CH-UA-Platform: "Linux" Sec-Fetch-Site: none Sec-Fetch-Mode: navigate Sec-Fetch-User: ?1 Sec-Fetch-Dest: document Priority: u=0, i
12. Syllogism a kind of logical argument that applies deductive reasoning to arrive at a conclusion based on two propositions that are asserted or assumed to be true. Valid means the structure works. If the premises are true, the conclusion has to follow. Whether the premises actually are true doesn’t matter for validity. Sound means valid and the premises really are true. Only a sound argument guarantees a true conclusion.
13. ORPHAN CHAIN a good argument with a weak spot If a page isn’t linked, it isn’t discovered. If it isn’t discovered, it isn’t crawled. If it isn’t crawled, it doesn’t rank. ∴ Orphan pages don’t rank. A leads to B, B leads to C, C leads to D, so A leads to D.
14. Shaky premise If a page isn’t linked, it isn’t discovered. Google can also find pages through XML sitemaps, external links, or If it isn’t discovered, it isn’t crawled. Redirects from pages it already knows If it isn’t crawled, it doesn’t rank. about. ∴ Orphan pages don’t rank. The logic is fine, but one premise is shaky.
15. FALSE Premise, Valid Structure All pages disallowed in robots.txt are not indexed MAJOR PREMISE - assumes A is TRUE /cart is disallowed in robots.txt MINOR PREMISE - A is a SUBSET of B ∴ /cart is not indexed Conclusion
16. Sound vs Valid So when an argument goes wrong, there are two places to look: 1. Is the structure broken? That’s a validity problem, and fallacies are this kind of problem. 2. Is a premise false? That’s a soundness problem.
17. Modus Tollens If Googlebot requested the page, it’s in the logs. MAJOR PREMISE - Conditional; If P, then Q (P → Q) It’s not in the logs. MINOR PREMISE - (Negation): Not Q (¬Q) ∴ Googlebot didn’t request it. Conclusion - Therefore, not P (¬P)
18. The logic is valid, but the first premise only holds if your logs are complete. If you’re pulling origin logs and the CDN served the request from cache, the request happened and your logs never saw it.
19. Let's practice If a request is from Googlebot, its user-agent says “Googlebot.” This user-agent says “Googlebot.” ∴ It’s Googlebot.
20. Fallacy: Affirms the Consequent If a request is from Googlebot, its user-agent says “Googlebot.” MAJOR PREMISE: Conditional; if P, then Q (P → Q) This user-agent says “Googlebot.” MINOR PREMISE: Affirmation; Q ∴ It’s Googlebot. CONCLUSION: Therefore, P (∴ P). Invalid: affirms the consequent
21. The king of the internet is bald Bertrand Russell analytic philosophy Created Theory of Definite Descriptions, a major tool in the philosophy of language and logic for analyzing how names and descriptions function in sentences. While we can't know for certain, we can know what is necessary or sufficient
22. Necessary vs Sufficient Evidence What it tells you Enough on its own? UA string says “Googlebot” A claim about which bot it is No Reverse DNS resolves to googlebot.com or google.com Unconfirmed, since anyone can set their own reverse record No IP is in Google’s published crawler ranges It’s from Google Yes, with the UA Reverse DNS plus a matching forward lookup It’s from Google Yes, with the UA Valid signature (Web Bot Auth), where supported It’s from the signing operator Yes, with the UA
23. The Real Bot will send specific headers Header Value accept */* accept-encoding gzip, br from gptbot(at)openai.com referer https://edgecomet.com/ user-agent Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) x-openai-host-hash 467038936 Request headers - what Googlebot and AI bots send with every request, Serge Bezborodov
24. Evidence gets messy Signal What it is What it can substantiate A claimed name What remains uncertain User-Agent What the client says it is IP / DNS Where the request came from The operator, if the IP matches published ranges or forward-confirmed DNS Which of the operator’s bots sent it, and why ASN / IP reputation Who owns the network and how it has behaved Who the client is. Shared IPs and residential proxies blur it TLS / HTTP/2 fingerprint (JA3/JA4) How the client negotiates the Inconsistencies, like a “Chrome” connection UA on a Python TLS stack Identity. A mismatch is a clue, not proof Header consistency Do Sec-CH-UA, Accept and header order match the UA? Contradictions with the claimed client Identity. Consistent headers can be copied Behavior Request rate, paths, POSTs, session shape Patterns that fit a type of bot Anything on its own. It only means something combined with other signals and compared to a baseline Network type and history (hosting vs. residential, past abuse) Whether the claim is true
25. Our new ally: Edge Security • Web Application Firewall (WAF) judges the request: is this request dangerous? A WAF is a syllogism engine. It runs rules: if the request matches this pattern, then block it. That’s modus ponens at the edge, so it’s fast and predictable. And like any syllogism, a bad premise gives you a confidently wrong answer. • A bot manager judges the requester: who or what sent it, and are they who they claim to be? Bot manager handles ontology
26. Vantage point changes the view • • • • Origin logs omit cache-served requests CDN edge observations have broader visibility WAF and bot analytics can count rule matches rather than unique requests Active browser telemetry only exists when the client executes the sensor. These populations cannot be combined naively. The denominator changes depending on the view.
27. 28-29.4% meta externalagent by comparative share August 2026 a360 and BotInsights studies saw similar shares but roughly 8.6× different absolute volumes. Data courtesy of Akamai Research. October 2026
28. Happy path view For data hoarding dragons CDN Analytics: Contains Edge and Origin data WAF Analytics: A subset of CDN analytics that triggered one or more rules
29. new Tool: Telemetry the evidence a bot manager collects to decide who you are. Passive telemetry Active telemetry How it’s collected Observed from the request itself Gathered by a script running in the client What it includes IP, ASN, TLS/HTTP/2 fingerprint, header order, Sec-CH-UA, request rate Mouse movement, keystroke timing, scrolling, screen size, canvas/WebGL fingerprint, automation flags like navigator.webdriver Who can see it Every request, including bots that never run JavaScript Only clients that execute JavaScript How hard it is to fake Moderate. It takes engineering Easier to replay, harder to fake convincingly
30. In practice, active telemetry works like this: 1. The page loads a bot manager sensor script. 2. The script collects browser and behavior signals. 3. It POSTs that data back to a sensor endpoint. 4. The verdict is stored in a cookie (like _abck) and checked on later requests. Telemetry is Russell’s knowledge by description, collected at scale.
31. AI Agents complicate measurement What is claimed Chrome/Linux Real Chromium in a cloud VM using a Cloudflare or Fastly IP What is observed What is requested, how the client-side environment behaves, and network conditions Edge Classificatio n 19.4M–63.7M requests/day classified as Muse Agent (Suspected) captured in North American views per Akamai internal reporting We adapt by moving from binary labels to confidence levels Data courtesy of Akamai Research. October 2026
32. Su ec sp Un d te rif Ve Pragmatic Classes of Bots n w kno ied d e r a l c e D
33. you SCALE
34. AI crawling grew 2.89×, like-for-like. BILLIONS OF RULE TRIGGERS / DAY 4 3 2.89× 2 0.90B → 2.59B Fixed 29-bot cohort 1 0 JUL 25 OCT Fixed cohort (29 bots) JAN 26 APR JUL 26 All tracked bots (coverage expands) Data courtesy of Akamai Research. October 2026
35. Marked increase in failed bot verification 2025 H1 2026 Q3 91.5% verified 7.7% declared 0.78% failed 11.0% declared 1.64% failed 87.4% verified Denominator: selected Akamai bot-rule triggers, not deduplicated requests; “impersonator” = failed verification. Data courtesy of Akamai Research. October 2026
36. “AI traffic” is not one behavior. Training crawlers 53.1% Search crawlers 23.1% Fetchers 11.0% Other / agents (approx.) 12.8% Data courtesy of Akamai Research. October 2026
37. Your WAF has never met GPTBot or any other AI crawler. It just has a really good description of it.
38. How AI Bots Declare themselves Silent Declared Signed The AI agent uses a generic user-agent without any agentic-specific identifiers User-agent and network details declare an operator or bot identity Cryptographic proofs establishes mathematical trust for autonomous systems Examples • Muse (uses a generic Chromium browser on Linux, IPs from Cloudflare or Fastly) • Grokbot Examples • Googlebot • Bingbot • Claudebot • GPTbot Examples • Verifiable Intent (Mastercard & Google) • Trusted Agent Protocol (Visa & Cloudflare)
39. Signed agents +----------------+ | User | | +-------+ | +--------+ | | Agent +--+----- signed request + URL ----->| Origin | | +---+---+ | +---+----+ +---------+------+ | | | | +-----------+ | +-- publishes -->| Directory |<-- resolves --+ +-----------+ Figure 1: Web Bot Auth architecture IETF Datatracker: HTTP Message Signatures for automated traffic
40. Web-Auth-Bot Header Breakdown Signature-Input: Defines the signature's metadata—which parts of the HTTP message are covered (e.g., "@method", "@path", "@authority", "date", or custom headers), the key identifier (keyid), the algorithm (typically ed25519 or ecdsa-p256-sha256), and creation/expiration timestamps (created, expires). Signature: Contains the raw cryptographic signature (often formatted as structured field dictionary data) proving that the private key corresponding to the stated keyid signed the component dictionary specified in Signature-Input. Signature-Agent: Identifies the software agent, platform, or gateway orchestrating the signing (e.g., signaling that the request originates from xAI/Grok or Muse infrastructure rather than an arbitrary client spoofing User-Agents).
41. Becoming a disinterested spectator Edmund Husserl Phenomenology Knowledge starts with careful description of experience exactly as it shows up, before you layer theories or assumptions on top of it. Suspend your assumptions and describe exactly what shows up.
42. An observation test It is Tuesday. Another AI company has launched yet another agent. There is no documentation. Your inbox is inflating rapidly.
43. Reproducible Observations 1. Use a test page with a unique harmless query marker, record time and expected behavior 2. Inspect CDN/edge and origin logs separately. Capture UA, source network, classification, and response outcome. 3. Where possible, compare observed requests against independently published bot verification guidance. 4. Avoid treating absence in one log source as a negative result. Document any missing edge visibility or cached requests before drawing conclusions.
44. Inventory your existing premises Google Search uses a Googlebot agent MAJOR PREMISE - assumes A is TRUE AI Mode is part of Google Search MINOR PREMISE - B is a SUBSET of A ∴ AI Mode uses a Googlebot agent Therefore, B (∴ B). Invalid: Fallacy of division
45. AI Mode live fetch Request UA: Google-Lens Gemini and AI Mode Fetch with Google and Google-Lens User-Agents, Dave Smart
46. Gemini PROMPT fetch the page "https://tamethebots.com/tools/lost-impressions-in dex-check?layout=auto" (the parameter is important) and tell me about the page Request UA: Google Gemini and AI Mode Fetch with Google and Google-Lens User-Agents, Dave Smart
48. Applied Philosophy Let's apply these concepts to a build a skill
49. Step 1: Start from pain, not from docs IF our premise is bad THEN our data is unsound
50. Step 1: Start from pain, not from docs 1. Table schemas for your log files a. Server logs will likely have a different schema than WAF logs 2. Bot taxonomy a. Define by platform - include links for resources like IP ranges and UAs b. Define by purpose - if you have a bot manager, this work may be done for you. Look for a bot_category value in your WAF table 3. Page Segments a. Matching regex patterns to group like URLs together. Check what URLs are returned by the definition Ask Claude to write the query, execute, and Refine
51. Step 1: Start from pain, not from docs Everything in a good skill is knowledge you can’t find anywhere else. Your skill’s critical rules are all failures you hit and fixed: • Using a column name from X in a query for table Y • Nullable strings breaking splitByChar • /path/ vs. /path variants • LIMIT capping rows when you wanted to cap series Every time a query breaks, write down the error and the fix. Ten of those make a skill.
52. Step 2: The anatomy logfile-queries/ ├── SKILL.md ← when to use it + the critical rules ├── references/ │ ├── schema-logs.md ← your log format & definitions │ ├── schema-waf.md ← your waf format & definitions │ ├── bot-taxonomy.md ← buckets for bot classifications │ ├── google-crawlers.md ← detailed profiles of platform-specific crawlers │ ├── query-patterns.md ← copy-paste templates │ └── page-segmentation.md ← buckets for page types └── evals/evals.json ← premise testers
53. Step 3: Write critical rules as premises Use one format for every rule: the rule, the failure it prevents, the fix. Wrap Nullable string columns in coalesce → otherwise you get Nested type Array(String) cannot be inside Nullable type → splitByChar('/', coalesce(reqPath,''))[2] Include the error text so the model recognizes it when someone pastes it in.
54. Step 5: Write evals to test soundness Query: “Show me GPTBot’s top URLs this week.” Expected: coalesce on reqPath, $__timeFilter, a SETTINGS clause. Query: “What segment is /michel-foucault/discipline-and-punish/?” Expected: Genealogy
55. Step 6: Keep it alive A skill is never finished. Every new break becomes a new rule. Keep an eye out for: • Changes to schema– Particularly in WAF tables as updates may change, rename, or remove values • Changes to crawlers • New, exciting ways queries break and how to adapt. This is BIG data.
56. Our reward: claim to query machine "To check whether Google lowered a site's crawl capacity limit, look at the number of connections Googlebot opens to the site: if it has dropped, the capacity limit changed." Audience Q&A - Day 1 - Google Search Central Deep Dive Search Central Live · Deep Dive Europe 2026 Knowledge Base, Ibrahim Anjro
57. What you see Most likely explanation TTFB, 5xx or 429 rise, then connections and peak RPS fall Capacity limit lowered. This is consistent with the claim Health stays flat while connections, peak RPS and unique URLs all fall Crawl demand fell. Google wants less, and the capacity limit didn’t change Connections fall while requests stay flat and requests per connection rise Connection reuse (HTTP/2). Nothing changed Requests fall, but health and unique URLs stay flat Inconclusive. Check Search Console’s Crawl Stats host status
58. ass claim um n o i s clu ita t on ob con lim pti s ser ion s ns o i t a ific ver Get the Deck: not-a-robot.com/tsc-lo gs vat ion s Get the starter skill not-a-robot.com/logskil l
59. |￣￣￣￣￣￣￣￣￣￣￣￣| DON'T BE AFRAID TO LEARN IN PUBLIC |＿＿＿＿＿＿＿＿＿＿＿＿| (\__/) || (•ㅅ•) || / づ