Log analysis
Parse server and CDN logs so automated traffic becomes visible outside analytics scripts.
AI
traffic visibility
But what is it actually?
See which AI bots are reading your website.
GoatScope analyzes logs and robots.txt to show how AI crawlers, search bots, and automated agents interact with your site. It helps you distinguish intentional access from accidental openness.
What this means for site owners
Default access is still a decision.
AI crawlers and agentic browsers are increasingly visiting websites. Many sites have no explicit policy for them, which means access may be technically allowed without the owner ever making a conscious choice.
AI crawler detection
Identify known AI crawlers, search bots, and browser-like automation patterns.
robots.txt interpretation
Show when access is explicit, blocked, or merely allowed by default.
Example workflow
From raw logs to a readable report.
Provide logs
Start with exported server or CDN logs from a recent traffic window.
Detect automated traffic
GoatScope classifies known crawlers and flags traffic that looks automated.
Make the choice explicit
Review detected bots, requested paths, robots.txt status, and recommended next steps.
example-report.html
18
AI crawler requests
4
known agents
?
explicit policy
Allowed by default
No explicit AI crawler rule found in robots.txt.
GPTBot
ClaudeBot
PerplexityBot
browser-like automation
The quiet default
Allowed by default does not always mean intentionally allowed.
GoatScope highlights when AI crawler access appears to be permitted simply because no explicit policy exists.
Philosophy
This is not just about blocking bots.
GoatScope is about understanding which bots visit, what they access, and whether that access is intentional.
Some websites may want AI crawlers. Some may want stronger limits. Some may want structured data, honeypots, feeds, or specific rules for agentic systems.
GoatScope does not aim to trap or exploit bots. The crawler test page is a transparent observation page designed to help study automated traffic.
Vibe-coded project
Messy in places, useful already, improving in public.
GoatScope is currently experimental and shaped by real-world feedback. The engine already works locally: it can parse logs, identify known AI crawlers and suspicious automated traffic, compare findings against robots.txt, and generate a report.
Feedback wanted
What should GoatScope become useful for?
Would you check your website logs for AI bot traffic?
Would you prefer a local-only tool, hosted dashboard, or browser extension?
What would you want to see in a report?
Should GoatScope help write robots.txt policies?
Should it help optimize websites for agentic visitors?
Proposed timeline
A practical path from local reports to regular visibility.
Local engine and static report generation.
Public website and example report.
Privacy-conscious upload or local analysis flow.
Browser extension or Cloudflare integration.
Repeated monitoring for regular reports.
Tools for making websites easier for useful agents to read.