AI Crawlers Visibility, Analytics & Control

AI Crawlers Visibility, Analytics & Control helps WordPress site owners understand and manage how AI systems access their content — from classic AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and 100+ known bots) to MCP / Abilities API clients calling tools on your site.

The plugin combines analytics, live AI Inventory, access control, and AI discoverability in one WordPress dashboard. Track crawler and MCP activity, catalog agents / servers / abilities, block unwanted traffic, publish llms.txt, run an AI Visibility Score, and export reports — without replacing your SEO plugin, firewall, or server-level bot protection.

AI SEO, GEO, and AEO for WordPress

Traditional SEO helps search engines index a site. AI SEO, Generative Engine Optimization (GEO), and Answer Engine Optimization (AEO) focus on making content easier for AI crawlers and answer engines to discover and understand.

This plugin supports the technical foundations of AI visibility by helping you:

• Measure which AI crawlers and MCP clients access your content.
• Inventory the agents, MCP servers, and abilities touching your site.
• Check discovery, structured-data, and content-readiness signals.
• Publish AI-readable discovery files and Markdown versions of public content.
• Control crawler and MCP access without changing the content human visitors see.

No plugin can guarantee inclusion, citations, rankings, or traffic from ChatGPT, Claude, Gemini, Perplexity, or other AI services.

Why monitor AI traffic on WordPress?

AI companies use automated AI crawlers to index, train on, or retrieve website content. Separately, MCP clients and Abilities API tools may call your WordPress REST endpoints. This plugin covers both worlds:

• See which AI crawlers visit your site and which pages they request.
• See which MCP servers, abilities, and clients interact with your site.
• Compare traffic across date ranges and spot trends over time.
• Block AI crawlers (robots.txt / PHP 403) and MCP tools or servers (MCP Control).
• Keep logs for compliance, debugging, and content licensing decisions.
• See a 7-day crawler summary on the WordPress admin dashboard.

How it works

1. Enable crawler tracking (and optionally MCP tracking) and choose log retention.
2. Review Analytics, AI Inventory, and the Activity Log.
3. Use Crawler Control and/or MCP Control for traffic you do not want.
4. Run an AI Visibility Score scan and enable Discoverability features that fit your site.

Overview: Analytics, AI Inventory, and Activity Log

Analytics

• After activation, a getting-started panel explains that crawler tracking is already on and points you to a homepage Visibility Score check.
• Dual crawler + MCP summary cards on one dashboard (MCP cards appear after AI tool traffic is recorded).
• Charts for crawler requests over time and traffic by category; MCP timeline and server charts.
• Top AI crawlers, top MCP tools, and most-crawled pages.
• Download CSV reports for crawler and MCP activity (preset or custom date ranges).

AI Inventory

Live catalog of everything AI-related touching this site:

• Agents — crawlers and MCP clients with last seen, requests (7d), and status.
• MCP Servers — namespaces with trust tier (core / known / unknown).
• Abilities / Tools — tools seen in MCP traffic with a simple read/write risk heuristic.
• Findings — items that may need review (for example unknown servers).
• Detail pages per agent, server, or tool with entity-specific activity (paginated).
• Block / unblock from the inventory list or detail page (crawler PHP block or MCP Control rules).
• MCP detail activity includes an eye-icon request details modal (request/response JSON), same idea as MCP Tracker.
• Live refresh of the inventory catalog while the page is open.

Activity Log

• Dedicated recent-activity view with Crawlers and MCP tabs.
• Filters for crawler, status, content type, MCP server, and MCP outcome.
• Blocked attempts for crawlers and MCP.
• Click a row to open the matching AI Inventory detail page.

AI Visibility Score

Find technical signals that can limit AI discovery and get prioritized recommendations.

• Analyze any page or your entire site for AI readiness.
• 28 automated checks across discovery, structured data, content structure, and technical signals.
• 0–100 score with letter grade and prioritized recommendations.
• Site-wide batch scan, scan history, comparison diff, and PDF export.

AI discoverability: llms.txt, Markdown, and JSON-LD

Discoverability is organized with in-page tabs (llms.txt, WooCommerce, Markdown for Agents, JSON-LD, AI Meta, IndexNow).

• Auto-generate /llms.txt and optional /llms-full.txt for AI systems.
• Choose public post types, set URL limits, and include or exclude individual content.
• Exclude content by URL pattern and automatically exclude noindex content.
• Respect noindex metadata from Yoast SEO, Rank Math, All in One SEO, SEOPress, and Genesis.
• Regenerate discovery files when content is published or on a schedule.
• Optionally add LLMs: and LLMs-full: references to robots.txt.
• Serve .md URL endpoints for public content and taxonomy archives.
• Per-content AI Meta: description, keywords, and exclude-from-AI on posts/pages (product data tab when WooCommerce is active).
• Optional <meta name="ai-description"> (and ai-keywords / llms) in the HTML head.
• Respond to Accept: text/markdown requests from compatible AI agents.
• Output Organization, WebSite, and optional Article JSON-LD with AI discovery pointers.
• IndexNow: notify Bing, Yandex, and other IndexNow partners when content is published or updated (Google does not use IndexNow).

Visits from AI

See whether assistants send real visitors — not just crawlers:

• Detect referrals from ChatGPT, Perplexity, Gemini, Claude, Copilot, and similar sources via HTTP referrer and UTM.
• Optional short-lived attribution cookie for AI referrals only (Settings → General).
• Analytics card, sources table, per-page landings, and CSV export.
• Referral logs stay in your WordPress database and follow the same retention settings.

WooCommerce AI Search Optimization

When WooCommerce is active, optimize products for AI search engines:

• Include products in llms.txt with price, stock status, ratings, and attributes.
• Variable product price ranges and sale price display.
• Product Markdown endpoints with YAML commerce frontmatter (SKU, price, availability).
• Optional Product JSON-LD schema (skipped when major SEO plugins already output it).
• Product data panel tab: exclude from AI and set a short AI description.
• HTTP Link header pointing to /llms.txt (rel="ai-content-index").
• Product-focused Visibility Score checks (schema, SKU, price).

100+ built-in AI crawlers

• Pre-configured detection for major AI, search, and scraper bots.
• Category labels (training, search, assistant, agentic, scraper, and more).
• Per-crawler info with user-agent pattern details.
• Enable or disable monitoring for individual AI crawlers.
• Includes crawlers associated with OpenAI, Anthropic, Google, Perplexity, Meta, Amazon, Apple, ByteDance, Common Crawl, and others.

Custom AI crawler detection

• Add your own user-agent patterns for bots not in the built-in list.
• Assign company name and category for consistent reporting.

Auto crawler detection

• When an unknown bot (not in the built-in or custom list) visits your site, it is added to the Crawlers list automatically.
• Auto-detected entries show an Auto detected tag and support the same monitor, robots.txt, and PHP block controls as known crawlers.
• Remove an auto-detected crawler from the list anytime from its options menu.

robots.txt control for AI crawlers

Publish clear crawler preferences and identify bots that continue visiting after being disallowed.

• Add Disallow rules for selected AI crawlers.
• Bulk templates to disallow or allow entire categories (training, scraper, and more).
• Compliance monitor shows top non-compliant crawlers that still visit.
• Rules are appended to your WordPress robots.txt output.
• Live preview before you publish changes.

Block AI crawlers with PHP (403)

robots.txt communicates a preference; PHP blocking enforces access control at the WordPress application level.

• Hard-block selected AI crawlers before WordPress serves a response.
• Block specific IP addresses with the same 403 PHP response.
• Customize the plain-text message returned to blocked bots.
• Blocked requests are logged in analytics / Activity Log.
• Works alongside robots.txt for stronger enforcement when needed.

MCP Tracker

MCP Tracker monitors and logs Model Context Protocol (MCP) requests related to the Abilities API — without modifying request flow.

• Automatic MCP detection via Mcp-Session-Id and tracked namespaces (wp-abilities, mcp, mcp-adapter).
• Logging: route, HTTP method, headers (sensitive values redacted), request payload, response status, and response body.
• User attribution — which WordPress user (or guest) initiated the request.
• Filters by date range, ability/tool, user, and server namespace.
• Detail modal with formatted JSON request/response bodies.
• Block / unblock tool or server rules from the tracker row actions.
• Stats for total requests, unique abilities, errors, and last-24-hour activity.
• Enable or disable tracking from the MCP Tracker page header.
• Privacy focused — logs stay in your WordPress database.
• Shared retention with crawler logs; delete all MCP logs from Settings → Danger zone.
• Export MCP activity as CSV from Settings (Reports section).
• Optional MCP summary in scheduled email reports.

MCP Control

MCP Control lets you restrict MCP / Abilities traffic with allow/deny rules:

• Block or allow by tool, server, user, or IP.
• Rules apply at the WordPress REST layer for matching MCP traffic.
• Manage rules from Manage MCP → MCP Control, or add/remove common tool/server rules from MCP Tracker and AI Inventory.

Data management

Crawler and MCP activity is stored in your WordPress database.

• Configurable log retention (7 days to 1 year, or keep all data) for crawler and MCP logs.
• Optional delete-all crawler data and delete-all MCP logs from Settings.
• Delete plugin data on uninstall (optional setting).
• Multi-select filters on Activity Log and analytics tables.
• Download reports for preset or custom date ranges.

Reports and integrations

Reports live under Settings (with General and Danger zone):

• Daily, weekly, or monthly HTML email reports with configurable recipients and alert types.
• Email summaries cover crawler activity, traffic changes, blocked attempts, new crawlers, robots.txt non-compliance, and MCP activity.
• Test email delivery and view the next scheduled report.
• Authenticated REST API for analytics summaries, recent activity, CSV exports, visibility scans, and blocked attempts.

Clean admin experience

• WordPress admin menu: AI Crawlers (by miniOrange).
• Grouped sidebar navigation and dark mode (theme toggle in Settings → General).
• Overview: Analytics, AI Inventory, Activity Log.
• Manage Crawlers: Crawlers, Control.
• Manage MCP: MCP Tracker, MCP Control.
• Optimize: Visibility Score, Discoverability.
• System: Settings, Support (contact popup).
• Built for site administrators — no code required.

Who is this plugin for?

• Bloggers and publishers who want to know when AI crawlers access their articles.
• Business and marketing sites tracking AI bot traffic to key landing pages.
• Developers and agencies managing AI crawler and MCP policy across client WordPress sites.
• Site owners exploring robots.txt, PHP blocking, and MCP Control.
• Teams using Abilities API / MCP who need inventory, audit logs, and allow/deny for tool calls.

Privacy and data ownership

• AI crawler, MCP, and AI-referral activity logs are stored locally in your WordPress database.
• Analytics tracks detected crawler and MCP requests plus optional AI-referred human visits — not a full visitor analytics suite.
• A short-lived cookie is set only when a visit is attributed to an AI assistant; it is not used for general tracking.
• MCP Tracker / Control do not send logs to external services. IndexNow submits published URLs to the IndexNow API when enabled.
• Log retention and deletion are controlled from the plugin settings.
• Scheduled reports are sent through the standard WordPress wp_mail() function.
• Core monitoring does not require a separate analytics account.