Robots.txt Parse & Crawler Policy Guard

Parse RFC 9309 groups, select crawler rules, evaluate one or many paths, diff policies, and apply explicit publication controls without fetching a URL.

Start free workspace OpenAPI →

CONTRACT

Predictable for agents. Legible for reviewers.

REST: POST /v1/robots/policy-check
MCP tool: check_robots_policy
Inputs: Caller-supplied robots.txt up to 512,000 UTF-8 bytes and 10,000 lines; path checks accept one HTTP(S) URL or origin-form path, batches accept up to 1,000 paths, and diffs accept up to 256 sample paths.
Returns: CLEAR, REVIEW_REQUIRED, or BLOCKED with parsed groups, exact line evidence, selected crawler tokens, longest-match rule decisions, policy changes, named findings, and an agent-readable next action.
Latency: Synchronous
Usage unit: Robots-policy operation

The suite also exposes /v1/robots/parse, /v1/robots/select, /v1/robots/can-fetch, /v1/robots/batch-check, and /v1/robots/diff plus /mcp/robots. It processes caller-supplied text only and never fetches a URL, crawls a site, bypasses access controls, verifies indexing behavior, protects private content, or stores the payload. Its fully configured private RapidAPI project is release-queued behind the provider's twenty-public-API cap; direct workspace access remains available.

AGENT-READABLE EXCEPTIONS

ROBOTS_PATH_DISALLOWEDROBOTS_BATCH_PATHS_DISALLOWEDROBOTS_NO_MATCHING_GROUPROBOTS_POLICY_CHANGEDROBOTS_WILDCARD_GROUP_MISSINGROBOTS_USER_AGENT_NOT_ALLOWEDROBOTS_ROOT_DISALLOWEDROBOTS_SITEMAP_NOT_ABSOLUTEROBOTS_SITEMAP_NOT_HTTPSROBOTS_EXTENSION_BLOCKEDROBOTS_PATH_REQUIREDROBOTS_URL_UNSUPPORTED

HTTP request

POST /v1/robots/policy-check
Authorization: Bearer apc_...
Content-Type: application/json

{
  "content": "User-agent: *\nDisallow: /private/\nAllow: /private/public.html\nSitemap: https://example.com/sitemap.xml\n",
  "require_wildcard_group": true,
  "require_https_sitemaps": true,
  "require_absolute_sitemaps": true,
  "disallow_root_is_blocking": true,
  "blocked_extensions": [
    "crawl-delay",
    "host"
  ]
}

cURL (bash)

curl --request POST \
  --url 'https://api-production-9502.up.railway.app/v1/robots/policy-check' \
  --header 'Authorization: Bearer ${AP_CONTROL_API_KEY}' \
  --header 'Content-Type: application/json' \
  --data '{
  "content": "User-agent: *\nDisallow: /private/\nAllow: /private/public.html\nSitemap: https://example.com/sitemap.xml\n",
  "require_wildcard_group": true,
  "require_https_sitemaps": true,
  "require_absolute_sitemaps": true,
  "disallow_root_is_blocking": true,
  "blocked_extensions": [
    "crawl-delay",
    "host"
  ]
}'

Representative response

{
  "status": "COMPLETED",
  "decision": "CLEAR",
  "allowed": true,
  "group_count": 1,
  "wildcard_group_present": true,
  "sitemap_count": 1,
  "extension_names": [],
  "findings": [],
  "ruleset_version": "RFC_9309_2022",
  "scope": "CALLER_SUPPLIED_ROBOTS_TEXT_ONLY",
  "network_queried": false,
  "url_fetched": false,
  "access_control_enforced": false,
  "indexing_outcome_verified": false,
  "payload_stored": false,
  "next_action": "The document satisfies the supplied static policy; test the deployed file and enforce real authorization separately."
}

RFC 9309 caller text onlyThe API parses the supplied document and returns exact group, rule, pattern, and line evidence without fetching a site.

Robots.txt is not access controlAllowed crawl guidance never grants authorization, bypasses authentication, or proves that private content is protected.

No indexing or crawler claimThe service cannot verify deployment, crawler compliance, indexing outcomes, runtime throttling, redirects, or destination safety.

Try Robots.txt Parse & Crawler Policy Guard with your own workflow.

Start with the workspace, inspect the typed result, then integrate with REST or the focused MCP server.

Start free workspace Read the integration guide →