WebDexy-Bot 1.0
WebDexy-Bot is the search engine web crawler for WebDexy. It visits public web pages and follows links to discover content for the search index.
Identify the crawler
Look for WebDexy-Bot/1.0 in the User-Agent field of your access log. Use WebDexy-Bot as the crawler name in your robots.txt rules.
Manage crawler access
Publish your rules at /robots.txt on the host you want to manage. Choose the example that matches your intent.
Exclude the entire host
User-agent: WebDexy-Bot
Disallow: /Exclude a directory
User-agent: WebDexy-Bot
Disallow: /archive/Report a crawl request
When you contact WebDexy, include the hostname, requested path, timestamp with time zone, and User-Agent from your access log.