| What is WebduniaBot? |
|
WebduniaBot is a user-agent for Webdunia's web crawler. Webdunia shops around the web to build the next generation search engine. It crawl the websites and gathers their content for search engine indexes.
|
|
|
|
|
|
How do I request that Webdunia not crawl my site?
|
|
By defining few rules in Robots.txt file, you can instruct WebduniaBot to not crawl and index certain files. The format of the Robots.txt is specified in the Robot Exclusion Standard. Following are the instructions about how to prevent
WebduniaBot from crawl your site:
|
|
|
|
|
• Webdunia follows the Standard Robot Exclusion Protocol.
• If the protocol disallows, your site can not be crawled.
• To prevent the WebduniaBot from crawling your "/source" directory, add the following in Robots.txt file:
|
| User-agent: WebduniaBot |
| Disallow: /Source |
|
|
|
|
| For more details about Robot Exclusion Standards, click here |
|
|
|
|
| If you face any kind of problem in crawling of site through WebduniaBot, please send your queries to crawler@webdunia.net. |