Jump to content

What is the Coding For Roboot.txt File..?

Recommended Posts

robots.txt files indicate whether certain user agents (web-crawling software) can or cannot crawl parts of a website. These crawl instructions are specified by “disallowing” or “allowing” the behavior of certain (or all) user agents. Basic format: User-agent: [user-agent name] Disallow:

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...