Jump to content

What are robots.txt file?


smartlinktechlab

Recommended Posts

The robotstxt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots which pages on your site to crawl. It also tells web robots which pages not to crawl.

Link to comment
Share on other sites

Robots. txt is a text file that instructs the search engine bots on how to crawl and index your website. Essentially, it consists of rules that are part of the REP (Robots Exclusion Protocol), a standard protocol to communicate with web crawlers and web bots.
 

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...