What is robot.txt file?

[Medical]

What is robot.txt file?
 

[shoppingswag]

The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl.

[sinelogixtech]

Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.