Jump to content

Where robots.txt file uploaded to hosting directory?


thaichacha

Recommended Posts

A robots text file or robots.txt file (often mistakenly referred to as a robot.txt file) is a must-have for every website. Adding a robots.txt file to the root folder of your site is a very simple process, and having this file is actually a ‘sign of quality’ to the search engines. Let’s look at the robots.txt options available to your site.

Link to comment
Share on other sites

Robots.txt is a text file located in the site’s root directory that specifies for search engines’ crawlers and spiders what website pages and files you want or don’t want them to visit. Usually, site owners strive to be noticed by search engines, but there are cases when it’s not needed: For instance, if you store sensitive data or you want to save bandwidth by not indexing excluding heavy pages with images. 

When a crawler accesses a site, it requests a file named '/robots.txt' in the first place. If such a file is found, the crawler checks it for the website indexation instructions. 
 

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...