Jump to content

Where do I put robots txt file?


eCommerceChamp

Recommended Posts

Hi Friends,

Where the file goes in your filesystem depends on what host you're using, so it's hard for us to give a specific answer about that.

The best description is: put it wherever the index.html (or index.php or whatever) file is that represents your homepage. If that's domainname/public_html/index.html, for example, put it in domainname/public_html/robots.txt.

Link to comment
Share on other sites

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door, you cannot prevent thieves from coming in but the good guys will not open to door and enter.

Link to comment
Share on other sites

A robots.txt file consists of one or more rules. Each rule blocks access for a given crawler to a specified file path in that website. Here is a simple robots.txt file with two rules, explained below: Explanation: 1. The user agent named "Googlebot" crawler should not crawl the folder http://example.com/nogooglebot/ or any subdirectories. 2. All other user agents can access the entire site.

Link to comment
Share on other sites

  • 2 weeks later...
On 5/29/2019 at 12:50 PM, eCommerceChamp said:

hi

 

Where do I put robots txt file?
 
A robots text file i s a text file .It is a must have for every website. Adding a robots.txt file to the root folder of your site is a very simple process, and having this file is actually a ‘sign of quality’ to the search engines.
 
 
 
 
 
 

 

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...