Jump to content

How do I add a robots txt file to my website?


eCommerceChamp

Recommended Posts

  1. Place your robots.txt file in the top-level directory of your website code to simplify crawling and indexing.
  2. Structure your robots.txt properly, like this: User-agent → Disallow → Allow → Host → Sitemap. ...
  3. Make sure that every URL you want to “Allow:” or “Disallow:” is placed on an individual line.
Link to comment
Share on other sites

Than go on to the search console account / webmaster tool and follow these steps

>> Select your site in which you have put your file.

>> In left side menu click on to the crawl menu

>> Here you will see a sub menu with robots.txt tester

>> Here your file will be shown any error or warning

So it is so easy to add robots.txt file…

Link to comment
Share on other sites

Identify which directories and files on your web server you want to block from being crawled by the crawler.
Identify weather or not you need to specify additional instructions for a particular search engine bot beyond a generic set of crawling directives.
Use a text editor to create the robots.txt file and directives to block content
Optional: Add a reference to your sitemap file (if you have one)
Check for errors by validating your robots.txt file
Upload the robots.txt file to the root directory of your site.

Link to comment
Share on other sites

Identify which directories and files on your web server you want to block from being crawled by the crawler.
Identify weather or not you need to specify additional instructions for a particular search engine bot beyond a generic set of crawling directives.
Use a text editor to create the robots.txt file and directives to block content
Optional: Add a reference to your sitemap file (if you have one)
Check for errors by validating your robots.txt file
Upload the robots.txt file to the root directory of your site.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...