Jump to content

What is the role of robot.txt In Seo..?


sonamsharma

Recommended Posts

The asterisk after “user-agent” means that the robots.txt file applies to all web robots that visit the site. The slash after “Disallow” tells the robot to not visit any pages on the site. ... After all, one of the major goals of SEO is to get search engines to crawl your site easily so they increase your ranking.

Link to comment
Share on other sites

  • 3 weeks later...

Robots.txt file tells the search engines crawlers which content, pages need to access/index on the website or which pages not to. As an example, if your website doesn't have a robot.txt file, then maybe search engines can crawl all your root files and many more backend pages. In every site there is some pages which you do not want to make visible for everyone.

Link to comment
Share on other sites

  • 4 weeks later...

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...