Jump to content

In the robots.txt file, what does the “Disallow:” line do?


DynaPro
 Share

Recommended Posts

Hello,

Pages that you disallow in your robots.txt file won't be indexed, and spiders won't crawl them either. Robots.txt Format. The format for a robots.txt file is a special format but it's very simple. It consists of a "User-agent:" line and a "Disallow:" line. The "User-agent:" line refers to the robot.

Link to comment
Share on other sites

Hello Friends,

Which page you have set disallow in robots, google will not crawl that page and the all back link which you had made by that url. Basically robots use to for the security for the website. If you have to hide some pages from the Google and the audience. you can disallow that pages on robots files..!!

Link to comment
Share on other sites

  • 1 month later...

If you want to instruct all robots to stay away from your site, then this is the code you should put in your robots.txt to disallow all:

User-agent: *
Disallow: /

 

The “User-agent: *” part means that it applies to all robots. The “Disallow: /” part means that it applies to your entire website.

In effect, this will tell all robots and web crawlers that they are not allowed to access or crawl your site.

Link to comment
Share on other sites

Hello,

Disallow: The command used to tell a user-agent not to crawl particular URL. Only one "Disallow:" line is allowed for each URL. The second line in any block of directives is the Disallow line. You can have one or more of these lines, specifying which parts of the site the specified spider can’t access. An empty Disallow line means you’re not disallowing anything, so basically it means that a spider can access all sections of your site.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...