Jump to content

Which is better — Meta Robot tags or robots.txt?


Gleam Edu

Recommended Posts

Robotstxt files are best for disallowing a whole section of a site, such as a category whereas a meta tag is more efficient at disallowing single files and pages. You could choose to use both a meta robots tag and a robots.

 

Link to comment
Share on other sites

Use robots.txt :

To stop engines from crawling certain sections of the site

There are two important considerations when using the robots <META> tag:

robots can ignore your <META> tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
the NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (perhaps on some other site), and so still arrives at your undesired page.

Link to comment
Share on other sites

  • 2 weeks later...

Robots.
Txt documents are perfect for disallowing a whole section of a site, like a category whereas a meta tag is significantly more effective at disallowing single files and pages. You could choose to use both a meta tag and robots.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...