Jump to content

What is the limit of a robot.txt file?


Recommended Posts

A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb. In terms of limitations (not sure if you meant that as well), there's the fact that therobots.txt file is merely a optional directive, not a mandate.

Link to comment
Share on other sites

Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb. In terms of limitations (not sure if you meant that as well), there's the fact that the robots.txt file is merely a optional directive, not a mandate. So search engines don't have to adhere to your guidelines.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...