Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Literally Google Webmaster Guidelines say: "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines."

So a couple of questions: - What is of value for a user? - Who is determining the value for the users in these cases?

As always, it's not always quite clear on what treatment you should use for search pages!



I'm pretty sure that having different search engines indexing each other would be bad. Don't cross the streams.


If I recall correctly, crossing the streams did kill marshmallow man!

In any case, I agree with you, search engine indexing search results would be bad, but the line is not that clear all the time!

Some vertical search engine result pages are a great and relevant result from a user perspective on the question they are trying to solve.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: