What do you guys think of adding a new directive to the sitemap?
Something like <index_exclusive>url</index_exclusive>.

Including this in the sitemap would direct the bot to only index the
pages listed and ignore any other internal links found on the pages
listed.
It would have to be understood by the user that including this
directive would mean they are taking responsibility for telling the
bot explicitly which URL’s to index and that anything left out would
not be indexed.

I came up with this ( “bright idea” ) while trying to use the nofollow
to eliminate the duplication WMT “errors” in a PHPBB forum, but this
would apply to most any forum or blog.

So rather than peppering the nofollow all over the place, one entry in
the sitemap would direct the bot to ignore all of the different ways
provided to the user to jump to content and eliminate the duplicate
Title and Meta Tag “errors”.

Bill

On Dec 1, 10:22 am, wreilly wrote:

That's something we've considered before and pretty much dropped. The
big problem is that it's just too easy to break things completely with
something like that. If you forget URLs in your Sitemap file or if you
forget to update your Sitemap file or even if you forget that you have
a Sitemap file, you could accidentally limit your site's indexing
without knowing it. I do however agree that the problem you mentioned
(duplicate content through URL parameters) is an important one - and
it's one that we (and all other search engines) are always working on
improving.

John

Thank you. I had figured this wasn’t the first time this came up;  the
idea was hatched from thinking “there’s gotta be a better way”.

It would be complicated to implement and fraught with peril for the
uninformed user, I agree.

Don’t suppose I could persuade you guys to take a look at what I did
with my forum? Basically there are some global restrictions in the
robots and then using nofollow, limiting the “path” for the bot to the
sitemap urls generated with ( a modified ) GSitecrawler’s phpbb sample
project.

It seems to be working for the bot, but I have been there before.

Bill

On Dec 3, 3:53 am, JohnMu wrote: