Comments on: Google Throttles Blog Search Indexing http://www.seo-theory.com/2009/02/12/google-throttles-blog-search-indexing/ Algorithm analysis, Web community relationship analysis, SEO practices and techniques, industry news, etc. Mon, 15 Mar 2010 10:48:35 -0400 http://wordpress.org/?v=2.8.4 hourly 1 By: Michael Martinez http://www.seo-theory.com/2009/02/12/google-throttles-blog-search-indexing/comment-page-1/#comment-1449 Michael Martinez Fri, 13 Feb 2009 17:47:41 +0000 http://www.seo-theory.com/?p=1409#comment-1449 Jeremy, thanks for the timely reply and the explanations. I do see some improvement although I have not checked out all the queries I normally monitor. I watch many blogs in Blogsearch and the overall trend has been for delays in seeing their content appear in Blogsearch. I'm still not happy with the change in the default behavior, but I realize I'm just one user out of millions. I appreciate your responsiveness to the issues I've been able to document. Jeremy, thanks for the timely reply and the explanations. I do see some improvement although I have not checked out all the queries I normally monitor. I watch many blogs in Blogsearch and the overall trend has been for delays in seeing their content appear in Blogsearch.

I’m still not happy with the change in the default behavior, but I realize I’m just one user out of millions. I appreciate your responsiveness to the issues I’ve been able to document.

]]> By: jhylton http://www.seo-theory.com/2009/02/12/google-throttles-blog-search-indexing/comment-page-1/#comment-1448 jhylton Fri, 13 Feb 2009 16:29:43 +0000 http://www.seo-theory.com/?p=1409#comment-1448 Michael, Thanks for you feedback on blogsearch. I read your post about two hours after you published it, and it was already in the our index. (I checked our backend systems. It looks like it took almost eight minutes for us to index it after receiving your ping. I'm not sure why it took so long to get indexed.) I tried the same queries you did and saw the same problems. There were a few different causes for these problems. One simple problem is that searchenginejournal.com was classified as a news site, so we weren't including it. That's just a mistake, and I'll make sure it gets fixed today. The problem with the other queries is a little more subtle. We have some algoritms that attempt to eliminate very similar results, which can be fairly aggressive at times. If you get to the last page of results, you'll sometimes see a message that says, "We have omitted some entries very similar to the NNN already displayed." There is a following the message that repeats the query with an extra filter=0 CGI param, which shows all the results. For [site:seo-theory.com] and [site:seroundtable.com], we omitted some results because we thought they were too similar. The results are filtered on a per-query basis, so posts will still show up for other queries. You can see the difference by adding the filter=0 param yourself: <a href="http://blogsearch.google.com/blogsearch?q=site:seroundtable.com" rel="nofollow">[site:seroundtable.com]</a> <a href="http://blogsearch.google.com/blogsearch?q=site:seroundtable.com&filter=0" rel="nofollow">[site:seroundtable.com] with filter=0</a> We were already planning to disable the duplicate suppression algorithm when you restrict your query to a single site or blog. I think that change will be live in a week or two. I didn't get a chance to look at the Sphinn query yesterday, but it looks okay today. I don't know what might have gone wrong when you ran your query yesterday. A variety of transient problems might have caused us to briefly serve stale results, but I think they are rare. It's not unusual for a particular blog to be slow, either, and then Googlebot may take longer than normal to crawl the new posts. Jeremy Hylton Google Michael,

Thanks for you feedback on blogsearch.

I read your post about two hours after you published it, and it was
already in the our index. (I checked our backend systems. It looks
like it took almost eight minutes for us to index it after receiving
your ping. I’m not sure why it took so long to get indexed.)

I tried the same queries you did and saw the same problems. There
were a few different causes for these problems. One simple problem is
that searchenginejournal.com was classified as a news site, so we
weren’t including it. That’s just a mistake, and I’ll make sure it
gets fixed today.

The problem with the other queries is a little more subtle. We have
some algoritms that attempt to eliminate very similar results, which
can be fairly aggressive at times. If you get to the last page of
results, you’ll sometimes see a message that says, “We have omitted
some entries very similar to the NNN already displayed.” There is a
following the message that repeats the query with an extra filter=0
CGI param, which shows all the results.

For [site:seo-theory.com] and [site:seroundtable.com], we omitted some
results because we thought they were too similar. The results are
filtered on a per-query basis, so posts will still show up for other
queries. You can see the difference by adding the filter=0 param
yourself:

[site:seroundtable.com]

[site:seroundtable.com] with filter=0

We were already planning to disable the duplicate suppression
algorithm when you restrict your query to a single site or blog. I
think that change will be live in a week or two.

I didn’t get a chance to look at the Sphinn query yesterday, but it
looks okay today. I don’t know what might have gone wrong when you
ran your query yesterday. A variety of transient problems might have
caused us to briefly serve stale results, but I think they are rare.
It’s not unusual for a particular blog to be slow, either, and then
Googlebot may take longer than normal to crawl the new posts.

Jeremy Hylton
Google

]]> By: Michael Martinez http://www.seo-theory.com/2009/02/12/google-throttles-blog-search-indexing/comment-page-1/#comment-1447 Michael Martinez Fri, 13 Feb 2009 05:54:12 +0000 http://www.seo-theory.com/?p=1409#comment-1447 I think it will help some people having crawl issues. I think the SEO community will also find ways to make up nonsense about the tag. And maybe there is a small chance someone will discover an unforeseen use for it. I think it will help some people having crawl issues. I think the SEO community will also find ways to make up nonsense about the tag. And maybe there is a small chance someone will discover an unforeseen use for it.

]]> By: olmei http://www.seo-theory.com/2009/02/12/google-throttles-blog-search-indexing/comment-page-1/#comment-1446 olmei Fri, 13 Feb 2009 03:29:27 +0000 http://www.seo-theory.com/?p=1409#comment-1446 Michael, Not exactly related to your article but I'm currently in a heavier mode of research and data accumulation hell than usual and am itching to know your initial take on what the effect the forth coming 'Canonical Concert' of msn, google and yahoo might have. Feel free to expand heavily with respect to the content repetition theme (which I also utilize and enjoy your take on), in copy, urls, etc... GOOD ARTICLE by the way. Though next time consider expanding your screen shots so I can better view your programs-of-choice icons. Mike Michael,
Not exactly related to your article but I’m currently in a heavier mode of research and data accumulation hell than usual and am itching to know your initial take on what the effect the forth coming ‘Canonical Concert’ of msn, google and yahoo might have.

Feel free to expand heavily with respect to the content repetition theme (which I also utilize and enjoy your take on), in copy, urls, etc…

GOOD ARTICLE by the way.
Though next time consider expanding your screen shots so I can better view your programs-of-choice icons.

Mike

]]>