Removing Links

Someone sent me an email recently asking for his site to be removed from the index (and claimed that he had been added against his will). I checked, and in fact, the site was not part of the index. What he was referring to were links that had been observed on other people's weblogs, and had subsequently generated traffic by way of Blogdex. This raises three questions:

1. Should websites request that their links not be included in Blogdex? The court cases to date have upheld all kinds of free linking, most of which can be found in the open directory category on linking law. But Blogdex is supposed to be a tool for the people, and if weblogs don't want to be found, they sould have that right. Which brings me to

2. If the answer to 1 is yes, then should Blogdex post a list of those who are asking to be censored? The other side of Blogdex is as a research tool, and freely removing sites could lead to flawed data. Providing a list of self-censored sites will allow individuals doing research to have a complete view of the data at hand.

3. Should some standard be instituted for all robots like Blogdex? Perhaps an addition to the robots meta tag could allow crawlers to know those people who do not want links to be indexed (such as a "NOLINKINDEX" parameter).

I'd be interested to hear people's thoughts on the subject. I'm currently writing a list of policies for Blogdex, and I think this is a crucial issue to address.

Posted by cameron on April 23, 2002 at 07:47 PM
Whoa! Where'd all the weblogs go?

You may have noticed the severe drop in the weblog count, down from 16,000 to 11,000. There was a bit of dead wood in our crawl list, those weblogs that have not been updated in over 6 months. This makes yet another unlisted policy (I'll get around to listing them someday): no updates for 6 months will render a weblog "out of date," which simply means it will not be crawled on a daily basis.

Posted by cameron on April 17, 2002 at 05:33 AM
Stephanie Clifford uses Blogdex as

Stephanie Clifford uses Blogdex as a tool to gauge trendiness: Lee Jeans Pitches Bare Bottoms. After exchanging emails with her, I realized how difficult it is to perform this sort of analysis. How popular is this Lee Jeans Ad compared to the release of IPod? Or Super Greg? At present, there really isn't any way to do comparative analysis, something that really should be part of the tool set.

Posted by cameron on April 08, 2002 at 09:29 AM