Thing is, it doesn’t stop you from browsing and continuing to browse tag intersections that don’t match any content (after all, there might be some content there later, right?). It just keeps giving you new intersection urls and new feeds.

The math lesson I learned was … well, I’m no good with permutations and combinations and factorials, but the lesson was that you can intersect twelve tags in a whole lot of ways. I guess since my blogging software doesn’t enforce any ordering of the tags when it generates the intersection url, you could theoretically have:

12! + 12!/1! + 12!/2! + 12!/3! + ... + 12!/11!

Something like that. Anyway, it’s a lot. And Google started crawling them all, plus the RSS feeds. (Actually, not all of the permutations were accessible because a given tag has to be related to one in the current intersection in order to show up as a choice to add. Still: a lot.) Combine that with the fact that I was caching the RSS feeds even when they were empty, and I had a problem.

I worked it out by: adding a meta “noindex,nofollow” tag to all the pages with no actual content, blocking Googlebot from all the tag RSS feeds it had already found via robots.txt, deleting the tens of thousands of cache files for the RSS feeds Googlebot had already hit, and turning off future RSS caching. All told, Googlebot crawled almost 200,000 pages using about one gigabyte of bandwidth before it stopped.

it’s also hard to deny that del.icio.us has zero spam protection built in.

All complex ecosystems have parasites. (Audio)

its hard to deny the usefulness of millions of handpicked links sorted with human verified meta-data.

True, but it’s also hard to deny that del.icio.us has zero spam protection built in. I believe that is the biggest issue in this Google + del.icio.us discussion, and one I’d consider a real stumbler: if Google were to try to derive value from del.icio.us, it is hard to conceive that the spammers would take longer than you need to say “potato” before they’d be all over it.

In that sense I would actually prefer that Google stay away from it.

Code Snippets (awesome link, btw, thanks Ryan) is very different, because it has actual content that stands on its own. I think Ian hit on the right spot in saying that del.icio.us is, itself, not a destination.

But anyway, isn’t access to /rss good enough? That contains 100% of the information del.icio.us carries; everything else is UI that would be distracting to search engines

Right. The only issue is that del.icio.us/rss moves so fast. You would need a 10-15 second poll on it to ensure you’re getting everything. Maybe you don’t need everything or maybe something like mnot’s feed history IETF memo could help but even then, del.icio.us might not be able to allow this level of access without incurring significant cost.

The Big G (or others), could probablly find space for that sort of knowledge in their ranking algorithms.

But anyway, isn’t access to /rss good enough? That contains 100% of the information del.icio.us carries; everything else is UI that would be distracting to search engines, not helpful. I suppose it would allow Google to point to http://del.icio.us links — but is that really useful? The whole point is to point to the real page, and while del.icio.us links can have a little commentary, it isn’t enough to make it a destination.

And the more use it gains the more useful it becomes. A great example of a Web 2.0 app.