This post is a bit all-over-the-place, sorry.
I’ve stumbled across Code Snippets at least 5 times in the past couple of days. It’s basically del.icio.us for small pieces of code. Each snippet gets a title, description, actual code snippet, a set of tags, and most importantly a URL. The result of this situation is obviously great search engine indexability because, as I said, I’ve happened upon it at least 5 times now through basic Google usage.
What’s interesting is that, as far as I can tell, they’ve placed NO limitations on what type of snippets can be posted. There’s a quick bash two-liner for Automatically adding a bunch of stuff to CVS next to a Generic XHTML Template next to a Python snippet for generating midi tones on a Series 60 cell.
Considering this from more abstract level, you might call this a demonstration of a few bits of theory laid out by Doc Searls and David Weinberger in World of Ends. Here we have two ends (Google and Code Snippets) that work well together due to a common level of understanding of what’s desirable in the larger system they both operate in. The value of each end seems to increase with each new end that it touches. I think this basically follows Metcalfe’s Law, which states that “the value of a network equals approximately the square of the number of users of the system (n2).” Only “users” in this context can mean “other systems”. In this case, Code Snippets enhances Google and Google enhances Code Snippets. You might also say that Code Snippets gets more value from Google than Google gets from Code Snippets and that the actual value each obtain is close to that predicted by Metcalfe (or not).
Anyway, the enhanced searchability this style of organization facilitates got me thinking about the quality of metadata at del.icio.us proper… Did you know that Joshua disallows access to / for all robots? and that Google is not spidering the amazing set of collobrative metadata available there? Having the Googlebot run through del.icio.us on a regular basis would be insanely expensive for del.icio.us - Technorati did it for about two weeks and then were cut off if I remember correctly.
But why shouldn’t Google/Yahoo/whoever purchase the bandwidth and other resources necessary to run their spiders over del.icio.us? If you did pay, spidering probably wouldn’t be the best method of getting at the meat on del.icio.us. If I were a search engine, I would try to convince Joshua to lease me a basically persistent stream from this RSS 1.0 Feed. You should be able to put something together that improved the quality of your search dramatically with the quality of data in that stream.
Finally, it’s interesting to note that “/rss” is the only resource robots are allowed to access:
User-agent: * Disallow: / Allow: /rss
If del.icio.us were so inclined, I think it’s reasonable to believe that they could start picking up revenue by leasing high quality access to that URL.