Norman Walsh’s post on XMLK reminded of a post I had been meaning to write. I think microformats are a non-backwards compatible, but very practical, variation on XMLK. In this XML subset, HTML named character entities (e.g. ) can be resolved without a DTD, having been grandfathered in.
Here’s a slightly elided log line from a system I work with:
WARN 2005-09-28 14:38:38 StandardContext[/pmr] {”mid”:”123″, “type”:”XYZ”, “msg”:”foo”, “T”:”2005-09-28 14:38″, “L”:”warn”}
It identifies an XML document passing through the system using some keys from the envelope. It may seem a strange looking way to log an event, but between the angle brackets lies a useful side effect - it’s a Python dictionary. Hence this (vastly oversimplified) example log processing code, that runs eval() on each log line to load the dictionaries into memory:
Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)] on win32
Type “help”, “copyright”, “credits” or “license” for more information.
>>> dicts =[]
>>> import re
>>> p = re.compile(”.*({.*?}).*”)
>>> file = open(”log.txt”).readlines()
>>> for line in file:
… m = p.match(line)
… if m:
… d = eval(m.group(1))
… dicts.append(d)
…
>>> for d in dicts:
… print d
…
{’L': ‘warn’, ‘mid’: ‘123′, ‘type’: ‘XYZ’, ‘T’: ‘2005-09-28 14:38′, ‘msg’: ‘frobnizer found’}
{’L': ‘warn’, ‘mid’: ‘124′, ‘type’: ‘XYZ’, ‘T’: ‘2005-09-28 14:38′, ‘msg’: ‘frobnizer found’}
{’L': ‘warn’, ‘mid’: ‘125′, ‘type’: ‘XYZ’, ‘T’: ‘2005-09-28 14:38′, ‘msg’: ‘frobnizer found’}
{’L': ‘warn’, ‘mid’: ‘126′, ‘type’: ‘XYZ’, ‘T’: ‘2005-09-28 14:38′, ‘msg’: ‘frobnizer found’}
>>>
What’s nice about doing this is that it’s cheap to write the log line in a way that maximises usefulness later on - logs are often an afterthought, but good logging is critical to operating production systems. And it’s one example where creating metadata is cheap.
As well as programming data structures (changing the above to a Ruby hash is straightforward), you can get log lines to output something like Turtle, which is a greppable RDF format. What’s good about Turtle in particular is that you can dump or stream a log file straight into an RDF query engine, such as ARQ without any transform step and start writing queries against the data. Or how about writing down inferencing rules that help indicate the business consequences of a software error? The log lines themselves don’t carry enough information, but combined with rules and some context as to where the log event came from you can do interesting and useful things - the biggest issue with any monitoring software is gathering up all those machine level events and contextualising them so that data (noise) can be turned it into information (signal). Think of a machine readable data stucture inside logfiles as just another microformat.
David Janes and friends over at microformats.org have written an inspiring greasemonkey script that will find microformat structures on web pages. They include some links to pages with hCard and xFolk content.
When content in a supported microformat is encountered on a page, the script embeds a visual indicator like this one for hCard 
which when clicked produces a context menu. The functions on on the context menu depend upon the microformat. The menu for an hCard includes an “add to address book” item that will invoke Microsoft Outlook to create a contact.
Perhaps the most interesting thing though, is the menu item to map the vCard address with Google Maps. How many times per week do you grovel around the web for some physical coordinates only to then select-copy-paste into the browser search box, the first line of the address, then select-cut-copy-paste the second line in order to generate a map page? The “show on Google Maps” item does it all in one swell foop, taking you directly to the map!
dschud’s COinS-PMH shares with microformats the ability to embed into a static page, sufficient metadata and structure, to support extended processing without any need to re-invoke the web application that produced the content. But what does “extended processing” mean in this context? It means doing something with that content that the originator could not foresee. In the case of the microformats example it means a greasemonkey script. In the case of COinS-PMH the primary example is a “hacked ‘COinS Details’ sidebar”.
So repurposing content, conveniently, in ways unforeseen by its originator is what its all about. The greasemonkey script and the sidebar provide a glimpse of what’s in store. How much unforeseen functionality do the browser add-ons really provide though? In terms of scalability of functionality, should we expect the plug-ins to keep growing to encompass more functionality? For instance, should the hCard script grow to support mash-ups with more web applications beyond Google Maps? How about adding a function to send hCard holder a gift from Amazon?
The beauty of the hCard-Google-Map-mashup-script is that it didn’t require Google’s web app to understand hCard. The script-mashup did the translation. But that translation is at the heart of the scalability shortcoming. If the script has to contain mappings from each microformat to each way I want to use content conformant to that microformat then the opportunity for explosive functionality growth is missed. If on the other hand, the target web application understood the microformat in question, and we had a way to feed the target app conformant content then the scalability bottleneck would be broken. The wheels would be greased.
In a previous post I proposed the notion of a web app clipboard. The basic idea was that AJAX could be used to request content from a source application. The content would land on the operating system clipboard in a simple format (the web app clipboard format) and could then be transmitted to a destination application. Now I’m thinking that the original proposal was overbold. As Ryan Tomayko intimated, there ought to be a way to marry the microformats approach with the web app clipboard. Perhaps its profitable to think initially in terms of the source content being scraped directly off the page, without requiring the additional request to the sourcing web app. We’ve got that functionality in hand. With that in hand we could focus on the other side of the equation — where to send the content. The effect would be that we would no longer have to create both producing and consuming applications — we could start with just consuming ones. Anything that increases the likelihood that two web apps will communicate is a Good Thing. I’m with Phil Bogle and others who’ve pointed out that we need to bootstrap by finding those first two applications.
If a menu item called “copy” was added to the hCard context menu of the greasemonkey script, and that item caused the hCard to land on the operating system clipboard in a simple standard format, then it would just be a question of where the content could be pasted. Then our task is reduced to inducing a few web apps to support the paste action. A really simple first approach to paste support doesn’t even require XMLHttpRequest at all. A receiving web app could simply provide a text box wired to understand the web app clipboard format wrapping an hCard for instance. Paste into a text box works “out of the box” with all browsers. It’s not all that elegant, but it would work.
Giving microformats a place to go (destination applications) and a simple way to get there (the web app clipboard) will break a key bottleneck hindering broad interoperability.
For some time now, I’ve wanted to increase my understanding of microformats. If you’re unfamilar with the term or want to understand the basic purpose of this technology better, I suggest reading Phil Windley’s Microformats: Paving the Cowpaths. I read it some time ago and was intrigued but had very little time to do further research.
I have had a chance to dive in a bit more over the past few weeks and am excited at what I’ve found. I’ve trawled the mailing list archives, spent some time on the wiki, and read what people are saying on the blogs. I have yet to spend a lot of time in the guts of the individual specifications (e.g., hCard, XOXO, hCalendar, etc.) because, frankly, the nitty-gritty is a very small potion of what’s really grabbing my interest here.
There seems to be a bit of confussion around what “microformats” actually are and I think I know why. From what I’m seeing, the term “microformats” has two separate meanings - one is obvious and one comes after interacting with the community a bit.
“Microformats” are a set of specifications describing interoperable machine formats that work on the web.
“Microformats” is a process for arriving at interoperable machine formats that work on the web.
In general, when someone says “microformats”, they are most likely talking about the specifications. What I’ve found after lurking on the mailing list and watching the community is that when someone very close to the project says “microformats”, they are more often talking about the process that is evolving there. This is much harder to explain but it’s definitely worth trying because the core values, the Zen, these guys are carving out have very strong parallels to those of the less code movement, I think.
Luckily, I don’t think I’ll have to do much explaining myself because there are some gems from the microformats-discuss mailing list that I think go a long way in describing what’s going on there:
Mark Pilgrim in RE: Educating Others:
The regulars on this list grok this intuitively, or at least have accepted it for so long that they’ve forgotten how to articulate it. We simply don’t care about the general case.
Some people (Scott, Lisa, others) look at this and say “what about this edge case?” or “how do you combine them?” or “I need something with rigid structure” or “how do you validate them” or whatever. And these are all obvious questions that form interesting permathreads on mailing lists around the world. And we just don’t care. Not because we’re lazy or sloppy or naive — in fact, just the opposite. Our apathy towards the edge case is born out of bitter experience. We all bear the scars of drawn-out battles over edge cases that satisfied someone’s sense of “completeness” or “aesthetics” or “perfection”, but ultimately made the common cases harder and solved no real problem.
Ryan said microformats are all about 80/20. He’s right, but unless you’ve share [sic] our common experience, he may as well be speaking in Zen koans. Most standards go like this:
- Solve 80% of the problem in a matter of weeks.
- Spend two years arguing about the last 20%. (cough Atom cough)
- Implement the 80% in a matter of weeks. Wonder why everything is so hard.
- Spend months implementing the last 20%. Realize why the first 80% was so hard. Curse a lot.
- Discover that the last 20% wasn’t really worth all the time spent arguing about it or implementing it.
Microformats, on the other hand, go like this:
- Solve the 80% in a matter of weeks.
- Promise (wink wink) to tackle the 20% “later”.
- Implement the 80% in a matter of days.
- Watch people use it for a few months. Maybe tweak a little here and there.
- Discover that the last 20% wasn’t really necessary after all. Breathe a sigh of relief that you never bothered. Move on to the next problem.
The regulars on this list have all been through the full standards cycle many times. We know about edge cases, we know about validators, we know about standards. We know. We’ve been there. We’ve all decided that this way is better. Not because it’s easier or faster or sloppier, but because it leads to a better result. Really. The fact that it happens to be easier and faster is just a karmic coincidence.
Mark’s description of the mainstream standardization process vs. that of microformats could easily be used to describe the difference in technique employed by the mainstream software industry vs. that of the less code crowd.
Ryan King in RE: Educating Others:
… we’ve proven that microformats (at least, the ones developed so far) work in practice, we just need to show that they work in theory.
The arguments in this thread are theoretical–in theory there’s no difference between theory and practice, but in practice there is.
Luke Arno in Evolution vs. Intelligent Design:
It’s evolution not “intelligent design.”
Tantek Çelik in Microformats and the Semantic Web:
Microformats essentially ask:
Can we do more (practical use and applications) with less (logical formalism, formats, namespaces, etc.)?
Tantek Çelik in Microformats and the Semantic Web , and this one’s a gem:
I hardly ever look at PDF docs.
Without even looking at the actual technical specifications, I think these quotes say a lot about microformats’ potential.
To me, what’s exciting about microformats is the same thing that’s exciting about dynamic languages, REST, F/OSS, and other seemingly unconnected technologies and concepts: they are all evolving under the fundamental principle that you cannot adequately plan/design large systems upfront and expect them to be successful. That we don’t know anything until we’ve observed it. Respect for this simple truth leads to dramatic changes in how one builds and evaluates technology and you can see this happening to great effect (and with equally great results) in each of these communities.
The Web Standards Project holds as a fundamental truth: that document structure should be separated from presentation or visual style. This two-tier division is embodied by the XHTML standard for document structure and the CSS standard for visual style specification. Adoption of this strict separation of concerns provides revolutionary opportunities for the economical production of highly accessible, aesthetically pleasing web sites. At the same time huge strides are being made by AJAX innovators like 37signals to provide web application user interfaces on a par with those of desktop applications.
Concurrent with these breakthroughs in web user interface technology and practice, there is a an explosion of XML vocabularies, exemplified by OASIS, OPML, Microsoft Office XML, Open Office XML. A zillion XML vocabularies are blooming. These vocabulares are not presentational in nature like CSS, nor are they addressing document structure like XHTML. These vocabularies represent various other domains, commerce domains like order management, personal collaboration domains like calendar and contact management, media distribution and syndication domains, and myriad technical domains such as remote procedure call, database query, spreadsheet structural model. These vocabularies constitute a huge and growing, free, “domain model” for our world. Development, refinement and adoption of these shared, free domain models presents an unprecedented opportunity for both producers and users of information systems.
Enter web applications. Not static content-oriented websites, but honest to goodness web applications. Think email, calendaring, authoring, supply chain management, customer relationship management. These web applications are manipulating at their core, domain models. Sure, these applications have to present an interface to the user, but the bulk of what the application does is manipulate and manage domain-specific information.
While these web applications are manipulating domain-specific information they are doing precious little to expose that information in interoperable form. At best a web application may support download or upload of files. A contact management web application may for instance support bulk import or export — but where is the ability to conveniently “pick” an address from that application and use it as a “ship to” address at an online buying site?
Each web application is an island. This monolithic approach to web applications favors those product vendors who can make big bets — vendors who can tackle and deliver huge applications. Mash Ups provide a glimmer of hope — but only for developers — not really for end users.
It is not a new situation, that applications manipulate domain-specific information — information that users need to share with other applications. It has ever been so. So what’s the difference with web applications? The difference is that in the case of web applications, the world of document structure and presentation has not been bridged to the world of domain models. If Web 2.0 truly represents a new platform then perhaps we should consider whether there is anything to be learned from the older platforms in this regard. Where was presentation bridged to domain model for instance on the Macintosh, Windows and X-Windows platforms?
On those predecessor platforms, the clipboard paradigm was one very important bridge. In the clipboard paradigm a platform-wide set of gestures is supported by all applications. Those gestures enable a user to designate content, to copy that content to a clipboard, and to later “paste” that content into a different location — either in the original application or in a separate application. The range of gestures was expanded on some platforms to include drag and drop with added semantics at the drop location to support the notion of linking in addition to the traditional copy-paste and cut-paste a.k.a. “move”.
Fundamental to the notion of clipboard is the idea that there are potentially many representations for a given piece of content. This is where the bridging between presentation and domain model takes place. The clipboard can capture many representations at once — allowing the receiving application, in concert with the user to select precisely the level of presentation versus domain meaning desired. When pasting a circuit diagram (model) into an engineering report a presentational representation may be chosen, whereas when pasting the same content into a circuit simulation tool, a domain-specific (circuit modeling) representation may be chosen.
Could the original notion of the clipboard be updated to fit the Web Application architecture? Clearly the benefits would be significant. A new model, allowing web applications to leverage one another under user control means that smaller bets and therefore more bets can be made (by product vendors). More bets means more vendors, more applications, and more functionality delivered sooner to the 2.0 Web. While less can certainly be a competitive advantage to a product vendor, that advantage is only amplified in an environment where users can combine the functions of multiple products. If cut and paste and even Compound Documents were useful in the old platforms then they’re potentially much more useful now given the explosion of open, standard XML vocabularies. The opportunity for interoperability is greater than ever. Yet here we all sit, uploading and downloading files between applications.
Some will say, “but my browser supports cut and paste”. Well yes, the browser does support a limited form of cut and paste. But without access to the domain model, the browser is incapable (without help) of providing a cut and paste capability that goes deeper than document structure. Since the browser sees only the style sheet and the document — and not the underlying domain objects, it has no way (on its own) of loading a clipboard with a RDF vCard for instance, because that RDF vCard structure is never seen in its native form by the browser at all. At best an XHTML representation of that RDF vCard is seen at the browser.
Even if the RDF vCard could make its way onto the clipboard somehow, perhaps along with an XHTML representation and maybe a styled representation as well — there is no standard for presenting the multifaceted contents of the clipboard to the receiving web application. Is there a way to bridge the gap between domain structures and document structure?
Hang on… doesn’t AJAX provide a way out? Doesn’t AJAX allow various gestures to be captured and acted upon? Doesn’t AJAX allow arbitrary requests and responses between the browser and a web application in response to those gestures? Well sure it does!
If a source application could somehow get a standard “web application clipboard” structure onto the desktop clipboard, and if that structure could be sent by the user to a destination web application then the opportunity would arise for users to create “on the fly” Mash Ups. Users would recapture their lost ability to bridge applications requiring various levels of meaning (remember the circuit diagram example?) Web Applications could be combined by mere mortals in ways unforeseen by the original designers.
So what do we need to do to make it happen? Do we need to form a technical committee in a standards body? Do we need to go make a pitch to the software industry Old Guard? Sounds like it could take a long time and involve a lot of risk.
No I don’t think we need to do anything so heavy weight. Who’s in charge here anyway? We don’t have to ask anyone’s permission, and we really don’t have to ask for anyone’s help. Here’s how I see it going down:
This basic scheme will support copy/paste and cut/paste gestures today and set the stage for drag with drop-copy, drop-move and drop-link later. By focusing first on “by value” cut/copy and paste we avoid thorny issues of distributed trust. The user is in control of the clipboard and there is no direct application to application communication required. Let Big Brains working on things like SAML tackle federated identity management issues. Addressing the clipboard now, and composite applications later constitutes walking before running — or flying. We leave the door open, however, for eventual application to application communication — both for simple efficiency, i.e. avoiding unnecessary routing content through the browser, and for composite application construction.
Picking up what was lost with the move to the Web platform sets the stage for going beyond the predecessor platforms: