Open Document Format: A future without MS Office?

Blogger's Note: The future of office applications is no longer a Microsoft-exclusive domain. This is the message of this article. I hope you readers will understand that relaying on MS Office is not only an expensive proposition (if you do not believe, just check the news on software piracy raids on internet cafes.) but a non-innovate way also. Competition breeds innovation. I remember I was still using WordPerfect (sige na nga, pati WordStar!) which was infinitely much better than MS Word. Because Microsoft controls the operating system, it was able to make advances based on their competitors' products, added to its own product, package office apps into MS Office, and viola, market domination. Pretty sneaky. But things are changing and hopefully for the better. This a long document. If you are in a hurry, read the Bottom Line section. Read on... 8-)

Why OpenDocument Won (and Microsoft Office Open XML Didn’t)
David A. Wheeler, September 2, 2005 (updated September 15, 2005)

As noted in Groklaw, FT.com, ZDNet, and other places, the State of Massachusetts is backing the OpenDocument standard as the standard format for office applications, text documents, spreadsheets, charts and graphical documents like drawings and presentations. All Massachusetts agencies are expected to migrate by January 1, 2007. This is instead of Microsoft’s new Office XML format (aka Microsoft Office Open XML File format).

This is big news. Currently most people exchange office documents using Microsoft’s binary formats (known as .doc, .ppt, and .xls), but now that the XML technologies are available and more mature, many people want to switch to an XML-based approach. There’s general acceptance in the information technology community that for office documents some XML format will eventually replace the obsolete binary formats. Most people, a few years ago, expected that whatever XML format Microsoft created would win. Yet Microsoft appears to have lost the war, due to its own poor decisions.

Microsoft is predictably howling about this news, saying they are a “bit stunned” and that the results were “unnecessarily exclusive”. Microsoft better be prepared to be more stunned. Government officials in Massachusetts, Europe, and elsewhere, have been repeatedly telling Microsoft to stop posturing and actually meet their customers’ needs for complete interoperability, with no restrictions. Yet Microsoft has steadfastly refused to meet their customers’ needs, and they’ve done it so long that customers have abandoned their format. (Microsoft says they're open, but people who have independently evaluated the situation have determined that it's not true.) I suspect Massachusetts is only the first of many; governments around the world are working out their standards, preparing for the leap to XML-based office formats. The rest of industry is likely to follow suit, because they have many of the same needs and desires for long-lived documents and competitive suppliers. The best information available suggests that everyone is switching to OpenDocument, for all the same reasons, leaving Microsoft with a proprietary format no one wants to use.

UPDATE: Almost immediately after I wrote this paragraph, Indonesia's Ministry of Research and Technology announced that it will implement Java Desktop System (JDS) on Linux as a national-standard desktop. It plans to install it across Indonesia, beginning with its government-affiliated offices. JDS includes StarOffice, which is expected to soon release its OpenDocument implementation as its default file format. It sure doesn't take long for the steamroller to get moving, and the big kicker will probably be the European Union.

This article explains why things currently look so grim for Microsoft’s proprietary XML format, and so bright for OpenDocument. In some sense, a declaration that OpenDocument “won” on September 2005 is very early; who knows what will happen? But this is more than a snappy title; the tea leaves are looking really bad right now for Microsoft’s dreams to solely control the format used by all future office documents. In fact, if they don’t hurry, Microsoft could conceivably find their Office suite slowly moving into the dumpster with WordStar, VisiCalc, Lotus 1-2-3, dBase, and other former office leaders. That would be mind-boggling, but it’s occurred many times before -- who would have thought that any of those predecessors would stumble? I don’t think that’ll happen in this case, at least not so quickly, but it’s certainly a risk unless Microsoft changes direction. That would be horrific for them; Office is 40% of their revenue, and one of the primary reasons people use their operating system (which accounts for most of the rest of their non-investment income).

I’m more hopeful; Microsoft has historically changed direction when it needed to embrace a standard, and they can easily do it in this case. For example, around 1995 it suddenly and finally embraced the Internet standards, dropping its own networking standards that no one wanted. I think (and hope) that good sense will prevail on Microsoft in this case too -- in other words, that they’ll embrace OpenDocument and continue to sell products. If you can only read one other piece on this topic, take a look at ZDNet's "Microsoft must drop its Office politics", which is a good article. ZDNet concludes "Microsoft has a very simple path open to it ... include OpenDocument compatibility in its software. ... it either adopts the industry standard or gets locked out. It may not like this -- it prefers to use this logic to cow its competitors -- but it should have no reason to avoid a level playing field."

It looks like Microsoft gambled, and lost. Let’s see why, by looking at what governments are looking for... and why Microsoft chose to not compete.

Why Would a Government Choose OpenDocument?
In many ways this decision was fairly obvious. OpenDocument appears, at this point, to be the way to go, with no realistic alternative, for any government. The only real contenders were:
Microsoft Office binary format, the current common interchange format. But this is being abandoned by Microsoft, fails to exploit the newer XML technologies (thus giving up their advantages), and because it’s undocumented it’s causing continuous information loss. Just try to read Office documents from 10 years ago -- you’ll often fail. Now remember that governments need them in future centuries. They’re not meeting the need, so an alternative is needed.

Going with Microsoft Office XML, which as shown below, doesn’t meet government minimum requirements such as allowing any supplier to implement it. And implementations aren’t even available yet, though at this point that probably doesn’t matter any more.

OpenDocument, the only official standard. It’s already implemented by multiple vendors (including some at no cost), and it’s the only one that really meets government needs... and with a massive lead time to boot.

Other formats aren’t really competitive. PDF is a very useful display format, but it has a different purpose -- while it’s great at preserving formatting, it doesn’t let you edit the data meaningfully. HTML is great for web pages, or short documents, but it’s just not capable enough for these kinds of tasks. And so on. Both HTML and PDF will continue to be used, but they cannot be used as a complete replacement; people need what OpenDocument (or its Microsoft competitor) provides.

So let’s examine in more detail to see why OpenDocument is such an obvious success, by first looking at the requirements governments have. Governments don’t create office documents so that they can be tossed in the shredder. They often have to be accessible decades or centuries later, and many of them have to be accessible to any citizen, regardless of what equipment they use or will use. Let’s look at the kinds of issues governments (like Massachusetts) finds itself confronted with, by looking at their typical requirements for a modern office format:

An XML-based format. Now that XML is available, governments want a single format that uses XML for its many advantages (e.g., easy standard processing, flexibility, easy growth to arbitrary sizes, ease of repair/recovery, and interoperability). Binary formats have real trouble with extensibility, for example; if they assign one byte to a value, and later discover that they need more than one byte, it’s difficult to change anything, while in XML you just write the larger number. Repair is hard too; in XML, if some data is scrambled, you can recover the rest, but a scrambled binary file is often unrecoverable. (Compressed XML files like OpenDocument have the disadvantage that recovering after the scrambled point is often very difficult, but you can typically recover the data before scrambled section, and sometimes you can do much better.

Unspecified binary formats are worse -- if the program says “failed to load” there may be nothing you can do, even if there’s some recoverable data before the scrambled point.)

But the biggest reason for XML is to make it extremely easy for anyone to quickly make tools that can read, write, and manipulate the data. If you only use one program, ever, to read a format, it’s livable if the format is bizarre (like Microsoft’s current binary format is). But now that everything is networked, people want to quickly take pieces of data from many sources and combine them in new ways, and that demands a data format that’s much more flexible and accessible to any tool. XML was designed to do just this, so people want some kind of XML format. For office data, the choice is either OpenDocument or Microsoft’s XML format.

A specification. In the long term, all formats disappear. WordStar was once what everyone used as their word processor; now, no one even has a filter to read the format. Luckily, WordStar format is similar to ASCII and is thus mostly recoverable. Today, I can’t read some important PowerPoint 4 files in today’s PowerPoint - unacceptable to me, and governments think in terms of decades and centuries. Yet it happens, because there’s still no specification for the (now obsolete) Microsoft Office format. If there’s no spec, there’s no standard.

An aside: Microsoft program manager Adam Barr has suggested releasing a specification for the older Microsoft office binary formats. That would go a long way to improving interoperability, and if they did that (in a way that allowed any competitor to implement them) I’d be delighted.
This would certainly be a big help to many as long as arbitrary competitors could use the format (as opposed to the current license for the Microsoft XML format). It’s not even clear that Microsoft could really limit the license, since you’re not supposed to be able to patent things that already exist in the public for more than a year. That would not provide the benefits of XML, obviously, but it might mean that a transition to XML could happen in a slower manner (since there would be less concern about data loss, a serious concern today). I have not seen any evidence that Microsoft will do that, unfortunately.

Neutral specification maintainer, preferably a respected pre-existing standards body. OpenDocument has been developed and is maintained by a vendor-neutral body (OASIS); OASIS is even authorized to submit its specifications straight to ISO. Heck, Microsoft is even a member of OASIS; they certainly can’t claim ignorance of OpenDocument. Microsoft hasn’t even begun a standards process for its format. In May 2004 the European Union specifically told Microsoft to get involved with the OpenDocument standards effort, and that they considered the "winner" to be the one that became an international standard. At this point they’re too late -- the standards train already left the station, and arrived at the destination called OpenDocument. Patrick Gannon, president and CEO of OASIS, noted that "many European governments are considering similar policies [to require OpenDocument, like Massachusetts]", and it will be a topic of discussion at OASIS' European Adoption Forum in London, 17 October 2005.

Multi-vendor/customer development. The only way to make sure that all critical user needs and supplier issues are addressed is to get many different organizations to co-develop the specification, along with public feedback. Microsoft’s XML format didn’t; its development was completely controlled by Microsoft. That’s a terrible misstep for something that is supposed to be used worldwide, in perpetuity, for trillions of dollars worth of documents. Though first draft specifications are often created by a single person or small group, you have to get widespread review to get a good final result.

Multiple implementations. There are now multiple implementations of OpenDocument, with probably more to be announced soon. Governments don’t want to be locked into a single vendor, nor to force their citizens to do so. The costs go sky-high, and support vanishes, when there’s no competition. Only one vendor really supports the Microsoft XML format. Note that having multiple implementations is the best way to ensure that specification actually provides interoperability; the IETF even requires this for its standards because of this. Microsoft has mouthed nonsense such as claiming OpenDocument is only designed for OpenOffice.org, but multiple implementations show it false.

Anyone can implement the specification. Anyone can implement OpenDocument, and that’s not true for Microsoft’s format. Today, there are too many people and too many programs that need access to the data in office documents. Thus, it’s critical that anyone be able to implement an office format, especially since it’s the whole point of using XML.

This point seems to be the hardest for some people to understand, so here’s more detail. The bottom line: Microsoft licenses this format in a way that says, “you can use it freely, unless you’re a competitor”. And that’s a poison pill for any format like this, because governments want competition.

Microsoft itself acknowledges the need for an open format, saying that “Moving to document formats that are open, documented, and royalty-free is actually really valuable.... [because it makes your files] totally belong to you [so] you have control over them.” Good words! But there are words, and there are actions; governments are not always stupid. To meet such requirements, any such format has to be implementable by any proprietary program and by any open source software, at least using the licenses typical for each. Fifteen years ago it was easy to ignore open source software, but now the market has all sorts of open source software.
Nowadays, governments cannot in fairness mandate a standard that forbids implementations that use the most popular licenses for open source software; blatantly discriminatory regulations like that can bring officials into court.

And let’s be blunt: the most common license for open source software is the GNU GPL version 2, so any office format must be implementable by a program released under the GNU GPL. See here for GPL stats. I looked up the data again on September 1, 2005; Freshmeat’s statistics report that 67.41% of branches used the GPL, with the next-closest being LGPL (6.06%) and original BSD (3.34%). Even if you pretended that all non-GPL licenses were identical, when combined they’re still the minority. Not all backers of open source software like or use the GPL, but making it illegal to use such a widely-used license, for no good reason in public policy, is lunacy.

Microsoft hasn’t been willing to license its products for absolutely anyone to use, so it’s been unwilling to release a specification that’s appropriate for government use. Instead, Microsoft has only been willing to release a specification as long as it can’t be used by some of Microsoft’s primary competition, by creating weird legal licensing clauses that prevent interoperation and competition. Microsoft’s XML format cannot be implemented by programs licensed under the GNU GPL, for example. Under U.S. law, Microsoft is allowed to write specifications that exclude competitors, but it shouldn’t be surprised if people choose to not use them. Especially since there are current office suite products that use licenses that appear to be excluded by
Microsoft's terms (Gnumeric and AbiWord at least use the GPL; OpenOffice.org uses the related LGPL, and it's not clear they can use it either). There's no reason to lock out these market players. Remember, the whole point of the XML formats is to let anyone read and write them, if they choose.

In contrast, the OpenDocument specification can be implemented by anyone who uses any license, proprietary or open source -- including the GNU GPL license and Microsoft’s current Office license. So OpenDocument is open for anyone to implement, including Microsoft... while Microsoft’s XML format isn’t. Microsoft’s claim that OpenDocument is “unnecessarily exclusive” is nonsense; the shoe is on the other foot. I’d say this reason, by itself, is sufficient to disqualify Microsoft’s XML format from any government consideration, no matter what its capabilities, because it fails to give users the option of choosing what program or system they can use. Microsoft is just trying to prevent competition here. The European Union, IBM's Bob Sutor and many others all warn against this. Other countries are even less likely to accept Microsoft’s XML format; while Massachusetts sees Microsoft as a domestic company, other countries will see Microsoft as a foreign company and be very uninterested in forbidding competition against a foreign company.

Massachusetts’ Kriss emphasized that Massachusetts is not moving to open standards for economic reasons, but to protect the right of the public to open and free access to public documents, permanently. “What we’ve backed away from at this point is the use of a proprietary standard and we want standards that are published and free of legal encumbrances, and we don’t want two standards.”

Perhaps an analogy would help explain this. I expect that Microsoft would be unhappy if Massachusetts mandated that only GPL’ed software could be used by Massachusetts. Yet if Massachusetts did that, they could at least argue the advantages of doing so in terms of transparency of the code. (No, I’m not arguing that Massachusetts should do that, I’m just trying to make a point.) In contrast, Microsoft wants Massachusetts to mandate that GPL’ed software be forbidden for use in office suites. There’s no good public policy reason to do that, and lots of competitive reasons to avoid doing so. Especially when there’s a ready alternative -- an international standard, already implemented multiple times, including some high-quality freely-available implementations (giving them a range of options). I get the impression that Massachusetts worked really hard with Microsoft to get them to change the license to something more reasonable, so that Microsoft wouldn’t so obviously disqualify its work. Yet Microsoft continues to try to promulgate a specification license that forbids competition. Expecting any government to perpetually forbid the use of competing office suites is ridiculous, and Microsoft should have known better. I think they did know better, and hoped no one would notice.

Low-cost implementations. Not everyone is made of money. Governments have to interact with people who have little money, and governments are often strapped themselves. For OpenDocument, this is a no-brainer. Some OpenDocument implementations are available at no cost (particularly OpenOffice.org and KDE KOffice) and have a licensing structure that allows that to continue that way indefinitely. And these are good programs, not poor quality demos.

Even if you choose to use a non-free implementation (say StarOffice or Microsoft Office with an OpenDocument plug-in), this is obviously a big advantage to you, because it constrains the office suite prices. No such luck with Microsoft’s XML format; Microsoft XML is only available in the latest version of Office. Indeed, their licensing is carefully designed to prevent the most likely kinds of competition (it’s “free” as long as you’re not a real competitor). So to get Microsoft’s XML, you’d have to upgrade huge groups of people at a corresponding huge cost. There really isn’t even a competition between these two formats. And it gets worse; since no one else is supporting Microsoft XML, Microsoft XML will probably stay expensive to deploy, due to a lack of competition, especially if more people try to use it. This becomes a self-fulfilling prophecy; if more people try to use it, Microsoft will have incentive to raise the price (to get more money out of it), which in the long term will cause people to stop using it.

Support is already available for OpenDocument. OpenDocument is already out, and already getting used, so that lowers the risk. In fact, OpenDocument was developed based on lessons learned from the older OpenOffice.org format (they aren’t the same, but they’re similar, and the changes were made based on widespread review.) Microsoft’s full XML format still hasn’t even been fielded (it’s coming soon); it’s based on previous Office 2003 work, but that was never used as Office’s primary format, didn’t support critical components like PowerPoint, and the older version didn’t support all the functionality of the product. OpenDocument support is already out, and it appears more mature, especially if you consider multivendor support. As of yet I’ve seen no evidence of significant multivendor office suite support for Microsoft’s XML format.

The story here seems clear. Microsoft gambled that, because most current office users use their Office program, customers would choose Microsoft’s XML format even though Microsoft’s format did not meet their requirements. It appears that they hoped that, by creating subtle license traps, they would foreclose future competition, but make it look good enough that government officials wouldn’t understand the issues. Perhaps they expected that people wouldn’t examine their options carefully; an odd assumption, since so much money and data is at stake. Government folks are often overworked, yes, but there are a lot of smart people in government.

Government people can act especially smartly when they get good tips from others. A few years ago, secret Microsoft documents now named Halloween I and Halloween II were exposed to the world. These documents were developed in collaboration with key people in Microsoft. Their bottom line was a recommendation that Microsoft suppress competition by “de-commoditizing” protocols (creating proprietary formats that could not be used by others) and by attacking competitors through patent lawsuits. Since that time, people have been watching carefully and warning when Microsoft tries to “release” formats whose conditions inhibit competition.
ZDNet came to a similar conclusion, saying, "[when] open standards exist which are capable of supporting the work the state does, this should be an unexceptional decision; accessibility for as broad a range of citizens and organisations as possible is a primary responsibility for any government." The European Union said, similarly, "Because of its specific role in society, the public sector must avoid that a specific product is forced on anyone interacting with it electronically. Conversely, any document format that does not discriminate against market actors and that can be implemented across platforms should be encouraged. Likewise, the public sector should avoid any format that does not safeguard equal opportunities to market actors to implement format-processing applications, especially where this might impose product selection on the side of citizens or businesses. In this respect standardisation initiatives will ensure not only a fair and competitive market but will also help safeguard the interoperability of implementing solutions whilst preserving competition and innovation."

Customers, in this case governments, didn’t just accept whatever terms Microsoft gave them. That makes sense; few people just sign a blank check! Instead, they looked at the alternatives, found one that actually met their requirements, and chose that alternative instead. Now governments are starting to formally state their requirements, in terms of industry specifications, so that any supplier meet their needs. Suppliers can now choose to implement the specification and compete on cost, functionality, flexibility, consistency with public policy, and so on... or they can choose to not compete, and automatically lose. In other words, governments can do what governments usually do -- they can set a fair requirement that anyone can meet, clearly justified by their needs, and then use whichever suppliers best meets their requirements (in this case, for an interoperable format). This is not anti-Microsoft; governments have been specifically asking Microsoft for years to co-develop formats anyone can use, and Microsoft can implement the resulting industry standard, OpenDocument.

While the rest of industry doesn’t have the same needs as government, they have to work with governments, so government decisions sometimes have a trickle-down effect. Also, industry also has documents that need to be retrievable in the future, and the certainly want the lower costs and higher quality that tend to come from competition. So, as governments start making and announcing decisions in this direction, it’s reasonable to expect in this specific area that much of industry will follow. It’s true that Massachusetts all by itself cannot change the world, but Massachusetts has the same problems of many large governments, and it’s reasonable to think that if Massachusetts makes this selection, other governments will too.

So is OpenDocument any Good?
Is OpenDocument any good? Yes. In short, OpenDocument is a really good specification. The Future Is Open: What OpenDocument Is And Why You Should Care by Daniel Carrera gives some information on the advantages of OpenDocument. Tim Bray also makes some interesting comments. Wikipedia’s OpenDocument article has some interesting information.

You can sometimes learn a lot about something by seeing how it was created, and by who -- and that’s true here. OpenDocument was developed by many office suite developers, including those who develop OpenOffice.org, StarOffice, KDE’s KOffice, and Word Perfect (Corel). But some major users were involved, too, to make sure that their needs were met. Boeing was there; they have large, complicated documents, so their participation helped make sure that complex documents could be handled well. A Bible translation group also participated; they have lots of complex language needs, including multi-language documents and unusual languages; as a result, OpenDocument generally handles internationalization issues in a stellar way. They also allowed review by the public; I took that opportunity and voluntarily sent in comments, as did others. And they used as a basis an existing XML format (OpenOffice’s); this gave them a big leg up on Microsoft, whose previous XML work for Office documents did not cover key areas (Powerpoint, for example). This is how you get a good specification lasts a long time; you start with pre-existing work, and get many participants with different needs to work out any of its problems.

Contrast OpenDocument with Microsoft’s XML format. Last I saw, perfectly normal office-only documents can contain binary objects that depend on MS Office and Windows (e.g., OLE) and those lack complete documentation. But most importantly, its license essentially disqualifies it.
Though I wasn't on the committee that developed it, during their public comment period I read it myself and sent in a few comments. I was generally pleased with it after I reviewed it; there’s a lot of good to say about it. It’s actually quite clear to read, as these things go. And the careful crafting, and review by many, shows in the result. It's smaller than it might otherwise be, because they reused pre-existing standards instead of rolling their own (which is also a good idea in most cases). Now, OpenDocument isn’t perfect; no human product is. In particular, it underspecifies formulas in spreadsheets. It covers spreadsheets, including formatting, pivot tables, data, and lots of other issues, and gives examples of correct formulas, but doesn’t define in enough rigor the actual format for calculated formulas in spreadsheets. But the problem is underspecification, not that the specification that's there is bad, and for simple spreadsheets the material that's available is enough to get started. I found that to be a weakness, so I voluntarily developed a document called OpenFormula to try to address this. In any case, this turns out to be relatively easy to address; it’s certainly easier to address than the mess Microsoft has created for itself. And since there are multiple implementations of OpenDocument today, these sorts of weaknesses have already been identified and are being addressed. In contrast, Microsoft’s constraining licenses have restrained the kind of multi-vendor testing that is needed for good, long-lasting standards.

What Can Microsoft Do?
Now Microsoft’s in a minor bind. The world is already switching to OpenDocument, and now that all the other suppliers have invested in OpenDocument and have it working, there’s really no real incentive to use an alternative.

If Microsoft wanted to suddenly get their work standardized now, they would have many
problems doing so. Since it wasn’t developed in a large multi-vendor community, it will probably take years to really vet it and fix the inevitable problems that creep in when you work in isolation, years it doesn’t have because OpenDocument is already here and has had those years of experience and vetting. They could hire a bunch of people and do a pretend “independent” analysis by many people all paid by the same vendor, but the results are not likely to result in good work. I doubt Microsoft will even get much interest in the standards community; they already have a standard, so there’s no need to do the work twice. And that assumes Microsoft fixes its licensing problems, which appears unlikely. It can be done, but they’ve delayed entry into the standards process for so long that it’s not clear if anyone cares about standardizing on their format any more. In particular, it’s hard to imagine other office suite vendors being very interested, because they’ve already invested years in a perfectly good standard. Vendors will probably give Microsoft’s XML format a Monty Python-like response -- “It’s very nice-a, but we already got one.” And without participation from multiple vendors who implement office suites, the standardization would just be viewed (correctly) as a sham.

Microsoft can go ahead and use only Microsoft XML, but since nobody else can read it (including many current deployments of Office), and people are standardizing on OpenDocument instead, customers may find that they just don’t want the latest version of Office unless Microsoft hurries up and implements the standard. Strong pressure might cause abandonment of the latest version of Microsoft Office, intead of getting people to switch to the product.

One major use for Microsoft’s XML format might be as a starting point for a separate program that converts their format into OpenDocument. If an office suite built that in, but uses licenses like the GPL that Microsoft disapproves of, then Microsoft could exploit that through its license and shut the office suite down. But if it’s a separate program, it’s not clear what Microsoft would do. They could shut down the converter, but that wouldn’t stop competitors, and it would expose their licensing conditions as anti-competitive without actually harming the office suite suppliers. (Exposing conditions as anti-competitive makes sense from a business perspective if you can drive your competitors out of business before the law can do anything, but it makes no sense if it only exposes you to liability without harming your competitors.) If they do not shut down the converter, it could help people to move away from Office even more rapidly.

I feel really sorry for people like Microsoft’s Brian Jones; they’ve poured part of their lives into getting an XML format into Microsoft Office, and tried to get it at least partly open. Yet Microsoft’s decision-makers appear to have shot themselves in the foot. I doubt very much that people like Brian Jones made the legal decisions that appear to have made Microsoft XML lose; he does not deserve condemnation for those decisions.

But really, this needn’t hurt Microsoft at all. Anyone can implement OpenDocument, so Microsoft can easily add code to Microsoft Office so that they fully support OpenDocument too.

It’s easy, but will they swallow their pride enough to make their customers happy? I hope so. Someone will do it; OpenOffice.org itself could be used as a filter, if nothing else. There are XML-to-XML tools (like XSLT) that should make it easy to do. I suspect if Microsoft added good OpenDocument interfaces to Office, a lot of people would buy it. On the other hand, if people end up having to use OpenOffice.org as a plug-in to use Microsoft Office, they may soon start asking why they need Microsoft Office.

Microsoft can choose to do what it likes, but so can customers. Customers want a completely open standard, and Microsoft has chosen to not meet their customers’ requirements. If Microsoft continues to do so, then Microsoft should expect to lose its customers; that’s how the market works. They can’t claim ignorance that their customers want fully open standards; governments have been asking for them for decades. They know the rules, they just chose to ignore them.

But this Can’t Happen!
Actually, it can. A market leader can find that it’s failed to listen to its customers, and then either meet its customers’ needs or lose the market. It’s happened many times before. Microsoft decided to ignore the standards in this case, and so they’ll have to play catch-up if they want to compete. But they can do it, if they choose to, and they can harmed, if they ignore their customers. History even gives us some ready examples, so let’s look at them.

In the late 1980s through 1995, Microsoft refused to accept the Internet TCP/IP (and later web) standards, and tried to get everyone to use Microsoft’s proprietary networking standards instead. Even though everyone used Microsoft clients, the market rejected Microsoft’s networking standards, and chose plug-ins or switched away from Microsoft. Microsoft suddenly realized that its customers were leaving, and that they were about to be completely sidelined.
Around 1995 Gates commanded his troops to do an about-face and rush to get TCP/IP and the WWW far better supported. It wasn’t pretty at the time, but in fairly short order they got at least some things working to remain competitive. Microsoft has been powerful for a long time, but not omnipotent; when the market moves toward important interoperability standards, Microsoft usually manages to support them eventually, even if Microsoft had been trying to sell an alternative.

In fact, market leaders in this particular market niche have been overrun before. WordStar, Word Perfect, VisiCalc, Lotus 1-2-3, and dBase were all market-dominating office software at one time, supported by companies who had great incentive to stay as market leader. Each one lost because they ignored their customers, through problems such as ignoring the transition to 16-bit computing, to graphical user interfaces, or the need for reliability. Customers want to transition to a standard XML format for office data that anyone can implement, and are getting serious about it. They are even more serious now, because this is a side-effect of massive networking: with massive networking, everyone needs to be able to take and extract different snippets of data, in novel ways, so they need a format that is general enough to support it. Microsoft hasn’t taken that seriously enough.

Obviously, having a lock on the current market is no guarantee of the future. David Halberstam’s book “The Reckoning” gives another example from carmaking: “The US Big Three automakers thought that they could dictate what their ‘captive’ market could buy, but the US public proved that assumption to be false, in the mid seventies. The only survivors from that era of heavy, rear wheel drive land yachts (albeit in much reduced and much improved forms) were the Ford Crown Victoria/ Mercury Grand Marquis and the Lincoln Town Car. Every other passenger vehicle is some variation of the K-car.”

Can a group of suppliers overtake a big company? Sure. Look at the videotape standards war of VHS vs. Sony Betamax in the 1980’s. Sony was a big company, trying to control an industry through a format it created. But the rest of the industry chose VHS, which allowed many different suppliers (Sony tightly controlled who could implement Betamax, while the VHS specification was far more open to implementation by others). The group of VHS suppliers quickly competed with each other, while staying true to the standard, customers preferred formats where there were competing vendors, and as a group the VHS vendors demolished Betamax’s market share. There’s even a slang term based on this; “to betamax” means “to deploy a proprietary technology format that gets overwhelmed in the market by a format that allows multiple competing manufacturers”. There’s more to that story, of course, but my point is that being big does not mean every product you make is accepted in the market. The fact that there’s a slang term tells you something else: occasionally suppliers try this stunt. Large companies are often lured (by greed) into trying to completely control a market and their customers via a proprietary format. Often competitors (who fear being driven out of business) then band together with customers (who fear having a sole supplier) to develop and promote a standard that is not as proprietary. Eventually the broadly-created standards tend to win, because customers make the final decisions, and few customers want to be controlled by a single vendor. Having broad input into the standard’s development also helps make sure that all important needs were addressed. Microsoft is betamaxing in this case, and that’s too bad.

Microsoft even freely admits that customers do not want to be locked into a single supplier for their office documents. The problem is that Microsoft has been repeatedly claiming that their format meets this requirement. No independent evaluator believes Microsoft has met this requirement, and in fact, several published reports have explained exactly why Microsoft has not met this minimum bar.

Do official standards always win in the market? Of course not! Many standards have failed in the past because they didn’t have any implementations, their implementations were poor, or because the implementations were far more expensive than their competitors. But no one pretends that any of these cases are true. Even Microsoft’s Form 10-K report ending June 2005 admits that OpenOffice.org’s competition, in particular, is now a risk factor for them. The main reason standards don’t win are not in evidence here. But the typical reasons for a market leader to fail are in full bloom, namely, failing to meet critical customer requirements and pushing a proprietary format against an open standard widely supported by competitors and embraced by users.

There’s no doubt that this will cost money. Any transition -- even a minor transition to a new version of the same product -- costs money. But these are one-time costs, whereas staying where they are will cause more data loss, and by telling everyone now where they are going they can get everyone moving in the same direction (with more lead time, the risks and costs go down). Saugus, Massachusetts’ website Saugus.net suggests this transition may not be as difficult as some fear: “It won’t affect most of the bigger Saugus web sites at all (as Saugus.net has been supporting open formats generally since 1998 when the Saugus By-Laws were first made available, and newer Saugus School sites like the TAHG project site are already building in support, too).” Certainly this rollout will require planning, as with any IT policy, but that’s why people get hired to manage IT infrastructures.

Bottom Line
At this time it appears that OpenDocument is the wisest and lowest-risk long-term decision, even though at first blush it seems surprising. Any market leader has lots of advantages, but it appears that Microsoft has far overplayed its hand here.

The old Microsoft Office format is unspecified and will cause continuing data loss, and it fails to take advantage of XML technology. Even Microsoft is abandoning it. Microsoft’s XML format will prevent instead of help interoperability; it simply fails to meet typical government requirements, since its restrictive license prevents real competition and it failed to enter the standardization process (as requested by Europe and others). People will try to do what’s easy, but only if actually meets their requirements.

By announcing their goal early, governments like Massachusetts make it easier to achieve them, because that gives people time to plan that transition, and suppliers more time to implement the requirement. It appears that many other governments (including European governments) are coming to the same conclusion, for all the same reasons. Massachusetts has not rejected Microsoft Office, the program; they’ve simply rejected Microsoft’s new XML format, and chosen the international standard instead. Even if Massachusetts backed down (always a possibility, especially since sometimes technical decisions get trumped by good ol’ boy networks), this certainly suggests that other organizations will do the same. In fact, this can’t help but move the eye to another battleground: Europe.

In 2004 Europe made its desires quite clear. In particular, they told Microsoft to consider “the merits of submitting XML formats to an international standards body of their choice” and issue “a public commitment to publish and provide non-discriminatory access to future versions of its WordML specifications”. Europe also stated that where “only a single revisable document format can be used, this should be for a format around which there is industry consensus, as demonstrated by the format’s adoption as a standard.” Microsoft failed to take the hint, by avoiding standards and failing to provide non-discriminatory action. Microsoft could try to rush in at the last minute and catch up, or customers could decide that they don’t need their requirements; but every day that seems more doubtful. Governments and now doing just what they said they’d do... they’re choosing the standard, based on what meets their need.

I’m no Microsoft-basher, I even have friends there, and I’ll freely praise good decisions they make. But in this particular case, I think they’ve made some poor decisions, and the result was fairly predictable. Predicting is hard, especially about the future (so it’s said), and this certainly isn’t as certain as the sun’s rising. But things sure don’t look good for Microsoft’s proprietary XML format right now. Tacking the word “open” into the name doesn’t make it open, and ’royalty-free but not to all my competitors‘ is simply not acceptable to people nowadays. Even the Microsoft’s old binary office formats didn’t have those kinds of onerous limitations!

I’m sure that there will be people who use Microsoft’s new format. That’s not my point. My point is that it’s clear that Microsoft’s format is unlikely to dominate the future of office formats in the same way that their old binary formats did, unless something dramatic happens. The old binary formats for Word, Excel, etc., have become so common that essentially every office suite must be able to read and write them. But this ubiquity, without a specification and based on limited binary formats, has become problematic. These old formats are essentially unspecified, hard to process, and depend far too much on low-level arbitrary structures of old versions of Microsoft Office. Even Microsoft’s latest versions of Office often fail to read this format, and as they update their programs, older documents are increasingly likely to become unreadable. Since even Microsoft can’t manage to read the older versions of the format, reliably, there’s clearly a fundamental flaw in the format. Which is why there’s a need for a standard XML format.

David Berlind of ZDNet has a suspicion similar to mine: “My hunch is that there are plenty of government agencies, both domestic and foreign watching this one and that, in this game of chicken, Microsoft will not win.”

This won’t be hard for Microsoft to deal with, technically; they can just add the code to support OpenDocument. They already support RTF, ASCII, HTML, Word Perfect, and lots of other formats. (Yes, the HTML is very poor; it generally fails when validated by the W3C validator, for example, and is excessively complex. But the point is, they at least try.) The real question is, can Microsoft swallow its pride to meet user needs... or are they willing to risk their entire company in the hopes that users don’t care about their own requirements? Hopefully cooler heads will prevail, and Microsoft will simply implement the only international standard. It’s only one format, after all.

No doubt Microsoft will press on for a little while, and try to make it so that “everyone accepts both”. Except that everyone can’t accept both, because their licensing still forbids it. It really doesn’t make sense to have two formats for the same thing anyway. And if there can be only one format, customers want anyone to be able to compete using that format, without any restrictions.

There’s no use in Microsoft complaining that their proprietary format wasn’t chosen. Microsoft claims that OpenDocument is specific to OpenOffice 2.0, but that’s ridiculous. Many programs implement OpenDocument, in fact, KOffice was first. In contrast, only one company currently implements Microsoft’ format. Besides, there seems to be universal agreement that Microsoft’ format is specific to Microsoft, and Microsoft made all decisions about the format. The accusation appears far more appropriate to be raised against Microsoft XML. Microsoft was told that what users wanted was an internationally-standardized format usable by all, without restriction. For years they were told that. Yet Microsoft was unwilling to provide what their customers demanded. Now other (hungrier) suppliers have stepped up to meet the customer needs, and the rest of the suppliers risk losing the market to the suppliers who do want to meet customer needs. That’s how the market works.

More information is available in articles such as ZDNet's "Microsoft must drop its Office politics", Tim Bray's Massachusetts Back-Room, “Open XML Incompatible With GPL” by Peter Galli (June 20, 2005, eWeek), and A Study in Framing. Broader information is available in Wikipedia’s article on OpenDocument. According to Paula Rooney of Information Week, this should have been no surprise: “The state and software company have locked horns on this issue for more than two years”; in January 2005 Microsoft said it would ease licensing restrictions, but although they made a few tweaks, they never changed the license so all competitors could use their format.

Jason Brooks’ article Masachusetts vs. Microsoft? (eWeek, Sep 9, 2005) opines that Massachusetts’ “Plan to move toward open formats isn’t foolhardy if the state thinks it can better serve taxpayers by escaping proprietary lock.” His piece makes similar points to mine: “I don’t blame Microsoft for doing what it believes is best for its business -- but let’s not blame Massachusetts policy makers for doing what they believe is best for their state, either... Massachusetts has chosen to shift its default file formats to those for which any developer or firm has an equal chance of building an equally good application to create and consume these documents, thereby ensuring choice and flexibility for itself and for its residents. Where’s the controversy or zealotry here?” Some discussion about recent Microsoft missteps (more generally) are noted in this CRN article; hopefully they can get it together and deliver to their customers what the customers are asking for.

Feel free to see my home page at http://www.dwheeler.com. You may also want to look at my paper More than a Gigabuck: Estimating GNU/Linux’s Size, my article Why OSS/FS? Look at the Numbers!, and my papers and book on how to develop secure programs.
(C) Copyright 2005 David A. Wheeler. All rights reserved. Some of this material was posted earlier in Groklaw.