Open Source Quality Revisited

Author’s Note: This post was resurrected from an archived version of my website, and has been updated to point to current sources.

A number of people have responded to my original article and have raised a few objections about my initial proposal.

Several pointed out the fact that commercial software vendors generally don’t offer availability guarantees in their support offerings, although some seem to imply otherwise. Commercial support is a “best effort,” and really only provide you with someone to complain to when something breaks. Both of these points are equally valid, but I think that these complaints might (perhaps unintentionally) overlook the value of this level of support straight from the vendor.

When I speak of “specific penalties,” I don’t mean to imply that you can get $5000 if they don’t answer your question right now or patch the bug (although I can see how this can easily be interpreted as such a statement). Even without a traditional SLA, there are business costs associated with running and maintaing those support desks. It is in the best interest of the software developer to ensure that their code is as bug-free as possible. If there are too many problems the cost of maintaing the support operations, the developer downtime spent hunting problems, and damaged reputations, will begin to eat away at the precious bottom line. Even corporations who haven’t really focused on quality in the past have begun to sing a different tune. As with everything in the world, this is a system of checks and balances. Companies will find a mix of quality and support that helps them deliver the highest return on investment. It is our job to stay informed and help identify those companies who might be producing inferior goods. People who make physical components know the value of quality, and generally you know when you’re buying from a vendor who has sacrificed some quality for the sake of cost, you can get that information by measuring the tolerances allowed in their manufacturing process. Perhaps it is time that we start to think of our software in the same way.

A few pointed out that commercial support contracts, at most, give you access to people who can look at the code. With open source you can look at the code all you want. If you feel better, you can even print it out. However, unlike commercial software and support, open source projects do not afford you guaranteed access to anyone that knows anything about the project, and there are relatively few penalties associated with continually producing an inferior product.

I am not convinced that is a bad thing. Open source software thrives on the individual efforts of some crack programmers who are challenging established ideas of what and how and where and when software should behave. Without unsuccessful experimentation, this model would never have become what it has today. Competition and cooperation (the latter of which is relatively unheard of in the commercial world) among open projects are some of the most powerful forces motivating open source development today.

I do not think that it is in our best interest to place the burden of testing/measurement on these free-range developers, for a fear of stifling their creativity, but establishing guidelines that reflect the quality that we, as consumers of software components, expect and comparing similar projects is a worthwhile activity. other industries have organizations devoted to attempting to rate or analyze the risks of certain decisions, ranging from the risks of offering someone credit to car saftey to who delivers the best tasting steak for the dollar. My question to you is this: why is providing third party analysis of these risks, in the context of open source software components, a bad idea? Sure, it isn’t cheap, but isn’t it worth knowing what you’re up against?

I have also heard from people who argue that the same risks exist in the corporate world. I couldn’t agree more with that observation! But when I think about the feasibility of instituting this sort of rating system on closed-source products, I see many more hurdles standing in the way of a fair assessment. I question our ability as an industry to make such a big leap without first thinking about the problem in a more practical environment. If we start this sort of movement with open source software, we eliminate some of the biggest hurdles (although introducing a few smaller ones). Over time, as the benefits and metrics that validate the success or failure of this endeavor begin to surface, we can start to think about how we can take this to the commercial world.

Some have taken the “build” approach to the old build vs buy question. This mentality certainally has a lot of merit when developing many critical systems, but for the rest of us who can’t afford to start multi-year, multi-million dollar, projects, we must rely on the pre-built components that can help us focus on the business problem at hand, instead of infrastructure development. The J2EE market, which effectively commoditized the middleware game, has shown us how successful this approach can be. As many of these products mature, they become increasingly reliable. It’s our job to translate and build upon their successes and failures to deliver more reliable software to the general market.

Our challenge does not end with open source software, but it can be a great beginning. Our ultimate goal is to see software, whether commercial or open source, become as reliable as many of the systems we use today.

Perhaps the best analogy I can think of is that of the car. I leave you with this question: If your car broke down as frequently as your software, would you drive it? More importantly, if car manufacturers hadn’t taken the time to improve the quality of their vehicles, would they have ever achieved the ubiquity that they enjoy today? With the widespread adoption of cheap general purpose computing devices, and their incorporation into our daily lives, the software industry (both open and closed) is again challenged by the average consumer who wants lower costs and higher quality. Will we wait for the public to mandate or legislate quality, creating a disaster that would make Y2K look like a spring rainshower?