Saturday, February 24, 2007

Squeaky wheels receive attention (both good and bad)

I spent the past few weeks traveling the country meeting with organizations that use OpenAFS and Kerberos for Windows.  I heard a number of really wonderful things:
  • "We haven't had a show stopper event in more than a year"
  • "The performance is so much better than it used to be.  We no longer receive complaints about how slow it is instead our users send us messages like this one, 'OH My gosh, afs is so fast now since i got my upgrade :)'"
At the same time the amount of funding spent on support and new development has been decreasing.  Budgets are always tight and management wants to spend its money on addressing the issues that cause on-going problems. 

Just a couple of years ago, the OpenAFS Windows client was so bad that not only were organizations sending money but individuals would send personal paypal payments and bottles of tequila as a "thank you for improving my life".   These days expectations have changed.  The assumption is that the OpenAFS Windows client just works.

In the 1.5.15 release of OpenAFS for Windows, a serious data corruption bug was fixed.  As it turns out this bug had been reported to IBM within the last year by an organization that was still using the IBM AFS Windows client.  When the organization switched to OpenAFS it never occurred to them that OpenAFS would have the same problem given their common heritage.  OpenAFS is so much better in so many ways that they "just assumed it had already been fixed."

The truth is that all of the low hanging fruit has already been picked.  Its not that there is no more work to be done but that all of the remaining work is big.  So big in fact that it cannot be paid for out of support budgets.  Instead strategic planning funds must be used and those are much harder to come by especially when the scope of the projects is in developer years and hundreds of thousands of U.S. dollars.  Its no longer possible for someone to ask "how much would it cost to fix xyz?" and receive a response indicating that the work could be done in a few hours or a day or two.

Instead, much of the longer term strategic work that was done to support the Windows Vista platform was unsupported.  Secure Endpoints contributed hundreds of hours of developer time to ensure that there would be an OpenAFS client for the new operating system.  This was done on the assumption that the costs would be re-couped in the future through interest in support contracts.  What a surprise it was to hear this week that existing support contract customers are questioning the need for the support.  The long hours spent improving the product have taken OpenAFS off the radar of senior management and as a result the funding is disappearing.

One large user described how there have been so few reported issues with the 1.4.2 client that he can't justify upgrading to 1.5.15 even though he is aware of all of the significant improvements in performance and stability.  Performance improvements just aren't a reason to upgrade when there are thousands of clients involved.  Stability doesn't matter if the end users are not being adversely affected.   Sure there are bugs and annoyances but the help desk knows how to address them and the users move on with life.   Management simply is not going to spend money on something that is faster or prettier.  If there isn't a critical show stopper issue, it won't be detected by their radar.

Our philosophy is that software is built to address the needs of its users with the goal of making their lives happier and more productive.  Good software doesn't attract unwanted attention.  In the case of a file system or other infrastructure, the end user should be able to take it for granted.  If it receives attention from the user, that is a bad thing.

A good support contract vendor is one that addresses issues promptly when they occur, but more importantly works to ensure that you do not have issues in the first place.  The question is, if support dollars are used to fund development that pro actively addresses issues before they are noticed by the customer, how does the customer know that the support dollars were well spent?  This is especially true when management does not believe that incremental improvements in performance and stability are worth paying for.

I am now beginning to understand the behaviors of large corporations providing support to Federal agencies.  I find them extremely frustrating to deal with because the apparent goal is to deploy software with just the right amount of bugs such that there are never issues that bring the entire system to a halt but that ensure that there is a constant stream of small issues that will keep them on the phone with the agency's help desk.  Every week a report is sent to the customer detailing the number of issues categorized by severity and whether or not the user's problem could be addressed.  Large numbers of low severity issues is encouraged whereas even a single Priority One issue is to be avoided. 

Fortunately for the clients of Secure Endpoints Inc, I believe that our role is to help prevent problems regardless of the severity.  Unfortunately, it is then harder to make the case for additional financial investment in products that are already deemed to be "good enough".