Thread Safety is hard to maintain

I am constantly amazed by how hard it is to write thread safe code and have it stay thread safe. This week I had the pleasure of debugging two very different thread safety problems with very different origins on the theme.

Actually as I think about it, the origin of the problems are fairly similar. In both cases, the root cause was a failure of the thread safe architecture to be appropriately documented. In the first case, the degradation was gradual. The bugs were introduced slowly over time as multiple developers who did not understand the architectural requirements added or modified functionality. The worst part about thread safety problems is that problems that result in reference counting errors in which references are not released are nearly impossible to find. The same holds true for race conditions due to a failure to apply read/write locks or mutexes in a consistent manner.

These errors are extremely easy to introduce and unless there is very thorough auditing of the code it is quite likely that even a well written test suite is not going to trigger the errors. Deadlock conditions and under counting of references are much easier to identify because they typically result in a hard error.

The second problem was even more hideous. The architect made an assumption about the order in which threads would be allocated in an application. It never occurred to her/him that the libraries that are linked into the application might one day be multi-threaded themselves. The end result was that a data structure that was meant to allow thread safe access to objects by allocating a single bit in a table to each thread ended up due to math truncation associating multiple threads with the same bits. The end result is that while not a single line in the program changed, the thread safety of the application went out the window.

I don't know if better documentation would have helped in this second instance but it is clear that heavily used production servers started to bounce because of an undocumented assumption.

My advice to anyone that is developing and maintaining multi-threaded systems is to be careful. What looks on the surface like a trivial change may very well disrupt a a delicately balanced system. Whenever possible find someone else who understands the architecture of the system to review each and every line being very careful to pain attention to the expansion of macros and the side-effects that function calls might have.

Kerberos for Windows 3.0 update

Release Candidate 1 of the OpenAFS plug-in for KFW 3.0's Network Identity Manager is now available from the Secure Endpoints web site. https://www.secure-endpoints.com.

This release provides the full set of functionality necessary to manage token acquisition for multiple cells from a single Kerberos 5 principal.

New Daily Builds of OpenAFS for Windows

Today there are new daily builds of OpenAFS for Windows in both 32-bit and 64-bit versions available from http://web.mit.edu/jaltman/Public/OpenAFS/. The builds are dated 20051219.

They include performance improvements when fcrypt is in use and better logging of messages to the Windows Event Log.

MIT Kerberos for Windows 3.0 and Network Identity Manager 1.0 finally ship

Its been a long week but MIT's Kerberos for Windows 3.0 and the NEW Network Identity Manager has finally been shipped. I cannot give enough praise to Asanka for all of his hard work on this project. It would not have happened without him.

OpenAFS for Windows 1.4.1 RC2 and MIT Kerberos for Windows 3.0 Beta 2

Yesterday was a busy day. OpenAFS for Windows version 1.4.1 RC2 has been announced as well as MIT Kerberos for Windows beta 2.

The OpenAFS release can be downloaded from http://dl.openafs.org/dl/openafs/candidate/1.4.1-rc2/winnt/

The MIT Kerberos release can be downloaded from http://web.mit.edu/kerberos/.

MIT Kerberos for Windows 3.0 marks a turning point in the products history. For the first time, KFW will allow, through the use of the new Network Identity Manager, the ability to manage multiple Kerberos 5 identities at once. The NetIdMgr is based on the Khimaira Identity Management Framework which was described in a talk at the 2005 AFS & Kerberos Best Practices Conference at CMU. A copy of the presentation can be found at: http://www.secure-endpoints.com/talks/AFS-BPW-2005-Khimaira.pdf

The Khimaira framework enables the concepts of "identity" and "credentials" to be managed separately. In the modules shipped with MIT KFW 3.0, a single Kerberos 5 identity manager is included that allows users to maintain identities based upon Kerberos 5 principal names represented by Kerberos 5 Ticket Granting Tickets. Credential Managers are then provided to manage policy and perform intial credential acquisition and renewals for each specific credential type. In KFW 3.0, two Credential Managers are provided, one for Kerberos 5 and one for Kerberos 4. The Kerberos 5 Credential Manager maintains policy such as whether or not TGTs obtained should be forwardable, renewable, with what lifetimes, and whether or not they should be renewed before expiration. The Kerberos 4 credential manager receives notifications whenever a Kerberos 5 TGT is obtained and based upon its policy determines whether or not to generate a Kerberos 4 TGT via krb524d.


Secure Endpoints is providing an AFS Credential Manager that can be used to obtain tokens for an arbitrary number of cells in response to a Kerberos 5 identity being updated with a new TGT.
It is hoped that other organizations will take advantage of this new framework to add support for automated X.509 Certificate Acquisition.

Happy Fifth Birthday OpenAFS!!!

OpenAFS 1.4.0 has been released on the 5th birthday of OpenAFS.org.

http://lists.openafs.org/pipermail/openafs-announce/2005/000129.html

OpenAFS vs ntvdm.exe wildcard searches

In the OpenAFS RT there has been a ticket open for over a year because 16-bit applications when executed out of AFS were crashing.  It turns out that ntvdm.exe will perform wildcard searches for files as FOO?????.C?? instead of FOO*.C*.  OpenAFS for Windows was failing to match this pattern to FOO.C as it implemented a semantic of '?' must match a single character other than (dot).  The real rule is that (dot) is the component separator and '?' matches a single character and may match no characters if it is at the end of a component and the input string is empty.
With this fix all of those people who wish to execute the DOS versions of Quatro or Microsoft Word out of AFS (you know who you are) can rejoice.