I/O Processing
The November blog entry finished with an enumeration of "to do" items that were necessary but unlikely to be implemented in the short term. Contrary to what was written, a new I/O processing pathway was implemented and made available in 1.7.22. The primary benefit of the new I/O pathway implementation is sustained throughput during file system store operations. The prior implementation would stall due to lost races between the afs redirector and afsd_service fighting for ownership over file extents. In addition to throughput improvements, the new I/O processing pathway permits applications to store data to the file server and bypass both the Windows System Cache and the AFSCache file by specifying the FILE_FLAG_NO_BUFFERING flag when opening the file with CreateFile.In the last six months a number of bugs were corrected that could result in data corruption when a mixture of file write's were issued to the same file with and without FILE_FLAG_NO_BUFFERING. One common scenario in which data corruption has been observed is when downloading files via Firefox or Internet Explorer. In order to make the most efficient use of clock time, the browser begins downloading a file from the web server as soon as the user selects the url. At this point the browser does not know where to write the data so it stores it in process memory while the user is presented a File Save ... dialog. Once the browser knows where the data should be written it creates the file in "no buffering" mode and instructs that the all of the data cached in memory be written at once to disk. The file is then closed and re-opened with normal buffering behavior. The data corruption would occur when the Windows File Cache received a request to store data in the middle of a 4K page but did not recognize that it must first read the prior contents of the file into memory. The error was due to a failure to set the ValidDataLength on the file during non-cached (non-buffered) I/O write operations.
Reparse Points and Symlinks
The March entry entitled Symbolic Links on Windows described in significant detail the challenges of working with Symlinks on Windows. Over the last six months the management of Symbolic Links via the AFS redirector has been altered. Instead of representing AFS symlinks with the Microsoft assigned reparse point tag value for OpenAFS, the AFS redirector now uses the reparse point tag used to represent NTFS Symbolic Links. The benefits of this are many:- The Win32 CreateSymbolicLink api can be used to create AFS symbolic links.
- Applications that are NTFS Symbolic Link aware recognize AFS symbolic links without modification.
- JP Software's Take Command
- Microsoft's robocopy
- Microsoft's Power Shell
- Cygwin
- Link Shell Extension
Reparse Points to Files
There are many applications that are not reparse point aware. As described in Microsoft's Symbolic Link Effects on File System Functions if a directory entry's attributes include the FILE_ATTRIBUTE_REPARSE_POINT flag the attributes, timestamps and size refer to the reparse point object and not the target of the reparse point even though a normal CreateFile request will open the target object. Applying the reparse point size to the target object's stream is likely to result in an incorrect end-of-file determination.What is most unfortunate is that all versions of .NET and all versions of Java through 1.6 ignore the FILE_ATTRIBUTE_REPARSE_POINT flag. Of course, from the perspective of application developers that use .NET and Java, the problem is not Microsoft's to solve but a flaw in AFS. It is viewed as a flaw in AFS because the OpenAFS AFS SMB gateway didn't support reparse points and it did not expose symlinks to the applications. As a result .NET and Java simply worked.
In 1.7.24 a new registry option has been provided that disable the reporting of Symbolic Links to Files as reparse points. The 0th bit of the ReparsePointPolicy value when set activates this behavior. When this policy is activated, directory entries for symbolic links do not contain the FILE_ATTRIBUTE_REPARSE_POINT flag and their timestamps and file size are that of the target file.
Other improvements
There are have a broad range of application compatibility improvements to the network provider interfaces, optimizations of the garbage collection operations, and compatibility with IBM AFS 3.6 file servers for those that still use them, and dozens of other small tweaks.The Summer months will be spent on Windows 8.1 support and a rewrite of the Authentication Group and Process tracking. The AuthGroup changes are desired so that Reparse Point Policies can be applied at run time to independent groups of applications.
