Home Current Projects Technical Training More about Us contact us
Critical Comments

Handles |Invalid Handle Values | Ex Functions | How Many Bytes | 64-Bits and Pointers | Missing Functionality | Pointers | Timeout/Infinity | Program Safety | File Links | Commands/Utilities | Releasing Memory | Little/Big Endian | Remove or Delete | 64-Bits and Addressing | Excess Functionality | UNIX Pseudoterminal Emulation | Threads/Synchronization | Program Appearance

This section contains a variety of comments criticizing different aspects of the Win32 design and functionality. These comments are my own or, in some cases, have come from readers and course participants. Disclaimer: I generally like Win32. The comments here should be taken in that spirit. After all, I took the time to write four book editions and have spent a great deal of time consulting and lecturing on Windows subjects. What is more, Windows is commercially successful and has been extremely stable since its first appearance in 1993. Nonetheless, it is important to be skeptical and critical in the hopes that, perhaps, Microsoft will improve a few things in the future. In fact, they have made numerous improvements over the years, and some of these comments will may appear dated, but I've left them in to give a sense of history and so that we can see how much has changed.

Updates September, 2014, March, 2010, July, 2005, and September, 2004. Some opinions have changed since this was first written in 1998 and are marked as such. Likewise, some critical comments are marked as obsolete, such as anything dealing with W95. Since I first wrote this, 64-bit issues and condition variables have been dealt with nicely.

Send your Opinions

Dissenting or additional opinions on any of these subjects are welcome, and I'd be happy to post them here (several are included in this version). UNIX (or Linux, or … ) vs. Win32 technical opinions would be especially welcome, but will go in a separate section. Send your comments directly to jmh_assoc@hotmail.com or (preferred) jmhart62@gmail.com.


Introduction

The Win32 API is, in general, fairly consistent, logical, and easy to learn. However, there are times when I wondered whether someone was asleep during the design reviews. I say this with all due respect for the Win32 designers and for the tremendous pressure under which they must have operated. Furthermore, as the length of this book's errata sheet proves (for all editions), perfection is difficult to achieve. Nonetheless, after you use the API for a while, you start to see things that could be more logical, consistent, and easier to use.

Now that the API is established, however, programmers will have to deal with it for many years to come, and is seems fair to make a few critical comments. These comments and observations might help programmers avoid some simple mistakes and assumptions. Some comments are rather minor, even petty, but some are more serious. Many of the comments are merely a matter of aesthetics and taste. Some can be fixed with new functions, but in other cases we are stuck with the design.

Next to each comment, I've put a personal "seriousness grade" of A (most serious) to C (least serious). The comments are grouped into several sections.

March, 2010. These comments still seem appropriate. I've eliminated those that concern Win 9x and other obsolete Windows versions.

Handles- A

BACK TO TOP

Win32 is to be praised for using the HANDLE data type to refer to nearly anything. In general, we call CreateWidget to get a HANDLE for a hypothetical "widget" object. After calls to ManipulateWidgetThisWay and ManipulateWidgetThatWay (always using the HANDLE), we finally call CloseHandle. We can secure these objects at the time of the Create call, change the security, and duplicate the handles. There are a few exceptions to this otherwise commendable consistency, however, and the exceptions are often inexplicable. In some cases, one might justify the inconsistencies by noting that the objects are not sharable across processes.

  1. Legacy Handles. DLL handles (see LoadLibrary and GetProcAddress) are an example of "legacy handles," left over from Win16, that are treated differently. HWND handles, are, of course, totally different beasts.
  2. Heap Handles. Heaps are created with HeapCreate (rather than CreateHeap, which would be more consistent) and closed with HeapDestroy.
  3. File Search Handles. FindFirstFile returns a handle, which is then closed with FindClose rather than CloseHandle.
  4. Lock Handles. LockFile and LockFileEx would be more convenient to program, in my opinion, if they returned handles. As it is, you are required to retain the lock range.
  5. Critical Sections. There is CRITICAL_SECTION data type, but there is never a HANDLE for them. And, since when is "delete" (as in DeleteCriticalSection) the opposite of "initialize" (??!!) (as in InitializeCriticalSection). Similar comments apply to the slim reader-writer locks and condition variables introduced with Vista (NT 6.0 kernel and later).
  6. The Registry. Here, we have HKEY throughout.

"Invalid Handle Values"

BACK TO TOP

A Create call returns a handle to a specified object, normally as a function value, but, as in the case of CreateProcess and DuplicateHandle, the returned handle can be in a parameter or structure. At any rate, it would be convenient if either NULL or INVALID_HANDLE_VALUE were used consistently in all cases. This could almost be rated an "A severity" problem as it is all too easy to test for the wrong value, thus letting errors slip by.

JDH reports that GetCurrentProcess(), should it fail, will return INVALID_HANDLE_VALUE. Note that the documentation does not say which value you should get in the case of failure. This reader, after the call, then called DuplicateHandle on the GetCurrentProcess result, obtaining INVALID_HANDLE_VALUE under W95 (now long obsolete) and a valid handle under NT.

Sept, 2004. Peter Bright (drpizza@thorin.battleaxe.net) pointed out that this behavior is documented, and it's in a fairly obvious place. See the CloseHandle MSN entry:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sysinfo/base/closehandle.asp 

"This function will throw an exception if either a handle value that is not valid (or 0) or pseudo-handle value is specified as the value of hObject and the application is running under a debugger. This includes closing a handle twice, and using CloseHandle on a handle returned by the FindFirstFile function. Note that this function will not throw an exception when a handle value that is not valid or pseudo-handle value is specified and the application is not running under a debugger."

OK, that explains it. I'm not quite sure why it was implemented this way. Perhaps someone can come up with a good explanation.

Peter adds some excellent points, and I'm largely in agreement:

"The exception behavior is useful, by the way. Closing an invalid handle is generally indicative of a bug; it's much like a double free/double delete, really. It's also readily detectable by the OS. But equally, it's also quite innocuous (*un*like a double-free/double delete, which stands a good chance of corrupting something). So throwing an exception in general would probably be quite destructive. Throwing an exception only when being debugged seems to me a quite reasonable compromise. 

"Finding a handle's type is (if memory serves) feasible if you're willing to monkey around with the native interfaces. That said, I'm not sure it's entirely useful; the inability to do so doesn't seem a huge omission. Likewise its reference count; it's not really an application's job to care about such details. 

"Async I/O is a horrible model to work with, but is supposed to provide better performance characteristics than threading in some circumstances. I/O completion ports seem to offer a better model, though, and better performance still, and are useful for generalized thread pools too. 

"CreateRemoteThread is perfectly safe and useful; it's safe because processes are securable objects, so you can only CreateRemoteThread into processes that you own or have otherwise been granted access to (e.g. through use of SeDebugPrivilege). 

"pthread_cancel on a PTHREAD_CANCEL_DEFERRED thread is essentially identical to an appropriate APC; the APC is more general in some senses (it can do more things, not just cancel a thread), pthread_cancel can, however, cancel at a few more API calls (though the only guaranteed cancellation points are quite similar to the Win32 ones, once Win32's polymorphism is taken into account). 

"pthread_cancel on a PTHREAD_CANCEL_ASYNCHRONOUS thread is essentially identical to TerminateThread, with all the problems that TerminateThread has. pthreads have cancellation handlers, but handling an async cancel properly is nigh impossible anyway, so it's not really winning a great deal. 

"Win32 I find immensely frustrating. I like certain design decisions (the polymorphic approach to handles, for example) but the lack of orthogonality and consistency in the API is a real problem. The internal interfaces are in some ways a great deal better--they're at least quite consistently designed (for example, all functions return a status code (rather than Win32, where some return errors, others return duff values and expect you to call GetLastError), all functions take their "return" value as a by-pointer argument). But Win32 is far too heavily influenced by Win16. 

"Sometimes this is merely inconvenient; other times it's a downright pain in the backside. For example, window procedures should have an arbitrary void* context parameter for user-defined data. This way you could very easily stash away a pointer to a class, and implement window procedures as class members. Other callbacks do this, on many platforms, it's a well-established and effective mechanism. But wndprocs don't do it. Stashing away pointers attached to windows *is* possible, but it's tortuous. It would have been simple to do things properly, but they didn't."

Ex Functions - B

BACK TO TOP

I can imagine the Win32 API designers having dialogues along the following lines:

Speaker A: "The DoThisToThat function can't perform a 'long this' operation. There is no parameter to specify the operation's range."

Speaker B: "No problem. Just specify a new DoThisToThatEx function and put the range in an additional parameter.

Speaker A: "Sounds good. The specification will be complete this afternoon. And, I'll be certain that the additional parameters are used in a totally new way that no one has ever seen before."

Speaker B: "Cool."

This scenario must have been enacted with ReadFile, WriteFile, Sleep, WaitForMultipleObjects, and WaitForSingleObject to give us "Extended I/O" for files which have the FILE_FLAG_OVERLAPPED flag set. LockFile and UnlockFile were similarly enhanced. The registry functions then received the full treatment to the point where you have to be very careful to remember when the "Ex" suffix is and is not required. It's worth mentioning, however, that many of the Ex functions are welcome additions, such as SetFilePointerEx.

Will there be a DoThisToThatExEx function in the future?

How Many Bytes (or Characters) Do You Need? - A

BACK TO TOP

Numerous functions return values whose size cannot be known prior to the call. Such values may be strings, SIDs, Security Descriptors, and so on. There are several ways in which the size can be returned to the calling program, including: 1) the function value, 2) a function "call by address" parameter where the actual buffer size provided by the user is a separate parameter value, 3) a function "call by address" parameter where the current size provided by the calling program is modified by the function, or 4) a separate structure. Win32 seems to mix these styles more or less randomly. By the way, you also need to be careful as to whether the returned value represents a number of bytes or number of characters. Using these functions incorrectly can result in buffer overruns, which are a serious security concern (see Chapter 5 of Howard and LeBlanc's "Writing Secure Code").

While this may be only an annoyance to some, I have rated it an "A" because of the problem's pervasiveness and the resulting lack of coherence. The security functions are the most seriously impacted.

Here are some examples:

Function Value

GetCurrentDirectory is a "pure" function of the "how many bytes do you need" variety.

Call by Address

LookupAccountName has two separate call by address parameters, one for the SID size and one for the domain name. LookupAccountSid is similar in this way. GetUserName has a single such parameter.

Mixed Call by Address and Call by Value

GetFileSecurity has two parameters for the security descriptor size.

Mixed Call by Address, Call by Value, and Function Value

SetFilePointer is my favorite, even though it relates to file pointers and not the size of returned objects. This function has something for everybody. The high-order file pointer value is set in a call by address parameter, and the function modifies the parameter with the resulting high-order pointer. The low-order pointer is a value, and the resulting pointer is returned as a function value. To top it all off, the return value is overloaded with the error indicator. GetFileSize shows some of the same strangeness. One could argue that the functions were designed to be simple when dealing with "short" files and difficult when dealing with "long" files. Will we see a new class of "Ex" functions in the future? 

Sept. 9, 2000. Yes! It happened just as predicted! Windows 2000 provides GetFileSizeEx and SetFilePointerEx. These functions, which use LARGE_INTEGER data types, are improvements over their predecessors. There is also a GetDiskFreeSpaceEx. July, 2005. Huge files have been, for some time, quite common, so these two extended functions are greatly appreciated.

Separate Structure

GetAclInformation specifies an ACL_INFORMATION_CLASS structure that has a member for the number of bytes already used by the Acl and one for the number of remaining bytes in the programmer-specified buffer.

Annoying Inconsistencies

GetTempPath and GetCurrentDirectory specify the buffer length first and the buffer second. Other functions, such as GetUserName, arrange the parameters in the opposite order.

64-Bit File Sizes and Pointers - B

BACK TO TOP
  1. See the comments above regarding SetFilePointer and GetFileSize. There really will need to be better versions of these two functions as large files become more common. Let's hope that they are not called SetFilePointerEx and GetFileSizeEx. Sept. 9, 2000: Sorry, Windows 2000 provides exactly these functions. March, 2010. OK, I'm over it. I like these new functions and use them nearly all the time.
  2. LockFileEx, CreateFileMapping, and MapViewOfFile each have two value parameters.
  3. Why not just use a single parameter (a LONGLONG, PULONGLONG, or PLONGLONG) with functions requiring file pointers.

Missing Functionality - A, B, and C

BACK TO TOP

Multiple Wait Semaphores - A

It really would be convenient to have the ability to wait atomically for multiple semaphore units. You need this functionality at times. Take a look at Multiple Wait Semaphore for my solution to this problem. A standard, Microsoft-supported Win32 functions set would be very desirable. It appears that the designers did not want to destroy the coherent architecture that applies to events, semaphores, and mutexes.

Sept. 9, 2000: I've come to feel that semaphores are rarely useful (there are exceptions). Queues such as those implemented in Edition 2's Chapter 10 are not only more general but much more useful. Furthermore, semaphores can be constructed from an event, mutex, and counter.

Thread Synchronization with Events - A

July, 2005. I've discussed the Windows events in detail in the book and elsewhere on this site. To re-iterate, however, the auto/manual reset events with Pulse/Set/Reset event functions are an invitation to bugs. SignalObjectAndWait() was a nice addition which somewhat alleviated the problem, unless you need to run on 9x (Windows 95, 98, was a nice addition which somewhat alleviated the problem, and Vista condition variables are even better. 

Interprocess Signaling - B share the same console using GenerateConsoleControlEvent. It really would be convenient to be able to send signals between unrelated processes, as the console-sharing requirement is far too restrictive. As it stands, cooperating processes can have threads listening on a mailslot or named pipe to shut down a process, but this is not the most convenient method.

Sept. 9, 2000: Windows 2000 provides QueueUserAPC, which allows one thread to specify a function to be executed by another "target" thread. The asynchronous procedure call (APC) will be made when the target thread is an an alertable wait state (Ed2: Ch 14, Ed1: Ch 13). The procedure can raise an exception in the target thread. I'm in the process of writing a paper to illustrate this idea and to compare the resulting capability with that offered by the pthread_cancel function in Pthreads.  

HANDLE Types - B

There appears to be no way to determine a HANDLE's type. That is, does a specific handle refer to a thread, process, file, etc? The GetHandleInformation function does not provide this information.

HANDLE Reference Count - C

There appears to be no method to determine how many handles are open on a given object. Additional information could distinguish between references in other processes and in the current process.

File Length by Name - C

There is no convenient way to get the file length by name, rather than handle. I've used GetCompressedFileSize to find out whether a file is empty, but in other cases I'd either have to get a file handle using CreateFile, call GetFileSize, and then close the handle. FindFirstFile is another alternative.

fork() - C

There is no equivalent to the UNIX lone fork() function, and it is not at all easy to emulate. This has not bothered me, as CreateProcess has met my needs (hence the C- rating), and threads can be used to get much the same effect as fork(). However, it is extremely difficult to obtain an accurate emulation of the UNIX semantics, and there may be some situations that really do require the lone fork(). Any opinions out there among you readers?

JDH is happy about this situation, reporting that, under UNIX, if you fork() when many threads are running and one or more hold locks, you get an incredible mess. NT does not have this problem.

Sept. 9, 2000. fork() is tricky in a multithreaded environment, and is definitely not recommended in UNIX if you are using Pthreads, so there really is no loss here. In fact, the functionality really is not well defined. For example, if a thread in a process owns a named mutex. how can the child process be an exact duplicate? What is the state of a process that has threads running on several processors in an SMP system?

Pointers - The Long and Short of it - B

BACK TO TOP

Win32 gives us a 32-bit operating system, which is good. There is no need at all for "short" 16-bit pointers or any of the Win16 memory models that grew out of the old Intel memory architecture. So why do we sometimes have LP (as in LPVOID, LPCTSTR, and LPSECURITY_ATTRIBUTES)? And why is the L sometimes omitted (my preference, as in PHANDLE and PSECURITY_DESCRIPTOR). Why not just get rid of LP once and for all and give us a consistent L for everything? Get rid of the legacy stuff and create an environment for programmers who don't care about the old 16-bit world. Do the same for the lp (lower case) that appears with so many parameter names.

Incidentally, let's hope that we don't enter a world of "long long pointers" (LLPVOID, perhaps?) as Win32 evolves to Win64.

Sept. 9, 2000. The Win64 solution is pretty good (see Chapter 16) and Edition 4 uses it.

How Many Ways Can You Say "Forever" - B (name of a pop song?).

It is a game you can play when you look at the various functions that may wait and require a timeout period. Here are three examples; you may also want to play the same game with no time out (0would be good, but what about NMPWAIT_NOWAIT?)

  1. INFINITY. This would be my preferred name, and we can use it with WaitForSingleObject and WaitForMultipleObjects.
  2. NMPWAIT_WAIT_FOREVER is used with WaitNamedPipe and CallNamedPipe.
  3. MAILSLOT_WAIT_FOREVER is used with CreateMailSlot.

Program Safety - B (Perhaps an A)

BACK TO TOP

As programmers, we are, quite properly, admonished to avoid any and all situations that could "crash" a program. Thus, for example, we should be certain that we do not de-reference a NULL pointer. However, many functions that take handles as arguments will cause an exception if the handle variable has not been initialized. Occasionally, you can get an exception if you close the same handle twice in succession. (This is, of course, a foolish thing to do, but Win32 should check the parameters and not generate an exception.)

JDH reports: "The exception that is obtained by closing the same handle twice is C0000008 and has the symbol EXCEPTION_INVALID_HANDLE, which can be found nowhere in the documentation (that I know of); it lives in the Win32 headers, of course. I have noted that I get this only when running in the debugger on WinNT, and I can't figure out why this should be so."

File Links - A

BACK TO TOP

File links of some sort are extremely useful, even necessary. The NTFS actually supports links, but Win32 does not take advantage of this capability. Lack of links is a common headache in UNIX to NT ports. Sept, 2004. We now have a hard link (CreateHardLink), which is good. A soft link would be nice, IMHO. 2010: It's here..

Commands and Utilities - B

BACK TO TOP

Fortunately, UNIX-style commands, utilities, and shells can be obtained inexpensively (or even for free: see www.cygwin.com), and perhaps it is just as well that they are not bundled with Windows. These programs really are indispensable. November, 2010. The Power Shell is pretty nice, and I'm just learning to use it.

How Do I Release the Memory? - C

Several function calls, such as CreateFileMapping and MapViewOfFile, require you to specify the two pointer components with the high-order component first.

Remove or Delete? - C

BACK TO TOP

This is one of those little inconsistencies, but it's worth mentioning at least one. I call DeleteFile when I want to delete a file, but I call RemoveDirectory to delete a directory. Why not have, instead, a DeleteDirectory or RemoveFile?

32-bit, 64-bit Addressing, and Windows NT Version 5.0 - Obsolete Comment as of Edition 4

BACK TO TOP

Large scale applications really will need a large virtual address space in the very near future, if not now. Many UNIX implementations already support true 64-bit memory architectures.

Windows 2000 takes some very small, but not very satisfactory, steps in the direction of supporting larger address spaces. Among other things, it will support "Very Large Memory" (VLM) so that a system can have large amounts of physical memory to divide up among processes. Process address spaces can also be larger (up to 8GB, I believe), but this extra address space is not swappable; it appears as if it must be hard-allocated to a process. A large database vendor is already promising to exploit this feature to achieve higher performance.

As I obtain and digest additional information, I'll add a section on NT 5.0. At this time, is looks as if we are going to be revisiting new forms of "extended" and "expanded" memory (remember them?). A true large virtual address space is the only solution in the long run. I'd go so far as to say the NT 5.0 is extremely disappointing in this respect (it does have some good features, however, especially the "zero administration" functionality).

Sept. 9, 2000: Win64 and Windows 2000 do the job right, it appears (See Ed 3 and Ed. 2, Chapter 16). The concerns above were not really justified. Note that NT Version 5.0 became Windows 2000 (Kernel versions are: W2000 - 5.0, XP - 5.1, and W2003 - 5.2)

Things I Could Do Without - C

BACK TO TOP

This is only a personal opinion, but I feel that the functionality of any system should be both complete and reasonably orthogonal (that is, there should not be several ways to?

Things I Could Do Without - C

BACK TO TOP

This is only a personal opinion, but I feel that the functionality of any system should be both complete and reasonably orthogonal (that is, there should not be several ways to do the same thing for no good reason). For example, the synchronization functions (see the section below) generally meet these criteria.

It is also desirable not to have capabilities that are not really useful to anyone; there is more to learn, more to maintain, and more ways to make errors. Here are my candidates for functionality that I could do without (feel free to disagree). IO and asynchronous IO. Suppose you have two threads sharing the same handle to a device. Each of these threads then make an IOCTL call (not necessarily the same type of call) to the device (via the handle, of course). This call is synchronous, because they are blocked in the call and cannot do anything else. However, the calls are overlapped, 'cause there's obviously more than one operation
going on at a time. They are "overlapping" each other. Asynchronous IO is where a single thread makes an IOCTL call (or any other, quite frankly) to a device (via it's handle) and is not too concerned about when this operations completes, or is in a hurry to do something else (including making more IOCTL calls to the same device, using the same handle). After a while, the thread then checks to see if one (or more) of these operations have completed and responds accordingly. At least, this is the distinction made by the comp.os.ms-windows.programmer.nt.kernel-mode newsgroup (if memory serves me). I, myself, have encountered this distinction (I merely wanted overlapped and discovered that I had to make it asynchronous) with my code and learned this off of the newsgroup."

_beginthreadex

It would be nice if the thread-safe standard C library were written is such a way that it could initialize its own thread-specific storage. As a result, we could just use CreateThread rather than the rather ugly _beginthreadex. July, 2005. Ignore the following Sept 2004 statement. The problem was due to a stupid programming error.

ID Kong <Id@inscriber.com> sent me the following on 11/8/2000 (used with permission): -selected address). Watch the process crash or otherwise misbehave.

Conclusion: TerminateProcess is bad enough, but CreateRemoteThread is a real security and robustness threat. Of course, processes can protect themselves by denying PROCESS_CREATE_THREAD rights, but this takes some effort and is not the default. Sooner or later, if Windows NT is to be truly secure in the enterprise, this issue will need to be addressed.

Warning: I really recommend that you DO NOT carry out the above experiment other than as a thought experiment. The resulting thread could do nearly anything; it might, for example, delete a few files that you are fond of.

JDH suggests, for additional entertainment, that you pass the address of ExitProcess as the thread function, even supplying a reasonable termination code. Notice, of course, that this is the address of ExitProcess in the target process, not the one executing.the CreateRemoteThread call.

HEAP_NO_SYNCHRONIZE

This is a preliminary judgement, subject to revision after I perform some additional tests. I have a colleague, whom I admire, who thinks that this flag (used with HeapAlloc and friends) is terrific. I have found that it has a minimal positive performance impact, at least in simple cases, and using this flag creates certain dangers. There may be situations where the flag is helpful, but there must be other ways to improve performance that are safer. Furthermore, no other functions have such a flag.

UNIX Pseudoterminal Emulation Frustrations - A

BACK TO TOP

October 24, 2000: Peter Eliades <peliades@sanderson.net.au> of Sanderson Computers has made the following very interesting point, which I had not given much thought to. Peter's message is as follows:

"I agree with your appraisal of the Win32 API as well as that that of Diomidis Spinellis in his "A Critique of the Windows Programming Interface" (see http://softlab.icsd.aegean.gr/~dspin/pubs/jrnl/1997-CSI-WinApi/html/win.html (first published in Computer Standards & Interfaces, 20:1-8, November 1998)." JH Comment: This is worth reading. Spinellis makes many of the same points that I have, and more, but he is much more forceful and considers more than just the system services.

"An additional criticism that I would like to add is the absence of any means of reliably emulating Unix pseudoterminal (pty) functionality. The options for programmatically controlling the I/O of a console application under Win32 are:

  1. anonymous pipes, and
  2. console screen buffer polling

"Unfortunately both methods are deficient. Some applications, namely those using the "low-level" console API functions to perform screen I/O simply cannot be redirected using unnamed pipes. Polling the screen console buffer, is expensive and unreliable. There is no notification mechanism to allow a process to know when the buffer has changed. This forces the use of a crude loop that performs a "snapshot/compare snapshot to previous snapshot" process. This will necessarily result in missed screen output; the polling thread will not always be running to catch the redirected process that may be producing screen output profusely. Further, the console screen buffer is a matrix rather than a queue, earlier screen data will be overwritten.

"I have yet to devise a system that somehow uses both methods (if this is at all possible, I don't know).

"The console API tends to be important to those companies that have character based (typically Unix) applications that they wish to migrate to NT/Win32.

"I have investigated some commercial remote shell and telnet services on NT to find that they rely on the console screen buffer polling method to emulate ptys. If you have a telnet client with a back paging capability, you will find that on a long listing; e.g., the output of an 'ls' that extends beyond one screen, the client will only have received the last screen of the listing."

Peter goes on to say, "The emulation of psuedo-terminal like I/O functionality is a recurring theme on comp.os.ms-windows.programmer.win32 and microsoft.public.win32.programmer.kernel in one form or another: redirecting stdin and stderr/stdout, capturing screen output, controlling another process etc. The answers vary in quality but the thread named "Capture DOS Screen Tex (first published in Computer Standards & Interfaces, 20:1-8, November 1998)." JH Comment: This is worth reading. Spinellis makes many of the same points that I have, and more, but he is much more forceful and considers more than just the system services. 2010: Sadly, this link is no longer available.

"An additional criticism that I would like to add is the absence of any means of reliably emulating Unix pseudoterminal (pty) functionality. The options for programmatically controlling the I/O of a console application under Win32 are: output or control the input of a textual application.
 
"On Windows NT, the case of DOS applications is actually a subset of the more general case of "console mode" applications, because DOS applications run as coroutines within a Win32 process (NTVDM) that translates their I/O to Win32 equivalents. 
 
"There are two classes of console mode applications.  The important difference between the two is whether they read from and write to their standard input and standard output in "glass TTY" fashion using ReadFile() and WriteFile() (what Win32 terms "high-level console I/O"), or whether they use "random access" APIs such as WriteConsoleOutput() (what Win32 terms "low-level console I/O").   Translating this to DOS terms: DOS programs that use INT 21h to read from and write to their standard input and standard output are in the former class; and DOS programs that use INT 10h or that write directly to video memory are in the latter class.
 
"To capture the output and control the input of programs that use "low-level console I/O", one sits in a loop whilst the child process is executing, continuously monitoring the contents of the console screen buffer using ReadConsoleOutput() and sending any keystrokes using WriteConsoleInput()
 
"There are several problems with this design.  One minor problem is that it doesn't cope at all well with Win32 programs that take full advantage of the features of the Win32 low-level console paradigm and use
alternate console screen buffers for their output.  A more major problem is that because it uses polling (Win32 not providing any handy event mechanism to latch on to so that the monitor could know when the buffer has been modified) it is always going to be both unreliable and expensive.  It is unreliable in that depending from the relative scheduling priorities of the various processes, something which is going to vary dynamically with system load, it may well be the case that the child program may be able to generate reams of output that the monitoring process will simply miss because its console monitoring thread won't be scheduled to run often enough.  It is expensive in that if the child process happens not to generate any output for a while, the monitoring process is going to consume CPU time needlessly.
 
"To capture the output and control the input of programs that use "high-level console I/O", one redirects their standard input and standard output through pipes, and reads from and writes to the other end of the pipes in the monitoring process. 
 
"The advantage of this method is that one doesn't need to worry about missing any output, since this approach doesn't use polling.  But, conversely, this method has a problem of its own, in that it won't capture any output generated by "low-level console I/O", and programs that use "low-level console I/O" for input will bypass the redirection entirely.  Alas, all too many DOS programs fall into this category.
 
"It is difficult to combine the two mechanisms into one for capturing output, and practically impossible to combine them for controlling input.  So really one must know ahead of time the type of textual application that is going to be run, /i.e./ whether it is going to be using "high-level" or "low-level" console I/O, and select the appropriate mechanism accordingly.
 
"The only perfect solution that would cope with both sorts of programs simultaneously would be for Win32 to provide functionality akin to what in the UNIX world is known as a "pseudo-TTY".  Win32 would need to provide some means of for a monitoring process to hook into the "back" of a console instead of the "front" as seen in normal use.  The monitor process would write data to the back of the console and the console would turn those data into keystrokes that applications reading from the front of the console, by either means, would see as input.  All output written to the console, by either means, would be translated into a single encoded bytestream that the monitor process could then read from the back of the console, in sequence, without needing to poll, and without missing any data. 
 
"Alas, Windows has no such mechanism."

Threads and Synchronization - A   Revised Sept. 9 and Nov. 10, 2000

BACK TO TOP

I'm now much less fond of Win32 threads and synchronization than I was initially. The biggest problem (in my opinion) is that events are far too complex and error prone, and there really should be some sort of condition variables (We now have them as described in Edition 4). Disclaimer: Earlier versions of this page (before Sept. 9, 2000) were more complimentary toward Win32 threads and synchronization. I once felt that the thread management and synchronization functions were well-designed and implemented; I changed my mind after using Pthreads on some UNIX and Linux systems.

NOTE: For more on this entire subject, see: http://world.std.com/~jmhart/opensource.htm.

The objections to Win32 threads and synchronization can be summed up as follows:

  1. Thread management is made more complex than necessary by the requirement for _beginthreadex() and the confusing _beginthread() when using the C library. This comment also applies to _endthreadex() and _endthread(). Severity B.
  2. There is no direct thread cancellation. This can make if difficult to terminate a threaded system in an orderly way. Severity A.
  3. Semaphores are redundant and not completely implemented. Severity B. 
  4. Events are poorly designed and the cause of all sorts of synchronization bugs. Severity A.

Even so, Win32 threads are very easy (even fun) to use, generally working as expected. A common opinion is that threads and synchronization are difficult subjects and the programming is complex. I disagree; the functions (other than events) all make sense, even though the functionality is incomplete (I would still like to see an atomic multiple semaphore wait function, for instance, and thread cancellation would be nice). Semaphores, mutexes, and events all have a role to play, even though they are not totally independent. (For example, it is possible to create a semaphore out of a mutex and an event. See the example.) The biggest criticism  is that the event semantics are too complex and are also incomplete. 

On the positive side, threads and synchronization are a standard part of Win32 and have been there from the beginning. By way of contrast, threads arrived very late in UNIX, and it took a long time for them to become standardized. Many early implementations were not really kernel threads but were more like fibers. Until I really used it, I did not appreciate POSIX Pthreads, and while I still don't like the Pthreads naming, I do like the architecture and implementation. In particular, I now appreciate why mutexes are combined with condition variables; the Pthreads design is correct. See Edition 2, Chapter 10 of Win32 System Programming for a discussion of the Win32 "condition variable model.".

Extensive comments, Sept. 9, 2000: 

I've had the opportunity to use Pthreads extensively and now feel that Win32 events, with the four set/pulse, auto/manual-reset combinations, are confusing and a source of all sorts of bugs (in my own programs and others that I've seen). Edition 2, Chapter 10, describes what I've called the "condition variable model" as it is modeled on Pthreads condition variables.

As an example of the confusion, the latest edition of a best selling book on Win32 system programming (you'll have to figure out which one) states in the preface that a major improvement of the new edition is its treatment of synchronization. The author, who is extremely knowledgeable and has written extensively on Windows programming, then goes on to state that he has never seen a good use of the PulseEvent function. The new edition did remove an example from earlier editions in which the multiple wait semaphore problem was "solved" using successive waits on the same semaphore (a deadlock-prone solution). Well, if you want to solve this problem, you really do need to use PulseEvent to get an efficient solution. The point of all this is that even very experienced writers and programmers do not use Win32 events properly. and attempts to correct old errors often result in new errors. In my own case, I posted a multiple wait semaphore "solution" that erroneously used the signal, rather than the broadcast, condition variable model; this has now been fixed. 

As an additonal example, in a book dedicated to Win32 multithreaded programming, the authors state that PulseEvent and SetEvent have the same effect for an auto-reset event, both releasing one thread. While the distinctions are subtle, they are important. There is definitely a lot of confusion out there about events, and there must be a lot of latent defects in threaded code. Multithreaded application development is difficult enough without the added burden of these confusions.

Finally, a general implementation of a condition variable that is to be used in broadcast mode requires a timeout on the event wait in order to avoid missed signals. In a sense, all you are really doing is polling the condition variable predicate, with some assistance from the event. Admittedly, SignalObjectAndWait addresses this issue, but the solution is not portable to all Win32 platforms.

Bottom Line: Win32 would have done well to have used the condition variable concept rather than events. See http://www.cs.wustl.edu/~schmidt/win32-cv-1.html for more insight into this important issue.

There is a LINUX Open Source POSIX Threads for Win32 project underway, and a snapshot was posted on September 8, 2000. See http://sources.redhat.com/pthreads-win32 for more information. I haven't looked at this work yet, but will soon. As a "proof" of Win32 event clumsiness, consider the following:

  • Pthreads condition variables are complete, natural, and easy to use, especially if you use them with the correct programming idiom, as in my Chapter 10.
  • A complete condition variable Win32 emulation is quite complex; see he open source implementation. It should not be so complex to do something so essential.

Other comments: Pthreads provides for thread cancellation, which is a big improvement over TerminateThread. However, you can achieve nearly the same effect in Win32 with asynchronous procedure calls (see QueueUserAPC); this is the topic of a future note. Nonetheless, thread cancellation should be provided more directly.

Really A Final Note - Win32 Program Appearance

BACK TO TOP

Sept. 9, 2000. This may just be a personal opinion and a matter of taste, but, I find that Win32 programs look sort of clunky, heavy, and graceless when compared to their UNIX equivalents. This impression is due mostly to the extensive use of upper case and long function names. Does it matter? Perhaps. I'd argue that compact, graceful code is just better and is easier to understand and a lot more satisfying to write.

Diomidis Spinellis dspin@aegean.gr sent me the following comment on 10/25/2000: "Regarding your note on Win32 program appearance, I think that one reason for Win32 programs looking clunky is the non-existent to bad formatting style found in Microsoft's example code. I am under the impression that those programs are written at Microsoft by summer interns and never reviewed. Programmers that want to use a Win32 feature will invariable stumble upon such example code (eg through MSDN) and get infected by bad programming style. In contrast, most Unix code that one gets exposed to (the kernel, utilities, Perl, GNU) is very well-written by experts who know how to program in the large and in the small. Sadly, some more recent applications and code (especially those coming from the Linux world) are exceptions to this rule, but generally the vast majority of free Unix code is exemplar. When programming Win32 I try to use the Berkeley formatting and coding style; for an example you can have a look at the outwit tool suite <http://softlab.icsd.aegean.gr/~dspin/sw/outwit> (please be kind, I did not write it as an example for programming style)." (Used with permission.)

Sept. 8, 2014. An Amazon reviewer remarks:

"However, the examples belie the fact that doing what should be the most straightforward things on Windows requires a tremendous amount of effort and attention to picayune details, whereas, on UNIX (but of course), 'twould be next to trivial. That's no surprise: I mean, you can make a UNIX system sing and dance with Perl programs and shell scripts and such--there isn't necessarily a need to code in C unless you require the most detailed of semantics, such as utterly non-standard TCP or UDP or RPC disciplines--whereas the "comparable" glue language on Windows is, I guess, WSH that still clings to 1960s-style syntax hailing from Kemeny-and-Kurtz BASIC (e.g., you use DIM--short for DIMENSION [!]--to define a variable)."

I do have comments in response:

  1. Perl is available on Windows, and you can use shell scripts (PowerShell is nice) of all sorts. No problem.
  2. By now it should be clear that I feel that there are some valid criticisms of the Windows API, and in many cases *IX is marginally easier to use (The converse is also sometimes true).
  3. Sometimes you need to use the Windows API, or, at least, it's the most convenient option. For example, you may need to port an existing *IX program (there are lots of them!). Incidentally, the *IX program will use C (or maybe C++) and will deal with an equivalent set of detailed semantics and picayune details.
footer