Handles |Invalid Handle
Values | Ex Functions |
How Many Bytes | 64-Bits and Pointers |
Missing Functionality | Pointers |
Timeout/Infinity | Program Safety |
File Links | Commands/Utilities |
Releasing Memory | Little/Big Endian |
Remove or Delete | 64-Bits and Addressing
| Excess Functionality |
UNIX Pseudoterminal Emulation |
Threads/Synchronization | Program Appearance
This section contains a variety of comments criticizing different aspects of the
Win32 design and functionality. These comments are my own or, in some cases,
have come from readers and course participants. Disclaimer: I generally like
Win32. The comments here should be taken in that spirit. After all, I took the
time to write four book editions and have spent a great deal of time consulting
and lecturing on Windows subjects. What is more, Windows is commercially
successful and has been extremely stable since its first appearance in 1993.
Nonetheless, it is important to be skeptical and critical in the hopes that,
perhaps, Microsoft will improve a few things in the future. In fact, they have
made numerous improvements over the years, and some of these comments will may
appear dated, but I've left them in to give a sense of history and so that we
can see how much has changed.
Updates September, 2014, March, 2010, July, 2005, and September, 2004.
Some opinions have changed since this was first written in 1998 and are marked
as such. Likewise, some critical comments are marked as obsolete, such as
anything dealing with W95. Since I first wrote this, 64-bit issues and condition
variables have been dealt with nicely.
Send your Opinions
Dissenting or additional opinions on any of these subjects are welcome, and I'd
be happy to post them here (several are included in this version). UNIX (or Linux, or … ) vs. Win32 technical opinions
would be especially welcome, but will go in a separate section. Send your
comments directly to
jmh_assoc@hotmail.com or (preferred) jmhart62@gmail.com.
Introduction
The Win32 API is, in general, fairly consistent, logical, and easy to learn.
However, there are times when I wondered whether someone was asleep during the
design reviews. I say this with all due respect for the Win32 designers and for
the tremendous pressure under which they must have operated. Furthermore, as the
length of this book's errata sheet proves (for all editions), perfection is
difficult to achieve. Nonetheless, after you use the API for a while, you start
to see things that could be more logical, consistent, and easier to use.
Now that the API is established, however, programmers will have to deal with it
for many years to come, and is seems fair to make a few critical comments. These
comments and observations might help programmers avoid some simple mistakes and
assumptions. Some comments are rather minor, even petty, but some are more
serious. Many of the comments are merely a matter of aesthetics and taste. Some
can be fixed with new functions, but in other cases we are stuck with the
design.
Next to each comment, I've put a personal "seriousness grade" of A (most serious)
to C (least serious). The comments are grouped into several sections.
March, 2010. These comments still seem appropriate. I've
eliminated those that concern Win 9x and other obsolete Windows versions.
Handles- A
BACK TO TOP
Win32 is to be praised for using the HANDLE data
type to refer to nearly anything. In general, we call
CreateWidget to get a HANDLE for a
hypothetical "widget" object. After calls to
ManipulateWidgetThisWay and
ManipulateWidgetThatWay (always using the HANDLE),
we finally call CloseHandle. We can secure these
objects at the time of the Create call, change
the security, and duplicate the handles. There are a few exceptions to this
otherwise commendable consistency, however, and the exceptions are often
inexplicable. In some cases, one might justify the inconsistencies by noting
that the objects are not sharable across processes.
- Legacy Handles. DLL handles (see LoadLibrary
and GetProcAddress) are an example of "legacy
handles," left over from Win16, that are treated differently.
HWND handles, are, of course, totally different beasts.
- Heap Handles. Heaps are created with HeapCreate
(rather than CreateHeap, which would be more
consistent) and closed with HeapDestroy.
- File Search Handles. FindFirstFile
returns a handle, which is then closed with FindClose
rather than CloseHandle.
- Lock Handles. LockFile and
LockFileEx would be more convenient to program, in my opinion, if they
returned handles. As it is, you are required to retain the lock range.
- Critical Sections. There is CRITICAL_SECTION
data type, but there is never a HANDLE for them.
And, since when is "delete" (as in
DeleteCriticalSection) the opposite of "initialize" (??!!) (as in
InitializeCriticalSection). Similar comments apply to the slim
reader-writer locks and condition variables introduced with Vista (NT 6.0 kernel
and later).
- The Registry. Here, we have HKEY
throughout.
"Invalid Handle Values"
BACK TO TOP
A Create call returns a handle to a specified
object, normally as a function value, but, as in the case of
CreateProcess and DuplicateHandle, the
returned handle can be in a parameter or structure. At any rate, it would be
convenient if either NULL or
INVALID_HANDLE_VALUE were used consistently in all cases. This could
almost be rated an "A severity" problem as it is all too easy to test for the
wrong value, thus letting errors slip by.
JDH reports that GetCurrentProcess(), should it
fail, will return INVALID_HANDLE_VALUE. Note
that the documentation does not say which value you should get in the case of
failure. This reader, after the call, then called
DuplicateHandle on the GetCurrentProcess
result, obtaining INVALID_HANDLE_VALUE under W95
(now long obsolete)
and a valid handle under NT.
Sept, 2004. Peter Bright (drpizza@thorin.battleaxe.net) pointed out that this
behavior is documented, and it's in a fairly obvious place. See the
CloseHandle MSN entry:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sysinfo/base/closehandle.asp
"This function will throw an exception if either a handle value that is not valid
(or 0) or pseudo-handle value is specified as the value of hObject and the
application is running under a debugger. This includes closing a handle twice,
and using CloseHandle on a handle returned by the FindFirstFile function. Note
that this function will not throw an exception when a handle value that is not
valid or pseudo-handle value is specified and the application is not running
under a debugger."
OK, that explains it. I'm not quite sure why it was implemented this way. Perhaps
someone can come up with a good explanation.
Peter adds some excellent points, and I'm largely in agreement:
"The exception behavior is useful, by the way. Closing an invalid handle is
generally indicative of a bug; it's much like a double free/double delete,
really. It's also readily detectable by the OS. But equally, it's also quite
innocuous (*un*like a double-free/double delete, which stands a good chance of
corrupting something). So throwing an exception in general would probably be
quite destructive. Throwing an exception only when being debugged seems to me a
quite reasonable compromise.
"Finding a handle's type is (if memory serves) feasible if you're willing to
monkey around with the native interfaces. That said, I'm not sure it's entirely
useful; the inability to do so doesn't seem a huge omission. Likewise its
reference count; it's not really an application's job to care about such
details.
"Async I/O is a horrible model to work with, but is supposed to provide better
performance characteristics than threading in some circumstances. I/O completion
ports seem to offer a better model, though, and better performance still, and
are useful for generalized thread pools too.
"CreateRemoteThread is perfectly safe and useful; it's safe because processes are
securable objects, so you can only CreateRemoteThread into processes that you
own or have otherwise been granted access to (e.g. through use of
SeDebugPrivilege).
"pthread_cancel on a PTHREAD_CANCEL_DEFERRED thread is essentially identical to
an appropriate APC; the APC is more general in some senses (it can do more
things, not just cancel a thread), pthread_cancel can, however, cancel at a few
more API calls (though the only guaranteed cancellation points are quite similar
to the Win32 ones, once Win32's polymorphism is taken into account).
"pthread_cancel on a PTHREAD_CANCEL_ASYNCHRONOUS thread is essentially identical
to TerminateThread, with all the problems that TerminateThread has. pthreads
have cancellation handlers, but handling an async cancel properly is nigh
impossible anyway, so it's not really winning a great deal.
"Win32 I find immensely frustrating. I like certain design decisions (the
polymorphic approach to handles, for example) but the lack of orthogonality and
consistency in the API is a real problem. The internal interfaces are in some
ways a great deal better--they're at least quite consistently designed (for
example, all functions return a status code (rather than Win32, where some
return errors, others return duff values and expect you to call GetLastError),
all functions take their "return" value as a by-pointer argument). But Win32 is
far too heavily influenced by Win16.
"Sometimes this is merely inconvenient; other times it's a downright pain in the
backside. For example, window procedures should have an arbitrary void* context
parameter for user-defined data. This way you could very easily stash away a
pointer to a class, and implement window procedures as class members. Other
callbacks do this, on many platforms, it's a well-established and effective
mechanism. But wndprocs don't do it. Stashing away pointers attached to windows
*is* possible, but it's tortuous. It would have been simple to do things
properly, but they didn't."
Ex Functions - B
BACK TO TOP
I can imagine the Win32 API designers having dialogues along the following lines:
Speaker A: "The DoThisToThat function can't
perform a 'long this' operation. There is no parameter to specify the
operation's range."
Speaker B: "No problem. Just specify a new
DoThisToThatEx function and put the range in an additional parameter.
Speaker A: "Sounds good. The specification will be complete this afternoon. And,
I'll be certain that the additional parameters are used in a totally new way
that no one has ever seen before."
Speaker B: "Cool."
This scenario must have been enacted with ReadFile,
WriteFile, Sleep,
WaitForMultipleObjects, and WaitForSingleObject
to give us "Extended I/O" for files which have the
FILE_FLAG_OVERLAPPED flag set. LockFile
and UnlockFile were similarly enhanced. The
registry functions then received the full treatment to the point where you have
to be very careful to remember when the "Ex" suffix is and is not required. It's
worth mentioning, however, that many of the Ex functions are welcome additions,
such as SetFilePointerEx.
Will there be a DoThisToThatExEx function in the
future?
How Many Bytes (or Characters)
Do You Need? - A
BACK TO TOP
Numerous functions return values whose size cannot be known prior to the call.
Such values may be strings, SIDs, Security Descriptors, and so on. There are
several ways in which the size can be returned to the calling program,
including: 1) the function value, 2) a function "call by address" parameter
where the actual buffer size provided by the user is a separate parameter value,
3) a function "call by address" parameter where the current size provided by the
calling program is modified by the function, or 4) a separate structure. Win32
seems to mix these styles more or less randomly. By the way, you also need to be
careful as to whether the returned value represents a number of bytes or number
of characters. Using these functions incorrectly can result in buffer overruns,
which are a serious security concern (see Chapter 5 of Howard and LeBlanc's
"Writing Secure Code").
While this may be only an annoyance to some, I have rated it an "A" because of
the problem's pervasiveness and the resulting lack of coherence. The security
functions are the most seriously impacted.
Here are some examples:
Function Value
GetCurrentDirectory is a "pure" function of the
"how many bytes do you need" variety.
Call by Address
LookupAccountName has two separate call by
address parameters, one for the SID size and one for the domain name.
LookupAccountSid is similar in this way.
GetUserName has a single such parameter.
Mixed Call by Address and Call by Value
GetFileSecurity has two parameters for the
security descriptor size.
Mixed Call by Address, Call by Value, and Function Value
SetFilePointer is my favorite, even though it
relates to file pointers and not the size of returned objects. This function has
something for everybody. The high-order file pointer value is set in a call by
address parameter, and the function modifies the parameter with the resulting
high-order pointer. The low-order pointer is a value, and the resulting pointer
is returned as a function value. To top it all off, the return value is
overloaded with the error indicator. GetFileSize
shows some of the same strangeness. One could argue that the functions were
designed to be simple when dealing with "short" files and difficult when dealing
with "long" files. Will we see a new class of "Ex" functions in the future?
Sept. 9, 2000. Yes! It happened just as predicted! Windows
2000 provides GetFileSizeEx and
SetFilePointerEx. These functions, which use
LARGE_INTEGER data types, are improvements over their predecessors. There
is also a GetDiskFreeSpaceEx.
July, 2005. Huge files have been, for some time, quite common, so these
two extended functions are greatly appreciated.
Separate Structure
GetAclInformation specifies an
ACL_INFORMATION_CLASS structure that has a member for the number of bytes
already used by the Acl and one for the number of remaining bytes in the
programmer-specified buffer.
Annoying Inconsistencies
GetTempPath and
GetCurrentDirectory specify the buffer length first and the buffer
second. Other functions, such as GetUserName,
arrange the parameters in the opposite order.
64-Bit File Sizes and
Pointers - B
BACK TO TOP
- See the comments above regarding
SetFilePointer and GetFileSize. There
really will need to be better versions of these two functions as large files
become more common. Let's hope that they are not called
SetFilePointerEx and GetFileSizeEx. Sept.
9, 2000: Sorry, Windows 2000 provides exactly these functions. March, 2010.
OK, I'm over it. I like these new functions and use them
nearly all the time.
- LockFileEx, CreateFileMapping, and
MapViewOfFile each have two value parameters.
- Why not just use a single parameter (a LONGLONG,
PULONGLONG, or PLONGLONG) with functions
requiring file pointers.
Missing Functionality -
A, B, and C
BACK TO TOP
Multiple Wait Semaphores - A
It really would be convenient to have the ability to wait atomically for multiple
semaphore units. You need this functionality at times. Take a look at
Multiple Wait Semaphore for my solution to this problem. A standard,
Microsoft-supported Win32 functions set would be very desirable. It appears that
the designers did not want to destroy the coherent architecture that applies to
events, semaphores, and mutexes.
Sept. 9, 2000: I've come to feel that semaphores are rarely useful (there are
exceptions). Queues such as those implemented in Edition
2's Chapter 10 are not only more general but much more useful. Furthermore,
semaphores can be constructed from an event, mutex, and counter.
Thread Synchronization with Events - A
July, 2005. I've discussed the Windows events in detail in the book and elsewhere
on this site. To re-iterate, however, the auto/manual reset events with
Pulse/Set/Reset event functions are an invitation to bugs.
SignalObjectAndWait() was a nice addition which somewhat alleviated the
problem, unless you need to run on 9x (Windows 95, 98, was a nice addition which
somewhat alleviated the problem, and Vista condition variables are even better.
Interprocess Signaling - B share the
same console using GenerateConsoleControlEvent.
It really would be convenient to be able to send signals between unrelated
processes, as the console-sharing requirement is far too restrictive. As it
stands, cooperating processes can have threads listening on a mailslot or named
pipe to shut down a process, but this is not the most convenient method.
Sept. 9, 2000: Windows 2000 provides
QueueUserAPC, which allows one thread to specify a function to be
executed by another "target" thread. The asynchronous procedure call (APC) will
be made when the target thread is an an alertable wait state (Ed2: Ch 14, Ed1:
Ch 13). The procedure can raise an exception in the target thread. I'm in the
process of writing a paper to illustrate this idea and to compare the resulting
capability with that offered by the pthread_cancel
function in Pthreads.
HANDLE Types - B
There appears to be no way to determine a HANDLE's
type. That is, does a specific handle refer to a thread, process, file, etc? The
GetHandleInformation function does not provide this information.
HANDLE Reference
Count - C
There appears to be no method to determine how many handles are open on a given
object. Additional information could distinguish between references in other
processes and in the current process.
File Length by Name - C
There is no convenient way to get the file length by name, rather than handle.
I've used GetCompressedFileSize to find out
whether a file is empty, but in other cases I'd either have to get a file handle
using CreateFile, call
GetFileSize, and then close the handle.
FindFirstFile is another alternative.
fork() - C
There is no equivalent to the UNIX lone fork()
function, and it is not at all easy to emulate. This has not bothered me, as
CreateProcess has met my needs (hence the C- rating), and threads can be
used to get much the same effect as fork().
However, it is extremely difficult to obtain an accurate emulation of the UNIX
semantics, and there may be some situations that really do require the lone
fork(). Any opinions out there among you readers?
JDH is happy about this situation, reporting that, under UNIX, if you
fork() when many threads are running and one or more hold locks, you get
an incredible mess. NT does not have this problem.
Sept. 9, 2000. fork()
is tricky in a multithreaded environment, and is definitely not recommended in
UNIX if you are using Pthreads, so there really is no loss here. In fact, the
functionality really is not well defined. For example, if a thread in a process
owns a named mutex. how can the child process be an exact duplicate? What is the
state of a process that has threads running on several processors in an SMP
system?
Pointers - The Long and Short of it -
B
BACK TO TOP
Win32 gives us a 32-bit operating system, which is good. There is no need at all
for "short" 16-bit pointers or any of the Win16 memory models that grew out of
the old Intel memory architecture. So why do we sometimes have
LP (as in LPVOID,
LPCTSTR, and LPSECURITY_ATTRIBUTES)? And
why is the L sometimes omitted (my preference,
as in PHANDLE and
PSECURITY_DESCRIPTOR). Why not just get rid of
LP once and for all and give us a consistent L
for everything? Get rid of the legacy stuff and create an environment for
programmers who don't care about the old 16-bit world. Do the same for the
lp (lower case) that appears with so many parameter names.
Incidentally, let's hope that we don't enter a world of "long long pointers" (LLPVOID,
perhaps?) as Win32 evolves to Win64.
Sept. 9, 2000. The Win64 solution is pretty good (see
Chapter 16) and Edition 4 uses it.
How Many Ways Can You Say
"Forever" - B (name of a pop song?).
It is a game you can
play when you look at the various functions that may wait and require a timeout
period. Here are three examples; you may also want to play the same game with no
time out (0would be good, but what about
NMPWAIT_NOWAIT?)
- INFINITY. This would be my preferred name, and
we can use it with WaitForSingleObject and
WaitForMultipleObjects.
- NMPWAIT_WAIT_FOREVER is used with
WaitNamedPipe and CallNamedPipe.
- MAILSLOT_WAIT_FOREVER is used with
CreateMailSlot.
Program Safety - B (Perhaps an
A)
BACK TO TOP
As programmers, we are, quite properly, admonished to avoid any and all
situations that could "crash" a program. Thus, for example, we should be certain
that we do not de-reference a NULL pointer.
However, many functions that take handles as arguments will cause an exception
if the handle variable has not been initialized. Occasionally, you can get an
exception if you close the same handle twice in succession. (This is, of course,
a foolish thing to do, but Win32 should check the parameters and not generate an
exception.)
JDH reports: "The exception that is obtained by closing the same handle twice is
C0000008 and has the symbol EXCEPTION_INVALID_HANDLE,
which can be found nowhere in the documentation (that I know of); it lives in
the Win32 headers, of course. I have noted that I get this only when running in
the debugger on WinNT, and I can't figure out why this should be so."
File Links - A
BACK TO TOP
File links of some sort are extremely useful, even necessary. The NTFS actually
supports links, but Win32 does not take advantage of this capability. Lack of
links is a common headache in UNIX to NT ports. Sept, 2004. We now have a hard
link (CreateHardLink), which is good. A soft
link would be nice, IMHO. 2010: It's here..
Commands and Utilities - B
BACK TO TOP
Fortunately, UNIX-style commands, utilities, and shells can be obtained
inexpensively (or even for free: see www.cygwin.com),
and perhaps it is just as well that they are not bundled with Windows. These
programs really are indispensable. November, 2010.
The Power Shell is pretty nice, and I'm just learning to use it.
How Do I Release the Memory? - C
Several function calls, such as CreateFileMapping and MapViewOfFile, require you to specify the two pointer
components with the high-order component first.
Remove or Delete? - C
BACK TO TOP
This is one of those little inconsistencies, but it's worth mentioning at least one. I
call DeleteFile when I want to delete a file, but I call RemoveDirectory to delete a directory. Why not have, instead, a DeleteDirectory or RemoveFile?
32-bit, 64-bit Addressing, and
Windows NT Version 5.0 - Obsolete Comment as of Edition 4
BACK TO TOP
Large scale applications really will need a large virtual address space in the very
near future, if not now. Many UNIX implementations already support true 64-bit memory
architectures.
Windows 2000 takes some very small, but not very satisfactory, steps in the direction
of supporting larger address spaces. Among other things, it will support "Very Large
Memory" (VLM) so that a system can have large amounts of physical memory to divide up
among processes. Process address spaces can also be larger (up to 8GB, I believe), but
this extra address space is not swappable; it appears as if it must be hard-allocated to a
process. A large database vendor is already promising to exploit this feature to achieve
higher performance.
As I obtain and digest additional information, I'll add a section on NT 5.0. At this
time, is looks as if we are going to be revisiting new forms of "extended" and
"expanded" memory (remember them?). A true large virtual address space is the
only solution in the long run. I'd go so far as to say the NT 5.0 is extremely
disappointing in this respect (it does have some good features, however, especially the
"zero administration" functionality).
Sept. 9, 2000: Win64 and Windows 2000 do the job right, it
appears (See Ed 3 and Ed. 2, Chapter 16). The concerns above were not really justified. Note that
NT Version 5.0 became Windows 2000 (Kernel versions
are: W2000 - 5.0, XP - 5.1, and W2003 - 5.2)
Things I Could Do Without - C
BACK TO TOP
This is only a personal opinion, but I feel that the functionality of any system should
be both complete and reasonably orthogonal (that is, there should not be several ways to?
Things I Could Do Without - C
BACK TO TOP
This is only a personal opinion, but I feel that the functionality of any system
should be both complete and reasonably orthogonal (that is, there should not be
several ways to do the same thing for no good reason). For example, the
synchronization functions (see the section below) generally meet these criteria.
It is also desirable not to have capabilities that are not really useful to
anyone; there is more to learn, more to maintain, and more ways to make errors.
Here are my candidates for functionality that I could do without (feel free to
disagree). IO and asynchronous IO. Suppose you have two threads
sharing the same handle to a device. Each of these threads then make an IOCTL call (not
necessarily the same type of call) to the device (via the handle, of course). This call is
synchronous, because they are blocked in the call and cannot do anything else. However,
the calls are overlapped, 'cause there's obviously more than one operation
going on at a time. They are "overlapping" each other. Asynchronous IO is where
a single thread makes an IOCTL call (or any other, quite frankly) to a device (via it's
handle) and is not too concerned about when this operations completes, or is in a hurry to
do something else (including making more IOCTL calls to the same device, using the same
handle). After a while, the thread then checks to see if one (or more) of these operations
have completed and responds accordingly. At least, this is the distinction made by the
comp.os.ms-windows.programmer.nt.kernel-mode newsgroup (if memory serves me). I, myself,
have encountered this distinction (I merely wanted overlapped and discovered that I had to
make it asynchronous) with my code and learned this off of the newsgroup."
_beginthreadex
It would be nice if the thread-safe standard C library were written is such a way that
it could initialize its own thread-specific storage. As a result, we could just use CreateThread rather than the rather ugly _beginthreadex.
July, 2005. Ignore the following Sept 2004 statement. The problem was due to a
stupid programming error.
ID Kong <Id@inscriber.com> sent me the following on
11/8/2000 (used with permission): -selected address). Watch the process crash or otherwise misbehave.
Conclusion: TerminateProcess is bad enough, but CreateRemoteThread is a real security and robustness threat. Of
course, processes can protect themselves by denying PROCESS_CREATE_THREAD rights, but this takes some effort and is not the default. Sooner or later, if Windows NT
is to be truly secure in the enterprise, this issue will need to be addressed.
Warning: I really recommend that you DO NOT carry out the above experiment other
than as a thought experiment. The resulting thread could do nearly anything; it might, for
example, delete a few files that you are fond of.
JDH suggests, for additional entertainment, that you pass the address of ExitProcess as the thread function, even supplying a reasonable
termination code. Notice, of course, that this is the address of ExitProcess in the target process, not the one executing.the CreateRemoteThread call.
HEAP_NO_SYNCHRONIZE
This is a preliminary judgement, subject to revision after I perform some additional
tests. I have a colleague, whom I admire, who thinks that this flag (used with HeapAlloc and friends) is terrific. I have found that it has a
minimal positive performance impact, at least in simple cases, and using this flag creates
certain dangers. There may be situations where the flag is helpful, but there must be
other ways to improve performance that are safer. Furthermore, no other functions have
such a flag.
UNIX Pseudoterminal Emulation
Frustrations - A
BACK TO TOP
October 24, 2000: Peter Eliades <peliades@sanderson.net.au>
of Sanderson Computers has made the following very interesting point, which I had not
given much thought to. Peter's message is as follows:
"I agree with your appraisal of the Win32 API as well as that that of Diomidis
Spinellis in his "A Critique of the Windows Programming Interface" (see http://softlab.icsd.aegean.gr/~dspin/pubs/jrnl/1997-CSI-WinApi/html/win.html (first published in Computer Standards & Interfaces, 20:1-8, November 1998)." JH
Comment: This is worth reading. Spinellis makes many of the same points that I have, and
more, but he is much more forceful and considers more than just the system services.
"An additional criticism that I would like to add is the absence of any means of
reliably emulating Unix pseudoterminal (pty) functionality. The options for
programmatically controlling the I/O of a console application under Win32 are:
- anonymous pipes, and
- console screen buffer polling
"Unfortunately both methods are deficient. Some applications, namely those using
the "low-level" console API functions to perform screen I/O simply cannot be
redirected using unnamed pipes. Polling the screen console buffer, is expensive and
unreliable. There is no notification mechanism to allow a process to know when the buffer
has changed. This forces the use of a crude loop that performs a "snapshot/compare
snapshot to previous snapshot" process. This will necessarily result in missed screen
output; the polling thread will not always be running to catch the redirected process that
may be producing screen output profusely. Further, the console screen buffer is a matrix
rather than a queue, earlier screen data will be overwritten.
"I have yet to devise a system that somehow uses both methods (if this is at all
possible, I don't know).
"The console API tends to be important to those companies that have character based
(typically Unix) applications that they wish to migrate to NT/Win32.
"I have investigated some commercial remote shell and telnet services on NT to find
that they rely on the console screen buffer polling method to emulate ptys. If you have a
telnet client with a back paging capability, you will find that on a long listing; e.g.,
the output of an 'ls' that extends beyond one screen, the client will only have received
the last screen of the listing."
Peter goes on to say, "The emulation of psuedo-terminal like I/O functionality is
a recurring theme on comp.os.ms-windows.programmer.win32
and microsoft.public.win32.programmer.kernel in one form or another: redirecting stdin
and stderr/stdout, capturing screen output, controlling another process etc. The answers
vary in quality but the thread named "Capture DOS Screen Tex
(first published in Computer Standards & Interfaces, 20:1-8, November 1998)." JH
Comment: This is worth reading. Spinellis makes many of the same points that I
have, and more, but he is much more forceful and considers more than just the
system services. 2010: Sadly, this link is no
longer available.
"An additional criticism that I would like to add is the absence of any means of
reliably emulating Unix pseudoterminal (pty) functionality. The options for
programmatically controlling the I/O of a console application under Win32 are: output or control the input of a textual application.
"On Windows NT, the case of DOS applications is actually a subset of the more general
case of "console mode" applications, because DOS applications run as coroutines
within a Win32 process (NTVDM) that translates their I/O to Win32 equivalents.
"There are two classes of console mode applications. The important difference
between the two is whether they read from and write to their standard input and standard
output in "glass TTY" fashion using ReadFile() and WriteFile() (what Win32 terms "high-level console
I/O"), or whether they use "random access" APIs such as WriteConsoleOutput() (what Win32 terms "low-level console
I/O"). Translating this to DOS terms: DOS programs that use INT 21h to read
from and write to their standard input and standard output are in the former class; and
DOS programs that use INT 10h or that write directly to video memory are in the latter
class.
"To capture the output and control the input of programs that use "low-level
console I/O", one sits in a loop whilst the child process is executing, continuously
monitoring the contents of the console screen buffer using ReadConsoleOutput() and sending any keystrokes using WriteConsoleInput().
"There are several problems with this design. One minor problem is that it
doesn't cope at all well with Win32 programs that take full advantage of the features of
the Win32 low-level console paradigm and use
alternate console screen buffers for their output. A more major problem is that
because it uses polling (Win32 not providing any handy event mechanism to latch on to so
that the monitor could know when the buffer has been modified) it is always going to be
both unreliable and expensive. It is unreliable in that depending from the relative
scheduling priorities of the various processes, something which is going to vary
dynamically with system load, it may well be the case that the child program may be able
to generate reams of output that the monitoring process will simply miss because its
console monitoring thread won't be scheduled to run often enough. It is expensive in
that if the child process happens not to generate any output for a while, the monitoring
process is going to consume CPU time needlessly.
"To capture the output and control the input of programs that use "high-level
console I/O", one redirects their standard input and standard output through pipes,
and reads from and writes to the other end of the pipes in the monitoring process.
"The advantage of this method is that one doesn't need to worry about missing any
output, since this approach doesn't use polling. But, conversely, this method has a
problem of its own, in that it won't capture any output generated by "low-level
console I/O", and programs that use "low-level console I/O" for input will
bypass the redirection entirely. Alas, all too many DOS programs fall into this
category.
"It is difficult to combine the two mechanisms into one for capturing output, and
practically impossible to combine them for controlling input. So really one must
know ahead of time the type of textual application that is going to be run, /i.e./ whether
it is going to be using "high-level" or "low-level" console I/O, and
select the appropriate mechanism accordingly.
"The only perfect solution that would cope with both sorts of programs simultaneously
would be for Win32 to provide functionality akin to what in the UNIX world is known as a
"pseudo-TTY". Win32 would need to provide some means of for a monitoring
process to hook into the "back" of a console instead of the "front" as
seen in normal use. The monitor process would write data to the back of the console
and the console would turn those data into keystrokes that applications reading from the
front of the console, by either means, would see as input. All output written to the
console, by either means, would be translated into a single encoded bytestream that the
monitor process could then read from the back of the console, in sequence, without needing
to poll, and without missing any data.
"Alas, Windows has no such mechanism."
Threads and Synchronization - A Revised Sept. 9 and Nov. 10, 2000
BACK TO TOP
I'm now much less fond of Win32 threads and synchronization than
I was initially. The biggest problem (in my opinion) is that events are far too complex
and error prone, and there really should be some sort of condition variables
(We now have them as described in Edition 4). Disclaimer:
Earlier versions of this page (before Sept. 9, 2000) were more
complimentary toward Win32 threads and synchronization. I once felt that the thread
management and synchronization functions were well-designed and implemented; I changed my
mind after using Pthreads on some UNIX and Linux systems.
NOTE: For more on this entire subject, see: http://world.std.com/~jmhart/opensource.htm.
The objections to Win32 threads and synchronization can be summed
up as follows:
- Thread management is made more complex than necessary by the
requirement for _beginthreadex() and the confusing _beginthread() when using the C library. This comment also
applies to _endthreadex() and _endthread().
Severity B.
- There is no direct thread cancellation. This can make if difficult
to terminate a threaded system in an orderly way. Severity A.
- Semaphores are redundant and not completely implemented.
Severity B.
- Events are poorly designed and the cause of all sorts of
synchronization bugs. Severity A.
Even so, Win32 threads are very easy (even fun) to use, generally working as expected.
A common opinion is that threads and synchronization are difficult subjects and the
programming is complex. I disagree; the functions (other than events) all make sense, even
though the functionality is incomplete (I would still like to see an atomic multiple
semaphore wait function, for instance, and thread cancellation would be nice). Semaphores,
mutexes, and events all have a role to play, even though they are not totally independent.
(For example, it is possible to create a semaphore out of a mutex and an event. See the example.) The biggest criticism is that the event semantics are too complex
and are also incomplete.
On the positive side, threads and synchronization are a standard part of Win32 and have
been there from the beginning. By way of contrast, threads arrived very late in UNIX, and
it took a long time for them to become standardized. Many early implementations were not
really kernel threads but were more like fibers. Until I really used
it, I did not appreciate POSIX Pthreads, and while I still don't like the Pthreads naming,
I do like the architecture and implementation. In particular, I now appreciate why mutexes
are combined with condition variables; the Pthreads design is correct. See Edition 2,
Chapter 10 of Win32 System Programming for a discussion of the Win32
"condition variable model.".
Extensive comments, Sept. 9, 2000:
I've had the opportunity to use Pthreads extensively and now feel
that Win32 events, with the four set/pulse, auto/manual-reset combinations, are confusing
and a source of all sorts of bugs (in my own programs and others that I've seen). Edition
2, Chapter 10, describes what I've called the "condition variable model" as it
is modeled on Pthreads condition variables.
As an example of the confusion, the latest edition of a best
selling book on Win32 system programming (you'll have to figure out which one) states in
the preface that a major improvement of the new edition is its treatment of
synchronization. The author, who is extremely knowledgeable and has written extensively on
Windows programming, then goes on to state that he has never seen a good use of the PulseEvent function. The new edition did remove an example from
earlier editions in which the multiple wait semaphore problem was "solved" using
successive waits on the same semaphore (a deadlock-prone solution). Well, if you want to
solve this problem, you really do need to use PulseEvent to get an efficient solution. The point of all this is that even very experienced writers
and programmers do not use Win32 events properly. and attempts to correct old errors often
result in new errors. In my own case, I posted a multiple wait semaphore
"solution" that erroneously used the signal, rather than the broadcast,
condition variable model; this has now been fixed.
As an additonal example, in a book dedicated to Win32
multithreaded programming, the authors state that PulseEvent and SetEvent have the same effect for an
auto-reset event, both releasing one thread. While the distinctions are subtle,
they are important. There is definitely a lot of confusion out there about
events, and there must be a lot of latent defects in threaded code.
Multithreaded application development is difficult enough without the added
burden of these confusions.
Finally, a general implementation of a condition variable that is
to be used in broadcast mode requires a timeout on the event wait in order to avoid missed
signals. In a sense, all you are really doing is polling the condition variable predicate,
with some assistance from the event. Admittedly, SignalObjectAndWait addresses this issue, but the solution is not portable to all Win32 platforms.
Bottom Line: Win32 would have done well to have used the condition variable
concept rather than events. See http://www.cs.wustl.edu/~schmidt/win32-cv-1.html for more insight into this important issue.
There is a LINUX Open Source POSIX Threads for Win32 project underway, and a snapshot
was posted on September 8, 2000. See http://sources.redhat.com/pthreads-win32 for more information. I haven't looked at this work yet, but will soon. As a
"proof" of Win32 event clumsiness, consider the following:
- Pthreads condition variables are complete, natural, and easy to use, especially if you
use them with the correct programming idiom, as in my Chapter 10.
- A complete condition variable Win32 emulation is quite complex; see he open source
implementation. It should not be so complex to do something so essential.
Other comments: Pthreads provides for thread cancellation, which is a big
improvement over TerminateThread. However, you can achieve
nearly the same effect in Win32 with asynchronous procedure calls (see QueueUserAPC); this is the topic of a future note. Nonetheless,
thread cancellation should be provided more directly.
Really A Final Note - Win32 Program Appearance
BACK TO TOP
Sept. 9, 2000. This may just be a personal opinion and a matter
of taste, but, I find that Win32 programs look sort of clunky, heavy, and graceless when
compared to their UNIX equivalents. This impression is due mostly to the extensive use of
upper case and long function names. Does it matter? Perhaps. I'd argue that compact,
graceful code is just better and is easier to understand and a lot more satisfying to
write.
Diomidis Spinellis dspin@aegean.gr sent me the
following comment on 10/25/2000: "Regarding your note on Win32 program appearance, I
think that one reason for Win32 programs looking clunky is the non-existent to bad
formatting style found in Microsoft's example code. I am under the impression that those
programs are written at Microsoft by summer interns and never reviewed. Programmers that
want to use a Win32 feature will invariable stumble upon such example code (eg through
MSDN) and get infected by bad programming style. In contrast, most Unix code that one gets
exposed to (the kernel, utilities, Perl, GNU) is very well-written by experts who know how
to program in the large and in the small. Sadly, some more recent applications and code
(especially those coming from the Linux world) are exceptions to this rule, but generally
the vast majority of free Unix code is exemplar. When programming Win32 I try to use the
Berkeley formatting and coding style; for an example you can have a look at the outwit
tool suite <http://softlab.icsd.aegean.gr/~dspin/sw/outwit>
(please be kind, I did not write it as an example for programming style)." (Used with
permission.)
Sept. 8, 2014. An
Amazon reviewer
remarks:
"However, the examples belie the fact that doing what should be the
most straightforward things on Windows requires a tremendous amount of effort
and attention to picayune details, whereas, on UNIX (but of course), 'twould be
next to trivial. That's no surprise: I mean, you can make a UNIX system sing and
dance with Perl programs and shell scripts and such--there isn't necessarily a
need to code in C unless you require the most detailed of semantics, such as
utterly non-standard TCP or UDP or RPC disciplines--whereas the "comparable"
glue language on Windows is, I guess, WSH that still clings to 1960s-style
syntax hailing from Kemeny-and-Kurtz BASIC (e.g., you use DIM--short for
DIMENSION [!]--to define a variable)."
I do have comments in response:
- Perl is available on Windows, and you can use shell scripts (PowerShell is nice) of all sorts. No problem.
- By now it should be clear that I feel that there are some valid criticisms of the Windows API, and
in many cases *IX is marginally easier to use (The converse is also sometimes true).
- Sometimes you need to use the Windows API, or, at least, it's the most convenient option. For example,
you may need to port an existing *IX program (there are lots of them!). Incidentally, the *IX program will use C (or maybe C++)
and will deal with an equivalent set of detailed semantics and picayune details.