Testing two different operating systems for timing

I have designed a test that is dependent upon the timing of the subject’s responses. To make it available to the largest number of colleagues possible, it has been written for the Mac and PC, with identical programming for each version Is there any way to test whether or not identical responses by two hypothetically identical subjects would be timed the same by both systems, PC (Windows XP) and Mac (OSX, Leopard), other than trying to press keys on two keyboards simultaneously, which obviously would not work. The test depends on the times between presses of keys of the keyboard of the laptop or desktop on which it is running.

Thanks,
Robert

The short answer is that they are not the same, and the keyboard matters too.

The long answer involves me uploading lots of files from the experimentation I’ve re-done over the past few days. Basically, I’ve repeated the test I performed in this thread–the test by which I determined that my Apple keyboard was precise to 8ms. I’ve also determined how accurate it’s possible this keyboard can be.

I’ve tested two different keyboard under three different operating systems on two different computers. (proof of theory, not extensive test to find the absolute best possible combination.)

Keyboards:

Kensington Slim Type Keyboard Mac
Apple slim keyboard that shipped with the Mac Pro

Computers:

8-core Mac Pro Early '08 w/6GB of RAM
HP Media Center PC m7160n (Pentium D w/1GB of RAM)

Operating Systems:

Mac OS X Leopard 10.5.3 – Installed on the Mac Pro
Windows Vista SP1 – Installed on the Mac Pro (bootcamp)
Windows XP SP3 – Installed on the HP

The process: I created a simple SuperLab experiment that waits until the user presses the space bar to continue. All key presses are recorded. Since USB HID devices are polled at regular intervals, the idea is to stress this polling functionality. While the experiment is waiting for the space bar to be pressed, I pressed other keys at varying speeds, but generally about as fast as I could.

The verbose log provides response times in increments of .01ms (Directions for how to enable this are included here for Windows and here for Mac OS X), so this was the preferred source of data for this experiment. After getting the entire contents of the verbose log into a text file, I ran commands similar to the following UNIX command to extract only the actual response times.

awk ‘/actual/ {print $16}’ Leopard_Kensington_USB.log >& Leopard_Kensington_USB_ms.txt

Using a spreadsheet program (Apple’s Numbers), I computed the differences in time between each of the key presses. In doing this, I found that all times on all operating systems for the Kensington keyboard come in multiples of 16ms. All times on all operating systems for the Apple keyboard come in multiples of 8ms. Therefore, to find the accuracy of a given combination, I performed a modulus of the differences relative to the keyboard’s polling period. These results have been graphed as a histogram.

I’ve also provided graphs of this modulus error with respect to the amount of time since the previous keystroke. This additional visual representation makes both the precision and the accuracy visible in a single graph.

Conclusions:

With respect to reaction times from a USB keyboard, the choice of keyboard makes a far larger difference than the choice of operating system. While OS X has roughly 1/10 the error of Vista, the vast majority of response times on Vista still fall within a 1ms radius of the expected time. Knowing response time with .1ms accuracy is meaningless on a keyboard that can’t get you better precision than 16ms.

Notes:

This only takes into account keyboard accuracy. There are many other factors in a computer system. An important one is whether the refresh time of the monitor is available to applications. Mac OS X fails to retrieve this information from the monitor used for this experiment (a 30" Dell LCD), but Vista succeeded in retrieving this information. In this specific hardware setup, this would introduce an error radius of about 8ms on the Mac which makes the keyboard precision irrelevant. Furthermore, on LCDs, the display’s response time is important, but this information is not available to software.

Leopard Test Results.pdf (45.3 KB)

Vista Test Results.pdf (98 KB)

XP Test Results.pdf (48.9 KB)

Keyboard Test and Data.zip (238 KB)

This should have been attached, too. I was having difficulty getting it to upload into the other message.

This is the spreadsheet. The data are all available in the other zip file. This just shows how everything was calculated. This requires iWork '08 for Mac OS X.

data.numbers.zip (806 KB)

Kelly insisted posting images instead of PDFs would help. Here are a few of the graphs from the PDFs:




Thank you

Thank you for the prompt and very helpful reply, Hank.

Robert

I also calculated the standard deviation on the error data and found the following:

Mac Pro, Leopard: .0334ms
Mac Pro, Vista: .333ms
HP, XP: .576

Unfortunately, I don’t currently have a direct comparison on the same machine between XP and Vista.