Thinking - the bottleneck?

Introduction

During the autumn of year 2000 I had to write my first report for the course "Cognitive psychology", which is part of my education in computer science (initiated in 1996) with information psychology (initiated in 2000) as minor.

The theme and/or subject of the report was optional, so one could discuss an article, do an experiment etc., as long as it was related to cognitive psychology. The extent of the report was "a few pages", I think it meant 6-12 pages - mine was 15, but the teacher did not comment on this. The report was not graded, but mandatory if one wanted to pass the course.

I chose to test a hypothesis which I formulated myself, and for that I needed a lot of test persons (and in a hurry, since I was two weeks behind, due to various circumstances). These were all volunteers from a few Amiga mailinglists, and this page is put up solely to provide these friendly individuals with some sort of feedback - without them I would not have been able to do the experiment, so thanks again to these nice and loyal people!

This page is not intended for students of psychology (nor computer science for that matter) which basically means that only a basic introduction and the core results are published on this page. If you have any questions (about the theory involved, validity and/or reliability of the experiment, reservations, testdata etc.) then feel free to write and ask. The original report is in Danish, but you are naturally welcome to get a copy.

The hypothesis

When answering a multiple choice question (on a computer) there are several ways to let the user provide his answer. I intend to test two methods, which differ in the way that the first method tries to minimize the physical work and the second method tries to minimize the mental work. It is my hypothesis that minimizing the mental work will give faster answers than obtained by minimizing the physical work.

Method A (minimizing physical work)

This method shows a number in front of each answer. All the user has to do is press the number of his desired answer. This should lead to one keystroke pr. answer, and since the test persons are all experienced computer users, it is assumed that they can use the numeric keyboard without being slowed down by having to look at the placement of the numbers (but this is not verified).
The GUI for this scenario can be seen on the right. Answering "Europe" is achieved simply by pressing "5".

Method B (minimizing mental work)

Here the user is also presented with all the answers, but they have no distracting numbers attached. Instead there is a cursor which the user can move with the cursor keys. The user must press return when he or she has placed the cursor on the desired answer.
This scenario is also depicted on the right. Here one would use "arrow down", "arrow down" and "return" to answer "Europe", i.e. 3 keystrokes (actually one can also press "arrow up" to make the cursor wrap around and settle with 2 keystrokes, but no-one did this, which again illustrates how users are more comfortable with a well-known intuitive behaviour).

Results

Below is a plot of the average response time for each of the 18 questions.

Below is a plot of the number of choices for each question, the average number of keystrokes used (for method B) and the difference between the average response time for the two methods.

It is interesting to notice that the difference in response time actually seems to grow together with the average number of keystrokes. I.e. the questions which require several keystrokes (with method B) are relatively faster for method B compared to method A, than the questions which only require a few keystrokes.

Significance

When doing this type of experiment (testing if one method is "better" than another) it is not sufficient simply to obtain a difference in the sampled data. One has to submit the data to a thorough significance test, where various factors are tested. I have done this with my data set, and the result is that my test is categorized as highly significant (which is very good!).

Conclusion

My experiment has confirmed my hypothesis (that method B is faster than method A).

However, method A and B are not the only ways to allow a user to provide his or her answer to a multiple choice question. One could use the mouse, let the user enter the first letter of the answer etc. And wether method B is faster than these schemes is subject for a new experiment.

Appendix

Below are the 18 questions asked.

Webmaster - Created: 19. December 2000 - Last updated: 23. December 2000.