View this PageEdit this PageAttachments to this PageHistory of this PageHomeRecent ChangesSearch the SwikiHelp Guide

Q2 - Innovation vs. User Testing

Question
2. Some people argue that if the intent of your research is to create innovate visions of future interactions, then user testing is a waste of time. The argument is that if you invent anything really innovative, users won’t know what to make of it.

a) What support from the literature of HCI research and your own research experience can you use to support this opinion?

b) What arguments (again from the literature or your own research experience) can you present that would still support the need for user testing in the face of innovation?

Response
Visions of the future serve to inspire researchers and provide them with guidance in determining what problems to solve. The field of computer science is no exception, its history littered with new visions of human-computer interaction. In the 1940’s, Vannevar Bush [1] recognized the overwhelming amount of information being produced in the world and the difficulty in navigating this information. To address these problems, Bush envisioned a “Memex” device that would assist in the capture, access, and navigation of these large stores of information. Licklider [2] and Engelbart [3] envisioned a symbiosis between man and machine, to augment intelligence, and both contributed in their own ways to move closer to their respective visions (Licklider through funding computer science research and Engelbart through the creation of new tools including a chording keyboard and the mouse). Alan Kay dreamed of a Dynabook, a multimedia-rich device to support creative expression, and has devoted much of his professional life to reaching this vision through the creation of such systems as the Smalltalk environment. More recently, Weiser [4] envisioned a world where computers disappeared into the environment around us, creating interactions that are effortless and seamless.

An argument has been made that when creating such visions of future interactions, user testing is a waste of time, because anything truly innovative can’t be tested; users simply won’t know what to make of it. If we examine the literature in HCI, we can see examples of where this is true and where it is not true. To center the discussion, it is helpful to clarify what we mean by “visions of future interactions”.

Interactions with computational systems can be regarded at several scales. At the smallest scale, we can consider the individual interacting at the physical level – pushing buttons, turning knobs, moving a mouse, etc. At a larger scale, we can consider interactions with computational systems in terms of the context of use, for example, the interactions typified by routinely using a capture and access system to support education, as in the case of the use of Classroom 2000. This larger scale examines interaction in the context of use, rather than interaction at the physical level. As pointed in [5], system designers have increasingly paid heed to these larger-scale interactions in their attempt to create systems successful in their context of use. Grudin, quoted in [5], describes this process as “an outward movement of the computers’ interface to its external environment, from hardware to software to increasingly higher-level cognitive capabilities and finally to social processes.” As we examine arguments against user testing of future visions, it is important to consider the scale of interaction. For the most part, we will consider larger-scale interactions in this essay.

a) In support of this argument, some visions of future interactions are so far different from current day practices that it is difficult to perform user testing. Bush [1], in describing the Memex, envisioned a hypertext-like database of information. At the time, it was difficult and unrealistic to perform user testing on such an idea: the technology did not exist, and Wizard of Oz-like experiments (where portions the system and its behavior are mimicked by people) would yield little useful data. It would be too long before such a vision could be realized, so any Wizard of Oz-type data may be outdated when the technology became available to build a Memex: user practices may have changed, and the technology used to create the Memex may affect the work practice in ways not anticipated.

Englebart’s work with his workstations also lends support to this argument. The systems he developed were not quickly adopted by society; it was decades before the advantages of the mouse, a chording keyboard, or networked computers were understood by the population. Ivan Sutherland’s Sketchpad offers a similar example; the interaction methods promoted by the Sketchpad (graphics, directly manipulating the graphical objects) took years before they took hold. For most, these systems were too different than current practices, so user testing may not have revealed much. However, it is important to note that while user testing in the general population may not have been feasible (ie, not yielding a lot of valuable data), user testing did occur, for example in Englebart’s lab. This user testing did help refine his tools as they were developed, but the user population was not what one might consider a typical user population.

Finally, we can examine concept videos like those produced by Sun and Apple, which show a future where we interact with context-aware systems through voice and gesture. These videos promote a vision of the future, but as with Bush’s Memex, when these videos were produced, the technology necessary to begin implementation was too immature to warrant user testing.

b) While we see historical examples of the impracticality of user testing for envisioned interactions, the literature provides a host of research that stresses the need for user testing as these visions are developed. Design methodologies such as Scandinavian Design [7] and Contextual Design [8] recognize that the introduction of new technologies affects work practices, and thus include, as part of the design process, the introduction of prototypes and new work practices into authentic situations to understand how the new technology will impact work. User testing thus grounds the vision, highlights the areas where it works and uncovers the aspects of it which fail in practice.

The PARCTab experiments in ubiquitous computing [6] and work in Media Spaces [9], provide excellent examples of the need to perform user testing in authentic situations to understand how true a vision stands in actual practice. The former project tested out visions of Weiser’s vision for ubiquitous computing [4] by developing handheld, mobile, context-aware communication devices, while the latter tested out visions of collaborative work for people who are geographically separated by creating open video and audio channels between offices hundreds of miles apart. In the PARCTab project, user testing in an authentic situation uncovered factors influencing the acceptance of the system, which included the size of the device, its appearance, the convenience of carrying it, peer pressure, and the applications available for it. The Media Spaces work uncovered the importance of awareness cues in promoting community across distance (for example, cues as simple as seeing people at the other location walking or chatting in a common area). Such user testing is necessary to understand the true implications of a vision.

While innovations serve to inspire and guide work, user testing helps focus these visions, ground them in reality, and show where they work and don’t work. Visions can’t capture the subtle nuances and issues found in real-world use; user-testing exposes these.

References
[1] Vannevar Bush. As We May Think. Atlantic Monthly, July 1945
[2] J.C.R. Licklider. Man-Computer Symbiosis. IRE Transactions of Human Factors in Electronics, Volume HFE-1, pp. 4-11, March 1960
[3] Douglas Engelbart. Augmenting Human Intellect: A Conceptual Framework. Stanford Research Institute, October 1962
[4] Mark Weiser. The Computer for the 21st Century, Scientific American, September 1991, pp. 94-104.
[5] Hughes, Sommerville,Bentley & Randall. (1993) Designing with ethnography: Making work visible. Interacting with computers. Vol 5:2. Pp. 239-253.
[6] WANT, R., SCHILIT, B., ADAMS, N., GOLD, R., PETERSEN, K., ELLIS, J., GOLDBERG, D., AND WEISER, M. 1995. The PARCTab ubiquitous computing experiment. Tech. Rep. CSL-95-1. Xerox
PARC, Palo Alto, CA.
[7] Kyng. M. "Scandinavian design: Users in product development", In the Proceedings of CHI'94, April 1994, pp. 3-9.
[8] Beyer, H & Holtzblatt, K. (1998) Contextual design: Defining customer-centered systems. San Francisco: Morgan Kaufmann.
[9] Media spaces bringing people together in a video, audio, and computing environment; Sara A. Bly, Steve R. Harrison and Susan Irwin; Commun. ACM 36, 1 (Jan. 1993), Pages 28 - 46

Link to this Page