View this PageEdit this PageAttachments to this PageHistory of this PageHomeRecent ChangesSearch the SwikiHelp Guide

Q2 - Ethnography and Evaluation

Test#16
Question2
Evaluation

Ethnography, in its original research intent, is intended to allow a researcher to make sense of an existing human social situation, human culture. Taking the perspective of the interested nave outsider the researcher is freed to ask questions of the society under investigation that would otherwise remain invisible to a fully vested member of that culture.
Since human culture evolves over centuries this technique is designed to make sense of relatively stable systems of human behavior. As such the fact that it takes an enormous amount of time to do well is not unexpected. Anthropology Ph.D. degrees take on average 11 years to complete for reasons that are obvious.

Ethnography is practiced to develop a rich, thick description of human culture.
While this helps us understand why people of a culture do what they do it says nothing about how to uncover possible problems or how to suggest changes to existing system.
Additionally it makes no mention of predicting what effect those changes might have on the culture.

System designers are faced with multiple dilemma.

Computing is moving off the desktop and into the world.
The initial low lying fruit of techniques for understanding human activity in the constrained, limited domain of the highly motivated expert has, to some degree, been picked.
Once we move out of the work place and into the world, tasks are not so well constrained.
Predicting how a task will be executed is not constrained to the most efficient , sequentially executed list of activities. Humans are opportunistic in the real world. They take advantage of what is on hand when it is on hand and thus are not forced into a strict sequential execution of tasks.

Temporal constraints.
Since ethnography exists to investigate and understand how humans do what they do it is attractive to consider it as a mechanism for uncovering what users do while carrying out their tasks. The technology that they are interested in deploying does not change on the same temporal scale that one finds in evolving human systems. Instead of centuries to evolve human systems, technology evolves on a time scale that allows several generations in one year. This being the case the system designer does not have the luxury of time to investigate how things are currently done. On the other hand, systems that are technically correct may fail miserably if the user and his task is not fully understood (Markus).

The incompatibility of this temporal difference is further accentuated when one considers that the introduction of a tool (a new systems design for instance) changes that system by changing the nature of the tasks. In other words the design that comes out of the investigation might be the right design for the system as it existed BEFORE its introduction. Introducing a new artifact into a system, waiting for the effect it will inevitably have on that system and then doing another ethnographic study is clearly out of the question.

To solve this problem system designers use iterative design techniques in which the new system is introduced in some form (mock up, wizard of oz others). It can be evaluated in some discounted kind of way to reduce the time it takes. Typically this kind of evaluation technique is not done in situ. This approach turns its back on the idea that the environment in which the human acts is important. While the workplace is more constrained than real life this type of evaluation technique is more appropriate for work related activity than it is for non-work related activity.

Example system: information appliance in the kitchen.
An information appliance used in the kitchen and supporting family communications initially brings to mind a variety of constraints. The kitchen can be messy, hands can be occupied, it can be noisy and there can be multiple people in the kitchen. Preparing a meal may seem like a task that all people undertake in a kitchen but you must be sure to find out what that preparation entails. Kosher kitchens, kitchens of vegetarians, and different ethnic backgrounds may well dictate different tasks in the kitchen, not to mention that they will dictate different styles of interaction between family members.

Perspectives of the designer and the ethnographer in understanding the use of this appliance.

Ethnographers would ideally want to study the information appliance after it has been in place and in use. They would want to study the users of the system to see how they make meaning with the system. What is this device being used to do?

System designers would want to do a number of iterations on the design and use some discounted means to evaluate it before it is deployed. They would reach for techniques such as cognitive walkthrough or contextual design to overcome the temporal difficulties. Cognitive walkthrough uses an expert and scenarios of use (prepared in advance) along with evaluation criteria to pass judgement on the design. If the experts are also considered to be users of the proposed system then their judgement is more relevant. After all, discovery of any problem without the investment of a great deal of time is an advantage. Contextual design may be a better choice for evaluating the design and proposing possible changes because it employs a cross disciplinary approach that includes end users in the process of design and evaluation.

How ethnography can help.
The system designer is probably not the user. Users are not confident like designers, there is the fear of destroying data, hurting the machine and of seeming stupid (Rheingold). Do not assume that you know what the user is doing with the appliance. Rather than going into the situation with specific questions an ethnographic approach would be to take the position of the nave interested party and observe what the user is doing. As described earlier real ethnography takes too long. A technique used by ethnographers in their investigations that could be co-opted for use in the shorter time frame is to use the questioning stance of the ethnographer. The ethnographer carefully avoids wording that might lead the user to an answer that, in the absence of that question, he would not otherwise give.


Rheingold interview of Don Norman found in The Art of HCI Design edited by Brenda Laurel.

Markus, M.L. & Keil, M. (1994) If we build it, they will come: Designing information systems people want to use. Sloan Management Review P. 11-25.

Link to this Page