M8 User Interface Evaluation

You have put in a lot of effort designing and implementing your project. This project milestone will have you focus on uncovering significant usability problems with your prototype and suggesting ways to improve upon it.

But there is a little bit of twist in this assignment. You are going to do a usability evaluation of another project team's prototype and report back to them. And when you receive the usability evaluation from the other team, we are going to ask you to reflect on how significant a design and implementation change would be required to address any problems uncovered.

Meet the other team

Your team will conduct a usability evaluation of another team's prototype. The team assignments are posted here:

You should make every effort to meet up with the team you are evaluating during the Milestone 6 presentations.

Passing-on / Getting code

It is essential that you get your code to the team that will be evaluating your project by Wednesday, July 9th. . Contact the team you are evaluating ASAP to get their code. Ideally, a person from that team should meet with a person from your team and (1) get you the Visualworks image / fileout / Store parcel, (2) make sure that you know how to use it, and (3) answer some of your questions. This is advantageous to the other team as the higher quality your understanding is, the more likely it is your evaluation will be informed. A more informed evaluation leads to a better rebuttal. Teams that do not pass on their code to their evaluating team before or on Thursday, July 10th, may get a 0 for the M7 assignment.

If you never got anything to work, pass to the other team UI mockups, or drawings that they can use to evaluate your UI conception. Remember this a usability evaluation, not a functionality one.

To complete this milestone, you must form a usability evaluation plan, execute it, submit a usability report to the team whose prototype you evaluated, and then react to the usability report you receive on your own project.

Plan your usability evaluation (10 points)

In lecture, you have been introduced to three different usability evaluation techniques: heuristic evaluation, cognitive walkthrough and think aloud protocol analysis. You are asked in this milestone to select one of these evaluation techniques and justify why you feel it would be appropriate to evaluate the prototype from the other team. You then need to make sure that you have prepared all of the required inputs to conduct the usability evaluation. This section should clearly indicate which method you chose, why you chose it and what you hope to learn from it.

Execute the plan (30 points)

Your team needs to conduct the usability evaluation that you defined in your plan and present the raw results in (e.g., the usability bug reports from h.e., the believability story for one or more canonical tasks for cognitive walkthrough, the transcript and observations done for the think aloud) in a form that can be understood by the instructor and TA. These results need to be produced in written format, so be sure to plan your evaluation in such as way as to make it easy to collect these results and share them. This section has all the details for your plan. Most teams include the raw data, any checklists and other information, and a discussion of the results.

Usability Report (30 points)

You need to write a report of your usability findings that will be submitted to the instructor as well as the team whose prototype you evaluated. This report should be no longer than 3 printed pages and should clearly highlight and explain the top 3 recommendations for altering the prototype to enhance its usability. You are writing this report for the design team to read and react to, so make sure you are clear and constructive in your comments so that they understand what you are asking them to change and why.

These first three parts of the milestone are due Tuesday, July 16th to be turned in by midnite to Bob's office in hard-copy. You must also provide a copy of the 3-page Usability Report to the team whose prototype you evaluated. This is to facilitate the final part of the project. Please be sure you get the other team the report in a timely manner.

Rebuttal (30 points)

We are not asking the design teams to do any additional implementations to the project. Instead, we are asking you to read the report from the evaluation team and respond to the way you would modify your design and implementation to accommodate the recommended changes. This report should be no longer than 3 pages and should focus on explaining how you would adapt your prototype. We also want you to reflect on whether the object-oriented approach to your design makes it easier or more difficult to make the recommended changes. Please do not be defensive. Your grade is not based on how bad the UI faults were, it is based on your corrective action. Don't try to convince me there are no problems with your UI and the other team was just stupid.

Checklist of Things to Complete

You are responsible for turning in two separate reports:

• Report 1: 70 points, due Wednesday, July 16th (by midnite to Bob's Office)

1. (10 points) Clearly identify the team whose prototype you evaluated and provide a one-page description of your evaluation plan and the rationale behind the plan, i.e., why you chose to conduct that evaluation on this prototype.
2. (30 points) Results of execution of usability plan, providing clearly documented printed evidence that you conducted the evaluation plan correctly. For instance, this includes the actual raw data sheets and evaluator notes in addition to the synthesized data. I need to verify you did the evaluation you claimed to do.
3. (30 points) 3-page usability report highlighting the three top concerns arising from the usability evaluation. The report should identify clearly what each usability concern is and the evidence you have to support this problem being a real problem that should be corrected by the design team. One copy of this part of the report goes to Bob, and one copy should be given to the team you evaluated.

• Report 2: 30 points, due Monday, July 21st (in class)

1. 3-page rebuttal of the usability report submitted to your project team. Be careful not to get defensive in this report (resist refuting that the findings of the evaluation team are incorrect). Make a clear argument for how you would address each usability concern. Assess whether the object-oriented design you developed for your prototype made it any easier to address these usability concerns.

REMEMBER: This is a usability evaluation NOT a functionality evaluation. Don't spend time checking whether the application actually works or not. Focus instead on overall usability of their system using the techniques we discussed in class. You can do a usability evaluation on screen drawings if you have to.