Posts Tagged ‘ usability testing ’

Class notes F14 – Usability metrics

This week we started the 5th and last module of the course, Evaluation. As a reminder, the modules are:

  1. Foundations
  2. User research
  3. Conceptual design
  4. Implementation
  5. Evaluation

We clarified the basics of why and when to conduct usability testing. Then I explained more detailed usability metrics and procedures, based on the the book Measuring the User Experience. I also introduced to you Website Experience Analysis. Slides are below.

I’d like to emphasize that it is very important to review class readings and notes, and to do a bit more secondary research as you work on your projects. You do need to read more about usability testing (metrics, procedures) before you conduct your own research. Otherwise, we will again see basic mistakes that could have been very easily avoided.

We then practiced a very quick and dirty usability test and looked at BlueJeans for next week’s class.

Next time we meet, we will look at what to do with the different data we collect from a usability test  – how to analyze and present it.


Usability reporting

Please find below the slides about usability reporting.

Please run your research plan by me and let me know if you would like to meet with me during class next week.

Class notes: Usability testing

In preparation for discussing usability testing, we overviewed ALL the research methods mentioned in this class. The summary we came up with looked like this:



User Research


Conceptual Design

Brainstorm > Prototypes


programming (writing computer code) and implementation



Field observation



Cultural Probes, Design probes

contextual inquiries

ethnographic interviews

card sorting




picture cards

Case Study

Task Analysis  

Focus Groups


Cultural Probes

Card Sorting

Design probes

usability testing

focus groups

cognitive walkthrough

Heuristic Evaluation

Usability Inspection

Cognitive Walkthroughs

Usability Testing

Think-aloud (^)

Cognitive Walkthroughs

Focus Groups

Retrospective think-aloud protocol

User Acceptance Test

Usability Testing

Usability Inspections

Heuristic Evaluation

Laboratory Usability Testing

Field usability testing



Exit Interviews


Formative Research – Forms the creation

Evaluative Research – Evaluates the product

I then gave you more details about usability resting and Website Experience Analysis, as in the slides below:

At this point in the semester, my goal for you is to integrate all the knowledge – put it all together, and see links among the many concepts and methods we have learned. Please use your class reflection blog post to help you do that.


Usability testing 101

Get a large overview of what usability testing is and how much it costs from this promotional video:

See how a very simple usability test (interview) can be conducted with paper prototypes. This can help you validate your initial concepts before even one line of code is written:

Then, read a bit on what a usability test is and how to plan it, as well as how to run it.

Check out how usability testing can be done online:

And see how it can be done with Morae, which is installed and running in Knoy 316A – and you should plan using it for your own usability tests for this class:

Now, head over and read the articles posted on Blackboard.




Task and time-on-task consistency across groups

[UPDATE] Please record each one of your participants’ time on task in this spreadsheet.

I have just reviewed the research plan for all groups. If you have not heard back from me, please email me ASAP.

I am noticing that groups are getting creative with writing the usability tasks and what they consider as task success. You are welcome to do so, as long as you record time on task for the initial moment when the user has found the page (has reached the URL) with that information. Record that time as T1 for each task. Then, if you ask the user to do something else on that page, that’s fine. You can record that time as T2 for each task.

It is important we keep the tasks consistent across groups, so we can see some comparisons across groups.

The tasks I wrote on the Google Doc for this assignment are:

  1. Find information about requirements for admission in the CGT graduate program.
  2. Find what courses are required for a MS in CGT degree.
  3. Find the main areas of study within CGT.
  4. Find contact information so you can request more information.
  5. Find information about receiving funding assistance for your graduate studies.
  6. Find information about CGT professors’ areas of research.
  7. Find the page where you can begin the application process for admission to graduate school.

Please keep them as they are (some MINOR wording modifications are fine, but not really necessary) and measure T1 for each task consistently for all groups.

Let me know what questions you have.

Class notes: Usability Metrics

Please find below the slides that summarize the types of metrics you can collect with usability testing and lay out the sequence of a usability testing session.

The metrics come from Tullis & Albert chapters 4-7. Please revisit those chapters as you prepare you usability materials and think about how to report your findings.

Must read: the $300 Million button

You absolutely need to read this post about how making a tiny change to a very simple Web form resulted in a yearly revenue increase of $300 million for an online retailer.

It is a story about the power of:

  • the importance of the tiniest interface detail
  • observation and usability testing

You also absolutely need to learn this name: Jared Spool – and here’s why.

Hat tips to Twitter users @dtelepathy, @uie, @kellyndesigns – the post reached me through them.

Did you read this post? Please indicate by commenting, ranking, liking, etc.