Guidelines

Every Web site is unique with a different set of parameters to consider when designing your test plan. Here are some general guidelines.  It is important that you remain flexible because often respondents, through problems and suggestions, come up with issues that force you to deviate from your initial plan. Make sure you listen to their input, recording it and following where it may lead. It is a good idea to keep detailed records of what occurred during the testing to use for reference when designing future usability studies.

When to do Testing

It varies, depending on the size and level of sophistication of the site. 

Some General Guidelines Include:

Before any code is committed to the page,
use a paper prototype with sketches of pages outlining content, layout, and navigational elements.
Early versions of the site to test functionality and site flow.
Pre-launching of the site to make sure all elements are functioning properly.

Logistics of Usability Testing

The following are estimates of the time and costs for usability testing taken from Jakob Nielsen's Web Site
"Doing Usability Tests". 

In takes 39 hours to usability test a website the     first time you try.

This time estimate includes planning the test, defining test tasks, recruiting test users, conducting a test with five users, analyzing the results, and writing the report.

With experience, Web user tests can be completed in two work days.

With experience it is possible to conduct much more rapid user tests of a site.

Good test tasks can be written in one or two days, the actual test can be done in a day, and the results can be analyzed in a few hours.

If you are a member of the design team, then there is no reason to write an extensive report which nobody will read, so reporting can be done in a one-hour meeting supplemented by a summary that takes 2-3 hours to write.

In total, a discount usability study takes only two work days once you know what you are doing.

Even though experts can do the work more efficiently (and usually with better results), it is encouraging that rank beginners could complete a full Web usability project in less than one week. This truly proves that "limited budget" and "lack of time" are not valid excuses for inflicting difficult sites on your users.

Usability Testing Process

Four Step Process

1. Develop the test plan.

2. Set up the test environment.

3. Conduct the evaluation.

4. Writing the report and recommendations.

Developing the Test Plan

First you need to have a clear definition of your sites goals and purposes.

Understand what the client hopes to accomplish and their expectations of the testing.

Create scenarios and profiles of your targeted users and what their goals and expectations may be.

Draft a list of reasons for the testing and results you want to extract at this level.

Consider what kind of data are you looking for and what questions you need to ask the respondents to collect when compiling this data.

Draft a list of questions that specifically address the site goals and purposes.

Design tests that are specific to those goals and purposes.

Suggestions of Questions and Tasks.

Ask test subjects to locate and download a particular file.
Look up information on a particular topic.
Identify which elements on a page are actionable/clickable.
Please describe your experience when trying to complete x task.
Design a scavenger hunt, asking your testers to find particular items and information.

Questions

General

What was the testers first impression when they entered the site?
What do the users hope to accomplish once they are on the site?
What were the testers original expectations of the site? 
Were their expectations met?
If not, how do they think the site could be designed to better meet their expectations?
What do they think could be added that is missing?
Describe the first items you notice on the page.
How long does it take for the pages to download?
Is the site easy to learn?
Do you find the site a pleasant place to be?
Does the user feel that they could accomplish what they expected to accomplish in an easy and efficient manner?
What things would you change about the site?
Does the site allow for people with disabilities? (Color blindness, poor sight etc.)

Tools

Does the site operate as they expected?
What can be changed to make it easier to do/use?
Should there be a help page or can you include help right in the pages?
Are the names of the links easy to comprehend?
What do you expect to find behind a link?
Are the tools easy to locate on the screen?
Are the same fundamental tools available from page to page?
Are the command names easy to understand (site map, search, etc)?
Are the fundamental menu items in the same location on every page?
Is the layout consistent throughout the site?
Is the processing speed for interaction fast enough?
Does it take too many keystrokes to accomplish a task?
Do you feel engaged when you are using the site or is it too slow?
Does the site give you enough feedback?

Content

Does the way the Web site categories are set up make sense to the tester?
Are they intuitive and did they find the information they expected too under these categories? 
Is there too much information/too little? (The mind can only process so much information at once. Throw to much content at them and they will walk away in confusion.) 
Did they find the content meaningful.
What kind of content were they looking for and did they find it? 
What other content do they think should be added? Is there anything that they thought was irrelevant to the site? 
Did they think that the pages were organized in such a way that they could easily find what they were looking for?

Aesthetics

Is the mood and the tone of the design consistent with the theme of the site?
Does the design enhance or impede the function of the site? 
Do the graphical elements impede the download speed?
Do the graphical elements help prioritize the information? (That is the only reason to have them on the page.) 
Do multi-media features enhance or distract from the site?
Do you find the screen cluttered? 
Do the icons/graphical elements communicate the ideas/words they were used for?

Participants

It takes only five users to uncover 80% of high-level usability problems.
10 to 12 people will catch 95%. 

It is important for you to understand who is the target audience of your site when you choose your testers. The results have no relevance if the testers are not reflective of the end users.

If the site being tested is one of a general nature, not one designed for a niche audience, make sure that you include testers who:

Testers

Who are representative of the ultimate users of the site.
Have different levels of computer competencies.
Are of various age groups.
Come from different culture groups.
Different genders.

Facilitators

The facilitators must have a clear understanding of the potential users. They must also know:

Who is the user of this site going to be?
Is this an intranet, extranet or an Internet site? 
How skilled are the users?
Will they get any on site training?
Is this site targeted to a particular user group and what is their level of competence?
Is this site one where the user is looking primarily for information, entertainment or are they there to accomplish some sort of task or a combination of all of the above?
Depending on the nature of the site (internet, extranet, intranet) recruit some testers outside the team that helped develop the site.
Also include some of the developers so they can see users reaction to the site.
If users know that testers are not responsible for site design they may be more honest in their appraisal.

Ways of Collecting Data

Observation

Watching the testers behavior is one of the most effective tools of the data collection methods. Often their body language is more revealing than what they say. It is not always easy for testers to articulate their reactions to particular elements of the site.

Explanation

Asking question of the testers, first individually and then in a group discussion setting. Sometimes individuals are reluctant to speak up in a group. However, it is useful to also have a group discussion because the testers memory of an experience may be triggered by another testers comment.

Thinking Aloud Data

If you encourage your testers to think aloud, you often get a more spontaneous impression of their reaction. It may be a better indication of your testers experience.

Questionnaires

Well thought out questionnaires provide a checklist of items for the user and an orientation of the sites purposes and features.

Video

Filming videos can be a powerful way of recording some of the subtle responses users have to the site. Often people reveal more through their body language then they do through their words. Video can also be viewed after the testing is over, giving the evaluation team a chance to review the testing process. Clients may want to see the testing and it helps to them to understand the process better and provide them with an opportunity to view the results from a more direct perspective.

Introduction                                                   Test Day 

Interface Design 
Usability Testing
User-Centric Philosophy
Why is Testing Important?
Web Design Mantra