No doubt, user testing increases the usability and acceptance of your website and can/should be done as early as possible, preferably during Prototyping.
The following blog entry discusses the advantages & disadvantages of remote user testing, describes time estimates & costs and explains how a session looks like using Techsmith's UserVue.
- Reduced (or no) travel time and expenses.
- Higher exposure through easy screen sharing (managers can sneak in easily).
- Actual user environment, familiar and comfortable.
- Possibly fewer drop-outs.
- Facilitator not physically present (degree of separation can be challenging).
- Can't see facial expressions or non-verbal cues.
- Difficult to build rapport and trust.
- Difficult to control environment.
- Possibly technical difficulties (firewall, etc.).
- Setup and use of software or usability lab might be challenging and requires a liaison.
Time estimates & Costs
The following time estimates are to be taken with a grain of salt, they can change significantly up or down depending on the project size, experience of the team and infrastructure.
- Preparation (18 hours)
- User screening: 8 hours
- Task creation: 8 hours
- Environment: 2 hours
- User testing (10 hours)
- 2 hours per user
- 5 users per round
- Post-test (32 hours)
- Test report: 16 hours
- Implementation: 12 hours
- Communication: 4 hours
Summary: Our Time
1 Round = 60 hours
2 Rounds = 110 hrs.
3 Rounds = 160 hrs.
+ User comps
+ Time additional observers
How it works
UserVue is a Remote user testing software that enables a Facilitator to remotely observe a Participant using a phone line for communication. Multiple Observers can passively join the Session and share their observations with the Facilitator.
Morae Manager uses the collected data (observation markers and notes, video, keyboard and mouse inputs) to analyze and calculate task times, error quotes and other common measurements.
A Session is initiated by the Facilitator. Invitation emails are being sent to the Participant and Observer(s).
At the announced time all the involved parties need to download a small software bundle that allows them to connect to the UserVue software. The Facilitator then calls the Participant and gives instructions on how to start the Session.
After the session has ended the installed software bundle will be removed from computers of the involved parties.
Microsoft Windows 2000 and Windows XP or later version of Windows.
Internet Explorer 5.0 or later, Firefox 1.0 or later.
All communication with the UserVue Web site is performed over an encrypted Secure Sockets Layer transport mechanism (HTTPS). All session data is encrypted with a 128-bit Blowfish cipher as it is sent over the network.
Can anyone "eavesdrop" on my session?
All session data (including audio, video, chat data, etc.) is encrypted with a 128-bit Blowfish cipher as it is sent over the network. This makes it exceedingly difficult for anyone to intercept and observe session data.
Are copies of my session data stored anywhere?
No copies of session data are stored on any server. The only recording happens directly on the facilitator's computer. Session data may pass through TechSmith's servers to facilitate firewall and NAT traversal. However, this data is never stored. Also, this data is undecipherable as it is in an encrypted form as it passes through TechSmith's servers.
What are your experiences with Remote User Testing?
- Bolt, N.; Guide to Remote Usability Testing; http://www.ok-cancel.com/archives/article/2006/07/guide-to-remote-usability-testing.html
- Gough, D., Phillips, H.; Remote Online Usability Testing: Why, How, and When to Use It; http://www.boxesandarrows.com/view/remote_online_usability_testing_why_how_and_when_to_use_it
What it is
A focus group is a moderated discussion that lasts about two hours and covers a range of pre-selected topics.
In traditional focus groups, a screened (qualified target audience) group of respondents gathers in the same room. A moderator guides the group through a loosely structured discussion that probes attitudes about a client's proposed products or services. The moderator is typically given a list of objectives or an anticipated outline. Additional questions might serve to initiate open-ended discussions.
When to use
Pro's / Gains
- Discover what users want/desire/belief
- Observe group dynamics and organizational issues
- Show users spontaneous reactions and ideas
Con's / Disadvantages
- Don’t trust what people say or pretend to do
- Possible bias through specialized groups
How to perform
- Select representative participants.
- Identify problem area (what you want to learn).
- Prepare a script for the moderator to follow.
- Hire a skilled moderator (facilitator).
- Allow flexibility during the test to keep the discussion flowing.
- Tape and/or observe the test.
- Create good notes of the test.