Undercover Boss is User-centered Design at its best
I'm amazed, though not surprised, how revealing and informative field trips can be. Three episodes into Undercover Boss, a reality TV series by CBS, it becomes obvious how easy it is for upper management to get disconnected from the base, for a CEO to know little to nothing about its employees and their work conditions, for a corporate policy being counterproductive. High-level assumptions don't live up to their expectations, prove to be ineffective, are misinterpreted or simply not useful at all.
Undercover Boss logo
For all that haven't seen the show yet, each episode of the show features a senior executive at a major corporation working incognito as a new entry-level hire in his or her company for one week, to find out how the company really works and identify some of the unsung heroes among the employees.
Setting the stage
The show usually starts with the CEO of the corporation entering a board meeting revealing his plan to go undercover. I enjoy seeing the directors' mouths dropping with shear fear in their faces of the possible outcome and its consequences. Could it be that they know something might be wrong? Or not perfect?
Learning the hard way
The undercover boss assumes a new identity and with the excuse of being filmed for a documentary on a person working entry-level jobs in several different industries (or similar) starts digging dirt for Waste Mangement, cleaning dishes at Hooters or serving coffee at 7-Eleven.
We see a woman that has to pee in a can for lack of time to make a toilet stop en route, store lights that cannot be replaced within a 30-day window, a restaurant manager having his employees competing in degrading games so the winner can leave early. All examples of failed management and policies with huge potential for change!
We also live with some incredible and heartbreaking stories of inspiring people that are on dialysis or waiting for a kidney transplant but still work hard and motivated. A guy that works long nights to finance his studies but without perspective within the company, something is not going right here.
All these people are invaluable to any company but unfortunately mostly overlooked, another missed chance of finding hidden potential. Some of them with incredible skills that are simply not needed at their current job position but might be essential in another. Where to go?
Overall, the experience is eye-opening and a dedication to humanity, the revelation that every "position" is held by a person, that every "customer relation" is an interaction between people, that every "policy" directly or indirectly affects the life of an employee.
Catharsis
Finally the CEO, overwhelmed by his experiences, seems cleansed by his experiences and vows to correct the disgrace before it becomes outrage. In the show the boss reveals his true identity to the employees and like a big family they watch clips from the show, laugh, cheer and live happily ever after.
Effect on UX
Field studies, a widely accepted and practiced method of user-centered design, seems to find its way into the conscience of the American (and British - the show is based on the 2009 British Channel 4 series of the same name) viewers. It appears as the obvious thing to do.
My very positive take-away from the show is that without knowing, managers, directors and CEO's will ask for more involvement of the base (real people), will involve field studies or user testing in their methodology which surely won't disappointed them. Their insights will improve the live of their employees (the society) and gain considerable business advantages over their competition in the long run.
Am I dreaming or reaching for the stars? Let me know what you think?
Remote User Testing – A Comprehensive Guide
No doubt, user testing increases the usability and acceptance of your website and can/should be done as early as possible, preferably during Prototyping.
The following blog entry discusses the advantages & disadvantages of remote user testing, describes time estimates & costs and explains how a session looks like using Techsmith's UserVue.
General Discussion
Advantages
- Reduced (or no) travel time and expenses.
- Higher exposure through easy screen sharing (managers can sneak in easily).
- Actual user environment, familiar and comfortable.
- Possibly fewer drop-outs.
Disadvantages
- Facilitator not physically present (degree of separation can be challenging).
- Can't see facial expressions or non-verbal cues.
- Difficult to build rapport and trust.
- Difficult to control environment.
- Possibly technical difficulties (firewall, etc.).
- Setup and use of software or usability lab might be challenging and requires a liaison.
Time estimates & Costs
Task Breakdown
The following time estimates are to be taken with a grain of salt, they can change significantly up or down depending on the project size, experience of the team and infrastructure.
- Preparation (18 hours)
- User screening: 8 hours
- Task creation: 8 hours
- Environment: 2 hours
- User testing (10 hours)
- 2 hours per user
- 5 users per round
- Post-test (32 hours)
- Test report: 16 hours
- Implementation: 12 hours
- Communication: 4 hours
Summary: Our Time
1 Round = 60 hours
2 Rounds = 110 hrs.
3 Rounds = 160 hrs.
+ User comps
+ Time additional observers
How it works
Software
UserVue
UserVue is a Remote user testing software that enables a Facilitator to remotely observe a Participant using a phone line for communication. Multiple Observers can passively join the Session and share their observations with the Facilitator.
Morae Manager
Morae Manager uses the collected data (observation markers and notes, video, keyboard and mouse inputs) to analyze and calculate task times, error quotes and other common measurements.
Session
A Session is initiated by the Facilitator. Invitation emails are being sent to the Participant and Observer(s).
At the announced time all the involved parties need to download a small software bundle that allows them to connect to the UserVue software. The Facilitator then calls the Participant and gives instructions on how to start the Session.
After the session has ended the installed software bundle will be removed from computers of the involved parties.
Participant Requirements
Operating System
Microsoft Windows 2000 and Windows XP or later version of Windows.
Web Browser
Internet Explorer 5.0 or later, Firefox 1.0 or later.
JavaScript
Enabling JavaScript is recommended.
Security?
All communication with the UserVue Web site is performed over an encrypted Secure Sockets Layer transport mechanism (HTTPS). All session data is encrypted with a 128-bit Blowfish cipher as it is sent over the network.
Can anyone "eavesdrop" on my session?
All session data (including audio, video, chat data, etc.) is encrypted with a 128-bit Blowfish cipher as it is sent over the network. This makes it exceedingly difficult for anyone to intercept and observe session data.
Are copies of my session data stored anywhere?
No copies of session data are stored on any server. The only recording happens directly on the facilitator's computer. Session data may pass through TechSmith's servers to facilitate firewall and NAT traversal. However, this data is never stored. Also, this data is undecipherable as it is in an encrypted form as it passes through TechSmith's servers.
What are your experiences with Remote User Testing?
References
- Bolt, N.; Guide to Remote Usability Testing; http://www.ok-cancel.com/archives/article/2006/07/guide-to-remote-usability-testing.html
- Gough, D., Phillips, H.; Remote Online Usability Testing: Why, How, and When to Use It; http://www.boxesandarrows.com/view/remote_online_usability_testing_why_how_and_when_to_use_it
Benefits & Principles of User-Centered Design
We, the people, have been around for quite some years now. Computers, software, applications and the web not so much. Therefore it is clear that applications have to adjust to the people and not the other way round.
Many design principles have developed throughout the decades, but the main difference of user-centered design to others is that
UCD tries to optimize the user interface around how people can, want, or need to work, rather than forcing the users to change how they work to accommodate the system or function.
Purpose of UCD
UCD answers questions about users and their tasks and goals, then use the findings to make decisions about development and design.
UCD seeks to answer the following questions:
- Who are the users of the application?
- What are the users’ main tasks and goals?
- What are the users’ experience levels with the application?
- What functions do the users need from the application?
- What information might the users need, and in what form do they need it?
- How do users think the application should work?
Benefits & Return of Investment
- Increased usability
- Higher degree of customer satisfaction
- Continued business
- Higher revenues
- Project management optimization
- Focus on important functionality early
- Unforeseen user requirements
- Reduced costs
- Training costs
- Help-Desk calls and service costs
UCD Principles
- Focus on users’ needs, tasks and goals
- Spend time on initial research and requirements
- Identify your target audience and observe them (accomplishing their tasks)
- Let users define product requirements
- Emphasis on iterative design process
- Evaluate system on real target users
Summary
Nobody could state it simpler than Susan Dray: "If the user can’t use it, it doesn’t work".
UCD Method: Focus Groups
What it is
A focus group is a moderated discussion that lasts about two hours and covers a range of pre-selected topics.
In traditional focus groups, a screened (qualified target audience) group of respondents gathers in the same room. A moderator guides the group through a loosely structured discussion that probes attitudes about a client's proposed products or services. The moderator is typically given a list of objectives or an anticipated outline. Additional questions might serve to initiate open-ended discussions.
When to use
Requirements phase
Cost
High
Output
Non-Statistical
Sample Size
6-9 users
Pro's / Gains
- Discover what users want/desire/belief
- Observe group dynamics and organizational issues
- Show users spontaneous reactions and ideas
Con's / Disadvantages
- Don’t trust what people say or pretend to do
- Possible bias through specialized groups
How to perform
- Select representative participants.
- Identify problem area (what you want to learn).
- Prepare a script for the moderator to follow.
- Hire a skilled moderator (facilitator).
- Allow flexibility during the test to keep the discussion flowing.
- Tape and/or observe the test.
- Create good notes of the test.
References
- Usability.gov; Focus Groups; http://www.usability.gov/methods/focusgroup.html
- Nielsen, J.; The Use and Misuse of Focus Groups; http://www.useit.com/papers/focusgroups.html
- wikipedia; Focus Group; http://en.wikipedia.org/wiki/Focus_group
User-Centered Design (UCD) Methods: Comparison and Overview
This is the first in a series of blogs describing User-centered Design Methods. My goal is to summarize my experience, insights and findings across multiple literature and compile them into easy and quick to digest pieces for you to consume. I want to encourage you to comment your own experiences and give me feedback on why your company applies certain methods differently or not at all or something else altogether.
I personally don't like the term Usability too much, it's an empty buzz word. It means SOMETHING to everybody but isn't scientific enough to be taken serious. It's often interpreted wrongly and purely misunderstood by most. It's kinda like Psychology, we know it is important to understand fundamental human behavior, their problems and remedies, but I wouldn't pay a dime to go to a Psychologist. But who knows, just as Psychology got its scientific relevance and acknowledgment - partly maybe through the 'invention' of the IQ - hopefully Usability rises up to similar levels (Jeff Sauro offers interesting metrics via SUM (Single Usability Metric).
That's why I like the term User-centered Design. It works wonders with Project Managers and the-like, probably because Design is such an important term in their daily work. And when asked about Usability testing I can conveniently point out that this is only one tool of many in my UCD toolbox. But the really important sales trick is to know which UCD method is best used at what time in the project management cycle.
The following chart compares the most common user-centered design methods, outlines their cost and shows when to use them:
Overview of user-centered design methods
| Method | Cost | Output | Sample Size | When to Use |
|---|---|---|---|---|
| Competitive Studies | Medium | Stat. & Non-Stat. | 5 | Requirements |
| Focus Groups | High | Non-Statistical | 6-9 | Requirements |
| Field Studies | High | Non-Statistical | 2-3 | Requirements |
| Heuristic Evaluation | Low | Statistical | 2-3 | Design |
| Paper Prototyping | Medium | Stat. & Non-Stat. | 5 | Design |
| Card Sorting | High | Statistical | 15-20 | Design |
| Participatory Design | Low | Non-Statistical | n/a | Design |
| User Testing | Medium | Stat. & Non-Stat. | 5 | Design & Evaluation |
| Surveys | Low | Statistical | 20+ | Requirements & Evaluation |
| Interviews | High | Non-Statistical | 3-5 | Requirements & Evaluation |
| Server Traffic Log Analysis | Low | Statistical | n/a | Evaluation |
| Search Log Analysis | Low | Statistical | n/a | Evaluation |
Not long ago, after having completed a full project management cycle (requirements, design, implementation and evaluation) the PM proudly announced to perform a Focus Group with his stakeholders. Showing the ready application, he thought, would surely impress them and lead to valuable feedback for the next milestone. This impulse isn't uncommon but has to be fought before it becomes reality. Does he really want to produce MORE and EXPENSIVE requirements? Because that's the output of Focus Groups. Wouldn't he be better off running 2 iterations of User Testing to reveal usability issues or a Survey to receive input from outside the development environment?
Recommendations
- Requirements:
- Competitive Studies
- Interviews
- Field Studies
- Design:
- Heuristic Evaluation
- Paper Prototyping
- User Testing
- Evaluation:
- Surveys
- Server Traffic Log Analysis
- Search Log Analysis
- User Testing
- The Usability effort is NOT proportional to the size of the project. Bigger projects spend less percentage on UCD with same effort. Regardless, as a rule of thumb assign 10% of the projects budget for UCD.
- Faster iterations of prototype design require less testers
References
- Fidgeon, T.; User-centered design (UCD) - 6 methods; Nov. 2005; http://www.webcredible.co.uk/user-friendly-resources/web-usability/user-centered-design.shtml
- IBM; User-Centered Design Principles; https://www-01.ibm.com/software/ucd/
- Nielsen, J.; Field Studies Done Right: Fast and Observational; 01/20/2002; http://www.useit.com/alertbox/20020120.html
- Nielsen, J.; How Big is the Difference Between Websites; 01/19/2004; http://www.useit.com/alertbox/20040119.html
- Usability in Practice: Three-Day Intensive Camp; Nielsen, J. et. al.; April 2006; Proceeding, Usability Week 2006
For which generation are we designing for?
I just came back from a webcast held by Human Factors International titled "Who Are We Designing For? The Generation Dilemma". It reminded me that even though most designers - me included - belong to the Generation X (or maybe especially because of that) we need to understand and recognize behavioral differences between generations when designing web sites.
Sidenote: This generalization must not prevent us from continuing to identify and define our target audience through personas or applying all the other UCD principles based on representative users. It is solely meant to raise awareness that we as designers need to be aware of these differences.
So, what are these generations?
"Baby Boomers"
- 78 Mio (US) / 1.11 Billion (worldwide)
- born between 1943-1960 (age 65-48)
- think of technology as a tool
- computer at work or at home (often not the latest model)
- use computer mainly for email or work-related activities
- use sites that help save time or serve relevant information
"Generation X"
- 55 Mio (US) / 1 Billion (worldwide)
- born between 1961-1981 (age 47-27)
- are technology savvy and career-focused
- keep their computer (desktop) in a separate room or office
- start their day with practical activities like reading news or checking stocks
- utilize computer for work but still keep balance of life and work
- use sites for practical tasks like online banking, news, travel preparation
"Net Gen"
- 80 Mio (US) / 2 Billion (worldwide)
- born between 1982-2000 (age 26-8)
- technology is inherent and part of life
- have their computer (laptop) by the bed
- start their day checking social activity online
- look for and do things that make them laugh, might be silly
- attention span is short, impatient
- use social networking sites like Facebook
How does the knowledge about these generational differences affect our designs?
The same guidelines for accessibility and usability like consistency or efficiency hold true but it is to be emphasized that certain guidelines must be enforced stronger for some generations while others can be loosened up. Here is a rough overview:
Baby Boomers:
- Simple look and feel
- Safe to use, more error-prone than usual
- Navigation in predictable places
- Hierarchies of information
- Step by step instructions
- Provide options to enlarge font sizes
- Things that don't move, flicker, or play automatically
Generation X:
- Focus on quality content
- Provide 'do-it-yourself' tasks
- Give control and allow customization
Net Gen:
- Content has to be attractive, innovative, can be quirky
- Design can be highly interactive
- Audio and video is not only accepted, it's expected
- Entertainment is important
- Avoid pictures of elderly people (yes!)
How can a design satisfy all generations?
The answer is (as always): it cannot. Designs have to be focused on and implemented towards your key audience. A university web site needs to satisfy its customers, the students. The press, teachers and parents are without doubt important as well, but without satisfied students the university doesn't even have to think about getting the press to visit the site.
There are ways to generalize your site's content though:
- Identify the user (login) and present a customized UI
- Identify the content (landing page) and predict its likely customer
- Identify user preferences by explicitely asking
About Me
Michael Gaigg is Lead UI Engineer in Esri's Professional Services Division.
He has been designing map application for over 10 years and can't stop thinking about improving them.
Categories
- Accessibility (16)
- Code Samples & Tips (15)
- Conclusions (6)
- Design Guidelines (12)
- Designing Map Interfaces (5)
- Go figure (9)
- Good News (25)
- Here and there (29)
- Map Applications (12)
- This Week's Highlights (53)
- Usability & UCD (27)
- Web Design (28)
Popular Posts
- 10 Webpage Design Lessons learned from a trip to Las Vegas… seriously!
- Web Content Accessiblity Guidelines (WCAG) 2.0: Overview and Structure
- Powerpoint Wireframe Stencils as Free Download
- Go figure: 10 Comic Strips that have Something in Common
- Job Posting: User Interface (UI) Engineer at ESRI
- Dashboard Design
