A year later, working with another organization, I was behind the two-way mirror myself and began to feel the itch of judgments accruing. Having had the earlier unhappy experience of observing a researcher dismiss focus group participants, I was able to recognize my own feelings and focus on trying to feel connected with the people in the room. Still, I much prefer to do research in a manner that fosters feelings of empathy rather than a methodology that requires making an effort to overcome apathy and disdain.
Gerry Gaffney: Right to Be Wrong
I was researching, with my colleague Patrizia Bordignon, how people thought about and dealt with home renovations.
One of the methods was a diary study (“cultural probe”), and we had carefully recruited—or so we believed—a small set of participants with whom we would work for several weeks.
Warning bells sounded fairly early with one of the participants, who showed up very late for the initial briefing. These things can happen, so we ran a separate briefing session for him, gave him his kit of reporting materials (camera, diary and so on), and sent him on his way. Let’s call him Mr. W.
Three days after the briefing, we telephoned each of the participants. It’s a good idea to do this to remind people about their commitment, to redirect as necessary, and to address any issues that arise. All our participants were on track, with the notable exception of Mr. W, who seemed somewhat evasive in his answers.
At the end of the first week, we visited the participants. Again, this is good practice; it’s an opportunity to see how the data is being gathered, and what changes might be needed to the process. We also use that opportunity to make a partial payment to the participants, which can serve as a nice motivation.
We were delighted with what we saw. Participants had kept bills and receipts, photographs, and magazine clippings; they showed us their renovations or their plans; and we were confident that we were getting plenty of highly relevant data.
When we visited Mr. W’s house, however, it was evident from the first moment that his home was different. The front gate didn’t work properly, and the hinges squeaked; the garden was unkempt, and the house had an overall sense of dilapidation. Inside it was a similar story. Every room was in dire need of immediate restorative work, but none was evident. I felt a tad depressed as we drank tea from cracked mugs and listened to Mr. W list the things that needed to be fixed.
PHOTO BY GERRY GAFFNEY
Mr. W was not an enthusiastic renovator. His house represented a series of urgent and necessary tasks, none of which had been tackled.
It looked like we would collect no useful data from Mr. W, and as we traveled back to the office, we talked about our disappointment and reexamined our recruiting strategy.
However, as we moved into data analysis, we found ourselves referring quite often to Mr. W, and gradually came to realize (no doubt this should have been obvious earlier) that Mr. W’s world was, in fact, directly relevant to our project. While the enthusiastic renovator was undoubtedly a key consideration, the unenthused or reluctant could also present great opportunities. Their needs and goals were different, their attitudes were different, and the way that we would design for those characteristics was different.
In many ways, Mr. W was an ideal participant specifically because he didn’t fit our expectations. He challenged the underpinnings of the project, and he forced us to examine our design decisions in a much more rigorous fashion.
I often reflect back on this experience when I’m doing user research, and I specifically watch out for negative reactions and experiences, because they can often teach us things that we might not otherwise learn.
I still believe it’s important to recruit carefully, but perhaps we should be more open to the idea that the “wrong” participant is sometimes precisely the right one.
Leo Frishberg: No, We Really Meant the User
Our team was embarking on an ambitious, multi-country contextual inquiry effort. We had created our sample cells, identified the right industries, established a great relationship with our sales team, and done All the Right Things Up Front to make the effort a success.
Working from Oregon with prospective participants in Bangalore is never an easy prospect; introducing a new research technique at the same time raised the stakes.
Several weeks in advance of the interviews, we had contacted our sales team in-country explaining the process: we needed individuals who were currently working with our equipment and willing to let us observe them working in their labs, in situ.
Everyone claimed to understand. We arrived in-country, and I confirmed the arrangements, on the telephone, with the sales team. “Yes,” they confirmed, “we’ve found exactly who you are looking for.”
We arrived at our first interview in a gorgeous sparkling new office building and were led to an upstairs glass-enclosed conference room. Presently, a manager-type entered, clearly expecting to hold court with us.
I began the discussion with a recap of our expectations and a quick sanity check with the individual.
“So,” I began, “We are looking forward to working with an actual user in the lab. Are you going to work with us today?”
“No,” he said, dismissively. “I’m the team manager. I can tell you everything that’s wrong with your equipment. I’ve polled the team and have collected answers from all of them.”
It’s at times like this, having flown 10,000 miles, having spent as much time as I had setting things up, that I lose a part of my conscious brain. I could feel the anger rising, but I knew that wouldn’t help improve the situation.
Instead, I signaled to the sales guy sitting next to me that as far as I was concerned, the interview was over, and we could pack up to go to our next meeting. Here’s where the details get sketchy, but I know he said something in English to the manager, and whatever magic words he uttered, the manager smiled and nodded, suggesting he could definitely get the lead engineer to help us. He left to find the guy.
A few minutes later, the engineer entered the room, curious as to what the group was doing there. We began the front part of the interview, and it was clear he was the right guy. After explaining what we were planning to do, we asked if he had any questions or needed any further explanation.
‘No,” he said. “You want to see me work with the equipment. I don’t have anything to do today, but I could show you what I was doing last week.”
That was fine, we agreed.
“OK. Just give me a few minutes, and I’ll bring you back. . . . ”
Imagining what he might be doing in those few minutes, I stopped him. “Uhhh, what would you be doing between now and then?”
“Oh,” he assured us. “I’m just going to get the equipment all set up.”
“Great!” We practically shouted. “That would be great! We’d be happy to watch you do that!”
He smiled, as if hoping we had taken our medication. “I’m not sure what you’ll find so interesting about my pulling the machines off the shelf, but come on along.”