Focus Groups

 

I’ve written often about the concerns I have with research techniques.  This column was inspired by my attendance at a focus group.

 

I was just asked to attend some focus groups and I was reminded of the dozen or so reasons I don’t like them.

 

First, rich and busy people usually won’t attend, skewing data to a narrow demographic.  This would taint research in a decent undergraduate social science class.  Worse, focus group companies know where they can get repeat attendees on short notice.  These are often students, the unemployed or at least people who live near the focus group facilities.  All of this taints the data, too.

 

In this particular focus group I feel a twinge as soon as the leader walks in the room.  She is too talkative, as if trying to entertain a client, not conduct research.  Conducting professional scientific research is a learned skill.  Notice I use the term scientific and do not make a distinction with social science.

 

About fifty years ago Heisenberg said the instrument used in a laboratory experiment affects the results of the experiment—the Heisenberg uncertainty principle.  A thermometer used to measure temperature changes the temperature of the thing it’s measuring.  In 1962, Kuhn pointed out that scientists have a tendency to collect data that supports their views and ignore that which conflicts.  He used the term paradigms to describe this phenomenon.  Merton and Feyerabend discussed how science is a value laden pursuit, driven by those performing the science.

 

So the focus group leader is in a difficult position.  Waltzing in like the star of the show, trying to impress, performing for the client behind the two-way mirror or any other false behaviour can make that position untenable and skew the data.

 

A focus group leader should dress and talk one step up from the subjects in the room.  S/he is there for scientific purposes to gather data, not make friends or entertain.

 

Most focus groups begin way too quickly.  The leader should allow discussion to evolve slowly with open ended questions to see what the respondents want to discuss.  Closed, specific questions, especially ones that can be answered with “yes” or “no” or a short sentence won’t reveal as much as an open ended questions encouraging discussion.

 

Closed questions yield what researchers call an aided response.  The questions tell respondents what to talk about.  Open questions yield an unaided response where the respondents can tell researchers what to think about—the way it should be!

 

In the focus group I was in recently, we were testing confidential matters.  But let’s say it was reactions to Candidate X.  One of the first questions was, “What kind of car do you think Candidate X drives?”

 

This sparks a discussion of myriad types of cars.  I’m reminded that some people follow car models more than others.  Some may name one car thinking it’s the sportiest on the market, while another person may name the same model because it represents fuel efficiency, economy or some other attribute.

 

In the end, without knowing what respondents mean by the car models they name, you end up with a mish-mash of information that could mean anything.  “Cadillac” means luxury, high-price, high fuel consumption and perhaps the ability of American industry to compete with anyone on the planet.  Who knows?

 

But if there really were value in the car question, it should have begun in an open ended fashion such as, ”How does Candidate X get around the district or campaign trail?”

 

Respondents might name trains, boats, planes and cars.  If we find out people think the candidate flies around in a private jet, that could be a problem.  If cars are really relevant, after an open discussion of other types of travel, the closed ended question could be asked about what kind of car the candidate uses.

 

In the focus group I watched, one respondent got up on his hind legs and said he didn’t think the candidate used a car.  It takes guts to challenge an authority figure and say the question is wrong.  If a respondent does that, it’s a very powerful message that you could be on the wrong track.

 

Next the leader held up pages from the candidate’s web page, passed them around and asked for a review.  That’s not how people use web pages, so it’s hard to tell what was being tested.

 

Next, the leader passed around literature that the candidate regularly sends out.  The question was, “What do you like about the brochures, householders and other communication?”

 

I have two problems with this line of questioning.  First, it misses the opportunity to find out if anybody remembers getting anything in the mail from the candidate.  People may feel boxed into saying they received and remembered the material, when the most valuable information might be that they didn’t.  Next, what if people hate the literature?  Asking what they like about it, cuts off the discussion about what they don’t like.

 

People like to be cooperative when being paid, so you have to be careful they aren’t so cooperative that you don’t find out what they’re really thinking.

 

Next came the Barbara Walters question.  She was famous for asking an interviewee, “If you were a tree, what kind of tree would you be?”

 

In this session, respondents were shown pictures of all kinds of people—young, old, various races, both genders and so on.  As they are looking at the pictures, the leader asks, “If the candidate’s campaign were a person, what person would it be?”

 

Not only is the question a bit odd, it’s hard to tell what people mean when they pick a picture of a fit, muscular looking man.  Does mean the campaign is intimidating, full of thugs or has staying power for the long run?  Next, respondents start answering questions that weren’t asked.  One picks the construction worker because the candidate comes from a part of the district where there’s lots of construction.  Another picks the young Asian woman because the district is becoming more multi-cultural. Some seem to be picking people they find attractive or would like to be with, not who embody the campaign.  This ends in another mish-mash of unusable information.

 

I innocently asked the client where the pictures came from?  It turns out employees of the research company picked them.  They may use them every session, for all I know.  Regardless, it does not make a lot of sense to me to have respondents judging pictures picked either at random or purposefully by a research company to make their sessions go more smoothly.

 

Next, the leader asks what does the person in the picture do for a living, and what does the person do on weekends?  (I assume the construction worker works in construction).

 

Ironically, after going to all this trouble to stimulate fairly irrelevant discussion, the leader cuts off dialogue by asking if anyone has any final thoughts?  When the leader leaves the room to get more instructions from the client, I listen in to the continuing discussion in the focus group room.  It’s actually a better discussion, unaided by the leader.  One older woman began waxing nostalgic about the district, the party and the candidate.

 

So, what’s the right approach?  There are lots of them.  Credible scientists use several different methods and compare them to obtain more reliable data.  Polls, questionnaires, elite interviews with opinion leaders and lay-elite dialogues to see where there are gaps between those in the know and regular folks can all help.  So can old-fashioned research.  Online databases now make is really easy to search everything from newspapers to academic journals.  For all a campaign knows five distinguished academics and journalists have written 10,000 excellent words on the topic over the years.

 

Focus groups can be helpful, if run properly.  But they also need to be augmented with sound, old-fashioned research, preferably in libraries, surrounded by musty books.

 

 

 

On Line At New York

Outside the Helen Hayes Theatre, waiting for the doors to open, a very agitated male teenager passed on the sidewalk.  He was walking briskly and talking in a loud voice with himself.  He was also gesticulating, and in so-doing was using up all the free airspace in all directions.

“What’s the matter with him?” asked the woman on the line ahead of me.

“’T’s an angry young man.  What can I tell you?” replied her husband, as indulgent as he could sound.

This was my introduction to blasé New Yorkers.

Slowly the line shortened and I was up at the wicket at Lincoln Center.  I was mostly interested in seeing the spectacular setting and inside of the 16 acre site—home of the New York Philharmonic, the Metropolitan Opera and the New York City Ballet.  I was ready to listen or watch.  As it became my turn at the wicket, I asked:

“What’s playing?”  

Wicket keeper:  “Joo-lee-us Cee-ah.”

I guess I paused for a New York beat too long, so wicket man added:

“’Ts an Apra”

I bought a ticket and achieved all my goals.  

I would regularly walk on near the Ed Sullivan Theatre, home of the old David Letterman Show.  I watched the show on hundreds of occasions, and was in the studio audience several times.  I’d  occasionally stop in to chat with Rupert Gee, the deli-operator who was also a regular guest on Letterman.  Rupert is as he appears on TV—a little startled with the attention, but sincere.

On one occasion there was a big crowd extending down the block.  I thought perhaps Rupert was having a sale of special deli sandwiches.

No, it was a lineup to see someone coming in or leaving the stage door of the theatre.  This is where the guests came and went.  This time I was the talkative New York street performer:

“Who’s all this for?”

“Wa-keen Phoenix” said my fellow street performer.

This was early in Mr Phoenix’s career.  I guess after all my trips to New York I still paused a New York moment too long, because an older woman in the crowd showed some empathy:

“Ever since Frank Sinatra died, I don’t know any of them.

Way out in the Rockaways, I ran out of gas.  My GPS told me there was a gas station close by and so I locked my wife in the car and began walking.  The GPS was wrong and the gas station had closed.  I thought of lining up at a bus stop, but didn’t know how often busses came or where they went.  I lined up at a red light and thought I’d hitch hike.  Up came a convertible Thunderbird.  I could see inside and the driver could see me.  I explained my situation and asked for a ride.

“Sure” he said as he started moving a mountain of stuff off his passenger seat to the small spaces in the back of the car and between the seats.

“Oh, don’t worry, if you see my gun, I’m a cop.”

I pointed to a spot of blood on my shirt, and said:

“Great, and if you notice the blood on my shirt, I’m not an axe murderer, I just reached into my shaving kit this morning and cut my finger on my razor.”

“No problem.  Get in.”

Unlike so many New York stories, this man was a cop, seconded to the Triborough Bridge, and I wasn’t an axe murderer.  Many people go to the Big Apple to reinvent themselves.  Others remain who they actually are.      

The City of the Future

 

I’ve always thought of Seattle as a symbol of modernity.  Growing up in Vancouver, I viewed the Space Needle as an example of American prosperity and progress.  It reached toward outer space while the monorail sleeked along the ground toward the future.  I now live in Toronto, but visit Seattle when I can, to keep in touch with the future. 

  

The world almost missed getting these two lasting symbols from the 1962 World’s Fair.  The Fair was originally going to be called The Festival of the West, as hard as it is to mix cowboys and technology—Broncos and Boeing.  More futuristic heads prevailed and the fair became known as the Century 21 Exposition with the motto “Living in the Space Age.”  The Fair turned a profit and left a legacy of civic amenities and revitalization. 

 

Edward E. Carlson, one of the great civic boosters of any time and any city, extracted a victory within a victory.  He famously drew the Space Needle on a napkin, obtained funding for the project, started it a little late, but finished on time.  The Space Needle retired it’s debt in 18 months.  It now has a new pod for more weddings and parties.

 

My book Safer Cities of the Future is a study of better urban experiences through better design.  I began the book with a picture of the Space Needle and the Seattle skyline.  I recounted the story of Mr Carlson and the world’s fair as an example that we get things right occasionally.

 

It’s a lifetime later and Mr. Carlson and Seattle should be remembered for success and foresight.  But now we have new challenges — terrorism, a sluggish economy, severe weather events and little faith in space, the future, or in anything.  This is where Seattle can lead the world again.

 

Seattle’s All Hazards Mitigation plan is one of the best I’ve studied. It offers a very realistic view of the risk of floods, earthquakes, and terrorism. The plan notes, for example, that 32.75% of Seattle’s housing stock was built before 1939, an astonishingly high percentage considering that the seismic building code dates from 1992. If there were a serious earthquake, the damage could be enormous and could come at a very high cost. Seattle’s plan notes that property damage claims against the city have totalled as much as $12 million since 2006. In the event of a serious storm or earthquake the cost would be exponentially higher.

 

Seattle’s plan also includes a detailed summary of activities to mitigate the impact of earthquakes. These include upgrades to buildings to correct structural deficiencies, disaster recovery needs for all IT systems, and replacement of older underground cables. And the plan notes that city departments, the police, and public utilities have all received briefings and training in order to prepare them for seismic events.

 

These are all steps in the right direction. But Seattle’s plan could be improved by including specific instructions for residents on what to do or where to go in the event of earthquakes, floods, or other disasters; what to take with them; and even how to ensure the safety of their pets.

 

Don’t get me wrong: no plan is perfect, and there are some very good plans out there that Seattle could borrow from. Auckland, New Zealand, for instance is the only plan I’ve seen that precisely quantifies the economic cost of a volcanic eruption. Kansas City has a unique plan for rescuing citizens’ pets. And Boston’s plan for dealing with climate change is an excellent example for all other coastal cities.
Most plans leave much to be desired.  But Seattle’s is a template for progress.  

Today in History: July 11th

2005: Deh Cho First Nations agree to a deal with the Canadian government to get participation in the environmental assessment and regulatory review of the $5.7 billion Mackenzie Valley Pipeline gas project.

For more on negotiations: Buy ‘Tough Love at the Table: Power, Culture and diversity in Negotiations, Mediation & Conflict Resolution’

1991: Carla Hills, American trade representative, says the North American free-trade deal NAFTA will not endanger auto pact or harm Canadian culture.

For more on politics: Buy ‘Political Conventions’

1955: Seven American teenagers die on Mount Temple, near the Valley of the Ten Peaks near Lake Louise, Alberta. It is Canada’s worst single mountaineering accident.

For more on crisis management: An Ounce of Prevention

Today in History: June 30th

1984: Pierre Trudeau steps down as Prime Minister of Canada and retires from politics.

1997: The United Kingdom hands over control of Hong Kong to the People’s Republic of China.

Political Columns – Pugnacious Pierre — Trudeau

1912: The deadliest tornado in Canadian history roars through downtown Regina at 4:50 pm, killing 28 people. In all, 2,500 people are left homeless. Mayor Peter McAra cancels Dominion Day celebrations.

Today in History: June 29th

1995: A department store in Seoul, South Korea, collapses, killing more than 500 people. The tragedy occurs due to a series of errors made by the designers and contractors who built the store and the criminal negligence of the store’s owner. Rescue efforts continue for weeks and one survivor is pulled out 16 days after the collapse.

1995: The American space shuttle Atlantis docks with the Russian space station Mir to form the largest man-made satellite ever to orbit the Earth. This was an historic moment of cooperation between former rival space programs. Daniel Goldin, chief of NASA, called it the beginning of “a new era of friendship and cooperation” between the U.S. and Russia.

1974: With Argentine President Juan Peron on his deathbed, Isabela Martinez de Peron, his wife and vice president, is sworn in as the leader of the South American country. President Isabela Peron, a former dancer and Peron’s third wife, was the Western Hemisphere’s first female head of government.

Today in History: June 27th

1984: Pierre Elliot Trudeau is named the winner of the Albert Einstein Peace Prize for his global campaign to ease East-West tensions.

1969:  U.S. President Richard Nixon begins a starts visit to Canada. He speaks in Montreal, and marks the tenth anniversary of the St. Lawrence Seaway, which he said is a lesson “ how two nations can work together; how they can dream together, and make those dreams come true.”

1949: Louis St. Laurent leads the Liberal Party to re-election  in 21st federal general election, winning 190 of 262 seats, an increase of 65 seats. It’s the fourth consecutive majority for the Liberals.

Today in History: June 21st

1990: An 7.7 magnitude earthquake in Iran kills more than 50,000 and injures another 135,000 people. An estimated 400,000 people were left homeless by the earthquake. Worldwide relief efforts are undertaken, but the Iranian government refuses help from Israel and South Africa. Many relief workers from western nations are sent home after a brief time and before critical assistance can be provided.

1977: A fire at a police lockup at City Hall in Saint John, New Brunswick, kills 21 prisoners and injures seven others, as well as six police officers and a fireman. A prisoner, John Kenney, is later convicted of arson and sentenced to five years in jail.

1957: John Diefenbaker is sworn in as Canada’s first Conservative Prime Minister in 22 years, replacing Louis St. Laurent. He serves as Prime Minister until April 1963.

Today in History: June 18th

1985: Ontario Conservative Premier Frank Miller is defeated on a series of confidence votes by an alliance of David Peterson’s Liberals and Bob Rae’s NDP. Miller resigns the next day, ending 42 years of Conservative rule in Ontario.

1979: During a summit meeting in Vienna, President Jimmy Carter  and Soviet leader Leonid Brezhnev sign the SALT-II agreement, dealing with limitations and guidelines for nuclear weapons. SALT-II thus remained signed, but unratified.

1972: A jetliner crashes after takeoff from Heathrow Airport in London, killing 118 people. Just after its wheels retracted after takeoff, it fell from the sky. An intense fireball from the plane’s fuel supply erupted. The official cause of this accident remains unknown, but it may have happened simply because the plane was carrying too much weight.

Today in History: June 11th

1985: Karen Ann Quinlan died this day from pneumonia after being in a vegetative state for more ten years; she was an important figure in the history of the right to die controversy in the United States.

1983: Brian Mulroney is chosen as party leader by Progressive Conservative delegates in the Ottawa Civic Centre. He gets 1,584 votes, while former leader Joe Clark get 1,325 votes. Mulroney is the first PC leader from Quebec since John Abbott in 1891. Mulroney is Prime Minister from 1984 to 1993.

1955: During the 24 Hours of LeMans car race in France, a car goes out of control and crashes into stands filled with spectators, killing 82 people. Spain and Mexico temporarily ban motor racing. Grand Prix races in Germany and Switzerland later that year were cancelled. A complete ban on racing in Switzerland remains to this day.