Research Design Issues


I have many concerns about the quality of research conducted in political campaigns, by those governing and by industrialists.  I even see those in the not-for-profit sector spending precious money on unproductive research. 

“Original” research in universities is often only conducted at the doctoral level.  Research that is done often involves relatively unproductive statistical tables or questionnaires that purport to be” scientific”.  Many students find searching through microfilm, microfiche and original texts in libraries to be passé, if they are aware of the technique at all.  This constricted research in universities translates into marginally beneficial or even irrelevant techniques and results in industry and government.      


How is it that the best educated generation the world has ever seen is relying on research techniques that would not achieve a pass in a second year social science course in a reputable university?  In government, and in the political campaigns designed to lead to governing, senior managers are making decisions based on flawed methodology.


But we are in unstable times when we need excellent public policy and politics.  America is polarized domestically and the European Union is beginning to show signs of eventually having similar economic and political clout in some parts of the world.


For those who look to the private sector for leadership and use the refrain of “running the government like a business”—please don’t.  Fully 82% of all mergers and acquisitions in private industry fail to produce new value.  There is a crisis of competence in all sectors, in part because of poor research.


Here are the top ten issues and comments on research techniques and challenges faced both in campaigns and then in governing:



The population is much more sophisticated than they were when the random sample telephone survey was invented.  A telephone call is now an intrusion, especially during dinner time.  Pollsters are experiencing up to 70% refuse rates.  I tell my clients that often the biggest message they are getting is that their constituents refuse to speak to them at all.


Compounding the problem is caller ID which tips people off that it’s a pollster calling.  The moment of silence before the questioner begins speaking is a further tip off, as is the robotic reading of questions from a computer screen.


Perhaps the biggest challenge is that up to 10% of the population has just one hand-held device or phone—higher in the crucial 18-24 age group.  Many will not participant in phone surveys because they have to pay the air time.


People miss-remember dates, events and attitudes—what researchers call “backward and forward telescoping”.   They also tell researchers what they wish had happened, or use answers to researchers’ questions as surrogates for other messages.  The classic example is that far more Americans reported that they voted for President Kennedy after his assassination than could have done so in the closest election the US had had to that date.


Social science is too imprecise to determine that 22.3% of people think or do anything—often referred to as “spurious accuracy”.


Citizens reserve the right to lie to pollsters and reserve the right to park their votes in the undecided category or tell pollsters they will vote for a party or candidate when they have no intention of doing so, in order to temporarily reward or punish candidates.


One joke about polls goes like this:  “If an election were held today, everybody would be really surprised because it’s scheduled for November 4”.  That kind of captures some of the unreality of polls these days.


  • Focus Groups.


Robert K. Merton is the inventor of focus groups.  He also coined the terms “role model” and “self-fulfilling prophecy”.  He disassociated himself from the way practitioners implemented his ideas about focus groups.


The dirty little secret about focus groups is the number of times companies rely on semi-professional attendees whom they know will show up on short notice to fulfill a client’s needs.  Students, the disadvantaged and others who need an honorarium or have time on their hands are often overrepresented.


There are ways to make focus groups more reliable.  What the Harvard Business Review calls “empathic testing” involves using a product or discussing an issue in real life conditions.  Putting respondents around a board table and having a formal focus group leader ask questions is not a normal life experience or venue and the results will thus be forced and false.


Anamatics is similar and involves making the experience realistic and having participants focus on the element to be tested.  Realism in the venue can be addressed by driving respondents around in a van while they listen to radio ads a politician wants tested.  This is closer to how voters would listen to an ad than sitting at a board table.


For TV ads, we have stripped rough cut ads into tapes of the actual TV show in which they will appear.  Testing can occur in shopping malls where hundreds or even thousands of people can view the potential ads and react to them.


For print ads and even editorial content, we have mocked up the copy and inserted it into real newspapers to see how respondents react.  We don’t tell them what we want them to react to, we first want to know if they care to look at the ad or story at all.  That’s the so-called “unaided” response.  If they don’t look or read, we have some valuable information.  Then we asked them to review the ad and get more valuable information in their “aided” response.


Campaigns and sitting politicians use lots of mail.  Direct mail raises money and mobilizes troops.  Newsletters and political “householders” let constituents know what their representative is doing.  But nobody opens the mail or reads a householder while sitting around a boardroom table.  These items should be thrown on the floor in a pile of other mail and magazines to see if anybody bothers to stoop down and pick it up.  If someone does, the next question is whether the political piece is interesting enough to cull out of the pile and read.  If not, that’s a valuable answer in itself.

While on the campaign literature theme, there’s always somebody in political meetings showing a mock up of a brochure or householder who points out that the candidate’s picture or name or other important information is off on the right-hand side “where the eye naturally goes”.  By this time in the meeting, I’m too exhausted from trivia and nonsensical issues to point out that we read from left to right in English, Spanish, French and most other languages prevalent in North America, and only read right to left in Arabic, Persian, and some other languages.  (I wonder where these perceived and received pieces of communication wisdom come from?)


With regard to video and TV production, audiences are very sophisticated.  Most people own video cameras and watch TV dozens of hours per week.  Research has shown that focus group attendees will review the production qualities of ads, rather than the content.  To counter this, advertisements can be mocked-up by a graphic artist and one can then test the voice-over or content separately.


Candidates can test debate one-liners, still pictures for brochures, slogans and any other communication element, without layers of clutter or testing of extraneous elements.


  • Graduated Questionnaires.


Self-administered questionnaires are not used much anymore, but are a valid technique.  One of the best examples of these is the old Bureau of Broadcast Measurement diaries that were mailed to households to survey radio listening and TV viewing.  People often put down their favourite station, not the one they actually watched most.


With telephone or in person surveys, respondents become easily and quickly fatigued with having to choose among:  strongly agree, mildly agree, somewhat agree, agree, mildly disagree, strongly disagree.  What does mildly agree mean, other than the fact that it’s stronger than just agreeing and weaker than strongly agreeing?  How does one compare one person’s strong agreement with another person’s?


The best model to determine the weight to put on a respondent’s report is to see if that person actually changes behaviour as a result.  People often report that they will change voting habits, but actually do not.  This makes their threat to do so a surrogate for other matters that should be probed.


In industry, it’s the same.  I have a telecommunications client which conducts quarterly research to determine how much its customers like them.  The results show that up to 30% of respondents say they are “agree”, “strongly agree” or “somewhat agree” with the notion of switching service to a new company.  Yet for years the so-called “churn rate”—the rate at which customers actually change telecommunications providers (phones, hand-helds, internet, etc.) is under 3%.


It is vital to distinguish between what people actually do and what they say they might do.


  • Elite Interviews.


It may not sound egalitarian these days, but elites are good respondents because of how they became elites—they know their demographic well.  These one-on-one, in-depth interviews can augment focus groups, polling and other techniques.


Who’s an elite?  That’s easy.  Ratepayer groups, condominium boards, religious groups, union leaders and even book club busy bodies all rose to the top of their little heap, in part through knowing what their demographic is like.  They can be a great source of information.



The term, taken from navigation, stands for gathering data from several different sources, or with numerous methodologies.  Where data intersect, results are more reliable.   


Researchers have identified several types of triangulation including:  within-method, between-method, data, investigator, theory and methodological triangulation.  Within-method means two separate polls, perhaps by different companies that say the same thing.  Between method might be a poll and a focus group that produce similar results.  Data triangulation might involve qualitative or quantitative results that are much the same.  If several investigators find out the same thing, that’s triangulation.  Theory triangulation might involve a psychological and sociological explanation of behavior.  Finally, these days, using mixed methods—both qualitative and quantitative—is increasingly the norm to avoid the errors that each alone can produce.


  • Mixed methods.


The distinction between qualitative and quantitative data has been blurred for at least fifty years.  Few branches of any science have the predictability of Newtonian physics.  Current thinking is to engage in a mixture of methodologies, mentioned above.  So, a reproducible poll with a large sample that claims to be “scientifically” accurate might be cross referenced with qualitative focus groups, elite interviews and such that plumb small samples more deeply.          


  • Question formulation.


Average People don’t speak the way telephone researchers do, or the way those who write questions think they should.    It’s hard to imagine anyone constructing a questionnaire where a response could be “some good” which is a common expression in the Canadian Maritimes, or “awesome” as is currently popular.  The California “Valley Girl” response of “gag me with a spoon” was probably not used, even in its hay-day.


  • Telephone interviewers.


In addition to the long pause, script reading and intrusion, some companies balk at long distance charges, skewing data to urban respondents.  For decades, first year social scientists have been warned that telephone surveys obviously only gather information from those with telephones.  Triangulation is the antidote.



Social scientists are supposed to keep notes, tapes and a reflexive diary to examine themselves as a scientific instrument while they are examining other people or issues.  Commercial researchers would rarely do this.  


  • The Heisenberg Uncertainty Principle.


The use of a particular research instrument has an effect on the outcome of the research.  Heisenberg stated, “[o]n the level of molecular research, the presence of the experimenter is certain to affect the results in a way which cannot be measured”.


The mere fact that a pollster calls up respondents has such affect.  Asking about certain topics that the respondent might not be concerned with, puts that matter on the public agenda.  Moreover, researchers cannot control for the myriad other variables in that respondent’s life.


In the end, perhaps my premise is flawed.  Perhaps we are not the best educated generation the world has ever seen.  We have more degrees and a multiplicity of choices in methods, but may lack the clarity and professionalism of previous generations.  Pity, we need that clarity.







Boston’s Wake Up Call

Where I live in Canada, Boston is well known for a few things. First, it’s “Maritimers’ Heaven.” Folks from the Canadian provinces bordering on the Atlantic Ocean, think Boston is pretty special and sophisticated. Boston is also known as a tough town.

Boston’s urban emergency plan and the Mayor’s recent Task Force on Climate Change are a couple of tough documents, befitting a tough town. Boston’s not fooling around or mincing words. Alone among the 100 city emergency plans I’ve studied, Boston notes that it only has food on hand for 3-5 days and many facilities are located in a flood zone. So a severe winter storm or torrential rain could cause widespread hunger.

Everybody talks about the weather, but no one does anything about it.

But it turns out that Boston is actually doing something about the weather, and that makes it a leader in urban emergency planning. The Mayor’s recent Task Force on Climate Change shows the way.   

A combination of sea level rising and soil erosion could spell big trouble for cities on water.

What’s the threat? Boston’s excellent emergency plan notes that the weather in Massachusetts may be more like the Carolinas by the end of this century. Since 1991 most of Boston’s emergencies have been caused by flooding.  There’s been a bad winter storm almost every year in Boston and winter precipitation could rise by 16% according to the Mayor’s climate change report.  One hundred year floods could occur every 2 or 3 years by 2050.  

There are already hundreds of deaths per year in the US because of extreme heat and that could get worse as Boston warms up. When the temperature goes up, crime in the streets goes down, but domestic violence goes up. Heat means more danger for the elderly and those with respiratory problems. Heat means more sedentary lifestyle, diabetes and other diseases.

There is an alarming inventory of vulnerable facilities in Boston, including those which will be needed to respond to emergencies — schools, 1500 units of public housing, 430 miles of roads, half the Centers for Youth & Families, one-third of all emergency shelters and more than 900 critical facilities are susceptible to flooding. Many ambulances are parked on streets and thus immobile in a snowstorm. Even the roof leaks in the Emergency Operations Center.     

Flooding is also a major health hazard since it circulates pollutants. Heat brings in new insects and diseases, such as West Nile.  

Somebody in Boston has done world class research, such as noting that the 1929 Newfoundland tsunami which killed 28 people—a fact many Canadians wouldn’t know.

Someone has actually noted the dangers of the weather in outer space. Solar storms could disable 300 large transformers in the US and cut power to 130 million people.  

Here’s where Boston is doing more than talking. A park on “Parcel 5” has been designed to help with drainage and surging tides. Boston’s plan calls for elevating and relocation boilers, electrical panels and computers. There’s also talk of modifying work schedules, spray mists and water stations at outdoor events. It’s not enough to use water pumps to keep roadways open during floods, Boston is talking about using absorbent paving materials.  New types of asphalt will absorb and even filter water so it can go back into the drinking water supply, not into people’s basements.

There’s always more work to be done. More trees and landscaping is a solution to both heat and flooding. About 35% of Bostonians don’t have a private vehicle, 10% are over 65, and 22% have disabilities. How will Boston evacuate neighbourhoods or move people around during an emergency? Where will the food come from?

Boston’s plan proves that simple, innovative solutions can save money and lives.

All Bound For Morningtown!

In the children’s songs, Morningtown ride (1957 Malvina Reynolds), girls and boys under their blankets are “All bound for Morningtown/Many miles away.”  This refrain is repeated until near the end when we hear that “Somewhere there is Morningtown/Many miles away.”

I sang this song many times (badly) to my boys.  The technique of indicating there’s a train voyage which will take time and cause the boys to arrive “[m]any miles away” is a nice mix of time and place.  So is the notion that the name of the place is a combination of a time of day (Morning) and a settlement of people (town).  

John Milton in his epic poem Paradise Lost uses a similar combination of precision and vagueness to describe Hell in his quest to “justify the ways of God to men.”

Metaphors are invaluable when soothing children, and when trying to describe and probe the almost unfathomable.

But why would emergency plan writers use this technique?  

Brisbane, Australia advocates “[c]areful planning” in advance of a decision on whether to evacuate.  Fair enough.  In fact the plan suggests ensuring that people “would be significantly safer at another location.”  Readers are also encouraged to determine if the risks of moving to another location are less than the risks of staying put.  

This all makes sense, until you think seriously about what this might mean to a person facing a risk.  Does it make sense to someone who will have to make the life and decision about whether to call for an urban evacuation?  In fact it just might delay information gathering or the decision.  What is careful planning?  Is there a lot of dangerous planning going on that we need to correct?  How much safer is “significantly” safer.  Is this a statistical significance or a qualitative one?  Is “safer” measured in discomfort, injuries, or death?  

Comparing risks is a good idea.  There’s no such thing as no risk.  Some evacuation orders have killed more people than the emergency.  Moving the elderly and hospital patients probably reduces life expectancy in many cases.  But how shall one measure the risk in one location, the risk of moving people, and the risk of locating people elsewhere?  We are none the wiser by reading Brisbane’s plan.

Waterloo advocates developing a list of “reliable” contractors, but doesn’t actually list them.  

In Tampa they’re pretty precise.  Their Emergency Operations Plan stipulates that “every attempt will be made to obtain the assigned driver or drivers who are familiar” with certain types of vehicles.  Good idea.  There are good reasons why heavy equipment operators, school bus drivers, and 18 wheeler drivers need training and special drivers’ licenses.  Putting a cabbie at the wheel of a school bus or highway coach may be more live-threatening than leaving people in the danger zone.  

Another way of creating the impression of precision is through using words with lots of syllables.  “Procuring” seems more precise than “buying” and yet the end result is still the same—obtaining something.  When someone says s/he will “utilize” a technique, the impression given is that more thought has gone into this than from someone who is just going to “use” a technique.  Wrong, but a strong impression.

Tampa will “ascertain” the type and location of “all” available transportation vehicles.  

Houston uses a combination of vague weakness and proscriptive strength in its dictum that “[l]aw enforcement will request wrecker services needed to promptly respond and clear disabled vehicle impediments …”  Leaving aside how a “disabled vehicle” differs from a “disabled vehicle impediment”, Houston will need an order not a “request.”

This mock-precision does not take the place of actually doing the work described.  Make a list.  Check it twice.  Stop nattering about what a good idea a list would be.

Yahoo! I’m Out To Dinner

“Kill the bum…” says the quintessential sports fan at a prize fight, football or hockey game.  “Are you blind?” yells another quintessential fan at an umpire or other official.  Fist bumps, loud yips of delight and almost as loud moans of disappointment fill arenas, basements, and living rooms in which TVs are tuned into the tragedy and ecstasy of sports.  Some of my fondest memories are of Les Carabiniers Tavern (The Carb) in Alexis Neon Plaza, near enough to the old Montreal Forum to seen the intermission commentators in for a beer and watching the game just like all the fans–on TV.  

Other good memories involve an Expos game at Jarry Park, the Alouettes at the old Autostadt and the BC Lions at Empire Stadium.  And the crowds did roar.  And did they roar for Ron Lancaster at Taylor field in Regina.  

The other urban experience is of quiet.  The quiet of the lounge in the Admiral Hotel on Lougheed  Highway in Burnaby in which I had some of my first underage alcohol rings in my ears.  So does the River Room and Maverick Room in the Beavebrook Hotel in Fredericton, in which I had some more.  Ditto Golf’s in Regina, but I was of age.  

But somewhere along the way an arms race of sorts began.  I don’t know who started it, but by the time I saw a Vancouver Grizzley’s basketball game, the yelling had mechanized into roving spotlights, strobes, and very loud music pumped into the pumped up crowd.  Then, in a box to see the Edmonton Oilers, the same thing had some to hockey games.  Buffalo and Toronto feature the same migraine inducing noise and light shows.  

Back in restaurants, many seemed to want to compete.  Some feature that noise that is currently popular in place of music—mainly a monotonous base line.  Others feature a type of music I don’t like.  If I’m alone in a restaurant (which I often am for obvious reasons), I tell the servers that since no one is demanding this music, could it please be turned down?  I sometimes pull out my decibel meter app in my smart phone and note that the noise level will cause permanent harm to hearing—especially the servers’.  Sometimes I note that if the restaurant could play Dion and the Belmonts, I’d not complain.  Most don’t get my point—noise is unwanted sound and taste in music is individual.

Even in nice, quiet restaurants, patrons often take the place of unwanted music with unwanted exclamations.  Really, this buffoonery knows no gender or age limitations.  Those who can’t exclaim loudly enough, or guffaw at high enough decibels, augment their failings with serious hand clapping to punctuate a point.

Perhaps these folks are all making a public statement.  They are out having a good time and don’t mind letting the world know it.  This is akin to the documented phenomenon of the mandatory standing ovation at live theatre.  Habitual attendees reserve shouting “Bravo!” or “Brava!” for extraordinary performances.  People who attend just a handful of live performances per year consider vigorous clapping, shouting, and standing as part of their evening’s entertainment, regardless of the quality of the performance.  Fair enough.  It’s a free country.

But your right to shout and clap in a restaurant, should really end at my ear, to paraphrase American jurist Oliver Wendell Holmes.  Have a good time.  Enjoy your friends.  Enjoy the food.  But you’re not at a ball game.  Occasionally put some of that good food in your mouth, close it, chew vigorously, and stop making noise.  Thank you.