Aug 312013

This post started life as a response to Ellen Woods’ well thought out Greenbook post on the paradox of risk, which you can read here. However, here is an extended version of my thoughts on the difference between the herd and good for the individual, in terms of risk, risk avoidance, and its implications for marketers and market researchers.

In this post I want to concentrate of two issues: 1) the difference between what is good for the economy and the individual, and, 2) the difference between the short-term and the long-term.

The difference between what is good for an economy and what is good for an individual.
Most innovations fail, most entrepreneurs fail, most new products fail – the failure rate is typically quoted as being in the 80% to 90% range. Given these high failure rates, the logic for each individual entrepreneur is that they should not risk everything on some new change, idea, or innovation. However, the benefits to the economy of the few innovators and entrepreneurs that succeed is massive. Indeed, I think the number of entrepreneurs in the US is one of the keys to its success over the last 100 years – a success which is the result of a massive number of failures, producing a large number of successes. However, this difference between the individual losses and the collective gain raises an issue for market research. If we are accurate in our work, we should be telling most of our clients that the chances of success are well below 50% – which means research will typically and most often be a drag on innovation, something which inhibits change. (I am not saying research always inhibits change, but, probabilistically, it will inhibit it more often than it promotes it.)

What should research do about its dismal science of predicting probable failure? Research should not deceive; it should not tell people who are going to probably fail that they will succeed. Research can help them minimise risks of failure, but, in many cases, risk minimising strategies also limit the potential upside. This is one of the paradoxes. We want everybody to research their ideas, but if they do, we would probably reduce the rate of growth and development in the economy. Clients need to assess their risk profile and fit the research to their needs.

The difference between what is best in the short term and what is best in the long term.
In the short term, for most profitable companies, the best financial return for the current year comes from not changing anything, i.e. business as usual. And, in a world of short-termism, the impact of the latest share figures and this year’s performance can be massive. However, as Ellen points out in her post, companies that do not change fail, in the long run (and the long run can come quite quickly as Kodak found out). One consequence of this paradox is that it is the failing companies that often adopt the most innovative and risk-taking approaches. For example, when P&G first appointed AG Lafley as their CEO they had just lost $85 billion of market capitalisation – this gave Lafley the permission and the freedom to change P&G in fundamental ways (as he recounts in his book The Game Changer).

When I look at the issues surrounding insight, consultancy, and market research I am reminded of the old joke about “How many psychiatrists does it take to change a light bulb? One, but the light bulb has got to want to change!” Can market research help foster innovation, change, and growth? Yes, but companies have to want to change. For example, if major clients were to decide to stop all the brand trackers, customer satisfaction, and NPS type projects on Jan 1st 2014, the research industry could have better, cheaper, more appropriate options in place. But, my prediction is that it will be the new companies who don’t know the old ways, and the failing companies, who feel they have to roll the dice, who will drive the changes. The bulk of the big spenders on market research will follow some years behind.

Aug 262013

In November I am presenting a paper to the ESOMAR Conference on Qualitative Research, in Valencia in Spain. My paper suggests that one threat to qualitative research is the potential for damage caused by people with no training in qualitative research using one of the many DIY tools that are appearing – especially those for online discussions and instant chats.

My suggestion is to create a simple set of notes that will help put newcomers to our world on the right path. Below is my initial draft if of my notes, and I would really appreciate your feedback.

The Playbook

The playbook needs to be short, relevant, and easy to use if it is going to be of value to people looking to conduct their own research. Therefore, this initial draft covers the following topics:

  • Evidence, not answers
  • Creating a narrative
  • Analysis begins at the start not the end of the project
  • Creating a discussion guide
  • Not everything that matters can be counted
  • Data does not mean numbers
  • Consider actors and agendas
  • We are poor witnesses to our own motivations
  • Memoing
  • Enabling the participants whenever possible
  • Grounding the story in the data
  • Examples that inform, not ones that entertain
  • The “But, I already knew that test!”

Evidence, not answers
Qualitative research, for example, online discussions, real-time chat, smartphone ethnography, or discussions gathered from social media, does not provide categorical, definitive answers. Qualitative research provides evidence, and the researcher has to interpret this evidence to produce the product of the research.

A quantitative study might discover that 10% of the population buy a product from ACME Corporation. This 10% is an answer, something discovered and provided by the research. A qualitative discussion might suggest that people seem willing to use words like respect, admire, trust about ACME, but were less willing to say love, like, associate with ACME. The researcher has to determine what that might mean and what the implications for ACME might be.

At the end of a qualitative project we can’t say things like “50% of the participants said they would try the product”, implying that 50% of the target group will buy the product. The qualitative participants are not numerous enough to forecast population-wide behaviour and the way the questions were asked will have affected the thinking and responses of the participants. A qualitative finding is more likely to describe what the people who said they would try it liked about the product, how they came to their decision that they might try it, and what was inhibiting those who did not want to try it.

A quantitative ad test might try to forecast how many people would recall it, how many would recommend the product, and how many would buy the product. A qualitative ad test tries to find out how the ad worked and to suggest how it might be improved.

Creating a narrative
The purpose of a qualitative market research project is to create a story that illuminates the topic under investigation. Qualitative researchers do not ‘discover’ the story, they create the story from what they find, potentially co-creating it with the participants and/or the client. The evidence they gather, the knowledge they have, the knowledge the client has, need to be woven together to produce the final narrative.

The narrative that is created needs to explain the evidence in a way that throws light on the subject so that it facilitates better business decisions. Qualitative researchers are aware that there is no one ‘correct’ story, there are usually many ways to tell a good/effective/useful story (and of course even more ways to tell it in ineffective or misleading ways).

Analysis begins at the start not the end of the project
Before conducting the research the researcher needs to think about what is already known, what needs to be known, and the sorts of evidence that will help create a narrative. During the data gathering phase, the text (e.g. the chat, the posts, the comments) should be reviewed to challenge the hypotheses the researcher already has and to help create new hypotheses. The researcher should seek to test hypotheses by posting questions, by assigning tasks, and by probing existing answers, in ways that will make or break the hypotheses.

For example, if the researcher feels that the participants do not trust a specific brand, the participants might be asked to write a list of all the things they like about that brand. The words that are not on the list are a clue to what people feel. The words not on the list can then be used to elicit which brands do have those characteristics.

Create a discussion guide
A discussion guide is a plan of what is going to be discussed during the research. Researchers vary in how detailed their guide will be. Some researchers spell out every question they plan to ask in their online chat, focus group, or discussion. Other researchers will simply map out the topics they plan to cover and the sequence which they initially expect to ask them in.

Without a discussion guide the research runs the risk of running out of time, of failing to cover all the necessary topics, or of bringing up the topics in an order that is likely to inappropriately bias the results. A discussion guide can also be a useful way of checking with other stakeholders that the research is likely to cover what is needed.

Not everything that matters can be counted
In most cases, the exact number of times a particular word is used is not directly relevant to the outcome of a qualitative research project. Simple tools, particularly word clouds, give a picture, of qualitative data, based simply on how often certain words occur. Whilst a word cloud can be a useful starting point, it is never enough. Qualitative research is conducted by reading and considering all the material. In a modern qualitative project, that might include words, pictures, videos, audio contributions and more.

The sequence in which things are said can often matter more than the frequency of words. In an online discussion, for example, it is not unusual for several participants to comment on why they like something, until one person raises a major drawback. When this happens the conversation on that point may simply stop, because the drawback is so clear. But a word count of that conversation would treat the drawback as one comment, and the many, previous, praises for it, as being more significant. The order words are said in matters as much as the content of what is said.

Data does not mean numbers
When a qualitative researcher says ‘data’ they mean the words, pictures, videos, notes, audio recordings, and objects that have been collected. They do not mean a list of numbers in some tabular format.

There are other words that qualitative researchers use, such as text, corpus, discourse, artefacts, objects, exhibits etc. However, all of these can be subsumed in the term data. Sometimes, to reduce confusion these materials are described as qualitative data.

Consider actors and agendas
When looking at a post, an upload, or a comment, the researcher should consider who said it and why. People play roles in discussions, some are trying to be experts, while some are trying to conceal their true feelings. The researcher needs to assess who the actors in the discussion are and what they are trying to achieve, in order to place their contributions in the narrative.

In a discussion about coffee we may identify baristas, amateur experts, people with a green agenda, traditionalists, and innovators. The words cannot be separated from who said them, and ideally who said them to whom. Linking a series of contributions to the same person can increase the insight generated about narrative that is being sought.

We are poor witnesses to our own motivations
Many of the questions that researchers would like to ask are impossible for participants to answer accurately. People tend not to know why they do things. They mostly do not know the drivers of their behaviour. And, they are fairly poor at forecasting what they will do in the future. So, questions that ask “Why are you overweight?”, “Why did you buy that gym membership, knowing you’d hardly use it?” and “What is it about the ACME brand that makes you feel safe and warm?” are likely to fail.

Questions that tend to work are:

  • Reporting questions – e.g. “Which cupboard do you store you cleaning products in?” and “How often do you eat in a restaurant?”
  • Choice based questions. Show three items, ask “Which is the odd one out?”, and which can then lead into discussions of why.
  • Asking about other people. For example, “Tell me all the reasons why some people who are on a diet drink milk shakes?”
  • Asking what sorts of people do things. For example, “Tell me who might bake their own bread?”
  • Lists – in online research the creation of lists can be a natural way to get participants to be active and to reveal some of their feelings and beliefs. For example, a researcher might ask “Thinking about the brand Coca-Cola, list all the non-drink things you think they would be good at making?” – again leading on to why, and asking who agrees, and who has alternative suggestions.

Asking the obvious questions, for example, “What do you like about this advert?” are always going to be part of the qualitative research process. They are often an easy way to start a discussion, and we want to know what the answers are. However, we do not place too much motivational and narrative importance on the answers to these sorts of questions. The answers should certainly not be reported as being the actual motivations and feelings.

When analysing non-trivial amounts of qualitative information, it is really useful to annotate the material. This can be called tagging, memoing, commenting, annotating, highlighting, marking-up and probably a variety of other things. The material is read through and key themes, ideas, quotes, examples, hypotheses etc are noted.

Traditionally, this memoing process was done with scissors, copies of the transcripts, and coloured highlighter pens. Now there are a variety of software tools to help, often referred to as CAQDAS (Computer Aided Qualitative Data Analysis). Some people use specific software, whilst others find they can use Word and/or Excel to achieve what they need.

The narrative is then, typically, constructed from the memos. The source documents are often only referred back to when the story emerging from the memos needs additional evidence or appears inconsistent.

In a collaborative project, participants or ‘the crowd’ are enlisted to add their comments, tags, annotations.

Enable the participants whenever possible
The researcher will be developing hypotheses before the research, during the research, and after the research. One great way of challenging, supporting, or enriching these hypotheses is to actively involve the participants in the process.

Participants can be enabled and encouraged to tag their own comments and uploads, to tag and/or reply to other people’s contributions, and they can feedback on ideas presented by the researcher. In ‘researcher talk’, the researcher provides an outsider’s view (the etic view) whilst the participants can provide an insider’s view (the emic view). A narrative that combines the insider and outsider views is often more powerful than just a single perspective.

Grounding the story in the data
When the narrative is being created, the researcher should check that everything they are claiming can be supported by something evidenced in the data. Whilst not everything in the data should be in the narrative, everything in the narrative should be supported by something in the data.

If the researcher believes something to be true and important, but they cannot support it from the data, they should seek to introduce it to the research, in order to elicit evidence. This is could be via posts in an online discussion, or through the discussion guide for later online focus groups.

Examples that inform, not ones that entertain
There can be a temptation, when creating the research story, to include a video clip, photo, or quote that is particularly powerful, even though it is not truly relevant to the message, or perhaps is even at odds with the main element of the story. This is a bad practice and researchers need to be on their guard against it.

The researcher has to be seen as a ‘truth teller’. The role of the researcher is to tell the customers’ story. This means having the discipline to only use materials that are true to the narrative that has been created.

The “But I already knew that test!”
One test of a powerful narrative from qualitative research is that the client, when presented with the story says “But, I already knew that!” The client did already know it, but they did not know they knew it till they heard the research narrative.

This test, the “I already knew it” test goes to the heart of what qualitative research is all about. The research gathers evidence and synthesises them into a narrative that illuminates the topic under investigation. The illumination, typically, comes from revealing things we already knew, but did not realise or could not access without the research.


So, what you your thoughts and suggestions? What should be added, removed, or amended? Indeed, is the project just pure folly?

Aug 062013

Traditionally the term agnostic has been applied to people who have not had the courage of their convictions to settle for belief or refutation. However, over the last few years the term agnostic has become increasingly used in the area of market research, and indeed agnosticism appears to be the new creed for many of the suppliers to our industry and the prediction for many of the pundits forecasting the future.

Among the key areas where agnosticism is becoming a driving principle are:

  • Mode agnosticism – especially between online, mobile phone, and tablet.
  • Pull-Push agnosticism – between apps and browser based mobile research.
  • User agnosticism – between DIY, assisted serve, partners, and full-service.
  • Code agnosticism – between classic market research conducted under research codes, and other forms of research, such as Big Data and Social Media Research, which are as likely to be offered by non-research companies as research companies.
  • Sampling agnosticism – market researchers used to be believers in ‘the way’ (aka random probability sampling), but now the largest single method is the convenience sample (aka online panel) and alternatives are picked according to their merit (especially availability, speed, and price), rather than on a priori beliefs.

Why the change?
To me it seems as though there are three main reasons for this shift to agnosticism:

  1. The least good reason is that there are growing numbers of people in the research industry, as buyers and sellers, who do not know the beliefs (why one way might be methodologically better than another), so their agnosticism is based on not knowing and/or not caring.
  2. The second is a recognition that many of the most important questions should be answered by the buyer, not the seller. In the market research world before 2000 it was not uncommon for researchers to assert that they knew best, that they should define what was right and what was wrong, and to create the standards for what clients would be allowed to buy. If vendors realise that buyers are the people who should be defining standards and making choices, then they realise that the vendors need to be more agnostic, and less belief driven.
  3. It is currently very, very hard to know what is going to happen, over the next few years, with issues such as mobile versus online, with DIY versus full-service, and apps versus browsers. Some small companies will probably want to bet their future on a particular outcome, but most organisations will want to hedge one option against another.

So, what are your views? Do you agree that agnosticism is growing? If so, is it a good thing or a bad thing?

Aug 022013

This post has been written in response to a query I receive fairly often about sampling. The phenomenon it looks at relates to the very weird effects that can occur when a researcher uses non-interlocking quotas, effects that I am calling unintentional quotas, for example when using an online access panel.

In many studies, quota controls are used to try to achieve a sample to match a) the population and/or b) the target groups needed for analysis. Quota controls fall into two categories, interlocking and non-interlocking.

The difference between the two types can be shown with a simple example, using gender (Male and Female) and colour preference (Red or Blue). If we know that 80% of Females prefer Red, if we know that 80% of Men prefer Blue, and if there are an equal number of Males and Females in our target population, then we can create interlocking quotas. In our example we will assume that the total sample size wanted is 200.

  • Males who prefer Red = 50% * 20% * 200 = 20
  • Males who prefer Blue = 50% * 80% * 200 = 80
  • Females who prefer Red = 50% * 80% * 200 = 80
  • Females who prefer Blue = 50% * 20% * 200 = 20

These quotas deliver the 200 people required, in the correct proportions.

The Problems with Interlocking Quotas
The problem with the interlocking quotas above is that it requires the researcher to know what the colour preference of Males versus Females is, before doing the research. In everyday market research the quotas are often more complex, for example: 4 regions, 4 age breaks, 2 gender breaks, 3 income breaks. This pattern (of region, age, gender, and income) would generate 96 interlocking cells, and the researcher would need to know the population data for each of these cells. If these characteristics were then to be combined with a quota related to some topic (such as coffee drinking, car driving, TV viewing etc) then the number of cells becomes very large, and it is very unlikely the researcher would know the proportions for each cell.

Non-Interlocking Quotas
When interlocking cells become too tricky, the answer tends to be non-interlocking cells.

In our example above, we would have quotas of:

  • Male 100
  • Female 100
  • Prefer Red 100
  • Prefer Blue 100

The first strength of this route is that it does not require the researcher to know the underlying interlocking structure of the characteristics in the population. The second strength is that it makes it simple for the sample to be designed for the researcher’s need. For example, if in the population we know that Red is preferred by 80% of the population, then a researcher might still collect 100 Red and 100 Blue, to ensure the Blue sample was large enough to analyse, and the total sample could be created by weighting the results (to down-weight Blue, and up-weight Red).

Unintentional Interlocking Quotas
However, non-interlocking quotas can have some very weird and unpleasant effects if there are differences in response rates in the sample. This is best shown by an example.

Let’s make the following assumptions about the population for this example:

  • Prefer Red 80%
  • Prefer Blue 20%
  • No differences in colour gender preferences, i.e. 80% of males and females prefer Red
  • Female response rate 20%
  • Male response rate 10%

The researcher knows that overall 80% of people prefer Red, but does not know what the figures are for males and females, indeed the researcher hopes this project will through some light on any differences.

The specification of the study is to collect 200 interviews, using the following non-interlocking quotas.

  • Male 100
  • Female 100
  • Prefer Red 100
  • Prefer Blue 100

A largish initial sample of respondents are invited, let’s assume 1000 males and 1000 females. Noting that 1000 males at 10% response rate should deliver 100 completes.

After 125 completes have been achieved the pattern of completed interview looks like this:

  • Female Red 67
  • Female Blue 17
  • Male Red 33
  • Male Blue 8

This is because the probability of each of the 125 interviews can be estimated by the combination of the chance it is male or female (10% male response rate and 20% female means that it is one-third likely to be a male and two-thirds likely to be a female) and the preference for Red (80%) and Blue 20%). Which to the nearest round percentages gives us the following odds: Female Red 53%, Female Blue 13%, Male Red 27%, Male Blue 7%.

The significance of 125 completes is that the Red Quota is complete. No more Reds can be collected. This, in turn, means:

  • The remaining 75 completes will all be people who prefer Blue
  • 17 of the remaining interviews will be Female (we already have 83 Females, so the Female quota will close when we have another 17)
  • 58 of the remaining interviews will be Male, Male Blues will be the only missing cell left to fill
  • The rapid filling of the Red quota, especially with Females, has resulted in interlocking quotas being created for the Blue cells.

The final result from this study will be:

  • Female Red 67
  • Female Blue 33
  • Male Red 33
  • Male Blue 67

Although there is no gender bias to colour preference in the population, in our study we have created a situation where two-thirds of Males prefer Blue, and two-thirds of the Females prefer Red.

In this example we are going to have to invite a lot more Males. We started by inviting 1000 Males, and with a response rate of 10% we might expect to collect our 100 completes. But, we have ended up needing to collect 67 Male Blues, because of the unintentional interlocking quotas. We can work out the number of invites it takes to collect 67 Male Blues by dividing 67 by the product of the response rate (10%) and the incidence of preferring Blue (20%), which gives us 67 / (10% * 20%) = 3,350. The 1000 male invites need to be boosted, by another 2,350, to 3,350 to fill the cells. Most researchers will have noticed that the last few cells in a project are hard to fill, that is because they have created unintentional interlocking quotas, locking the hardest cells together, which makes them even harder.

This, of course, is a very simple example. We only have two variables, each with two levels, and the only varying factor is the response rates between Male and Female. In an everyday project we would have more variables, and response rates will often vary by age, gender, and region. So, the scale of the problem in typical interlocking samples is likely to be larger than in this example, at least for the harder cells to complete.

Improving the Sampling/Quota Controlling Process
Once we realise we have a problem, and with the right information, there is plenty we can do to remove or ameliorate the problem.

  • Match the invites to the response rates. If, in the example above, we had invited twice as many Males as Females the cells would have completed perfectly.
  • Use interlocking cells. To do this you might run an omnibus before the main survey to determine what the cells targets should be.
  • Use the first part of the data collection to inform the process. So, in the example above we could have set the quotas to 50 for each of the four cells. As soon as one cell fills we look at the distribution of the data and amend the structure of the quotas, making some of them interlocking, perhaps relaxing (i.e. make bigger) some of the others, and invite more of the sorts of people we are missing. This does not fix the problem, but it can greatly reduce it, especially if you bite the bullet and increase the sample size at your expense.

Working with panel companies. Tell the panel company that you want them to phase their invites to match likely response rates. They will know which demographics respond better. For the demographic cells, watch to see that they are advancing in step. For example, watch to see that Young Males, Young Females, Older Males, and Older Females are all filling at the same rate and shout if this is not happening.

It is a good idea to make sure that the fieldwork is not going to happen so fast that you won’t have time to review it and make adjustments. As a rule of thumb, you want to review the data when one of the cells is about 50% full. At that stage you can do something about it. This means you do not want the survey to start after you leave the office, if there is a risk of 50% of the data being collected before the start of the next day.

Questions? Is this a problem you have comes across? Do you have other suggestions for dealing with it?