Jan 072014
 

As part of the book on mobile market research that Sue York, Navin Williams and I are writing we need to give an overview of mobile qual, before going into depth. Do you think the image below helps?

Thoughts?

  • What are we missing?
  • What would you change?
  • What about the titles for the segments?

By WE-research we mean projects where participants are recruited to capture a variety of qualitative data about their lives. They might be asked to capture images of waste, or videos of travel problems, or audio comments about the school run, for example. The term WE-research was introduced by Mark Earls and John Kearnon a few years ago. But, do you have a better term for this type of research?

We are planning on three chapters on qualitative research, indeed we have written three chapters, an overview which covers all four segments above, followed by chapters specifically on the top two segments. Our view on techniques like passive tracking, Google Glass etc is that there is simply not enough material yet to have a chapter on it, there is too little experience around.

If you’d like to help by reviewing one or more of these three chapters, please email ray.poynter@thefutureplace.com – we’ll acknowledge you contribution in the book :-)

Aug 262013
 

In November I am presenting a paper to the ESOMAR Conference on Qualitative Research, in Valencia in Spain. My paper suggests that one threat to qualitative research is the potential for damage caused by people with no training in qualitative research using one of the many DIY tools that are appearing – especially those for online discussions and instant chats.

My suggestion is to create a simple set of notes that will help put newcomers to our world on the right path. Below is my initial draft if of my notes, and I would really appreciate your feedback.


The Playbook

The playbook needs to be short, relevant, and easy to use if it is going to be of value to people looking to conduct their own research. Therefore, this initial draft covers the following topics:

  • Evidence, not answers
  • Creating a narrative
  • Analysis begins at the start not the end of the project
  • Creating a discussion guide
  • Not everything that matters can be counted
  • Data does not mean numbers
  • Consider actors and agendas
  • We are poor witnesses to our own motivations
  • Memoing
  • Enabling the participants whenever possible
  • Grounding the story in the data
  • Examples that inform, not ones that entertain
  • The “But, I already knew that test!”

Evidence, not answers
Qualitative research, for example, online discussions, real-time chat, smartphone ethnography, or discussions gathered from social media, does not provide categorical, definitive answers. Qualitative research provides evidence, and the researcher has to interpret this evidence to produce the product of the research.

A quantitative study might discover that 10% of the population buy a product from ACME Corporation. This 10% is an answer, something discovered and provided by the research. A qualitative discussion might suggest that people seem willing to use words like respect, admire, trust about ACME, but were less willing to say love, like, associate with ACME. The researcher has to determine what that might mean and what the implications for ACME might be.

At the end of a qualitative project we can’t say things like “50% of the participants said they would try the product”, implying that 50% of the target group will buy the product. The qualitative participants are not numerous enough to forecast population-wide behaviour and the way the questions were asked will have affected the thinking and responses of the participants. A qualitative finding is more likely to describe what the people who said they would try it liked about the product, how they came to their decision that they might try it, and what was inhibiting those who did not want to try it.

A quantitative ad test might try to forecast how many people would recall it, how many would recommend the product, and how many would buy the product. A qualitative ad test tries to find out how the ad worked and to suggest how it might be improved.

Creating a narrative
The purpose of a qualitative market research project is to create a story that illuminates the topic under investigation. Qualitative researchers do not ‘discover’ the story, they create the story from what they find, potentially co-creating it with the participants and/or the client. The evidence they gather, the knowledge they have, the knowledge the client has, need to be woven together to produce the final narrative.

The narrative that is created needs to explain the evidence in a way that throws light on the subject so that it facilitates better business decisions. Qualitative researchers are aware that there is no one ‘correct’ story, there are usually many ways to tell a good/effective/useful story (and of course even more ways to tell it in ineffective or misleading ways).

Analysis begins at the start not the end of the project
Before conducting the research the researcher needs to think about what is already known, what needs to be known, and the sorts of evidence that will help create a narrative. During the data gathering phase, the text (e.g. the chat, the posts, the comments) should be reviewed to challenge the hypotheses the researcher already has and to help create new hypotheses. The researcher should seek to test hypotheses by posting questions, by assigning tasks, and by probing existing answers, in ways that will make or break the hypotheses.

For example, if the researcher feels that the participants do not trust a specific brand, the participants might be asked to write a list of all the things they like about that brand. The words that are not on the list are a clue to what people feel. The words not on the list can then be used to elicit which brands do have those characteristics.

Create a discussion guide
A discussion guide is a plan of what is going to be discussed during the research. Researchers vary in how detailed their guide will be. Some researchers spell out every question they plan to ask in their online chat, focus group, or discussion. Other researchers will simply map out the topics they plan to cover and the sequence which they initially expect to ask them in.

Without a discussion guide the research runs the risk of running out of time, of failing to cover all the necessary topics, or of bringing up the topics in an order that is likely to inappropriately bias the results. A discussion guide can also be a useful way of checking with other stakeholders that the research is likely to cover what is needed.

Not everything that matters can be counted
In most cases, the exact number of times a particular word is used is not directly relevant to the outcome of a qualitative research project. Simple tools, particularly word clouds, give a picture, of qualitative data, based simply on how often certain words occur. Whilst a word cloud can be a useful starting point, it is never enough. Qualitative research is conducted by reading and considering all the material. In a modern qualitative project, that might include words, pictures, videos, audio contributions and more.

The sequence in which things are said can often matter more than the frequency of words. In an online discussion, for example, it is not unusual for several participants to comment on why they like something, until one person raises a major drawback. When this happens the conversation on that point may simply stop, because the drawback is so clear. But a word count of that conversation would treat the drawback as one comment, and the many, previous, praises for it, as being more significant. The order words are said in matters as much as the content of what is said.

Data does not mean numbers
When a qualitative researcher says ‘data’ they mean the words, pictures, videos, notes, audio recordings, and objects that have been collected. They do not mean a list of numbers in some tabular format.

There are other words that qualitative researchers use, such as text, corpus, discourse, artefacts, objects, exhibits etc. However, all of these can be subsumed in the term data. Sometimes, to reduce confusion these materials are described as qualitative data.

Consider actors and agendas
When looking at a post, an upload, or a comment, the researcher should consider who said it and why. People play roles in discussions, some are trying to be experts, while some are trying to conceal their true feelings. The researcher needs to assess who the actors in the discussion are and what they are trying to achieve, in order to place their contributions in the narrative.

In a discussion about coffee we may identify baristas, amateur experts, people with a green agenda, traditionalists, and innovators. The words cannot be separated from who said them, and ideally who said them to whom. Linking a series of contributions to the same person can increase the insight generated about narrative that is being sought.

We are poor witnesses to our own motivations
Many of the questions that researchers would like to ask are impossible for participants to answer accurately. People tend not to know why they do things. They mostly do not know the drivers of their behaviour. And, they are fairly poor at forecasting what they will do in the future. So, questions that ask “Why are you overweight?”, “Why did you buy that gym membership, knowing you’d hardly use it?” and “What is it about the ACME brand that makes you feel safe and warm?” are likely to fail.

Questions that tend to work are:

  • Reporting questions – e.g. “Which cupboard do you store you cleaning products in?” and “How often do you eat in a restaurant?”
  • Choice based questions. Show three items, ask “Which is the odd one out?”, and which can then lead into discussions of why.
  • Asking about other people. For example, “Tell me all the reasons why some people who are on a diet drink milk shakes?”
  • Asking what sorts of people do things. For example, “Tell me who might bake their own bread?”
  • Lists – in online research the creation of lists can be a natural way to get participants to be active and to reveal some of their feelings and beliefs. For example, a researcher might ask “Thinking about the brand Coca-Cola, list all the non-drink things you think they would be good at making?” – again leading on to why, and asking who agrees, and who has alternative suggestions.

Asking the obvious questions, for example, “What do you like about this advert?” are always going to be part of the qualitative research process. They are often an easy way to start a discussion, and we want to know what the answers are. However, we do not place too much motivational and narrative importance on the answers to these sorts of questions. The answers should certainly not be reported as being the actual motivations and feelings.

Memoing
When analysing non-trivial amounts of qualitative information, it is really useful to annotate the material. This can be called tagging, memoing, commenting, annotating, highlighting, marking-up and probably a variety of other things. The material is read through and key themes, ideas, quotes, examples, hypotheses etc are noted.

Traditionally, this memoing process was done with scissors, copies of the transcripts, and coloured highlighter pens. Now there are a variety of software tools to help, often referred to as CAQDAS (Computer Aided Qualitative Data Analysis). Some people use specific software, whilst others find they can use Word and/or Excel to achieve what they need.

The narrative is then, typically, constructed from the memos. The source documents are often only referred back to when the story emerging from the memos needs additional evidence or appears inconsistent.

In a collaborative project, participants or ‘the crowd’ are enlisted to add their comments, tags, annotations.

Enable the participants whenever possible
The researcher will be developing hypotheses before the research, during the research, and after the research. One great way of challenging, supporting, or enriching these hypotheses is to actively involve the participants in the process.

Participants can be enabled and encouraged to tag their own comments and uploads, to tag and/or reply to other people’s contributions, and they can feedback on ideas presented by the researcher. In ‘researcher talk’, the researcher provides an outsider’s view (the etic view) whilst the participants can provide an insider’s view (the emic view). A narrative that combines the insider and outsider views is often more powerful than just a single perspective.

Grounding the story in the data
When the narrative is being created, the researcher should check that everything they are claiming can be supported by something evidenced in the data. Whilst not everything in the data should be in the narrative, everything in the narrative should be supported by something in the data.

If the researcher believes something to be true and important, but they cannot support it from the data, they should seek to introduce it to the research, in order to elicit evidence. This is could be via posts in an online discussion, or through the discussion guide for later online focus groups.

Examples that inform, not ones that entertain
There can be a temptation, when creating the research story, to include a video clip, photo, or quote that is particularly powerful, even though it is not truly relevant to the message, or perhaps is even at odds with the main element of the story. This is a bad practice and researchers need to be on their guard against it.

The researcher has to be seen as a ‘truth teller’. The role of the researcher is to tell the customers’ story. This means having the discipline to only use materials that are true to the narrative that has been created.

The “But I already knew that test!”
One test of a powerful narrative from qualitative research is that the client, when presented with the story says “But, I already knew that!” The client did already know it, but they did not know they knew it till they heard the research narrative.

This test, the “I already knew it” test goes to the heart of what qualitative research is all about. The research gathers evidence and synthesises them into a narrative that illuminates the topic under investigation. The illumination, typically, comes from revealing things we already knew, but did not realise or could not access without the research.


Suggestions

So, what you your thoughts and suggestions? What should be added, removed, or amended? Indeed, is the project just pure folly?

Apr 132013
 

As mentioned before (here and here), Navin William, Reg Baker, and I are producing a mobile marketing research module for the University of Georgia’s Principles of Marketing Research course. I have bounced some ideas off the readers of this blog, and here is another topic where I’d love to hear your views.

Some of the most interesting work, to date, in terms of MMR (mobile market research) has been in the area of qualitative research and this is a key point for students of MMR to be aware of.

The key areas of qualitative MMR:
My feeling is that the key uses of mobile in qualitative research are:

  1. Smartphone Ethnography, recruiting participants to capture slices of the own lives and the lives of people around them to produce ethnographic data and in some cases to engage citizens in mass or auto ethnography.
  2. Mobile blogging, where participants use their mobile device (which can be as basic as SMS) to record or comment on some aspect of their lives. This can also include asking the participants to record their own vox pops.
  3. Mobile focus groups, where participants can use a mobile device to take part in focus groups. At one extreme this means voice only, at the other end it can mean using a web-enabled tablet to show all the participants on the screen, with full audio-visual connectivity.
  4. Discussions, allowing participants to take part in asynchronous discussions from their mobile devices, typically via internet access.
  5. Homework, where the participant is sent tasks via their mobile device, often in advance of a discussion, and often including the participants using their mobile devices to capture artefacts (e.g. pictures of your pantry).
  6. Tracking, where a small number of participants agree to be tracked for a period of time, for example 24 hours or a week, and the researcher uses the participant’s mobile device (typically a smartphone) to record some or all of: location, internet usage, voice calls, when and how the phone used (e.g. to check time of day), who the phone contacted (e.g. Bluetooth and WiFi), and much more. Qualitative tracking is based on looking in-depth at traces, often in conjunction with the participant themselves to gain insight into what is revealed by the data.
  7. As a tool in a qualitative session, for example tablets can be used in focus groups or in a one-on-one interviews, to show images and video, and to allow the participant to access materials and respond, for example by sorting items or creating pictures.

In addition, mobile devices are used to organise and coordinate qualitative activities, ensuring people receive instructions, helping them find locations, and generally communicating with participants. Also, mobile devices are often used with insight communities as part of the overall method of communication with members, for qual, quant, and administrative purposes.

What do you think? Have I missed some important areas? Are some of my items marginal?

Jan 072013
 

Market research is being deluged with new sources of data, from social media, from electronic communications, and from research communities. Whilst some of this information is suitable for quantitative analysis, large parts of it are unstructured, for example tweets, posts, comments, and uploads. Whilst this data presents an interesting opportunity for market research, it presents a sequence of inter-connected problems and challenges:

  1. Many of the market researchers who are most proficient with unstructured data, the qualitative researchers, are not instinctively drawn towards online data, preferring to deal with people in a face-to-face environment.
  2. Many of the researchers most attracted to large amounts of online data, the hard core quantitative researchers, have little appreciation of the different epistemologies of quantitative and qualitative research.
  3. Many of the software vendors, perhaps in a rush to market, have released products before they were really ready and with massive over-claims.

In order for market research to fully leverage the potential benefits of the discourses being generated, market research needs to address the qualitative deficit. The qualitative deficit is the shortage of talent, software, and approaches designed to utilise massive amounts of qualitative data.

Some of the new approaches and skillsets that are needed relate to the process of quantifying messages within discourses – Google’s Flu Trends being a straightforward example (where Google use phrases that indicate people are searching for flu remedies to quantify the incidence of flu). This use of unstructured data will be of value in areas such as brand and ad tracking.

However, the bigger deficit relates to taking qualitative information and extracting qualitative findings. In online discussions the meaning is often unrelated to the frequency of words and phrases, the meaning rests in the structure of the conversation and the outcomes of the conversation. Feedback for product design, the identification of opportunities, core reasons for product dissatisfaction are likely to be found in the meaning of discourses, as opposed to counts of terms and phrases.

Whilst part of the answer will be new software, my feeling is that research urgently needs to expand the number of researchers with a good understanding of qualitative methods and epistemology.