Thoughts on Validity

 Posted by on September 9, 2012  Uncategorized  No Responses »
Sep 092012

Recently Kevin Gray asked, in the LinkedIn NewMR group, “When you hear a claim that a marketing research methodology is “valid”, what does that mean to you?” The question elicited a range of views and here is a tidied up version of my thoughts.

My feeling is that we need to start not with science or the theory of research methodology (both passions of mine BTW), but with what the users of research want and need the research to deliver. I think that what a research user means by valid is that what the research tells her/him is true*, that the results do not exclude important information, and that the information useful.

In my experience, much of the criticism of market research focuses on the second two criteria, i.e. that research does not include everything that the user needs to know and that the information is not sufficiently useful. Much of the drive towards new research (e.g. mass mobile ethnographics) is driven by a desire to produce findings that are more complete and more useful.

The tests for qual and quant in terms of complete and useful are very similar.

*The BIG problem with my definition is the reference to ‘true’. The traditional quant epistemology (positivism) held that absolute truths can be established, and that information that was not reliable (i.e. replicable) and not consistent with a theory was not valid. Quant research has established a family of validities, ranging from face validity at the weakest end of the spectrum i.e. does it look right) through to construct validity (is it consistent with theory and can it be tested).

Move the clock forwards, past Karl Popper and Thomas Kuhn, and into post-positivism and we see that research has had to soften its definitions. Most quant researchers accept that when measuring and researching human feedback and society there can be no absolute truth. The reflexivity of the researcher, the biases of observation, and the feedback loop in human systems mean that there are no absolute truths, just associations that for a period of time are useful. In modern market research the term valid tends to be used in a sense closest to Convergent Validity – i.e. to what extent does a method produce results that are similar to ones that it is supposed to be similar to (e.g. predicting sales).

Qual research occupies a much wider spectrum, in terms of epistemology, than quant research. There are post-positivists who aim for something similar to quant researchers, i.e. replicable research that predict real world events. There are naturalists who liken themselves to 19th Century botanists, as opposed to 20th Century physicists, who seek to accurately describe what they see, even though they can’t explain the underlying mechanisms. However, there are various modern schools who reject the notion of objective truth completely, who typically seek to create a narrative that is credible and useful.

Whenever methodology and theory is discussed, somebody (or several somebodies) chime in with comments that say clients do not want to know about methodology or even that clients do not care about methodology. Whilst I am sure that only a minority of clients want to delve into the minutiae (although I have dealt with three clients who have PhDs in the area and who were very interested), I am sure that most clients want to know that the research they are buying/using is in some sense ‘fit for purpose’, i.e. that it is likely to ‘work’. This means somebody has to be interested in methodology, somebody has to put the hard yards in, so clients don’t have to.

If you want to join the debate on validity, why not visit Kevin Gray’s discussion on LinkedIn? []

Welcome back to the NewMR Blog!

 Posted by on September 9, 2012  Uncategorized  1 Response »
Sep 092012

The NewMR blog returns from its sabbatical today. We envisage that the blog will host a wide range of content, but it will be focused on the theme of NewMR. The NewMR blog will be a place where we can present more detailed thinking on industry and NewMR topics; complement our LinkedIn discussions; discuss themes and learnings from our events in more detail; and put forward provocative thoughts about where the industry is (or perhaps should) be going.

We are interested in your thoughts on what you like to see in the NewMR blog and we are looking for guest bloggers. If you have specific suggestions about topics you think we should cover or if you would like to guest blog or contribute in any other way please let me know at