Essays: Changing Definition of Validity in the Social Sciences
I wrote this paper for a class on participatory action research that I took last semester — it ranks among my favorite courses. Rather than focus on practice or methods, this was a review of different strands of participatory research, action research and community-based research, with an emphasis on understanding the similarities and differences between different approaches situating our own approaches and philosophies.
The concept of “validity” is a tricky one for action researchers and something I’ll continue to grapple with as I embark on my dissertation. I do the work that I do because I’m part of a movement that wants to change our current food system to be more diverse, locally-integrated, environmentally resilient, and equitable. I happen to think that a combination of informed action/experimentation, and rigorous documentation/analysis/evaluation is part of the way to effect that change. But what makes this “research” research? By what criteria should my “results” be judged and deemed “valid”?
Written for a “mini-paper” reading response 4/13/11:
The concept of validity in social science research originally developed within a positivist paradigm. Since the advent of new approaches to human inquiry that challenge positivist assumptions about the nature of reality and the purpose of research (such as critical theory, constructivism, and participatory research), the concept of validity has stretched beyond its original meaning.
The positivist approach to inquiry assumes truth is observable and testable, that the purpose of research is to explain and predict, and that social science should be objective, value-free, and clearly separated from practice. Within this mode of thinking, asking about the validity of research means asking whether our tests or methods accurately measure “whatever it is that is supposed to be measured” (p. 343, Wolcott, 1990). On the other hand, “transgressive” forms of validity like the crystalline or situated validity embraced by researchers like Laurel Richardson and Patti Lather, seek instead to intentionally “problematize reliability, validity and truth” (Richardson qtd in Guba & Lincoln, 2005). In an article examining both contradictions and blurring between old and new research paradigms, Lincoln and Guba (2005) suggest that in all cases, validity seeks to address the question:
Are these findings sufficiently authentic (isomorphic to some reality, trustworthy, related to the way others construct their social worlds) that I may trust myself in action on their implications? More to the point, would I feel sufficiently secure about these findings to construct social policy of legislation based on them? (p. 205)
Lincoln and Guba (2005) separate validity into two parts: validity of method and validity of interpretation. They posit that traditional positivist definitions of validity like the kind described by Litwin (1995) in “How to Measure Survey Reliability and Validity,” deal mostly in the “rigor in the application of method” (Lincoln & Guba, 2005, p. 205). While critical theorists, constructionists, and other “new-paradigm” researchers are not exempt from questions about their methods of observation, they also grapple with questions of how, what, and why we interpret observations.
Wolcott’s 1990 article, “On Seeking – and Rejecting – Validity in Qualitative Research,” is an early example of a struggle to look beyond a concept of validity tied to methodological rigor or procedure and get at valid interpretation, or rigor “in ascribing salience to one interpretation over another and for framing and bounding an interpretative study itself” (Lincoln & Guba, 2005, p. 205). Wolcott starts off describing the tactics he employs to “satisfy the implicit challenge of validity” and “not get it all wrong” (1990, p. 347). He then pushes beyond the concept of validity tied to criteria like internal consistency and the capacity to predict, and proposes instead that ethnographic research should seek to understand social structures that we humans construct. In 1990, Wolcott calls this a ‘rejection’ of validity, but fifteen years later, Lincoln and Guba describe how other new-paradigm researchers have chosen to stretch rather than reject the concept of validity and ask not only about what constitute valid methods of measurement and observation, but also what constitutes valid interpretation. “Can our cocreated constructions be trusted to provide some purchase on some important human phenomenon [what Wolcott might call understanding]?” (2005, p. 206)
The shift in focus from methodological validity to questions about interpretive validity is ultimately rooted in a shift in the ontology and epistemology of new modes of social science. In order to determine whether research findings are authentic to ‘reality’ and to know whether and how our findings engage with ‘reality,’ we must first understand how we view the nature of reality (ontology) and how we acquire knowledge about this reality (epistemology). For example, foundationalists who believe in a transcendental reality might say “real phenomena necessarily imply certain final, ultimate criteria for testing them as truthful” (Lincoln & Guba, 2005, p. 204). On the other hand, antifoundationalists who refute the idea of a truth separate from human perception might argue “agreement regarding what is valid knowledge arises from the relationship between members of some stake-holding community” (Lincoln & Guba, 2005, p. 204). In the latter case, what is valid must always be negotiated because reality only exists as it is constructed between people.
Certain definitions of what is valid go beyond questions about the nature of reality and knowledge into the purpose and ethical obligations associated with inquiry (axiology).
When social inquiry becomes the practice of a form of practical philosophy – a deep questioning about how we shall get on in the world and what we conceive to be the potentials and limits of human knowledge and functioning – then we have some preliminary understanding of what entirely different criteria might be for judging social inquiry. (Lincoln and Guba, 2005, p. 206)
Ontological, educative, catalytic and tactical validities, for example, ask about the outcome of inquiry; specifically, increased awareness or increased capacity or tendency for individual or collective action. These types of validity play a major role in participatory research and action research because these modes of inquiry tend to make social transformation or change their explicit end-goal. Here what research is ‘valid’ becomes less about what mirrors reality and more about what has the capacity to change, form or shape reality.
Guba, Egon G. and Lincoln, Yvonna S. 2005. Paradigmatic Controversies, Contradictions, and Emerging Confluences. In: Handbook of Qualitative Research, Third Edition, edited by N. Denzin and Y Lincoln. Thousand Oaks, CA: Sage Publications.
Litwin, M.W. 1995. How to Measure Survey Reliability and Validity. In The Survey Kit, edited by Arlene Fink. Thousand Oaks, CA: Sage Publications.
Wolcott, H. F. 1994. On Seeking – and Rejecting – Validity in Qualitative Research. Chapter 11, in: Transforming Qualitative Data: Description, Analysis, and Interpretation. Thousand Oaks, CA: Sage Publications.