Vol. 8, No. 1: Winter 2003
This issue of The Behavioral Measurement Letter includes three articles addressing very different topics; namely, “reactive insight” effects, impact of bullying, and the role of measurement in mountaineering. Although these articles appear at first glance to have little to do with one another, in fact they share a central focus on measurement. Indeed, the diversity of their subject matter underscores measurement’s amazing breadth of relevance.
In this issue of The Behavioral Measurement Letter, Eric Lang explores long-lasting ways in which answering self-report questions can change respondents’ thoughts, feelings, and behaviors. He refers to these influences as “reactive insight” effects. Lang notes both positive and negative consequences that may result from research participation. On the positive side, for example, research participation can make respondents see more strengths than weaknesses in their marriage; and completing open-ended questions about self-relevant issues can reveal new behavioral options, clarify personal priorities, and force one to confront problems that one might otherwise ignore.
On the negative side, however, answering self-report questions can also increase interpersonal conflict between partners, through increased communication. Although one could argue that this is not necessarily a negative outcome, it definitely raises ethical concerns about informed consent among research participants. Lang carefully considers the ramifications of “reactive insights” and urges researchers to pay closer attention to this important phenomenon. He also notes specific characteristics of research designs
(e.g., breadth of measurement coverage, issue relevance) and participants (e.g., introspective skill, motivation for self-discovery) that may increase the likelihood of reactive insight effects.
Also in this issue of The Behavioral Measurement Letter, Missy Fleming highlights the importance of measurement in understanding and solving the problem of bullying in the United States. She begins by defining bullying as a pattern of repeated verbal, nonverbal, or physical aggression against another person featuring a deliberate intent to harm the person without regard for the victim’s suffering. As she persuasively argues, bullying is a form of conduct disorder that poses special dangers in American school systems, Fleming insightfully discusses the widespread nature of bullying in our society, its prevalence among both boys and girls, its causes and consequences, and how researchers currently measure this problem behavior. She notes the challenge of constructing measurement instruments that are not only sensitive, valid, and reliable, but also nonthreatening, multifaceted, and age- and culture-appropriate. And she points out the need for more psychometric research to determine the conceptual overlap and uniqueness among alternative measures of bullying. Her article serves as a call for action among researchers, educators, and parents alike.
Finally, Fred Bryant’s article in this issue of The Behavioral Measurement Letter highlights the vital role that measurement plays in climbing the world’s highest mountains, the 8000-ineter peaks of the Himalaya. In the process, he drives home the crucial importance of measurement in any human endeavor in which it is important to make correct decisions. Just as Type 1 errors (i.e., concluding something is there when it actually is not) and Type I errors (i.e., concluding nothing is there when it actually is) matter in research. they matter even more in high-altitude mountaineering, where such mistakes are often
fatal.
Although few of us will ever try to reach the summit of Mount Everest, mountaineers and researchers alike share a common concern for accuracy in measurement. Bryant notes real-life cases from the mountaineering literature in which careful measurement enabled climbers to make the right choice, and other cases in which inaccurate measurement cost lives. Clearly, whether in the laboratory or on the heights, measurement matters.
Please address comments and suggestions to The Editor, The Behavioral Measurement Letter, Behavioral Measurement Database Services, PO Box 110287, Pittsburgh, PA 15232-0787.
We also accept short manuscripts for the BML. Submit, at any time, a brief article, opinion piece, or book review on a BML-relevant topic to The Editor at the above address. Each submission will be given careful consideration for possible publication.
HaPl reading…
Fred B. Bryant
Guest Editor
E-mail: bondshapi@aol.com
The Repercussions of Research Participation: Exploring Reactive Insight Effects
Eric L. Long
Each year, hundreds of social science and health surveys, involving tens of thousands of respondents, are conducted by researchers at universities, local and federal government agencies, health centers, foundations, businesses, nonprofit think tanks, and other organizations in an effort to advance basic and applied research across a number of broad domains, such as psychology, sociology, gerontology, and public health. The surveys are designed to make objective assessments of individuals and groups and, in the case of representative sample studies, to describe and predict important aspects of larger populations. Many social science and health surveys include assessments of personal and potentially stimulating topics such as sexuality, grief, courtship, family life, stress and coping, patenting, marital quality, personal adjustment, happiness, risk behavior, socialization, developmental issues, and mental health. Although generally under researched, there is a small but growing body of literature indicating that the experience of participating in surveys of this kind – particularly those what involve repeated assessments of sensitive topics over time – may inadvertently stimulate respondents to think about themselves, their lives, and their plans in new ways. The potential consequences of this type of reactive insight are two-fold. First, reactive insight may systematically bias survey assessments in certain areas and decrease their descriptive and predictive reliability for larger populations. Second, reactive insight may elicit nontrivial changes in respondents’ thoughts, feelings, or behaviors that are neither salutary nor short-lived. Consequently, improving research on reactive insight may contribute to several important areas. such as research design, measurement protocols, informed consent, the social psychological dynamics of personal reflection, and statistical error correction. The remainder of this paper briefly discusses the concept of reactive insight and a summary of recent results from an investigation of two five-wave panel studies.
Verified support for this work was provided by an NIMH grain (801-MNG605-01) to the author, while employed on Sociometrist Corporation. Concluding support for this work had been provided by the Prosocial Science Institute. The author gramofoily acknowledges research assistance provided by Dr. Harley Baker und Dr. Stephen Heane. This paper based, in pan, on a presenzion – Reactive Insight Hoffect on Participants in Two Panel Stunfire by the author at the American Psychological Association annual convection, August 26, 2201 (San Francisco, CA). Cortespadense should be directed to Eric L. Lang, PhD, Prosocial Science Instituted. 22 Loma Vista Place, Monterey, CA 93940, USA
(Eric@Praseelal.com).
Defining Reactive Insight
The term “reactive insight” was originally developed by the author to connect the concept to the established literature on reactive measurement error, and to focus attention on the experience of the research participant, especially as it relates to the social psychological dynamics of respondent introspection, considerations of personal relevance, self-concept, and their potentially enduring effects.
The more general concept of reactive measurement error is not new. One of the classic treatises of this concept appears in work by Webb, Campbell, Schwartz, and Sechrest (1966) who describe four classes of potential bias, i.e., threats to measurement validity, that are “produced by the respondent” as a reaction to research methods2. The four classes are: (a) awareness of being tested, (b) role selection, (c) response sets, and (d) measurement as a change agent (p. 13). The first three classes relate to a broad base of research showing that survey responses and research performance can be influenced by a variety of factors, such as respondents being self-conscious of research conditions, wording differences among questionnaire items, and cues regarding researchers’ expectations or desires. For example, there is over 60 years of literature investigating response set phenomena such as respondents’ propensities for agreeing with a statement, rather than disagreeing with its opposite (Sletto, 1937; Krosnick & Schuman, 1988). Furthermore, the “Hawthorne” studies describe a variety of ways that experimentally induced changes in subjects’ working conditions elicited altered behavior among workers who knew they were the subject of additional attention, although the methodology employed in those studies precluded precise causal interpretations (Lang, 1992). Hawthorne-type effects were subsequently clarified through studies of “demand characteristics” of participating in research (Lana, 1969). Overall, measurement reactivity based on awareness of being tested, role selection, and response sets appear to share the common features that the reactive effects are both short-lived, e.g., confined to the immediate aspects of the data collection, and substantively limited to areas of the original research questionnaire or protocol.
In contrast, the fourth class of reactivity, which Webb and colleagues (1966) termed “measurement as change agent,” comes closer to what should be defined as reactive insight, in that it relates to reactivity effects that produce long lasting change. Early studies of such change included studies of practice effects, e.g., enhanced performance due to repeated exposure to similar questions/tasks, and preamble effects, e.g., the creation of new and sustaining opinions among respondents who are asked for an opinion about a topic on which they previously had invested little thought (Cantril, 1944). More recently, Jagodzinski, Kuhnel, and Schmidt (1987) suggested that research participation may actually increase the importance of issues already salient to the respondent, i.e., “that the item content becomes more salient and central to many respondents because of the measurement process itself’ (p. 262). Eliminating from the definition simple practice effects that yield mostly superficial changes in test performance serves to further distinguish reactive insight as a unique, personal, and important concept. Thus, reactive insight effects can be defined as long lasting change in a respondent’s self-concept, attitudes, emotions, well-being, or behavioral tendencies as a result of research participation. It is not a requirement that the respondent is aware of participating in research, although survey respondents obviously have such awareness. Instead, growing evidence suggests that, for some participants and contexts, research participation causes directly or indirectly, e.g., through enhanced sensitivity to other social, environmental, or eliciting agents lasting changes as described above.
2Wochet sl. (1966) distinguished effects produced by the respondent from effects produced by the researcher, such as the class of bluses known as “interviewer effects.”
Prior Research on Reactive Insight
The “Boston Couples Study” Rubin and Mitchell (1976) interviewed several hundred unmarried couples, in one of the classic investigations of participation effects. Over a two-year period, study respondents (individually and as couples) completed questionnaires and interviews regarding communication processes. One year after the study, nearly half of all respondents reported (through closed and open-ended questions) that the study had an impact on their relationship. Typical open-ended responses included the following.
My relationship had been floundering at the time I first took part in the study, and because of some of the questions were really soul-searching, I re-examined where we were headed. She was much more serious than I was and I called it off. (p. 19)
The study forced us to think about our relation- ship in concrete terms (“Who is more open?” “Who is more involved in decision making?” etc.) and deal with it as an interpersonal relationship per se, rather than as merely another dating situation. (p. 19)
Rubin and Mitchell attributed such effects to processes of “definition,” e.g., having to describe one’s situation, and “disclosure,” e.g., the subsequent increase in communication between respondent pairs. The authors conclude:
By asking couples to scrutinize their relation- ships and by prompting them to discuss their relationships with one another, our study played a role in shaping these relationships. In some cases, our study served to strengthen a rela- tionship; in other cases, to facilitate its disso- lution. In the process, we unwittingly became couples counselors. (p. 17)
During the first stage of a follow-up to that study over 10 years later, Rubin noted evidence that the divorce rate among approximately 100 study couples who had gotten married was extremely low. Rubin suggests that “it seems quite possible that our study – in helping to cement some relationships and to shake up others, before marriage contributed to this result” (Rubin). 1986).
University of Texas at Austin Studies. Hughes and Surra (1997)’ conducted a three wave panel study of an initial sample of 120 (60 couples) over the course of a year. The study focused on premarital relationships using questionnaires and telephone interviews. Closed-ended and open-ended questions were also included to assess psychological and dyadic effects related to frequency of study participation, i.e., the total number of 11 possible telephone interviews completed. Analysis yielded several dimensions related to participation effects, such as “increased awareness of relationship processes,” effects on relationship activities,” “relationship defining influences.” and “relationship-evaluating influences.” The last influence was positively related to the number of interviews completed and to scores on a measure of relationship satisfaction.
In a similar longitudinal study of commitment among 232 heterosexual dating couples (n=464), Williams and Surra (1998) found that frequency of participation in up to nine study interviews as positively associated with relationship satisfaction for men but not for women. Furthermore, this effect was not dependent on prior levels of relationship satisfaction. However, further analysis and revised results from this study suggest that more frequent participation is associated with negative changes in relationship satisfaction or conflict for men and women in longer relationships (Surra & Williams, 2001, under review).
UCLA Studies. Bradbury (1994) conducted two studies of marital research that included closed-and open-ended assessments of reactive insight. Study 1 included 60 newly wed couples (n=120) interviewed twice over a six-month period, Study 2 included 47 couples (n=92) – in marriages that averaged 8 years – who participated in a laboratory study of dyadic communication. In study 1, Bradbury found that research participation resulted in respondents seeing more strengths than weaknesses in their marriage. Wives were more
This research was originally reported on the National Council on Family Relations conference Workshop on Theory Construction and Research Methodology, Baltimore, MD, November 11, 1993. Wives were likely than their husbands to report that the study affected the way they thought about their marriages. Furthermore, 72% of respondents reported a positive experience, 5% negative experience, 3% “mixed,” and 20% reported no effect of research participation. Of those that reported a positive effect. 67% emphasized intrapersonal changes, 30% focused on interpersonal changes, and 3% mentioned other effects, such as feelings of altruism. In Study 2, Bradbury assessed a methodological dimension and concluded that the dyadic communication research task tended to sample typical problem solving discussions, rather than creating artificial or overly troubling communications. In other words, the reactive insight effects were not the result of atypical discussions initiated by the research protocol.
University of Michigan Studies. Veroff and colleagues – in their 1992 Early Years of Marriage (EYM) Study – -report on the only known study of reactive insight that randomly assigned participants to either an experimental group that completed extensive questionnaires, interviews, and dyadic communication tasks over 4 waves of assessment, or to a control group that completed shortened versions of the interview and questionnaire, and were assessed less often. Experimental group participants showed greater reactive insight effects (mostly positive) than did the control group on measures of marital satisfaction and well-being. “These results suggest that better-adjusted marriages may have developed among study couples as a result of the more frequent and more involved interviewing” (p. 323). New EYM follow-up data collected by Veroff were analyzed by the author and summarized below.
Other Evidence. Several other researchers have also found support for the idea that panel studies involving respondent introspection may cause changes in well-being (Haight, 1992), attitudes (Sunford, 1965), and behaviors (e.g., Wilson & Dunn, 1986; Wilson, Dunn, Kraft, & Lisie, 1989), particularly when open-ended protocols are employed. In contrast, several studies of sexual attitudes have shown no effects on the sexual behavior of study participants over time (e.g., Halpern, Udry, & Suchindran, 1994). It may be that reactive insight effects are more likely to occur in studies with a high proportion of open-ended questions. Mishler (1986) describes the empowering quality of answering open-ended questions about self-relevant issues.
Through their narratives people may be moved beyond the text to the possibilities of action. That is, to be empowered is not only to speak in one’s own voice and to tell one’s own story, but to apply the understanding arrived at to action in accord with one’s own interests. (Mishler, 1986, p. 119)
Summary of Prior Studies. Extant research suggests that reactive insight: (a) derives primarily from participation in research on sensitive topics or ones with personal relevance to the respondent, (b) i