Designing Surveys for a Diverse Society

            Recently I ended up in a debate with a friend about survey administration in India. He was horrified to realize that we allowed our interviewers to administer survey questions by substituting local terminology as needed. He worried about the errors being introduced by interviewers as they translated on the fly and suggested all instruments should be translated. I tried to explain that we translate the instruments in 12 different Indian languages but there is so much local variation in language, often between neighboring districts, that it is impossible to undertake translations that capture these variations. Census of India lists 50 dialects in Hindi alone.

            His answer was that with rising education standard language is sure to become more prevalent and instead of interviewer translating the Hindi questionnaire in Braj Bhasha in Mathura district,  one can just use Khari Boli (standard Hindi). Having done fieldwork in Mathura, all I could say was that the educated respondents will understand Khari Boli but the rest will be scratching their heads about what these strange interviewers are asking.

            This discussion has led me to reflect on the challenges of designing surveys that balance a variety of biases and errors and get as close to the lived realities of our respondents as possible. In survey methodological discourse, these differences are often characterized by differences between standardized surveys and conversational surveys.

            This is a complex problem in any setting, linguistic and cultural diversity of India poses additional difficulties. Even identifying the nature of the beast facing us is sometimes difficult, let alone estimating the magnitude of errors.

            Interviewer Errors: The greatest fear of any survey designer is that interviewers will not ask the questions in a standardized fashion, making comparisons across respondents difficult. When the interviewer tries to substitute a local word for family planning, is she translating correctly or is she referring to family limitation, leaving out spacing methods? Moreover, when respondents don’t understand a question, standard methodology would prohibit interviewers from providing any explanation and just ask the respondent to answer the questions as they understand it. However, if we allow interviewers to deviate from the standard questionnaire, will they then bias the response?

            Comprehension Errors: The other side of the coin, which has received very little attention, has to do with comprehension errors. Standardized surveys force interviewers to read out questions exactly as written with no clarification. This forces the respondents to try to judge what the questions mean and if they ask for clarification, the standard response is, “whatever it means to you.” Studies have shown that the nature of response depends on whether interviewers are allowed to provide clarificatory answers. For complicated questions about sexual behavior, it is not surprising that clarification makes a difference.  However, even for simpler concepts like hours of work, once the workers are not in standardized work situations, conversational interviews provide very different results from standardized interviews.

            I find it difficult to tell anyone how many hours I work in a week. If you asked me how many hours I spend in my university office, that is easy. However, does the time I am spending right now writing this blog, curled up on my couch count? What about the time I spend internet browsing while hiding in a corner during faculty meetings? Many workers have unique situations, this includes women simultaneously raising chickens and children and domestic workers residing on premises with highly permeable boundary between work and personal time. It would be hard for them to identify their work hours without some guidelines.

             Frederick Conrad and Michael Schober have conducted a series of experiments where they find that both standardized and flexible interviewing produce high levels of accuracy when respondents have no doubts about how concepts in a question map onto their circumstances. However, flexible interviewing produces higher response accuracy in cases where respondents are unsure about these mappings. They also studied the amount of error introduced by interviewer flexibility and found it to be relatively small.

            My strange experiences with questionnaire design for India Human Development Survey (IHDS) has taught me that what is common knowledge and what is not is hard to know a priori. Widely used survey questions from World Value Survey, “How much confidence do you have in the government” produced puzzled counter questions in our interviews, “What do you mean, confidence about what?” Even in World Value Survey, Indian respondents, particularly those with low levels of education, have greater difficulty responding to questions about confidence in institutions. Whereas less than 2% of the respondents in other countries (e.g. United States, Germany and Peru) were unable to answer the question about confidence in government, in India almost 10% responded “Don’t know”. In contrast, complex and potentially sensitive question like, “Does anyone in your household practice untouchability?” was crystal clear and few respondents had trouble answering it. Untouchability is clearly something almost all Indians seem to understand; confidence in institutions seems not to be an easy concept for Indian respondents with low levels of education.

            Cultural Constraints: It is hard enough to develop a rapport with rural respondents as an urban, educated interviewer, being handcuffed by a standard questionnaire is even worse. When my mother, who was raised in South Gujarat became a gynecologist in a tiny town in Saurashtra, barely a couple of hundred kilometers away from where she was raised, initially she couldn’t understand who her patients referred to when they would say “Your brother wants me to deliver at home,” or “Your brother is busy now but will buy my medicines tomorrow.” Finally, she figured out that women who almost never referred to their husbands by name, also found it very awkward to say the phrase 'my husband'. Creating a fictive kinship and referring to their husbands as 'your brother' was their normal mode of conversation. I also find it much easier to ask about bhaisab (brother) and bhabhiji (sister-in-law) instead of ‘your husband’ and ‘your wife’ in rural areas. No such camouflage is needed among urban professionals. This is of course a trivial example but think about asking about menstruation and sexual relations without using locally acceptable terms!

Research Needs:

            When beset by competing considerations, the standard response of social scientists is to conduct experiments to examine the magnitude of various biases. However, few such experiments have been conducted in India that test the role of interviewer flexibility in making interviewes more comprehensible to speakers of different dialects. How difficult it is for someone speaking Awadhi or Bhojpuri to understand standard Hindi? If they were administered a questionnaire in standard Hindi, would it increase the interview length? How would it affect completion rate or quality of response?  This would be a highly fruitful area of investigation for future research.

             

© Sonalde Desai 2012