Over the past 30 years, I’ve conducted my share of multicultural research studies and learned a few things about sound research practices. The elements of sound multicultural research are generally not learned at academic institutions since much of their curriculum and textbooks devote little attention to this topic. Instead, research professionals are more likely to learn through the “school of hard knocks” and may even grasp these elements over time. In my opinion, many research professionals still don’t get it and appear increasingly indifferent to the consequences of their misguided practices. Following is a sampling of some questionable practices that have become rather commonplace in studies of multicultural populations:
- Incorrect usage of race/ethnic labels to screen respondents
- The use of monolingual surveys with known bilingual audiences
- Sample sizes that are too small to detect statistically significant differences
- Use of online surveys that exclude large proportions of consumers who are not online
- Consumer segments defined using unreliable language data
- Over-sampling of foreign-born respondents which leads to biased indicators
- Interviewers translating questions “on-the-fly”
- Adjusting survey data with unreliable self-reported language data
- Use of predictive dialers that lower respondent cooperation rates
One might conclude that these practices are more characteristic of small research organizations with limited resources; large organizations, however, are not immune. J.D. Power & Associates, for example, conducts their U.S. customer satisfaction research in just one language – English – which systematically excludes feedback from many customers who prefer a survey in their native language. Nielsen and Arbitron continue to use self-reported language information to adjust their radio and television ratings, despite evidence that such information is unreliable. Arbitron, in particular, is currently under heavy criticism in regards to their PPM methodology and sampling strategies that allegedly under-estimate Hispanic and African-American radio audiences.
The Census Bureau tells us that by the year 2010, African-Americans, Hispanics and Asians will collectively comprise one-third of the nation’s population – an astonishing 102 million persons. You would think that this seismic demographic transformation would alone motivate industry research organizations to abandon their outdated practices in favor of methods that more accurately capture the experiences of multicultural populations. Unfortunately, that has not been the case.
To get the research industry moving in this direction, we clearly need to expand the dialogue on these issues since academia is moving at a snail’s pace in this arena, too many important deliberations about methodology are taking place behind closed doors, and currently available books on multicultural marketing provide minimal guidance on measurement issues. Thus, I have dedicated this blog as a forum to (a) discuss methodological issues in regards to research with multicultural populations, (b) review selected studies, articles, books, and white papers that address multicultural issues, and (c) create a community of researchers and non-researchers who share similar concerns about the need to improve the quality and transparency of multicultural research. While I do not pretend to know all of the answers to these issues, I will commit to an objective and passionate discussion with members of my blog community.