As you may have heard, mail surveys are dead, obsolete, going the way of the dinosaurs, and have been since the late 1990’s when mail response rates started to decline. Even in the age of online panels and email surveys many ask, is there still a place for these “mythical” mail surveys in the current world of market research?
In many ways, web and mail surveys are very similar. In the realm of quantitative research, both modes can deliver large amounts of highly reliable data and can increase the anonymity of the survey taking process. Mail and web surveys are considered self-administered data collection modes, which means the questionnaire is not being directed by an interviewer. Self-administered surveys lend themselves well to research regarding sensitive topics and other questions skewed by societal norms, including health-related questions. The potential for a social desirability bias, in which respondents tend to take social norms into account when responding, tends to be low for self-administered questionnaires. (1) Respondent reporting of positive health-related questions, such as engaging in physical activity or medication compliance, can be inflated in interviewer lead surveys while behaviors that are less socially desirable, such as binge-eating, are more likely to be under-reported by respondents. (2)
Despite the similarities between mail and web surveys there are distinct differences which should be considered before selecting one or the other. In general, mail survey response rates are much higher than web surveys. The response rates of web surveys rarely exceed 30%. On the other hand, mail survey response rates average closer to 30-40% and can climb into the 80%-90% range. Particularly when a non-response bias is vital, attaining a high response rate can be of the upmost importance. When a study has a lower response rate there is a greater chance that the non-respondents are different than those who responded, thereby reducing the generalizability of the results.
While a high response rate is important, so too is attaining a low item non-response rate. This rate refers to the amount of questions that are left unanswered by a respondent, and is a significant problem in mail surveys. It has been shown that respondents do not answer significantly more questions in mailed surveys than in web surveys when responses were not forced. (3) In many cases, a higher response rate still outweighs the cost of having fewer answered questions in each survey. This could especially be the case when trying to reach populations that are more likely to answer mail surveys than via the web. Respondents of mail surveys tend to be of older, less-educated and lower income populations. (4) In terms of healthcare related research, this is one reason mail surveys lend themselves well to surveying Medicare and Medicaid populations.
Many other factors not covered here can also introduce bias into survey research. Clearly, web surveys with their quicker turnaround time, lower cost, and reduced chance of both a social desirability bias and an item non-response bias, have many advantages. Yet, mail surveys are just as suited for health-related topics, and are likely to achieve a higher response rate and therefore, could be more generalizable. When deciding on the mode or modes of survey administration there are advantages and disadvantages of each mode that need to be carefully weighed and anticipated. No one mode can outperform another. Nonetheless, selecting the best mode for your survey research can mean the difference between quality data and low value data.
References:
1 Tourangeau R, Smith TW. Asking sensitive questions: the impact of data collection mode, question format, and question context.
Pub Opin Quart 1996:60:275-304.
2 Siemiatycki J. A comparison of mail, telephone and home interview strategies for household health surveys. Am J Pub Hlth 1979; 69: 238–245.
3 Messer B, Edwards M, & Don Dillman. 2012. “Determinants of item nonresponse to web and mail respondents in three address-based mixed-mode surveys of the general public.” Technical Report 12-001, Pullman, WA: Social and Econimic Science Research Center.
http://www.sesrc.wsu.edu/dillman/papersweb/2012.html
4 Kwak, Nojin & Barry Radler. 2002. “A comparison between mail and Web surveys: Response pattern, respondent profile, and data quality.” Journal of Official Statistics 18(2):257-273.