The accuracy of research depends on rigorously following proven methods of reducing error in sampling, response, and questionnaire design.
The basics of sampling are obvious: sample among those you are trying to describe and don’t introduce bias.
This is achieved by sampling randomly. Randomness of who is asked to participate eliminates biases that arise using other selection methods. However, this is not convenient, easy, or cheap. A convenience sample of only the readers of a particular magazine, or only those who visit your website will only describe those unique populations. They do not represent all magazine readers or all internet users, much less all churches or all Americans. Convenience samples may save time or money, but they cannot describe the full population of the groups of church leaders, laity, or the unchurched that we seek to research.
Sampling is the one type of error that can be measured statistically. For example, we may state that a sample has a margin of error of +5% at the 95% confidence interval. This simply means if we repeated this study with a random sample 100 times, 95 of those times we would get answers not more than 5% higher or lower than the findings we are reporting. Sampling error never is completely eliminated. However a sample of sufficient size reduces the sampling error to the point we are confident that the differences in answers are real and not a function of how respondents were selected.
LifeWay Research uses proven methods of sampling that ensure that the various audiences we frequently describe reflect their population. When we utilize “panels” (individuals who have agreed to participate in surveys), we have ensured that the invitation to participate in the panel has previously been extended to all or a randomly selected subset of the population we are seeking to survey. Surveys conducted by phone or mail are conducted with random samples selected at the time of the survey.
Response bias occurs when portions of the randomly sampled population systematically do not respond. LifeWay Research uses various techniques to reduce and to measure the presence of “known” bias. Incentives are offered to those completing a survey as one means of encouraging reluctant respondents to participate. Another is contacting those in the random sample more than once seeking their participation. When we discover a response bias that impacts the use of a finding, LifeWay Research will state this in our findings.
Wording and sequence of questions within a survey can influence response. LifeWay Research utilizes experienced survey writers who can navigate many of these pitfalls based on prior experience. Surveys are reviewed by multiple researchers to search for potential problem spots. For new topics, a pre-test is conducted to ensure that wording used is understandable among the respondents who will complete the survey.