Warning Participants about Limited Generalizability

A researcher and his student submitted an IRB application for a simple survey about perceptions of incarcerated individuals. The questionnaire contained new questions the student had created specifically for this study (the topic was fairly specific and no existing measures were usable). The IRB application at this institution contains a section requiring applicants to indicate the validity of any measures developed by the applicants. Since this was a new measure, no validity data existed, and this was noted on the application. The IRB requested a number of minor modifications. Among them was a request that the researcher and their student "add a statement, saying the survey has limited generalizability due to the unknown validity of the instrument, to the informed consent, so potential participants know that prior to engaging in the study."

This resulted in a number of emails and one phone conversation, during which the researcher explained the difference between validity and generalizability, as well as the fact that it is common practice across many social science fields to use novel questions in survey research. It also became apparent that the IRB member who requested this particular change was unable to explain exactly what purpose such a statement would serve from a research ethics standpoint. Additionally, the researcher explained that such a statement might bias the data collected, as participants would be suspicious of the questions in the survey, which could have a number of downstream effects on their responses.