What is a good number of questions for a survey?

Saudi J Anaesth. 2022 Jan-Mar; 16(1): 65–68.

Abstract

A questionnaire plays a pivotal role in various surveys. Within the realm of biomedical research, questionnaires serve a role in epidemiological surveys and mental health surveys and to obtain information about knowledge, attitude, and practice (KAP) on various topics of interest. Questionnaire in border perspective can be of different types like self-administered or professionally administered and according to the mode of delivery paper-based or electronic media–based. Various studies have been conducted to assess the appropriateness of a questionnaire in a particular field and methods to translate and validate them. But very little is known regarding the appropriate length and number of questions in a questionnaire and what role it has in data quality, reliability, and response rates. Hence, this narrative review is to explore the critical issue of appropriate length and number of questions in a questionnaire while questionnaire designing.

Keywords: Biomedical research, demography, population surveillance, reproducibility of results, surveys, and questionnaires

Introduction

A questionnaire is an essential tool in epidemiological surveys and mental health surveys and to assess knowledge, attitude, and practice (KAP) on a particular topic of interest. In general, it is a set of predefined questions based on the aim of the research.[1]

Designing a questionnaire is an art which unfortunately is neglected by most researchers.[2] A well-designed questionnaire not only saves time for a researcher but helps to obtain relevant information most efficiently, but designing such a questionnaire is complex and time-consuming.[3,4]

The quality of the data obtained by a specific questionnaire depends on the length and number of questions in the questionnaire, the language, and the ease of comprehension of the questions, relevance of the population to which it is administered, and the mode of administration, i.e., the self-administered or paper method or the electronic method [Figure 1].[5,6]

Qualities of a well-designed questionnaire

Response rate is defined as the number of people who responded to a question asked divided by the number of total potential respondents. Response rate which is a crucial factor in determining the quality and generalizability of the outcome of the survey depends indirectly on the length and number of questions in a questionnaire.[7,8]

Several studies have been conducted to assess the appropriateness of the questionnaire in a particular field and methods to translate and validate them. But very little is known regarding the appropriate length and number of questions in a questionnaire and what role it has in data quality and reliability. Hence, this narrative review is to explore the critical issue of appropriate length and number of questions in a questionnaire while questionnaire designing.

What is a questionnaire

Merriam Webster defines the questionnaire as “a set of questions for obtaining statistically useful or personal information from individuals,” whereas Collins defines a questionnaire as “a questionnaire is a written list of questions which are answered by a lot of people to provide information for a report or a survey.” The oxford learners’ dictionaries also give a somewhat similar definition which states that a questionnaire is “a written list of questions that are answered by several people so that information can be collected from the answers.”[9,10,11]

Thus, this provides a simpler meaning that a questionnaire in simpler terms is a collection of questions that can be used to collect information from various individuals relevant to the research aims.

Where are questionnaires generally applied?

A questionnaire, in general, can be applied to a wide variety of research which can either be quantitative or qualitative research which completely depends on how and in which a number of open-ended questions are asked.[12]

Questionnaires are generally applied when a large population has to be assessed or surveyed with relative ease where they play a crucial role in gathering information on the perspectives of individuals in the population.

There is a variety of applications of questionnaire in opinion polls, marketing surveys, and in politics, wherein the context of biomedical research questionnaires are generally used in epidemiological surveys, mental health surveys, surveys on attitudes to a health service or health service utilization, to conduction knowledge, attitude, and practice (KAP) studies on a particular issue or topic of interest.[13,14]

What are the types of questionnaire?

Questionnaires in general are of two types those which are in paper format and those which are in electronic format. The questionnaire can further be of two types i.e., self-administered or professionally administered via interview. The paper format can be administered easily both in self-administered mode or professional administered mode via direct administration when the population is relatively small as it is cumbersome to manage and store the physical questionnaire, paper format can also be administered to a larger population via postal surveys. Electronic questionnaires can be easily administered to a larger population in self-administered mode via Internet-based services like google forms, e-mails, SurveyMonkey, or Survey Junkie, etc. When administering professional-administered questionnaires professional telephonic services must be utilized to interview a larger population in a shorter duration of time.[15,16,17]

What it is required to answer individual questions in the questionnaire or the burden imparted on respondents

As mentioned by Bowling, in general, there are at least four intricated steps required in answering a particular question in a questionnaire, these steps are comprehension, recall of information asked by the question from the memory, judgment on the link between the asked question and the recall of information, and at last communication of the information to the questionnaire or evaluator [Figure 2].[18]

Steps involved for answering a particular question in the questionnaire

In the case of a self-administered questionnaire, there is also a need for critical reading skills which is not required in one-to-one or face-to-face interview which only requires listening and verbal skills to respond to questions in the same language in which they are being asked or interviewed.[18]

There are many other crucial factors which play an important role in deciding the utility of questionnaire in various research, one such factor is the literacy of the participants which is a major limiting factor in self-administered questionnaires. Whereas, the other factors include the respondent's age, maturity, and level of understanding and cognition, which are some of the other ways related to the comprehension of the questions.[19]

Do the length of the questionnaire matters?

Length and number of items in the questionnaire play a crucial role in questionnaire-based studies or surveys, it has a direct effect on the time taken by the respondent to complete the questionnaire, cost of the survey or study, response rate, and quality of data obtained.[20]

As evident from the study conducted by Iglesias and Torgerson in 2000, on the response rate of a mailed questionnaire, an increase in the length of the questionnaire from five pages to seven pages reduces the response rate from women aged 70 years and over but on contrary does not seems to affect the quality of response to questions.[21]

Another study conducted by Similar Koitsalu et al.[22] in 2018 reported that they were able to increase overall participation and information gathered through a long questionnaire with the help of prenotification and the use of a reminder without risking a lower response rate.

Whereas Sahlqvist, et al.[23] in 2011 reported that participants were more likely to respond to the short version of the questionnaire as compared to a long questionnaire.

Testing of ultrashort, short, and long surveys of 13, 25, and 75 questions, respectively by Kost et al.[24] in 2018, revealed that a shorter survey utilizing a short questionnaire was reliable and produce high response and completion rates than a long survey.

Bolt, on the other hand, in 2014, found a surprising find that reducing the length of a long questionnaire in a physician survey does not mean that it will necessarily improve response rate hence to improve the response rate in nonresponders’ researchers may think to utilize a drastically shortened version of the questionnaire to obtain some relevant information rather than no information.[25]

But the most interesting find comes from the web-based survey giant “Survey Monkey,” which states that there is a nonlinear relationship between the number of questions in a survey and the time spent answering each question. Which in other words can be explained as more there are questions in a survey lesser time respondent spend answering each question which is known as “speeding up” or “satisficing” through the questions. It is also observed that as the length of and the number of questions asked increased there is an increase in a nonresponse rate. This in term affects the quantity and reliability of the data gathered.[26]

What happens when respondents lose interest?

When there is a loss of interest, in the case of a long length questionnaire or extensive interviews, the bored respondents provide unconsidered and unreliable answers, or in other scenarios, it may lead to high nonresponse to questions. Where on one side a high nonresponse rate may lead to difficulty in data analysis or an unacceptable reduction in sample size, whereas on the other side, unconsidered or unreliable answers may defeat the whole purpose of the research [Figure 3].[19]

Consequences of Loss of interest in research participant

Considerations while using a long questionnaire

While using a long questionnaire, a high nonresponse rate should always be expected hence appropriate measures to address the missing data should be considered such as data trimming or data imputation depending on the amount of data missing.[27,28]

While the loss of interest can be administering counteracted by dividing the questionnaire into sections and administering each section separating to avoid respondents’ fatigue or boredom.[19]

It is always advised that the administration of telephonic interview–based questionnaire should be kept short in general about 30 min to prevent fatigue or inattention which may adversely affect the quality of data. In the case of a very long telephonic interview, questions can be divided into sections, and each section can be administered on separate days or shifts lasting 30 min each. A long questionnaire should preferably be administered through face-to-face interviews.

Conclusion

Designing a questionnaire is an art and requires time and dedication, which in turn leads to the easiest way to measure the relevant information on a desired topic of interest. But many a times, this crucial step in biomedical research is ignored by researchers. With this narrative review, we were able to provide a glimpse of the importance of a good questionnaire. A good questionnaire can be of 25 to 30 questions and should be able to be administered within 30 min to keep the interest and attention of the participants intact. It is observed that as the number of questions increases there is a tendency of the participants speeding up or satisficing through the questions, which severely affect the quality, reliability, and response rates. In case a long questionnaire is essential, it should be divided into sections of 25 to 30 questions each to be delivered at a different time or day. In the case of a long questionnaire i.e., more than 30 questions, a larger amount of missing data or nonresponse rates must be anticipated and provisions should be made to address them. At last, it is always advised that shortening a relatively lengthy questionnaire significantly increases the response.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

References

1. Slattery EL, Voelker CC, Nussenbaum B, Rich JT, Paniello RC, Neely JG. A practical guide to surveys and questionnaires. Otolaryngol Head Neck Surg. 2011;144:831–7. [PubMed] [Google Scholar]

2. Minto C, Vriz GB, Martinato M, Gregori D. Electronic questionnaires design and implementation. Open Nurs J. 2017;11:157–202. [PMC free article] [PubMed] [Google Scholar]

4. Artino AR, Jr, La Rochelle JS, Dezee KJ, Gehlbach H. Developing questionnaires for educational research: AMEE Guide No.87. Med Teach. 2014;36:463–74. [PMC free article] [PubMed] [Google Scholar]

5. Boynton PM, Greenhalgh T. Selecting, designing, and developing your questionnaire. BMJ. 2004;328:1312–5. [PMC free article] [PubMed] [Google Scholar]

6. Setia MS. Methodology series module 8: Designing questionnaires and clinical record forms. Indian J Dermatol. 2017;62:130–4. [PMC free article] [PubMed] [Google Scholar]

7. Phillips AW, Reddy S, Durning SJ. Improving response rates and evaluating nonresponse bias in surveys: AMEE Guide No.102. Med Teach. 2016;38:217–28. [PubMed] [Google Scholar]

8. Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, et al. Increasing response rates to postal questionnaires: Systematic review. BMJ. 2002;324:1183. [PMC free article] [PubMed] [Google Scholar]

14. Jones TL, Baxter MA, Khanduja V. A quick guide to survey research. Ann R Coll Surg Engl. 2013;95:5–7. [PMC free article] [PubMed] [Google Scholar]

15. Bojcic JL, Sue VM, Huon TS, Maletis GB, Inacio MC. Comparison of paper and electronic surveys for measuring patient-reported outcomes after anterior cruciate ligament reconstruction. Perm J. 2014;18:22–6. [PMC free article] [PubMed] [Google Scholar]

16. Regmi PR, Waithaka E, Paudyal A, Simkhada P, van Teijlingen E. Guide to the design and application of online questionnaire surveys. Nepal J Epidemiol. 2016;6:640–4. [PMC free article] [PubMed] [Google Scholar]

17. Mukasa O, Mushi HP, Maire N, Ross A, de Savigny D. Do surveys with paper and electronic devices differ in quality and cost. Experience from the Rufiji Health and demographic surveillance system in Tanzania?? Glob Health Action. 2017;10:1387984. doi: 10.1080/16549716.2017.1387984. [PMC free article] [PubMed] [Google Scholar]

18. Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Public Health (Oxf) 2005;27:281–91. [PubMed] [Google Scholar]

19. Vaus DD. Surveys in Social Research (Social Research Today) 6th ed. Routledge; 2013. [Google Scholar]

20. Lavrakas PJ. Encyclopedia of Survey Research Methods. 1-0. Thousand Oaks, CA: Sage Publications, Inc; 2008. [Google Scholar]

21. Iglesias C, Torgerson D. Does length of questionnaire matter. A randomised trial of response rates to a mailed questionnaire? J Health Serv Res Policy. 2000;5:219–21. [PubMed] [Google Scholar]

22. Koitsalu M, Eklund M, Adolfsson J, Grönberg H, Brandberg Y. Effects of pre-notification, invitation length, questionnaire length and reminder on participation rate: A quasi-randomised controlled trial. BMC Med Res Methodol. 2018;18:3. [PMC free article] [PubMed] [Google Scholar]

23. Sahlqvist S, Song Y, Bull F, Adams E, Preston J, Ogilvie D, et al. Effect of questionnaire length, personalisation and reminder type on response rate to a complex postal survey: Randomised controlled trial. BMC Med Res Methodol. 2011;11:62. [PMC free article] [PubMed] [Google Scholar]

24. Kost RG, de Rosa JC. Impact of survey length and compensation on validity, reliability, and sample characteristics for ultrashort-, short-, and long-research participant perception surveys. J Clin Transl Sci. 2018;2:31–7. [PMC free article] [PubMed] [Google Scholar]

25. Bolt EE, van der Heide A, Onwuteaka-Philipsen BD. Reducing questionnaire length did not improve physician response rate: A randomized trial. J Clin Epidemiol. 2014;67:477–81. [PubMed] [Google Scholar]

27. Kwak SK, Kim JH. Statistical data preparation: Management of missing values and outliers. Korean J Anesthesiol. 2017;70:407–11. [PMC free article] [PubMed] [Google Scholar]

28. Abellana Sangra R, Farran Codina A. The identification, impact and management of missing values and outlier data in nutritional epidemiology. Nutr Hosp. 2015;31:189–95. [PubMed] [Google Scholar]

Articles from Saudi Journal of Anaesthesia are provided here courtesy of Wolters Kluwer -- Medknow Publications