Abstract

Background: In the face of knowledge regarding HPV being an important determining factor for HPV vaccine acceptance and uptake, only few studies have assessed HP-related knowledge in young adults. The aim of this study was to develop and validate the HPV questionnaire regarding knowledge, attitude and perception (KAP) among the age eligible young adults for HPV vaccination.

Methods: The development and validation of the HPV questionnaire was carried out in various stages among small cohorts of potential respondents for the main study. The pilot test for reliability of questionnaire was administered to 48 young adults of both genders aged between 18 and 26 years using close-ended questions, except for the socio-demographic characteristics. The questionnaire consisted of two sections, 16 knowledge items and 19 attitude/perception based items respectively. Participation in the study was voluntary and willing for repeated measures. Descriptive statistics were performed for categorical variables using frequency, percentage, median and interquartile range (IQR).

Results: The final draft of questionnaire was subjected to text readability, agreement of translation, known-group's validity, test-retest and intra-class correlation coefficient (ICC) to compare total knowledge score and KAP scores. In order to test reliability, the internal consistency and corrected item-total coefficient were assessed using Cronbach's alpha. The Cronbach's alpha coefficient for internal consistency (N=48) was .862, p<.001 (Mdn=51, IQR=7) for the final 35 item KAP questionnaire. There was a strong, positive correlation observed and the questionnaire showed good reliability and stability.

Conclusion: This pilot study demonstrated that the questionnaire had very good reliability and stability properties and this can be taken into account for future studies.

Keywords: Development; Translation; Validation; HPV questionnaire; Young adult; Cronbach's alpha.

Introduction

Despite Human Papilloma Virus (HPV) knowledge being an important determinant for vaccine acceptance and uptake [1, 2] only few studies have assessed HPV-related knowledge among young adults, age eligible for HPV vaccination [1, 3]. The prevalence of HPV infection in women, according to a data from 59 countries, ranges from 2% to 44% based on geographic location, population sampled and analytical method used [4]. Genital HPV infection is relatively high with a prevalence of 24.1% among sexually active women aged less than 25 years [5]. About a dozen high-risk HPV types have been identified of which, HPV types 16 and 18 are responsible for most HPV-related cancers [6,7]. Although HPV related cervical cancer is preventable, nearly 530,000 women contact the disease and almost 275,000 die annually, worldwide [8] and mostly affects women under 50 years of age [9]. This makes cervical cancer the second most common cancer and third in terms of cancer-caused deaths among women suffering from gynaecological neoplasms, worldwide [10]. Annual cervical cancer cases in Malaysia are 2145 and mortality rate is 621, the second highest among cancer deaths [11]. It is important to understand that prophylaxis and early detection play a vital role in prevention. However, to implement preventive measures, one must be aware of the seriousness of the problem. Therefore, there is a need to obtain accurate data on the current knowledge about HPV infection, cervical cancer and vaccination attitude among young adults. All these efforts are aimed at launching campaigns that would encourage HPV vaccination and cytological examination for prevention. Many studies have been conducted to assess knowledge and attitudes towards cervical cancer [12,13]. Unfortunately, recent study shows that public awareness on HPV and cervical cancer is insufficient [14]. Regardless of country or continent, there is a pressing need for wider and better education on the subject of HPV infection, as well as cervical cancer screening and prevention. Young women with low education and poor economic background should be the first line target for educational campaigns. Very few studies have been reported on the development and validation of questionnaire, assessing young adult's knowledge, attitude and perception (KAP) about HPV infection and vaccination [15,16,17]. Research is needed to fill the gaps in knowledge and perception about the HPV vaccination. It is not clear if young women understand the purpose of the Pap test, regardless of recent advancements and national attention over cervical cancer prevention. The aim of this pilot study was to develop and validate the reliability of the questionnaire used in young adults of both genders (18-26 years) as a tool to assess knowledge, attitudes and perception towards HPV infection, awareness and HPV vaccination for prevention.

Method

Development of Questionnaire

The questions were mostly adapted from other published studies and slightly modified to meet the objective of the present study [18-23]. The questions were designed to test the three domains, namely knowledge, attitude, and perception. Questions in knowledge section tested the respondents' awareness or knowledge using close-ended questions. Items developed in the knowledge section comprised of definition and epidemiology; progression, signs and symptoms; diagnosis and risk factors of HPV infection and cervical cancer; availability, side effects, number of dose and duration of HPV vaccine. One point was given for correct answer and zero point for incorrect answer. A higher score indicated better knowledge with maximum possible total knowledge score of 16 points.

The attitude and perception section measured the current beliefs, misconceptions and attitudes about HPV infection and HPV vaccination. This was effectively designed using a five point Likert scale to mark their best agreement to the 19 items, attitude and perception testing statements. The respondents were asked to indicate the extent to which they agree with the statements like, strongly disagree to strongly agree which was given 1 to 5 points respectively. The attitude and perception domains included health seeking behaviour; importance and follow-up procedure of HPV prevention; importance, significance and severity of HPV infection; importance of vaccination and referrals; cost, affordability, willingness for HPV vaccination and HPV management [24-29].

Face and Content Validation of the Questionnaire

The face validation of the questionnaire was aimed to assess the ease of comprehension, relevance, effectiveness in providing useful information, degree to which the questions are interpreted and understood correctly by the potential respondents and its integrity to address the study objectives. The content validation was done among two practising physicians (community medicine), six academicians from pharmacology unit and ten academicians from clinical pharmacy and pharmacy practice unit from a private university, Malaysia. All recommendations found appropriate were considered for the final draft of the questionnaire. The questionnaire was also subjected to KnownGroup's validity [30,31].

Readability

Analysis was performed using well-validated scales that have been commonly used to evaluate healthcare-related materials to assess the simplicity and readability using on-line ‘Text Readability Consensus Calculator' which checks for reading levels, reading assessment and reading grade levels using eight readability formulas [32,33].

Translation of the questionnaire

The questionnaire was forward and backward translated using lecturers from linguistic studies well versed in both English and Bahasa Malaysia from a private university and compared to the original questionnaire and the second version was prepared. The final version of the Malay KAP questionnaire was completed after making necessary corrections based on pilot study after translation (N=20) and made available for reliability and validity studies. These participants were not included in the pilot study.

Agreement of translation between English and Malay Version

Cohen's κ statistics was run to determine inter-rater reliability between the two versions, to see if there was an agreement between the English and Malay version of the KAP questionnaire.

Known-group's Validity

The Spearman's rank-order correlation test was run to check the correlation between total knowledge score and total KAP scores among two known groups, and Mann-Whitney U test was run to predict any statistically significant knowledge score difference between pre- and post-test.

Test and retest study

A test-retest study was conducted among two groups of young adult students; one group from medical background (expected good knowledge) and the other from non-medical students (expected poor knowledge). The Spearman's rho correlation test was run to see the correlation in knowledge and KAP scores at pre- and post-test. Further, Mann-Whitney U test was run to check the total knowledge score differences at pre- and post-test. Further Intra-class Correlation Coefficient (ICC) test for significant correlations at all three scales was run using one way random effect model, single measurement. Finally, the Cronbach's alpha test for internal consistency was run to check the reliability and stability of the questionnaire.

Ethical Considerations

The research proposal along with the study instruments (questionnaire) and informed consent form was submitted to the AIMST University Human & Animal Ethical Committee and the ethical clearance was obtained before the study was conducted. Further, informed consent forms were signed from all the participants before distributing the questionnaire.

Statistical Analyses of Data

The survey data was tabulated using Microsoft Excel workbook and analysed using Statistical Package for Social Sciences ‘SPSS version - 23’ (IBM SPSS Statistical software) for windows. The categorical variables were analysed using descriptive statistics for frequency, percentage, median and IQR and p-value was computed using Pearson's Chi Square test. A p value < .05 was considered significant.

Result

Development and validation of the KAP Questionnaire

The final draft of questionnaire was subjected to readability consensus, agreement of translation and Known-Group's validity. In order to test reliability, the internal consistency and corrected item-total coefficient were assessed using Cronbach's alpha.

Text Readability Consensus

The readability was found to be easy to read and understand for the age group of 18 to 26 years [32,33] whereas the results reported are for 13 to 15 years. The consensus report is shown in table 1.

Translation of the Questionnaire

The KAP questionnaire was translated to the local Bahasa Malaysia language, with the help of linguistic school lecturers in a private university. In the first step, the questionnaire in English was given to three different linguistic specialists and their translated version was cross matched and verified. The first version of the questionnaire was arrived. The second step involved the reverse translation of the finalized questionnaire in Bahasa Malaysia which was again given to three different linguistic specialists to re-translate to English. This was done to make sure the translated questionnaire is clear and easy to understand and does not carry any misrepresentations. The reverse translated questionnaire was compared to the original questionnaire and the second version was prepared. The final stage involved the pilot testing of the questionnaires using 20 young adults, aged 18 to 26 years who completed and commented on the questionnaire. Necessary amendments were made after comparing the first and the second versions along with the participants' comments. The final version of the Malay KAP questionnaire on HPV infection and HPV vaccination was completed and made available for reliability and validity studies. These participants were not included in the pilot study.

Agreement of translation between English and Malay Version

For inter rater reliability, Cohen's κ was run to determine if there was an agreement between the English and Malay version of the KAP questionnaire among30 young adult participants. There was substantial agreement between the two versions of the questionnaires, as shown in table 2, κ=.69 (95% CI), p<.001

Known-group's Validity

The Spearman's rank-order correlation test between total knowledge score and total KAP scores showed a significant strong positive correlation. Table 3 depicts the knowledge score correlation between two known groups, 1 (N=12) and 2 (N=12). The median knowledge score of group 1, (Mdn=10.5, IQR=4) and KAP Score (Mdn=61.5, IQR=26), were positively correlated, r (24)=.595, p=.002. The median knowledge score of group 2, (Mdn=14, IQR=3) and KAP Score (Mdn=76, IQR=19) were also strongly positively correlated, r (24)=.754, p<.001.

Further, Mann-Whitney U test showed a statistically significant knowledge score differences between groups at pre- and post-test, U (24)=12, p<.001 and U (24)=15.5, p<.001. As hypothesized, when the respondents' knowledge level increased, their KAP score significantly improved.

Test and retest study

A test-retest study was conducted with two groups of young adult students; one group of medical students in (N=12) and the other, non-medical students (N=12). All 24 students completed both test and retest. Spearman's rho correlation test showed a strong correlation in knowledge and KAP scores at pre- and post-test which was statistically significant, r (24)=.929, p<.001 and r(24)=.985, p<.001 respectively (Table 4a). The Mann-Whitney test indicated that the total knowledge score was higher for post-test, U(24)=6, p<.001 than for pre-test, U(24) = 2, p<.001.

Intra-class Correlation Coefficient (ICC) for the questionnaire showed significant correlations at all three scales as shown in table 4b.

Reliability
Socio-demographic data of the Adult Respondents (Pilot Study)

Reliability test for KAP Questionnaire

The Cronbach's alpha test for internal consistency was .862, p<.001 (Mdn=51, IQR=7) for the final 35 item KAP questionnaire. There was a strong, positive correlationand the questionnaire showed good reliability and stability (table 6).

Discussion

This study aimed to develop and validate the adult KAP questionnaire regarding HPV infection and vaccination. The questionnaire covers a wide range of issues relating to HPV infection and HPV vaccination and the focus was not limited to the general history. Other issues covered include risk factors, primary prevention and screening information about Pap test, cancer risk, vaccination, side effects, doses and many more. A comprehensive overview of HPV infection and vaccination was possible to achieve using the said approach. The questionnaires stability and reliability was established through a series of pilot tests among the young Malaysian adult population from Kedah state. The results obtained reflect the performance of the KAP questionnaire which was found encouraging. The contents incorporated in the questionnaire were based on extensive literature review, discussion with healthcare providers, subject experts and the potential respondents. The interviewee feedback for the questionnaire revealed, most of the respondents neither had difficulty nor confusion, embarrassing or displeasing with any of the KAP questions/ statements regarding HPV questionnaire pilot tested. The participants completed the questionnaires within 15 to 20 minutes. This study was very careful not to touch the subject about sexuality as it is still tabooed and the subject most reluctant to talk about sexual lifestyle. Older people, especially the elderly are still displeased with questions relating to sexuality [34].

The validation of the questionnaire was done through a number of processes as discussed in detail earlier. The first to go was the readability of the text material in the questionnaire as the questionnaire was intended to be used in a diverse population of varying educational background. The text readability accomplishes significance in giving individuals' information, the prerequisites and the implications of accomplishing an effective readability text are as differing as the utilization of text itself. In this study, we focused on all parts for clarity, similar to text dimension or font size difference, readers' capacity to decipher the text, and more on the semantic components of the text. The globally accepted readability score similar to the Flesch Reading Ease score was used to assess the readability of the text contents in questionnaire. The studies have reported that the readability assessment scale for learners of English as second language [35]. This study intended to balance the existing readability-related surveys, which have inclined to focus on education and psychological aspects of readability measures. Using on-line text readability consensus calculator, we found the adult questionnaire, to be easy to read and understand among the study population aged 18-26 years. Had there been any difficulty in the questionnaire in the less educated, the research personnel was always available to explain the questions

Next, the questionnaire was translated to the local Bahasa Malaysia language from English version, after making all the preliminary revisions. The purpose of translation was to reduce cognitive burden and facilitate respondents' comprehension. The translation to local language, Bahasa Malaysia was of fantastic advantage for the precise apprehension of the respondents. The report of the translated material was presented to the experts for their feedback. Upon the agreement of experts with both versions for easy and acceptable comprehension and meaningful translation, the Cohen kappa test for concordance was applied to check the agreement of translation between English and Bahasa Malaysia versions. The test results showed good to substantial agreement between the questionnaires intended for the study population. Cohen's kappa statistics was interpreted as suggested by McHugh (2012). Although kappa score of .41 might be acceptable, a value below .6 indicates inadequate agreement among raters [36]. Regardless of the interpretation models used, the Bahasa Malaysia versions of the KAP questionnaire in this study scored above .6 with substantial to good agreement as shown in results section.

Known group validity (or extreme-groups validity), otherwise called construct validity is used if no yardstick exists for validating the survey questionnaire. In this, the questionnaire was distributed to two recognized clusters who have dissimilar knowledge levels of the concept to confirm whether the expected difference is replicated in the scores of both groups [37]. The study utilized non-medical and medical students for young adult respondents.

Though some studies report “correlation is a measure of association, and does not measure agreement satisfactorily and tends to show higher than true reliability” values [38], this study used kappa and ICC correlation to establish “test-retest reliability.”The test-retest reproducibility of a questionnaire was applied repeatedly in the same population over time. Parmenter & Wardle, recommended “The time lag between the measures should be long enough for precise answers to be forgotten and short enough to minimize real change of knowledge” [39]. The time lag was three to four weeks between the test and retest during university holidays. The test and retest reliability was measured using Pearson's or Spearman's rho correlation. The Spearman's rho correlation showed a strong correlation among knowledge and KAP scores at pre- and post-test which was statistically significant, r(24)=.929, p<.001; r(24)=.985, p<.001 among young adults [39,40]. Further the Mann-Whitney test indicated that the total knowledge score differences was higher for post-test, U(24)=6, p<.001 than for pre-test, U(24)=2, p<.001. After the retest, the expert panel took time to review to confirm the number of items to be retained in each questionnaire for the final KAP study on HPV infection and HPV vaccination.

The test retest reliability was further strengthened as many researchers proposethe measure of agreement between two dichotomous ratings. Blackman and Koval (1993) explainedthe measurement shall be performed by two different or by the same rater at two points. Entailing stability across time, Cohen kappa test was intended to measure agreementas a measure of association [41,42]. The most common factors that influence the magnitude of kappa, include non-independence of ratings, bias and prevalence [43]. To address these issues, Intraclass Correlation Coefficient (ICC) for ordinal data which uses weighted kappa was used to address the issue when the categories are equally spaced and in one dimension [44,45]. The ICC produce reliability interval between .70 to .90 [46]. The results indicated evidence for repeatability for construct measurements between two time points for the questionnaire and the intra-rater reliability test results showed .722 and .851, ICC coefficient for knowledge and KAP scales of the questionnaire and reliability was thus established.

The questionnaires were content validated by a panel of experts in the subject area. Reliability and clarity was tested with the same inclusion and exclusion criteria using 48 adults participant as intended for original study. The construct analysis of the KAP questionnaire confirmed reliability with Cronbach alpha value, .862 for the questionnaire. An “α = .7” is generally considered acceptable according to interpretation guidelines, however, this study questionnaire showed good reliability [47,48,49].

The study carried out using the KAP questionnaire demonstrated insufficient knowledge regarding HPV infection and vaccination among the study cohort. HPV infection was not a major aetiological risk factor for cervical cancer. Vaccination is uncommon among age eligible young adults, and poor secondary prevention using pap screening prevailed among the respondents. The questionnaire will be of priceless assistance and provide opportunities to plan and organize social campaigns intended for increasing primary and secondary prevention awareness about HPV related illness which possibly reduces the mortality rates. This survey tool also gave the opening to confirm changes in knowledge and KAP at post educational interventions.

Conclusion

Our findings highlight the need to increase education regarding HPV infection related cervical or other cancers and counselling regarding HPV vaccination tailored to young adults. This pilot study demonstrates that the ad hoc questionnaire as a tool to examine knowledge, attitudes and perception or behaviours towards primary (prevention of HPV infection) and secondary for prevention of cervical cancer in young women in Malaysia has very good reliability properties and can be taken into account for future studies.

References

1. Gottvall M, Larsson M, Höglund AT, Tydén T. High HPV vaccine acceptance despite low awareness among Swedish upper secondary school students. The European Journal of Contraception & Reproductive Health Care. 2009; 14(6): 399-405. doi: 10.3109/13625180903229605   
2. Dempsey AF, Abraham LM, Dalton V, Ruffin M. Understanding the reasons why mothers do or do not have their adolescent daughters vaccinated against human papillomavirus. Ann Epidemiol. 2009; 19(8): 531-538. doi: 10.1016/j.annepidem.2009.03.011   
3. Gerend MA, Magloire ZF. Awareness, knowledge, and beliefs about human papillomavirus in a racially diverse sample of young adults. Journal of Adolescent Health. 2008; 42(3): 237-242. doi: 10.1016/j.jadohealth.2007.08.022   
4. Bruni L, Diaz M, Castellsagué M, Ferrer E, Bosch FX, de Sanjosé S. Cervical human papillomavirus prevalence in 5 continents: meta-analysis of 1 million women with normal cytological findings. Journal of Infectious Diseases. 2010; 202(12): 1789-1799. doi: 10.1086/657321   
5. Ammatuna P, Giovannelli L, Matranga D, Ciriminna S, Perino A. Prevalence of genital human papilloma virus infection and genotypes among young women in Sicily, South Italy. Cancer Epidemiology and Prevention Biomarkers. 2008; 17(8): 2002-2006. doi: 10.1158/1055-9965   
6. Lowy DR, Schiller JT. Reducing HPV-associated cancer globally. Cancer Prev Res. 2012; 5(1): 18-23. doi: 10.1158/1940-6207   
7. Centers for Disease Control and Prevention (CDC). Human papilloma virus-associated cancers-United States, 2004-2008. MMWR. Morbidity and mortality weekly report. 2012; 61(15): 258-261.   
8. Arbyn M, Castellsagué X, de Sanjosé S, et al. Worldwide burden of cervical cancer in 2008. Ann Oncol. 2011; 22(12): 2675-2686. doi: 10.1093/annonc/mdr015   
9. Wender R, Fontham ET, Barrera E, et al. American Cancer Society lung cancer screening guidelines. CA Cancer J Clin. 2013; 63(2): 106-117. doi: 10.3322/caac.21172   
10. Armstrong EP. Prophylaxis of cervical cancer and related cervical disease: a review of the cost-effectiveness of vaccination against oncogenic HPV types. J Manag Care Pharm. 2010; 16(3): 217-230.   
11. Bruni L, BarrionuevoRosas L, Albero G, et al. ICO Information Centre on HPV and Cancer (HPV Information Centre). Human Papillomavirus and Related Diseases in the World. Summary Report 2015. 2017.   
12. Blödt S, Holmberg C, Möller-Nordhorn J, Rieckmann N. Human Papillomavirus awareness, knowledge and vaccine acceptance: A survey among 18-25 year old male and female vocational school students in Berlin, Germany. The European Journal of Public Health. 2012; 22(6): 808-813. doi: 10.1093/eurpub/ckr188   
13. Pitts M, Clarke T. Human papillomavirus infections and risks of cervical cancer: what do women know? Health Education Research. 2002; 17(6): 706-714. doi: 10.1093/her/17.6.706   
14. Kamzol W, Jaglarz K, Tomaszewski KA, Puskulluoglu M, Krzemieniecki K. Assessment of knowledge about cervical cancer and its prevention among female students aged 17-26 years. European Journal of Obstetrics & Gynecology and Reproductive Biology. 2013; 166(2): 196-203. doi: 10.1016/j.ejogrb.2012.10.019   
15. Urrutia MT, Hall R. Beliefs about cervical cancer and Pap test: a new Chilean questionnaire. Journal of Nursing Scholarship. 2013; 45(2): 126-131. doi: 10.1111/jnu.12009   
16. Saulle R, Miccoli S, Unim B, et al. Validation of a questionnaire for young women to assess knowledge, attitudes and behaviors towards cervical screening and vaccination against HPV: survey among an Italian sample. Epidemiol Biostat Public Health. 2014; 11(2). doi: 10.2427/8913   
17. Jaglarz K, Tomaszewski KA, Kamzol W, Puskulluoglu M, Krzemieniecki K. Creating and field-testing the questionnaire for the assessment of knowledge about cervical cancer and its prevention among schoolgirls and female students. J Gynecol Oncol. 2014; 25(2): 81-89. doi: 10.3802/jgo.2014.25.2.81   
18. Wetzel C, Tissot A, Kollar LM, Hillard PA, Stone R, Kahn JA. Development of an HPV educational protocol for adolescents. J Pediatr Adolesc Gynecol. 2007; 20(5): 281-287. doi: 10.1016/j.jpag.2006.12.002   
19. Ragin CC, Edwards RP, Jones J, et al. Knowledge about human papilloma virus and the HPV vaccine–a survey of the general population. Infect Agent Cancer. 2009; 4(1). doi: 10.1186/1750-9378-4-S1-S10   
20. Blazeby J, Sprangers MA, Cull A, Grønvold M, Bottomley A. EORTC Quality of Life Group Guidelines for Developing Questionnaire Modules, 3'rd Edition. Cancer, questionnaire, validity. 2001; 34.   
21. Al-Naggar RA, Bobryshev YV. Practice towards human papilloma virus vaccine among Malaysian women: a survey of the general population. Asian Pac J Cancer Prev. 2011; 12(8): 2045-2049.   
22. Blödt S, Holmberg C, Möller-Nordhorn J, Rieckmann N. Human Papilloma virus awareness, knowledge and vaccine acceptance: A survey among 18-25 year old male and female vocational school students in Berlin, Germany. The European Journal of Public Health. 2011; 22(6): 808-813.   
23. Ozyer S, Uzunlar O, Ozler S, et al. Awareness of Turkish female adolescents and young women about HPV and their attitudes towards HPV vaccination. Asian Pac J Cancer Prev. 2013; 14(8): 4877-4881. doi: 10.7314/APJCP.2013.14.8.4877   
24. Ezenwa BN, Balogun MR, Okafor IP. Mothers' human papilloma virus knowledge and willingness to vaccinate their adolescent daughters in Lagos, Nigeria. Int J Womens Health 2013; 5: 371-377. doi: 10.2147/IJWH.S44483   
25. Thomas TL, Strickland OL, DiClemente R, et al. Parental Human Papilloma virus Vaccine Survey (PHPVS): Nurse-led instrument development and psychometric testing for use in research and primary care screening. J Nurs Meas. 2013; 22(1): 96-109.   
26. Urrutia MT, Hall R. Beliefs about cervical cancer and Pap test: a new Chilean questionnaire. Journal of Nursing Scholarship. 2013; 45(2): 126-131. doi: 10.1111/jnu.12009   
27. Chang IJ, Huang R, He W, et al. Effect of an educational intervention on HPV knowledge and vaccine attitudes among urban employed women and female undergraduate students in China: a cross-sectional study. BMC Public Health. 2013; 13(1): 916. doi: 10.1186/1471-2458-13-916   
28. Stridh S, Hammar S. Knowledge of Human papilloma virus (HPV) and attitudes towards HPV-vaccine among Thai female university students. Uppsala University. 2014.   
29. Fu LY, Bonhomme LA, Cooper SC, Joseph JG, Zimet GD. Educational interventions to increase HPV vaccination acceptance: a systematic review. Vaccine. 2014; 32(17): 1901-1920. doi: 10.1016/j.vaccine.2014.01.091   
30. Rubio DM, Berg-Weger M, Tebb SS, Lee ES, Rauch S. Objectifying content validity: Conducting a content validity study in social work research. Soc Work Res. 2003; 27(2): 94-104. doi: 10.1093/swr/27.2.94   
31. Al-Qazaz HK, Hassali MA, Shafie AA, Sulaiman SAS, Sundram S. The 14-item Michigan Diabetes Knowledge Test: translation and validation study of the Malaysian version. Practical Diabetes. 2010; 27(6): 238. doi: 10.1002/pdi.1495   
32. Flesch R. How to write plain English: Let's start with the formula. University of Canterbury. 1979.   
33. Chen HH. How to Use Readability Formulas to Access and Select English Reading Materials. Journal of Educational Media & Library Sciences. 2012; 50(2): 229-254. doi: 10.6120-JoEMLS.2012.502/0513.RS.AM   
34. Paradowska D, Tomaszewski KA, Bałajewicz-Nowak M, et al. Validation of the Polish version of the EORTC QLQ-CX24 module for the assessment of health-related quality of life in women with cervical cancer. Eur J Cancer Care (Engl). 2014; 23(2): 214-220. doi: 10.1111/ecc.12103   
35. Schwarm SE, Ostendorf M. Reading level assessment using support vector machines and statistical language models. Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. 2005; 523-530. doi: 10.3115/1219840.1219905   
36. McHugh ML. Interrater reliability: the kappa statistic. Biochemia medica. 2012; 22(3): 276-282.   
37. Davenport SJ, de Morton NA. Clinimetric properties of the de Morton Mobility Index in healthy, community-dwelling older adults. Archives of physical medicine and rehabilitation. 2011; 92(1): 51-58. doi: 10.1016/j.apmr.2010.08.023   
38. Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use. Oxford University Press, USA. 2015.   
39. Parmenter K, Wardle J. Development of a general nutrition knowledge questionnaire for adults. European Journal of Clinical Nutrition. 1999; 53(4): 298-308. doi: 10.1038/sj.ejcn.1600726   
40. Kline P. Handbook of psychological testing. Routledge. 2013.   
41. Blackman NJ, Koval JJ. Estimating rater agreement in 2 x 2 tables: Correction for chance and intraclass correlation. Applied Psychological Measurement. 1993; 17(3): 211-223.   
42. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960; 20(1): 37-46. doi: 10.1177/001316446002000104   
43. Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical therapy. 2005; 85(3): 257-268. doi: 10.1093/ptj/85.3.257   
44. Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement. 1973; 33(3): 613-619. doi: 10.1177/001316447303300309   
45. Portney LG, Watkins MP. Foundations of clinical research: applications to practice. Prentice Hall. 2000.   
46. Spitzer RL, Cohen J, Fleiss JL, Endicott J. Quantification of agreement in psychiatric diagnosis: A new approach. Archives of General Psychiatry. 1967; 17(1): 83-87. doi: 10.1001/archpsyc.1967.01730250085012   
47. George D, Mallery M. Using SPSS for Windows step by step: a simple guide and reference. 2003.   
48. Henson RK. Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development 2001; 34(3): 177.   
49. Kahn JA, Xu J, Zimet GD, et al. Risk perceptions after human papillomavirus vaccination in HIV-infected adolescents and young adult women. J Adolesc Health. 2012; 50(5): 464-470. doi: 10.1016/j.jadohealth.2011.09.005