Abstract

Cognitive functioning is a salient issue among people in the late adulthood stage where mental health declines with age. A common cognitive disability among elders is Dementia including Alzheimerʼs disease. Cognitive screening tools such as the MiniMental Status Examination and the Montreal Cognitive Assessment are most commonly used to measure the cognitive ability areas leading to a diagnostic evaluation. COGNISTAT as a neuropsychological instrument is a recent screener being introduced to a few outpatient clinics. In this study, COGNISTAT is translated to Filipino for the first time using a rigorous procedure and pilot tested on elderly volunteers in local settings. The translation was done by an interdisciplinary team of a Geriatric Physician, a Speech Pathologist, and a Clinical Psychologist. The Filipino-translated COGNISTAT was administered to 22 elders in two batches using the cognitive interview method. Challenges were identified in the initial testing phase which has implications for future adaptations of foreign instruments.

Keywords: Cognitive Interview; Cognitive Screening; COGNISTAT; Elderly; Translation

Abbreviations Used

MCI – Mild Cognitive Impairment

MMSE – Mini-Mental Status Exam

MoCA – Montreal Cognitive Assessment

Introduction

An aging population is a major public health challenge confronting the Western Pacific Region including the Philippines. Globally and in the region, the proportion of people over 60 years is growing faster than any age group due to declining fertility rates and longer life expectancy. According to the World Health Organization, the burden of mental illness is likewise increasing among older people who have to live with dementia, depression, anxiety, and cognitive decline.

Brain dysfunctions such as dementia develop insidiously and progress gradually over time, therefore it often goes undetected for many years before symptoms that merit medical attention starts to surface. Dementia or any cognitive impairment may go unnoticed as people continue to regard cognitive decline as a normal part of aging. Morley and colleagues1 emphasized the importance of recognizing cognitive impairment to arrest the deterioration of this condition.

Statistics show that among those who are above the age of 70, 16% already have mild cognitive impairment (MCI), and 14% have Dementia[1]. In community health settings, primary care physicians, unfortunately, fail to recognize MCI and dementia in more than half of their patients that have the condition[2]. About 25% to 30% of people 85 or older experience some degree of cognitive decline. As mentioned, early signs of dementia are often mistaken as a normative sign of aging, which delays diagnosis and treatment[3]. It was also observed that primary care physicians practicing in the Philippines showed poor knowledge and lack of confidence in making a diagnosis of cognitive impairment in general[4].

Existing Screening Tools

Cognitive appraisal involves the measurement of several discrete abilities that establish an individualʼs unique pattern of weaknesses and strengths. Popular measures that are regularly used by hospitals in the Philippines are the MiniMental Status Examination (MMSE) and the Montreal Cognitive Assessment (MoCA). The MMSE[5] is a test of orientation, memory, attention, and language. The MMSE is interpreted through score categories to indicate mild to extreme conditions of dementia. An MMSE Filipino translation by Ligsay[6], although unpublished, is widely used locally in clinical settings.

The MMSE, however, was found to be useful only in ruling out dementia[7]. The MMSE does not provide a detailed description of other soft symptoms as it only reports global scores for the presence or absence of a specific disease such as dementia.

The MoCA[8] is one of the chosen tools for its brevity, simplicity, and reliability as a screening test for Alzheimerʼs disease. The MoCA has already been validated in Filipino[9]. The MoCA assesses eight brain functions namely short-term memory, visuospatial abilities, executive functions, attention, concentration, working memory, language, and lastly, orientation to time and place. To this day, the MoCA is being used and recommended as the screening instrument not only for Alzheimerʼs disease but also for other types of dementia such as vascular cognitive impairment[9].

The third screening tool, COGNISTAT, has been introduced to the Philippines in recent years and applied in limited settings although it is already being used in other countries[7,10,11]. The advantage of COGNISTAT is that it provides a graphic profile of impairments and a metric profile of cognitive strengths and weaknesses[12]. It has been developed and validated to detect cognitive deficits among patients with neurological conditions such as stroke and TBI, besides Alzheimerʼs disease[13]. COGNISTAT examines a broader range of cognitive functioning such as language (speech, comprehension, repetition, and naming), reasoning (similarities and judgment), orientation (to person, place, and time), construction, memory registration, and calculation, consciousness, and attention[14].

Due to socio-cultural diversities, COGNISTAT was translated into 13 foreign languages such as Arabic[13], Chinese[15], Japanese, and Bahasa[12], to name a few. COGNISTAT in English is applicable in major cities in the Philippines but impractical in rural areas where English is not the second language. Among African Americans in a study[16] for example, the metric portion of the Naming subtest where there is only one acceptable answer is potentially problematic. An equivalent word for the original COGNISTAT might have a different frequency of being used from one region to another resulting in an error rate due not to the lack of naming ability but to the synonym range answers that are scored as incorrect based on the scoring guidelines. Hence, adapting items that will respond to the language and cultural nuances and consequently, the scoring guidelines are necessary. Its relevance for the specific Filipino population underscores the need to modify items that were not culturally, linguistically, or educationally appropriate.

Translation and Cognitive Interview

Several types of research have been done on the translation of a psychological measure of one language to another[17]. It aimed to develop a translation of the original scale that is culturally sympathetic and as psychometrically good as the original version. Some translation initiatives have adopted the cognitive interview method following the basic steps of translation[17]. Typically, translations involve preparation, forward translation (by two independent translators of the target language) and reconciliation (of the two forward translations), backward translation (of the target language back to the original language of the scale), and harmonization (to make the necessary corrections to the terminology used in the new version to ensure that it is true to the intent of the original language of the scale). However, in the processes of translation, there are terms in the original language of the scale that do not have a definite equivalent in the target language. This poses a challenge to the translators[18]. The scale is then pre-tested to find out and make the necessary modification to the specific items that present difficulty among respondents in understanding the items[19]. This was the value offered by the method of the cognitive interview. It is to improve the cultural adaptation of foreign tests to a local indigenous group and validate the equivalence of the translation. In this sense, a cognitive interview is a form of pretesting a questionnaire or survey in languages other than English[17].

COGNISTAT in the Philippines

COGNISTAT, at present, has no Philippine version that is pilot tested and validated, which limits the use of this tool in the country. There has been, however, an earlier attempt to translate it to a local dialect[20]. The primary aim of this paper is to conduct the preliminary steps of translation of this instrument and use the cognitive interview method with the target elderly participants to evaluate the translation in Filipino. After this study, it will eventually go into pilot testing of the translated COGNISTAT to a larger sample size. We aim to describe the process of translation and the adaptation of the 2015 COGNISTAT Paper version to the Philippine language and the difficulties we encountered as translators. Based on the results of this study, the challenges are identified in adapting and translating COGNISTAT; and what measures can be introduced to improve the translation process.

Method

Participants

A total of 22 elderly participants gave informed consent to participate in the cognitive interview phase of the study. Participants were required to be 60 years or older and can comprehend and speak Filipino. Many of the participants were referred by their attending Geriatrics Specialists. Respondents were required to go through all the subtests of the Filipino-translated version. The first test was administered to four participants. Revisions were made on the questionnaire after evaluating the items with these participants before it was administered to the second batch of participants.

Materials

The 2015 COGNISTAT Paper Test was used as the main instrument that went through the process of translation. The test assessed intellectual functioning such as Language, Spatial Skills, Memory, Calculations, and Reasoning. The level of Consciousness, Attention, Memory Registration and Orientation was also assessed.

The various subscales of COGNISTAT are modeled after the more extensive and well-validated neuropsychological tests but in an abbreviated form. It follows the traditional approach to personality testing with different, domain-specific subtests[12]. Unlike other screening procedures that yield a single summary score, COGNISTAT is designed to yield a score for each domain and thus produce a differentiated profile of cognitive abilities. COGNISTAT also employs an adaptive testing approach (referred to as a screen and metric approach) to decrease the time spent in administration. The raw scores for each subscale are plotted on a standard profile form, and performance is classified as being in the average range or as indicative of mild, moderate, or severe impairment[14]. A web-based COGNISTAT also known as the Cognitive Assessment System is the other option for administering the test[12].

Cognitive Interviewers

Four cognitive interviewers, all of whom were graduate students in Psychology were trained on the use of the cognitive interview method. They had prior training in the administration of cognitive tests similar to COGNISTAT. All of them were proficient in at least two languages and one major dialect.

Procedures for Translation

For English instruments to be conceptually equivalent in each of the target countries or cultures, the process of translation and adaptation must be done following a systematic procedure. The focus of this study is on cross-cultural and conceptual adaptation, rather than on linguistic equivalence.

Forward Translation

The goal of the first step of translation was to identify and resolve the inadequate expressions and concepts of the translation as well as any cultural discrepancies. The result of the process was an output of a draft translated version of the instrument.

An expert panel consisting of a geriatric specialist, a speech-language pathologist, and a clinical psychologist was convened to do the first task of forwarding translation. The first three members had an average of 15 years of clinical practice. All panel members were articulate in both English and Filipino. The panel met for three meetings to agree on the first draft of the translated COGNISTAT. A psychologist then further translated the general instructions of each of the COGNISTAT subtests.

Back Translation

The goal of the second step is a back-translation of the Filipino version to English. Back translation was done by two graduate students in Psychology who were both fluent in English and Filipino. Both were not familiar with COGNISTAT.

Cognitive Interview of Participants

Cognitive interviewing aims to minimize errors arising from respondent misunderstanding during data collection by assessing the clarity of terminology, phrasing, and format. Following item translation, developers designed and developed cognitive interview questions to evaluate and revise the instrument. The qualitative interview allowed for unscripted probing of participants by interviewers.

The first cognitive interview was conducted with four elderly female patients in a geriatric center in the city. The first draft of the translated COGNISTAT was administered using an orally administered paper and pencil method. After getting their consent and personal information, instructions were read to the participants, and were asked how they understood each question. The participants would respond by saying it in their own words. All subtests were administered and the same set of questions was repeated on whether they understood the items. At the end of the test, they were asked about what they thought about the entire screening tool. This procedure was followed in all cognitive interviews.

Additional revisions were made to the first draft based on the feedback of participants and these were integrated into the revised instrument. The second round of test administration was conducted using the cognitive interview procedure earlier described. Fifteen male and female participants in the age range of 60 to 71 years old took the test. All participants either resided in a facility for the elderly or lived with their families in Manila and the province.

Results

Translation Results

The Translation Procedure:

All items in all subtests were translated using a conversational language format which considered the familiarity and contextual meaning of a particular word or item. Moreover, two major decision markers influenced the translation process of the COGNISTAT. First is a cautious usage of direct translation which tends to be counter-intuitive and runs the risk of making it difficult or complex to understand. Second, mindful awareness of the possibility that direct translation can lead to the complicated nature of Filipino morphosyntax where adding or altering a syllable in Filipino changes its meaning. For instance, the translation of “reside or “live” could be long and heavy if translated to formal Filipino (6 syllables, low frequency) as against a shorter informal word (4 syllables, high frequency, conversational). Other factors considered were the cultural familiarity of a given word; the length and complexity of instructions; exposure to a concept and contextual equivalence of difficult terms and paragraphs.

English words with no Filipino translations were retained. An example would be “clip” which was modified and we used “paper clip” in the Filipino version instead because the latter phrase was more familiar. The word “painting” was retained because of its common usage in typical conversations among Filipinos. The word “bus” was also retained. Moreover, all images in the Stimulus booklet were retained. Likewise, a respectful word of greeting was suggested to be introduced only during the main instructions at the start of the test and need not be used in subtests.

The choice for a word or phrase translation in some subtests was explained. Table 1 shows the translation rationale for some selected items. For example, in Part III, Memory Registration, the word ‘robinʼ was translated to a local bird popularly known in the Philippines. Both birds when spelled have 2 syllables and were high-frequency words. In Part IV, Repetition, the sentence ‘The honeycomb drew a swarm of beesʼ was translated so that the local language matched the 10 syllables found in the original English sentence and included 2 low-frequency words.

Cognitive Interview Results

The results of the cognitive interview of elder participants were consolidated with highlights on significant responses of participants. It listed the comments and recommendations during the first and second test administration of the Filipino COGNISTAT. It combined the results of all pilot participants. The final draft of the test reviewed and considered some of the recommendations.

Instruction Comprehension

In general, the participants were able to perform the tasks in COGNISTAT. Instructions were given, but when asked if and how they understood the instructions, they could not give a complete answer. For longer, more complex instructions, a few participants were unable to retain all the information, claiming, “I understand but I tend to forget the tail end of it.” Some were challenged, “Oh my, it doesnʼt enter my head,” or “I canʼt explain.”

Many participants were able to understand the stimulus instructions but they found the sentences too long. They recommended that some words should be simplified by using conversational terms rather than the formal textbook vocabulary.

One participant commented that it would be easily understood if the test administrator started with the instruction and will not proceed with the next instruction until the participant was done with the first task.

Attention

English numbers were preferred over translated numbers in Digit Repetition. The reasons are identified in Table 1.

Memory Registration

Four participants failed to repeat the word “piano” on the first try and had difficulty recalling the word. Additionally, the word “green” if translated in Filipino was found difficult because its category prompt, color, was confused with the category prompt of ‘carrotsʼ translated as a vegetable.

Language

In naming objects, participants responded with a variety but closely related answers. For example, the word “clip” was referred to as “hanger,” “clip paper,” and “clip together”. Similarly, the word “tip” was referred to as “to write with”; “for writing,” and “writing with.” The fact that in the Filipino language, an objectʼs name can be its function indicates a need to determine whether these answers are accepted as correct.

Constructional Ability

Four participants experienced difficulty in working with the tiles to recreate the design. Performance on spatial tasks decreased with age[21], but also because the construction task was a novel task for many participants.

Similarities

Most participants had a hard time finding the similarities or relationships between the concepts in the questionnaire. A participant commented that the similarity between painting and music was only comprehensible among those who were educated and well-versed in the arts.

Reasoning

For judgment, the ‘stranded in an airportʼ scenario where the character was short of money was found to be not relatable to the participants. When asked what they would do, the participants answered that they would walk home, or take the bus home. However, in the hypothetical situation, they are supposed to be in a different country that was 10,000 km (about the distance between the Philippines and Brazil) away from home. The main challenge of this item is that the participants may not have had frequent direct experiences of being alone in airports or being unaccompanied in another country. An alternative but the parallel situation may be: “What will you do if the bus you are riding to a distant province broke down and you only have 20 Pesos?”

Stimulus Booklet

For naming pictures, five out of the ten drawings were unfamiliar to the participants. These were: horseshoe, anchor, xylophone, harmonica, and abacus. The picture card showing the xylophone was unfamiliar to respondents who belonged to a lower socioeconomic status. It was suggested that this picture be changed to a general instrument everyone was more familiar with. The abacus was also another hardly recognizable picture card. The octopus was also found difficult. The local translation of octopus was hardly used particularly by those who lived in a landlocked province. Possible alternative images that were more culturally relevant could be a horse-drawn carriage instead of a horseshoe and a guitar instead of a xylophone.

The Experience of Taking the Test

During the administration of COGNISTAT, the participants expressed their sentiments on their declining cognitive health. One expressed appreciation for being part of the study because “my brain was exercised.” In the instances when a participant had a lapse in her memory during the test, she would often say, “I am bound to fail” or this is dangerous!” At the end of the session, the participants were asked if they had any questions about the study. One inquired about when the next session would be scheduled because she wanted to learn the tasks, saying, “I am not highly educated,ʼ in self-defense. Another participant confidently said, “What is important is that I was able to use my head.”

Discussion

The Translation Process

In this study, the translation process and the administration of the translated COGNISTAT to the elderly were described. The implications of specific translations on the screening and the identification of cognitive deficits were of particular concern to the team of researchers. There were psychometric challenges that we must deal with. First is our lack of pre-morbid knowledge of the stock of information that target users of a specific background possessed. Is a particular concept learned in earlier life and faded in memory through time? Or is a particular concept not ever learned in life at all? There is no clear standard where the expressive local language would be typical of aging and be part of the cognitive norm. Hence, the translation process must include a standard scoring scheme that is adaptive to the newly translated version and should be done in consultation with the original test developers.

In Philippine culture, labeling some stimulus objects in terms of their use is a linguistic norm. If the norms provided by the English COGNISTAT were the basis for oneʼs level of cognitive functioning, someone who identifies an object by its function rather than by its name would gain a less than perfect score and might indicate a mild symptom of cognitive weakness. Again, we propose a revised scoring scheme if this happens in the Filipino population of test-takers.

A further illustration of the aforementioned observations was the uneven responses we gathered during the cognitive review. For instance, some translated words and pictures were found difficult and did not register to an older adult as “familiar.” Typically, we can hypothesize that the visual concept was learned in earlier life but is now forgotten. The basic question is how do we know that an image of an object is completely unfamiliar and was never seen or learned in the lifetime of a person. A “xylophone” in the stimulus booklet was hardly recognized by some and it was suggested to be replaced with a more familiar instrument like “guitar.” However, the risk of changing the xylophone is that it may be intended as a difficult stimulus i.e., it will expectedly elicit a failure among those with cognitive decline despite early learning. In other words, if this object is replaced with “guitar” to make it a relevant item, we might satisfy the examinee but not the test developer.

The challenge is therefore finding the balance of the structural and meaning equivalence between the two languages and the neuropsychological measure of cognitive functioning as intended by the original COGNISTAT. Structurally, this means finding the appropriate length of a particular stimulus word i.e. number of syllables, that will match the English version. Meaning equivalence means finding the level of familiarity of a concept equivalent to the original instrument. The frequent learning exposure of a target respondent to a particular concept in the original English COGNISTAT will facilitate translation but there are other factors to consider such as the structural equivalence of the translated word.

English words in the subtests were noticeably brief and instructions were short. Consequently, the short instructions easily deliver the meaning as intended. In contrast, Filipino or the Philippine language is less deliberate, and local words have more syllables. It was an effort to make the length of Filipino translation equally similar and at the same time make the meaning as near to the original concept as possible. In the administration phase, the participants still found the instructions too “long” despite our effort to keep them short to facilitate comprehension and recall. This affirms our observation that the average span of receiving and comprehending local concepts among Filipino older adults may be different from their foreign counterparts because of the length of Filipino words and phrases. Chunking the long concepts embedded in a single instruction could be an option.

As a result, finding the fit between a valid measure of a cognitive deficit that ranges from “mild” to “moderate” to “extreme” status of deficiency in the translated version is another formidable challenge. The risk of violating the psychometric properties of items of subtests amidst the translation process is an important technical concern.

The use of a cognitive interview with a geriatric population merits some attention. A test of cultural adaptation, to be more productive, should be administered to participants with ages 60 years to 65 years or with elderlies who have relatively good cognitive status. This way, the translated instrument could be better evaluated by relatively well participants. During the pre-test phase, the performance of prospective participants could be tested on an existing translated instrument like the MoCA and compare their results with their performance on the COGNISTAT.

We suggest that future research simplify the stringent process of cognitive interviewing. The entire process of interviewing is viewed by elderly participants as intellectually exhausting. As a result, they showed frustration and perceived themselves as unable to “pass” the test which is not the objective of the interview. The step-by-step procedure of cognitive interviewing should be modified to adjust to the level of intellectual energy and capacity of participants to assist the translators in the full evaluation of the items. Alternatively, the entire test could be split into two parts where one part is received by one set of participants while the other by another set. The distribution of interviewees in terms of educational background, regional residence, or socio-economic background to achieve wider representation in performance is also recommended. The next step is for the expert panel to review the results of all interviews and identify appropriate changes in the translation. The Philippine version can be subjected to another brief round of pilot testing focusing on these iterations before introducing it to target users.

Conclusion

The significance of this translation experience is that we want to minimize the misinterpretation of data and error in screening and later diagnosis when the Filipino COGNISTAT is eventually used. The error in the measurement tool itself must be carefully considered since it becomes more vulnerable as it stretches the language of the original screener. As earlier described in this paper, the goal of translation is not a linguistic translation. It is finding a conceptual and cultural equivalence. Some subtests that meet difficulty in terms of cultural appropriateness should be translated but validated well. An adapted and validated version of Filipino COGNISTAT will be a subject of further study when it goes through pilot testing with geriatric participants. We consider the translation incomplete until we have gone through thorough validation of the cognitive test.

Overall, this study discovered challenges in translating an English neurocognitive test which points to several things that should be highlighted. This includes:

1. The importance of being aware of the limitations of a local translation of an English instrument in the sense that translation efforts do not always come to a perfect equivalence in meaning. The literal translation provides a neat word-for-word matching but it does not always approach a clear context of the construct being measured.

2. The COGNISTAT is a neuropsychological instrument that assesses or screens for deficits in brain functioning through the use of a battery of subtests. It is important, therefore, that translations of words, phrases, sentences, or paragraphs must take into account the underlying assumptions of a choice of the latter verbal stimuli or any concept in measuring certain functions that may be consistent with central nervous system involvement.

3. Among the domains that were identified as impacting particular neurocognitive soft functions of affected persons were learning of stimuli, memory about the stimuli, attention to the length of words and sentences, recognition of concepts, and several others. The challenge of translation must take into account a good understanding of the reason behind the choices of particular subtest terminologies which includes the structural property or psychometric properties of subtests and the extensive research behind its relationship with functional impairment.

4. Similar to other psychological tests, moderator variables such as age, education, geographical origin, or pre-morbid intelligence are related to neuropsychological test performance. The COGNISTAT uses the quantitative cutoff scores indicative of impairment. It is recommended that norms should be corrected for the aforementioned factors such as age and education to achieve better classification[22].

In the near future, due to the advances in the diagnosis of organic and functional difficulties, there will be less emphasis on the value of measurement[22]. The primary objective of tests like the COGNISTAT is not expected to focus on diagnosis but rather on the cognitive behavior of a client who has already been diagnosed with the disability. Hence, the translation of the instrument earns a practical value when used by physicians or specialists in hospitals and health centers. It still holds an instrumental value by providing supportive evidence for the progress or deterioration of particular neuro-cognitive behaviors. The amount of information that could be gathered through COGNISTAT could be integrated into other sources of data with reference to diagnosis, chronicity, and client resources which makes treatment and care more client-centered. No doubt, the translated COGNISTAT is still relevant and highly beneficial in monitoring clients who reside in rural communities.

Acknowledgment

We thank COGNISTAT for allowing access to the Paper test and the subtest tools.

Conflicts of Interest Statement

The authors whose names are listed immediately below certify that they have NO affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakersʼ bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.

Maria Isabel E. Melgar, PhD

Miguel A. Ramos Jr., MD

Carla Krishan A. Cuadro, MRS-SP, CSP-PASP

Camille Therese C. Yusay, Ph.D. Candidate

Erica Kimberly R. King, MPsy

Denise Hazelyn Maramba, MPsy

References

1. Morley JE, Morris JC, Berg-Weger M, Borson S, Carpenter BD, Campo ND, et al. Brain health: The importance of recognizing cognitive impairment: An IAGG consensus conference. J Am Med Dir Assoc. 2015; 16(9): 731-739. doi:10.1016/j.jamda.2015.06.017   
2. Knopman DS, Petersen RC. Mild cognitive impairment and mild dementia: A clinical perspective. Mayo Clin Proc. 2014; 89(10): 1452-1459. doi:10.1016/j.mayocp.2014.06.019   
3. Wimo A, Prince M. World Alzheimer Report 2010: the global economic impact of dementia. Alzheimerʼs Disease International. Accessed May 2019.   
4. Capino P, Ramos M, Gatchalian E. Belief and Attitudes of General Practitioners on Alzheimerʼs Disease Dementia. Philippine Journal of Internal Medicine. 2010; 38(4): 183-197.   
5. Kurlowicz L, Wallace M. The mini-mental state examination (MMSE). J Gerontol Nurs. 1999; 25(5): 8-9. doi:10.3928/0098-9134-19990501-08   
6. Ligsay A. Validation of the Mini Mental State Examination in the Philippines. University of the Philippines. 2003; In Press.   
7. Johansson MM, Kvitting AS, Wressle E, Marcusson J. Clinical utility of COGNISTAT in multiprofessional team evaluations of patients with cognitive impairment in Swedish primary care. International Journal of Family Medicine. 2014; 649253. doi:10.1155/2014/649253   
8. Nasreddine ZS, Phillips NA, Bédirian V, Charbonneau S, Whitehead V, Collin I, et al. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc. 2005; 53(4): 695-9. doi:10.1111/j.1532-5415.2005.53221.x   
9. Dominguez JC, Orquiza MG, Soriano, JR. et al. Adaptation of the Montreal Cognitive Assessment for elderly Filipino patients. East Asian Archives of Psychiatry. 2013; 23(3): 80-85.   
10. Fouty HE, Mullen CM, Weitzner DS, Mulcahy DJ. Correcting for gender on the COGNISTAT judgment subtest. Appl Neuropsychol: Adult. 2013; 20(2): 152-154.   
11. Tsuruoka Y, Takahashi M, Suzuki M, Sato K, Shirayama Y. Utility of the Neurobehavioral Cognitive Status Examination (COGNISTAT) in differentiating between depressive states in late-life depression and late-onset Alzheimerʼs disease: a preliminary study. Annals of General Psychiatry. 2016; 15(1): 1-8. doi:10.1186/s12991-016-0091-5   
12. Mueller J, Kiernan J, Langston J, Flanagan R. COGNISTAT Manual. Fairfax, CA: Novatek International; 2018.   
13. Almubark BM, Cattani A, Floccia C. Translation, cultural adaptation, and validation of Cognistat for its use in Arabic-speaking population with acquired brain injury. Eur J Phys Rehabil Med. 2019; 55(5): 595-604. doi:10.23736/S1973-9087.18.05530-2   
14. Allen D. COGNISTAT. Encyclopedia of clinical neuropsychology. 2011; 614-5. In Print.   
15. Chan CC, Lee TM, Fong KN, Lee C, Wong V. Cognitive profile for Chinese patients with stroke. Brain Inj. 2002; 16(10): 873-884. doi:10.1080/02699050210131975   
16. OʼBryant SE, Schrimsher GW, OʼJile JR. Discrepancies between selfreported years of education and estimated reading level: Potential implications for neuropsychologists. Appl Neuropsychol. 2005; 12(1): 5-11. doi:10.1207/s15324826an1201_2   
17. Willis GB. Cognitive Interviewing Revisited: A Useful Technique, in Theory?. Methods for Testing and Evaluating Survey Questionnaires. 2004; 25: 23-43. doi:10.1002/0471654728.ch2   
18. Brogaard T, Neergaard MA, Guldin MB, Sokolowski I, Vedsted P. Translation, adaptation and data quality of a Danish version of the Burden Scale for Family Caregivers. Scand J Caring Sci. 2013; 27(4): 1018-1026. doi:10.1111/j.1471-6712.2012.01092.x   
19. Gupta R, Lahan V, Goel D. Translation and validation of International estless leg syndrome study group rating scale in Hindi language. Ann Indian Acad Neurol. 2011; 14(4): 257–261. doi:10.4103/0972-2327.91939   
20. Melgar MI, Flores M. Translation of the English COGNISTAT to Cebuano. Ateneo de Manila University. 2015; In Press. 21. Hale S, Rose NS, Myerson J, et al. The structure of working memory abilities across the adult life span. Psychol Aging. 2011; 26(1): 92-110. doi:10.1037/a0021483   
21. Groth-Marnat G. Handbook of psychological assessment. USA: John Wiley & Sons; 1997. In Print.