Psychometric Properties of the NEI-RQL-42 Questionnaire in Keratoconus

Abstract

Purpose.: To assess the psychometric properties of the National Eye Institute Refractive Error Quality of Life (NEI-RQL-42) questionnaire in keratoconus and compare these findings to patients with refractive error correction alone.

Methods.: The Portuguese version of the NEI-RQL-42 Quality of Life questionnaire was completed by 44 patients who had keratoconus before and after implantation of intracorneal ring segments. Rasch analysis was used to assess the use of response categories, success in measuring a single trait per subscale (unidimensionality), ability to discriminate person ability (precision), and targeting of questions to person quality of life (QoL).

Results.: Rasch analysis was performed for the questionnaire subscales using stacked preoperative and postoperative data. Three subscales (Symptoms, Dependence on correction, and Suboptimal correction) contained response categories that were not used as intended. Six subscales contained misfitting items indicating multidimensionality. Eleven subscales exhibited inadequate measurement precision. Only the Near vision subscale demonstrated adequate precision with a person separation greater than 2.0. Targeting of items to person QoL was adequate in 11 of the 12 subscales with a mean item location of less than 1 logit.

Conclusions.: Only one NEI-RQL-42 subscale (Near vision) performed adequately in keratoconus. Targeting was better in patients with keratoconus than in patients with refractive error correction alone, but 11 of the 12 subscales remain manifestly inadequate. Better instruments exist for measuring patient-reported outcomes in keratoconus.

Introduction

Quality of life (QoL) is a well-established trait measured in clinical and research settings across medicine, including ophthalmology. ^1,2 Indeed, patient-reported outcomes have become an essential component required by funding bodies, ethics committees, third-party papers, and regulatory agencies. ³ It is therefore imperative that any patient-reported outcome, including QoL questionnaires, be well developed, of high quality, and meet standard psychometric properties. 

The National Eye Institute Refractive Error Quality of Life (NEI-RQL-42) questionnaire was developed at RAND Corporation under the sponsorship of the National Eye Institute and is a commonly used questionnaire in the ophthalmic community to assess QoL related to refractive error correction. ⁴ The questionnaire consists of 42 items (questions) combined to provide 13 subscale scores. Previous studies have evaluated the NEI-RQL-42 questionnaire using more traditional validation techniques. ^5–7 These techniques fail to assess key psychometric properties such as response category ordering, dimensionality, person separation, and targeting of questions to persons. The NEI-RQL-42 also uses Likert scaling with a simplistic summary scoring method that is prone to bias and makes assumptions on item difficulty and unequal response option spacing. ⁸ Rasch analysis overcomes these disadvantages and provides a measure that is linear with the latent variable, with measurements conforming to a Guttman scale. ^9,10 Rasch analysis is recognized as the gold standard for questionnaire methodology in ophthalmology. We recently investigated the psychometric properties of the NEI-RQL-42 in spectacle and contact lens wearers undergoing laser refractive surgery using Rasch analysis. ¹¹ Rasch analysis is a simple yet effective psychometric model with two main characteristics. ⁹ Firstly, it estimates interval-level measurement on a continuous scale from raw questionnaire data (ordinal responses). This estimation reduces measurement noise and enables valid parametric statistical analysis of the output data. Secondly, Rasch analysis delivers unparalleled insight into the psychometric properties of questionnaires, including the appropriateness of the response options, precision of the measurement, the fit of questions to the construct under investigation, and targeting of the questions to the respondents. ¹² This has led to its growing use in the development of questionnaires in ophthalmology. ^13–16  

Our study revealed serious psychometric deficiencies with the NEI-RQL-42 QoL questionnaire, including response categories not performing as expected, questions that did not fit with the trait represented by a subscale, and poor precision. Precision problems occurred because the subscales possessed too few items (e.g., two items); some of these items were irrelevant to people with refractive error correction. For example, the questionnaire contains many questions related to activity limitations, yet very few people had such problems. We hypothesised that the NEI-RQL-42 questionnaire may perform better in disease groups where activity limitations may be more relevant. ¹¹  

To test this hypothesis, we draw upon data from a recent study investigating QoL in people with keratoconus before and after implantation of intracorneal ring segments. ¹⁷ This study was a collaborative project between a research team based in Australia and Brazil. The Brazilian group were conducting a larger project on keratoconus surgery outcomes, and the present study was an additional arm of the main project. The patients in Brazil spoke Portuguese; thus a Portuguese version of the NEI-RQL-42 questionnaire was used. The purpose of this study was to assess the psychometric properties of the NEI-RQL-42 QoL questionnaire in keratoconus, to determine if it functions better in patients with keratoconus than in those with refractive error correction alone. 

Methods

The National Eye Institute Refractive Error Quality of Life Questionnaire (NEI-RQL-42)

The NEI-RQL-42 contains 42 items across 13 subscales (Clarity of vision, Expectations, Near vision, Far vision, Diurnal fluctuation, Activity limitations, Glare, Symptoms, Dependence on correction, Worry, Suboptimal correction, Appearance, Satisfaction with correction). The 42 items of the NEI-RQL-42 are designed with 16 different question/response category formats. Question format is defined as the way in which the questions are asked: “How much difficulty . . . ?” or “Do you suffer . . . ?” The response scales indicate the number of different category response options such as one to six or one to three response options. The question/response category formats are not unique within each of the 13 subscales of the questionnaire, with some subscales incorporating multiple formats. Scoring of the questionnaire involves two steps. Firstly, the original numeric values are recoded following a set of scoring rules across a 0% to 100% range, with higher scores indicating better QoL. Secondly, subscales are scored by averaging together the items within each subscale. The number of items in each subscale varies from one to seven. 

Subjects

The translation and validation of the questionnaire have been previously published. ¹⁸ In brief, the translation involved a forward and back translation with native ophthalmologists and professional translators. The back-translated version was compared to the original English version, and the questionnaire underwent pilot testing in 20 patients undergoing refractive surgery at the Federal University of São Paulo. The final Portuguese version of the NEI-RQL-42 was administered to 102 patients who underwent refractive surgery (before and after surgery), with 20 patients answering the questionnaire at two different postoperative visits. 

Patients with keratoconus with no ocular comorbidities who were undergoing intracorneal ring segments at the refractive sector of the Ophthalmologic Department of Federal University of São Paulo (UNIFESP) were invited to take part in this study. The questionnaire was self-administered by 44 patients. The same 44 patients completed the questionnaire following implantation of intracorneal ring segments. Twenty-nine patients had bilateral implantation, and 15 had monocular implantation. Full details for the surgery have been previously reported. ¹⁷ Postoperatively, the questionnaire was completed when the patient had been wearing his or her best correction for at least 40 days, corresponding to between 4.5 and 8 months postoperatively. All patients were 18 years or older, were speakers of Portuguese, and had no severe cognitive impairment. The study was approved by the ethics committee of the Federal University of São Paulo (Rua Botucatu, 572 1° andar conj 14, CEP 04023-062, São Paulo, Brazil). All patients read and signed a consent form, and research was conducted in accordance with the Declaration of Helsinki. 

Rasch Analysis

Twelve of the 13 subscales underwent a separate Rasch analysis. The Satisfaction with correction subscale has only one item so cannot undergo Rasch analysis. Items that contained the response options “Never do these activities for other reasons/Don't drive at night for other reasons/Don't drive at dusk for other reasons” were treated as missing data. The response polarity was matched according to the recoding scores in the NEI-RQL-42 manual. Items 36 to 42 have an “a” part and a “b” part. Part “a” asks for a yes/no response, and part “b” is completed only if the response to part “a” is yes. We considered this a five-response scale. 

The Rasch model is based on a probabilistic relationship between item difficulty and person ability, with the difference known as the functional reserve or functional ability. The functional reserve expresses the probability of any person being successful on any item. In the dichotomous model, the probability of success is expressed as a function of the size of the difference between the ability (B) of the person (n) and the difficulty (D) of the item (i). Raw scores are converted into odds of success, the ratio of person percentage success (p) to person percentage failure (1 – p). The natural log of this ratio is the difference between the person ability estimate (Bn) and the item difficulty estimates (Di). Therefore, both person ability (Bn) and item difficulty (Di) are expressed on a logit scale with an average logit of zero. Positive logits indicate higher than average probabilities of person success on an item, and negative logits indicate a lower than average probability of success. A polytomous Andrich rating scale model using joint maximum likelihood estimation was applied to each question format using Winsteps (version 3.70.0.2; Winsteps, Chicago, IL). ¹⁹ The Wolfe and Chiu stacking procedure for pre- and postoperative evaluation was performed. ²⁰ This involves combining pre- and postoperative data in a single analysis. Rasch analysis was used to assess each subscale for response category performance, dimensionality, precision, and targeting. 

Category Threshold Order

Response category performance in terms of being used in the order intended was evaluated by observing if the category calibration increased in an orderly fashion in the probability curves. The threshold is the midpoint between response categories and indicates the point where the likelihood of choosing either response category is the same. Items in the questionnaire have between two and six choices, which translates to one to five thresholds, respectively. Each threshold has a location on the logit scale, and each item has a mean location. Hence, one would expect that with decreasing ability, the probability of selecting each category would increase in an ordered fashion from least to most difficult. Disordered thresholds may occur in the event of an underused category or unclear descriptive wording, or if the number of categories exceeds the number of levels the participants can distinguish. ²¹ Disordered thresholds can be a source of item misfit. Therefore, in cases of disordered thresholds, response categories were collapsed (adjacent categories combined together) until thresholds were ordered; this was performed prior to any further analysis. Category probability curves were inspected to identify which adjacent categories could be combined in the presence of disordered thresholds. Categories that showed the greatest overlap of curves are usually the most appropriate to combine; however, where an underutilized category could be combined with two adjacent categories, both were performed in turn, with the impact on the fit to the model assessed. The combination that provided the largest improvement in fit was accepted. Categories were considered for combining only when labels made it logical to do so. Ideally, categories should be evenly spaced and should advance step calibrations by at least 1.4 logits. ²¹  

Dimensionality

Item fit statistics are used in the assessment of unidimensionality, which demonstrates whether the questionnaire or subscale is measuring a single concept. Fit statistics (infit and outfit) focus on two aspects, which can be reported as a mean square (MNSQ) with expected values of 1. The MNSQ residual statistic is normalized to the average expected variance such that a residual of less than 0.70 indicates at least 30% less variance than expected, suggesting a high level of predictability or possible redundancy. Residuals greater than 1.30 indicate at least 30% more variance than expected, suggesting that items may be measuring something different from the overall scale. Therefore, an acceptable infit and outfit is within the range of 0.70 to 1.30. ²² Misfitting items were removed from subscales until all items demonstrated unidimensionality, if possible, prior to further analysis. 

Precision

Precision refers to whether a measure is able to discriminate along its scale, and person separation statistics indicate the overall precision of the instrument. The ratio of the true variance in the estimated measures to the observed variance can be used to determine how many groups or strata of person ability an instrument can discriminate. Greater precision is indicated with a greater value for the person separation statistics. Instrument precision enables a greater distinction between levels of function. A minimal acceptable cutoff value for the person separation ratio was set at 2.0 for this study. ²³  

Targeting

Targeting refers to the extent to which the difficulty of the items matches the abilities of the persons in the sample, and is assessed numerically by comparison of the person and item mean values. A perfect targeting instrument would have a difference of zero, whereas a difference of more than 1 logit indicates significant mistargeting. ²⁴  

Results

The questionnaire was self-administered by 44 patients (mean age 24.1 ± 5.2 years; age range 18–39 years; 25 female, 19 male). A separate Rasch analysis of stacked preoperative and postoperative data was performed for each of the 12 subscales. Category response thresholds were assessed first, before further analysis. Three response category formats had disordered thresholds requiring response categories to be reordered. The three affected subscales were Symptoms, Dependence on correction, and Suboptimal correction (Table). The Figure illustrates the category probability curves for the Symptoms subscale. It can be seen from the Figure that category 3 was underused; at no point was it more likely to be chosen over any other category. 

Rasch analysis was repeated with the newly reorganized response formats, and the subscales were assessed for unidimensionality with item fit statistics. Six of the 12 subscales contained misfitting items (Table). Misfitting items were removed prior to further analysis. There were two misfitting items in the Appearance subscale, and once these were removed, only one item remained. Therefore, further analysis was not possible for this subscale. 

Precision was assessed by person separation, with values greater than 2.0 deemed adequate (Table). Only the Near vision subscale demonstrated adequate person separation (2.81). All remaining subscales had a person separation below acceptable standards. 

Targeting of person ability to item difficulty was adequate in 11 of the 12 subscales, with a mean item location of less than 1 logit. The Worry subscale had poor targeting, with a mean item location of −1.50 logits (items too difficult). 

Discussion

Keratoconus is a noninflammatory, progressive, ectatic disease of the cornea. It has well-described clinical signs including stromal thinning, protrusion, irregular astigmatism, and scarring. It can lead to significant visual impairment and decreased QoL. ²⁵ There are a range of treatment and management modalities available for keratoconus. ²⁶ QoL is an important outcome measure in the assessment of the various treatment options; however, QoL measures have rarely been applied. The NEI-RQL-42 questionnaire was developed to measure refractive error–related QoL; however, the questionnaire's psychometric properties for use with patients with refractive error correction have been found to be grossly inadequate. ¹¹ It was hypothesized that the questionnaire may function better in disease groups where the latent trait of activity limitation may be more important. Hence this study investigated the psychometric properties of the NEI-RQL-42 QoL questionnaire in a keratoconus population. 

The results revealed that the questionnaire was deficient in the assessment of QoL in keratoconus, with only one valid subscale of the 12 assessed. This was the four-item Near vision subscale, with an adequate response category, non-misfitting items, adequate person separation, and adequate targeting. The remaining 11 subscales were deficient in one or more ways. Three subscales (Symptoms, Dependence on correction, and Suboptimal correction) had disordered thresholds that required reordering of response categories (e.g., Fig.). This commonly occurs because there are too many response options, as it has been shown that respondents can generally distinguish only four or five response options. ²¹ The number of response options for items in the questionnaire varied between 1-2, 1-3, 1-4, 1-5, and 1-6 across the subscales. Subscales with disordered response categories needed to be repaired prior to further analysis; hence all disordered subscales underwent category collapsing to eliminate disordering. 

Unidimensionality was assessed with item fit statistics, which indicated that six of the 12 subscales had misfitting items. This indicates that these six subscales were not assessing a single latent trait but rather multiple traits, with the questions within each subscale not measuring what they purport to measure. This is an important issue in measurement that also occurs with clinical measurement. For example, a device that measures intraocular pressure (IOP) and central corneal thickness (CCT) might be very useful if it produces two unidimensional scores (one for IOP and one for CCT). However, if it produces a single multidimensional score, this might have something to do with glaucoma, but it has no clinical utility. The six multidimensional subscales of the NEI-RQL-42 QoL questionnaire have this exact problem. In the previous study investigating the psychometric properties of this questionnaire in refractive error correction, six subscales were also found to have misfitting items, although only three subscales were the same as in the present investigation: Activity limitations, Symptoms, and Dependence on correction. ¹¹ The reasons for item misfit are probably related to the design of the questionnaire, particularly the appropriateness of the questions for the subscale and the format of the response options. Misfitting items were removed from the appropriate subscales and Rasch analysis was repeated. 

Person separation was found to be inadequate for all except one of the 12 subscales: Near vision. The remaining subscales had person separation values significantly less than the minimum accepted value of 2.0. ²³ This indicates that these subscales could not adequately discriminate between the individuals in the sample population. Targeting was adequate for 11 of the subscales. Targeting refers to the matching of the mean person ability estimates and mean item difficulty estimates. This finding suggests that these items are important to the keratoconus disease group. In comparison, targeting in the refractive correction group was poor, as many subjects had no such problems (items were too easy for the patients' ability level). 

While the Near vision subscale could be recommended for measuring visual functioning in keratoconus, there are many other visual functioning instruments that could be considered for use. ¹⁶ Many researchers who are unfamiliar with the properties of a good-quality questionnaire choose the NEI-RQL-42, as they assume it would have been developed appropriately under the sponsorship of the National Eye Institute and the subsequent naming of the questionnaire. A number of good visual functioning questionnaires have been revalidated with Rasch analysis for use in a cataract population which have better psychometric properties. ^27–29 There is also the Quality of Vision Questionnaire, which was developed using Rasch analysis for refractive surgery and cataract surgery. ^15,30 It is likely that these instruments will measure visual functioning adequately in keratoconus also. We have previously attempted to repair the NEI-RQL-42 questionnaire in the refractive surgery group without success, ¹¹ and it would not be advantageous to attempt repair in this keratoconus data set. Even if it were possible to create some sort of valid measure with adequate psychometric properties, we could not advocate it for keratoconus. The questionnaire was not developed for keratoconus, and it is likely to be content deficient. There are also significant problems with the design of the questionnaire, such as the differences in the wording and the number of response options across the items. Moreover, the Keratoconus Outcomes Research Questionnaire was specifically developed to measure visual functioning and symptoms in keratoconus patients, with good psychometric properties as determined by Rasch analysis, and is a better alternative (Pesudovs K, et al. IOVS 2007;48:ARVO E-Abstract 1841). 

The main limitations of the present study include possible cross-cultural issues with this analysis, as the questionnaire used was a Portuguese version of the NEI-RQL-42 and the results may be different with the original, English version. The study also contains a small sample size and reports findings similar to those from our previous study in a group of refractive surgery patients. ¹¹ However, it has the key benefit of utilizing modern psychometric theory to highlight the shortcomings of this questionnaire. 

In conclusion, this study further highlights the deficiencies of the NEI-RQL-42, this time in keratoconus, and demonstrates that the NEI-RQL-42 should not be used in keratoconus studies. Only one subscale was found to be functional in this group, with all other subscales deficient on a number of counts. Better instruments are available, such as the Keratoconus Outcomes Research Questionnaire or the Quality of Vision Questionnaire, depending on which latent trait one wishes to measure. 

Figure. 

 

Response category probability curve for the Symptoms subscale. The x-axis represents the difference between item and person calibration, and the y-axis represents the probability of the category's being chosen. It can be seen that category 3 was underused; at no point was it more likely to be chosen over any other category.

Table. 

 

Overall Performance of the NEI-RQL-42 Quality of Life (QoL) Questionnaire in Keratoconus using Rasch Analysis

Subscales	No. of Items	Thresholds Needing Reordering	No. of Misfitting Items	Misfitting Items	Person Separation	Mean Item Location
Clarity of vision	4	None	1	40	1.44	0.06
Expectations	2	None	None	NA	0.00	−0.43
Near vision	4	None	None	NA	2.81	0.03
Far vision	5	None	3	6, 9, 10	0.87	0.47
Diurnal fluctuation	2	None	None	NA	1.67	−0.44
Activity limitations	4	None	2	33, 34	0.66	0.16
Glare	2	None	None	NA	1.53	−0.05
Symptoms	7	1 (item 25)	3	25, 41, 42	1.91	0.76
Dependence on correction	4	2 (items 15, 16)	1	13	0.00	−0.58
Worry	2	None	None	NA	1.01	−1.50
Suboptimal correction	2	2 (items 31, 32)	None	NA	0.00	0.65
Appearance	3	None	2	27, 29	NA	NA
Satisfaction with correction	1	NA	NA	NA	NA	NA

Footnotes

 Disclosure: C. McAlinden, None; J. Khadka, None; J. de Freitas Santos Paranhos, None; P. Schor, None; K. Pesudovs, None