Moving from PROMs to POEMs for Glaucoma Care: A Qualitative Scoping Exercise

Abstract

Purpose.: Patient-reported outcome measures (PROMs) are accepted widely as important outcomes in comparative effectiveness research. Over 30 PROMs have been described for use in glaucoma research, but their role in routine care is undefined. Our study explores the views of patients with glaucoma and their carers on the feasibility and content of a glaucoma PROM for use in day-to-day clinical practice.

Methods.: Focus groups were held with 71 participants facilitated by 23 staff working in the field of ophthalmology. Comparisons were made among themes derived from this exercise using simple thematic analysis and currently available health-related quality of life (HRQoL) instruments.

Results.: Participants were supportive of reporting their outcomes if the instrument was short, practical, and useful. Potential barriers to the use of PROMs, including bureaucratic overload and accessibility issues, were identified. Measuring health outcome and patient experience was important to participants. No freely available instrument covers all the domains identified, particularly knowledge and understanding. A novel instrument, a glaucoma patient-reported outcome and experience measure (POEM) is proposed. This addresses three aspects of outcome (fear of blindness, acceptability of treatment/side effects, and impact on daily life), and three aspects of experience (safety, respect, and understanding).

Conclusions.: Using PROMs routinely in the management of glaucoma presents significant challenges. Although current instruments (questionnaires) cover many of the participants' key domains, no single instrument covered them all. Further research is required to determine the feasibility (resources required), and validity and reliability of the proposed glaucoma POEM in clinical care.

Introduction

 

The ultimate measure by which to judge the quality of a medical effort is whether it helps patients as they see it. ¹

 

Important aims of quality health care are patient centeredness ² and a focus on patient experience. ³ Traditional outcome measures are based on clinical signs; however, since the 1980s the use of consumer focused, patient-reported outcome measures (PROMs) has grown. ⁴ The United States Food and Drug Administration (FDA) reported that between 1997 and 2002, 30% of drug labels reported patient-reported outcome (PRO) information and 11% of products were approved on the basis of PROs alone. ⁵ Most PROs measure satisfaction with care received or health-related quality of life (HRQoL). HRQoL is difficult to define, but the World Health Organization (WHO) International Classification of Functioning, Disability and Health ⁶ (ICF) is a good starting place to describe health and disability by considering these states in terms of body functions, activities, participation, and the environment in which they occur. Although many instruments are described under the broad term “HRQoL instruments,” when compared to this definition it is clear that many only measure aspects of HRQoL, or different concepts altogether. ⁷ Some only record function (e.g., Glaucoma Quality of Life 15 [GQL15] ⁸ ), while others assess patient-reported disability and/or quality of life, and can be disease-specific (e.g., National Eye Institute Visual Function Questionnaire [NEI-VFQ-25] ⁹ ) or generic (e.g., EuroQol EQ5D ¹⁰ ). Utility measures value health status, are preference weighted, can be derived from questionnaires, and allow the estimation of generic (EQ5D) and program-specific (Glaucoma utility index ¹¹ ) Quality Adjusted Life Years (QALYs), respectively. 

PROMs were developed for use in research, but the UK National Health Service (NHS) now measures and reports on disease-specific and generic PROs routinely for four surgical conditions, using paper questionnaires at an approximate cost per patient of £6.50 (€8, $10). With advances in technology, PROMs can be administered by tablet personal computer ¹² or on the internet, ¹³ potentially reducing costs and increasing opportunities to use PROMs in routine practice. ¹⁴ Patient-based valuation of health status and the care process may provide useful information for commissioning optimal care pathways in developed and developing nations, ¹⁵ but such scores are difficult to interpret. 

Typically, groups of patients have been assessed with these instruments as they are insufficiently sensitive for use in individuals. ¹⁶ However, in other contexts PROMs have been useful to guide the care of individuals by increasing the identification and discussion of moderate-to-severe health problems. ¹⁷ Examples include pain measures in emergency departments, scales for urinary obstruction, and functional status measures for managing cancer. Measuring such outcomes in glaucoma is particularly challenging because it is a chronic disease, lasting decades, without any symptoms in the early stages. Nevertheless, the impact of a diagnosis of glaucoma on quality of life, even in the absence of symptoms, can be significant, making the development of an instrument to measure PROs longitudinally desirable. 

There has been a proliferation of instruments for use in ophthalmic research, both disease-specific and generic. ^18–21 Two reviews of the use of PROMs in glaucoma identified 27 and 33 instruments, respectively. ^19,20 This raises several questions about the collection of PROs in day-to-day glaucoma care as opposed to the research environment; which of these instruments should be used, ²² can instruments designed for use in the research setting adequately measure quality of care in routine health service delivery, and can such data be used in pragmatic randomized controlled trials ²³ and quality improvement initiatives evaluating comparative effectiveness of alternative treatments or programs of care? Many instruments have been developed according to basic psychometric principles, ²⁴ but a conceptual framework building on patient views was absent in more than 50% of instruments. ²⁰ The 2011 consensus meeting of the World Glaucoma Association concluded that there are numerous valid PRO instruments, but their role in clinical practice and trials requires further investigation. Questions remain about which patients to assess and how often, what type of instrument to use (generic, and/or disease-specific), how to collect the data, how to target hard-to-reach groups, and where to collect the data. Although it may be practical to collect responses in the clinic setting, there are potential problems of bias and coercion, and once data are collected, there remain the substantial tasks of analysis, interpretation, action, and dissemination. Our study aimed to develop a PRO instrument for use in clinical practice as opposed to research by assessing the patient perspective on PROMS, what a PROM for glaucoma should measure, and the feasibility of such measurement. The participants' PROM specification was compared to the items included in existing instruments available in the public domain to establish if an ideal instrument already exists or, if not, to develop one. 

Methods

The context of the study was the National Glaucoma Think Tank, an interactive Patient and Public Involvement (PPI) event hosted by the International Glaucoma Association (IGA), University College London Partners, and the NIHR Biomedical Research Centre for Ophthalmology. Public and patient participation was invited through word of mouth and through the IGA website. Participation and attendance was voluntary and open to all. A total of 71 participants attended the event (3 health care workers, 42 patients, 11 carers, 15 unspecified) and were divided into 7 focus groups. Of these participants 56 (79%) consented to give demographic data. Although participation was open to all, efforts were made to invite participants representing a wide spectrum of society covering sex (28 males, 43 females), all ages (mean age 54 years, range 11–82), multiple ethnicities (30% from non-White ethnicities, including Bangladeshi, Indian, African-Caribbean, Pakistani, and mixed background), multiple glaucoma diagnoses (55% primary open angle glaucoma, 14% secondary glaucoma, 12% angle closure glaucoma, 7% congenital/juvenile glaucoma, 7% normal tension glaucoma, 5% ocular hypertension). Of the glaucoma patients 76% were taking drops and 71% had undergone some form of surgical procedure (including laser procedures). Most of the participants with glaucoma had good visual function, but 7% were registered sight impaired and 10% were registered severely sight impaired. Informed consent was gained from all participants. The study adhered to the tenets of the Declaration of Helsinki and ethical approval was gained from the National Ethics Research Service (11/SW/0289). 

A range of educational, clinical care, and research-related issues were addressed during the event. Participants' viewpoints on the use of PROMs in glaucoma were considered in focus groups with facilitation by members of the multidisciplinary faculty of 23 healthcare professionals (doctors, nurses, orthoptists, operating theater practitioners, optometrists, pharmacists, representatives of support groups, and the IGA). The facilitators guided discussions in the focus groups and recorded discussions by note taking. After reflection, verbal summaries were fed back to the larger group and recorded. A semi-structured topic guide was used to direct discussions toward the impact of glaucoma on quality of life (in terms of the WHO ICF domains of functions, activities, participation, and environment), the purpose and use of PROMs and patient reported experience measures (PREMs), the mechanics of filling out forms, accessing results, and anticipated problems. Narrative data were collected through audio and video recording, written notes, and an online forum. Recordings were transcribed and analyzed with simple thematic analysis facilitated by NVivo software (QSR International, Cambridge, MA) using the protocol described by Braun and Clarke. ²⁵ Coding was done independently by JS and VC, followed by team discussion to refine the coding categories and develop a shared analytic framework. The identified themes were sorted into domains with subcategories formulated as questions by JS and VC. To assess the extent that participant's concerns were reflected in existing instruments, these domains and subcategories were mapped onto a range of existing instruments. The health outcome subcategories were compared to a database of items derived from 20 existing instruments available in the public domain developed in a previous study. ²⁶ The EQ5D can attract a licensing fee, but was included as it has been adopted in the NHS outcomes framework and by NICE. ²⁷ The howRu instrument was included as it was designed specifically for use in routine practice, and despite its brevity has shown good correlation with established generic health status measures, ²⁸ for a total of 22 instruments. The patient experience subcategories were compared to the NICE-QS on patient experience, the NHS outpatient survey questionnaire, and the howRwe questionnaire. It became apparent that the knowledge and understanding domain is a grey area between outcome and experience. Patients' understanding of their condition could reflect their experience of care, but also might be considered an outcome that may influence factors, such as adherence to treatments that alter HRQoL outcomes. This ambiguity is reflected by the presence of questions on this domain in outcome and experience measures (Table 1). This overlap led to the concept of combining measures of health outcome and experience. Group discussion conducted by the multidisciplinary research team (PS, VC, JS, FS) informed by the literature and research findings led to the formulation of a six-item instrument, a glaucoma patient-reported outcome and experience measure (POEM, Tables 1 and 2). 

Table 1. 

 

Narrative Data on the Rationale for and Barriers to the Collection of Patient-Reported Data

Domain	Category	Example Quotes
Drivers	Care improvement	That it's (the feedback) been acted on so that a follow up would say “we've listened, this is what we've found so we've adapted our clinical structure to reflect what you and others have said.” I think it would be a very useful tool.
	Variation	Glaucoma sufferers need to know that we are getting the best treatment possible, not the haphazard and often risky treatment that is doled out to us at present.
	Outcome measurement	Should be treated as a whole person.
	Purpose	It's going away from targets and it's going away from waiting lists and things like that, and going more towards the patient experience.
Cautions	Focus on different aspect of care	What are we trying to deliver?
		Do you want to really open up those floodgates?
		… surgeon should concentrate on getting treatment correct rather than on QoL issues.
Suggestions	Delivery	Must be short, practical and useful – there must be an action point, timeline, and deadline.
		Experience … there isn't time for that really.
		As long as it's not too in depth and too thorough, you need something that's basic and easy to fill in.
Barriers	Accessibility	The people here today are informed, motivated, and proactive. How do you reach those who are not?
		I think filling in forms with the lack of sight that I have now … my vision is very down … I think I would have the best of intentions, but it might never get filled in … . That's the truth … nothing like the spoken word and keeping it obviously as brief as possible …
	Bureaucracy	I'm all in favor of measurements … but … the problem is … the management will tend to produce more and more bits of paper to justify the situation you're in.

Table 2. 

 

Outcome and Experience Measures of Importance to Participants

Type of Patient-Reported Measure	Domain	Statement (Item No. Used in Fig. 1)	Glaucoma POEM	N of Comments	Inclusion in Existing Instruments
Outcome	Peace of mind	I am not worried about going blind in future (1)	Are you frightened of going blind from glaucoma? (PRO)	38	36%
		I don't think glaucoma will stop me from doing what I want to do in future (2)		15	45%
	Activities of daily living	I can use my drops properly without any help (3)	On balance is your glaucoma treatment (and any side-effects) acceptable to you? (PRO)	13	18%
		I'm not bothered by side effects from my drops (4)		23	18%
		I can cope with any mobility problems without any help (5)	Does your glaucoma interfere with your daily life? (PRO)	14	73%
		My eyes are never painful (6)		2	23%
		I can see well enough to do the things I enjoy (7)		15	55%
	Social relationships	I am still able to enjoy doing things with friends and family (8)		23	68%
		I still feel a valuable member of the team at work (9)		10	23%
Experience	Administration	I get convenient appointment times	Do you feel safe under the care of your glaucoma team? (PREM)	2	*†
		Changes to my appointments are kept to a minimum		6	†
		I am happy with the intervals between my appointments		5	*†
		The time I have to wait in the clinic is reasonable		5	†‡
		I have access to my clinical details if I want to see them		7	*
		Different members of the team are easily identifiable		3	*†‡
	Environment	The clinic is hygienic		2	†
		It is easy to find my way to and around the clinic		7
	Patient-centered care	Each time I visit the clinic the doctor knows about me and my condition		13	*†
		I trust my glaucoma care team to act in my best interest		16	†‡
		I feel confident that I am receiving the best possible care		19	†
		The doctor cares about me as an individual	Do you think your glaucoma is getting worse? (PREM)	20	*†‡
		My opinion is treated with respect		1	*†‡
		Sufficient time is available to address my concerns		14	*†‡
	Knowledge and understanding	I am confident I understand what glaucoma is (10)	Do you understand your diagnosis and treatment plan? (PREM)	24	9%*†‡
		I am confident I understand how my glaucoma will be treated (11)		4	9%*,†,‡
		I understand why putting my drops in every day is important (12)		1	9%*,†,‡
	Communication	In the clinic I am given the information I need		52	*†
		I am encouraged to ask the questions I want to ask		13	*†
		I have a contact number to use if I had any concerns at a later time		4	*†
		My GP and optometrist are kept informed about my condition		7	*†‡
		I know how to give feedback about my experience in the clinic		3

*  Those elements that are covered by planned NICE quality standard of patient experience.

†  Those elements that are covered by the Picker NHS outpatient questionnaire.

‡  Those elements that are covered in the howRwe patient experience instrument.

Results

A total of 42 comments related to the feasibility of collecting PROs for glaucoma. Participants were prepared to support the use of PROs if the instrument was short, practical, and useful: however, they identified several barriers (Table 3). 

Table 3. 

 

Glaucoma POEM

Question Number	Glaucoma POEM
1	Are you frightened of going blind from glaucoma? (PRO)
2	On balance is your glaucoma treatment (and any side-effects) acceptable to you? (PRO)
3	Does your glaucoma interfere with your daily life? (PRO)
4	Do you feel safe under the care of your glaucoma team? (PREM)
5	Do you think your glaucoma is getting worse? (PREM)
6	Do you understand your diagnosis and treatment plan? (PREM)

The focus groups generated 153 statements on PROM content mapping to three domains: Peace of mind (53), daily living (67), and social relations (33) summarized in nine subcategories. A further 228 statements on experience emerged mapping to 5 domains: Administration (28), communication (79), environment (9), patient centered care (83), and knowledge and understanding (29), with 22 subcategories (Table 1). 

Comparison of the domains and subcategories generated by this qualitative research exercise with 22 existing PRO instruments indicated that no publicly available instrument deals with all the domains identified (Fig. 1). It was notable that only 9% of the surveyed instruments collected data on the domain of knowledge and understanding, while 13% of the participants' comments were on this theme. The domains highlighted in the NICE-QS covered 14 of the 22 patient experience issues highlighted by the participants, with the NICE-QS also highlighting requirements to enquire about physical or learning difficulties, education of staff in relation to communication skills, collection of patient experience data, and provision of shared decision support tools. Of the 22 experience issues identified 19 are addressed in the 56-question Picker NHS outpatient questionnaire ²⁹ with the only omissions relating to an ability to access clinical details, such as medical records, knowing about the feedback process, and of particular relevance to patients with visual impairment, navigating to and around the clinic. The howRwe questionnaire, although shorter than the Picker questionnaire at just four questions, covered 10 of the 22 experience subcategories. Although such broad coverage can make interpretation and response to results more difficult, this observation led us to develop the six-question POEM, with each item supported by 9 to 28% of the original statements (Tables 2 and 3). 

Figure 1. 

 

Coverage of National Glaucoma Think Tank–derived domains and items in existing HRQoL instruments (where item is covered relevant box is shaded).

Discussion

Our study demonstrated that participants would support the measurement of patient-reported outcome and experience in routine practice if it promoted improvement, and was quick and easy to administer. However, there still are considerable obstacles to overcome to ensure that data collection is equitable, practical, and affordable, and the data produced are valid to guide patient care improvements and prioritization of healthcare resources. Our study represents a first step in developing PRO's for day-to-day clinical practice. Next steps must begin with the selection or development of an appropriate instrument for widespread use. 

A 2008 assessment of 11 HRQoL instruments for glaucoma concluded that there was no best PRO instrument, ²² a finding mirrored by our study. Comparing domains and subcategories identified in our study to those contained in the 22 HRQoL instruments indicated that none covered all of the domains of interest to the participants. The Impact of Vision Impairment questionnaire (IVI), EQ5D, and howRu had the best coverage, while only the Odberg ³⁰ and Low Vision Quality of Life (LVQOL) questionnaires ³¹ featured questions relating to knowledge and understanding. Previous work on the IVI, a 28-item instrument ³² initially designed to assess participation in daily activities in visually impaired people, indicated that it was poor at assessing patients with glaucoma and relatively good vision, ³³ and probably is too long for routine use. The EQ5D, a tried and trusted measure in widespread use, ²⁷ also provided reasonable coverage of issues of importance to the participants. A review of the use of EQ5D in glaucoma indicated that it can stratify patients according to Snellen score and does appear to correlate with other disease measures. ²¹ However, utility scores derived from the EQ5D are unlikely to reflect the benefits of current treatments for glaucoma, as they are related to mean deviation (MD) in a nonlinear fashion and are affected only when MD falls below −25 in the better eye. ³⁴ Therefore, EQ5D is unlikely to provide useful data for the majority of patients seen for glaucoma who have good central vision with predominant loss of peripheral vision. Using disease-specific measures, which correlate better with clinical disease measures, appears to be a sensible compromise, but most are not preference-based and cannot be used to calculate QALYs. ³⁵ The Odberg questionnaire, while targeting an area of importance to patients (knowledge and understanding) not covered in other questionnaires, does not conform to the standards of modern questionnaire design and at 33 questions probably is impractical for routine use. The LVQOL, although developed according to psychometric principles and shorter at 21 items, did not consider the views of patients with glaucoma when it was designed ¹⁹ and does not cover issues about peace of mind, which concern between 50% ³⁶ and 80% ³⁰ of patients newly diagnosed with glaucoma. 

One solution to this lack of a best fit instrument may be the third generation questionnaire approach of item banking, and adaptive, computer-based testing. ^37,38 Prior et al. outlined an alternative approach where items in existing instruments are pooled to create instruments with improved content validity. ²⁶ A third solution may be to combine one or more instruments and a final option would be the development of a bespoke instrument, such as the POEM outlined in our study. 

Once an instrument has been selected, it then is necessary to consider the logistics of data collection and analysis. A large scale attempt to collect PRO data in routine ophthalmic practice was disappointing. Using the VF14 to assess cataract surgery outcomes revealed a range of concerns not only with the content validity and responsiveness of the questionnaire, which may be rectified by using other questionnaires, ³⁹ but also because visual problems made it impossible to self-fill the paper questionnaire for 30% of patients. ⁴⁰ The participants in our study identified the need for a short, quick instrument. The developers of the howRu instrument reached the same conclusion, and developed it to be generic, practical and suitable for electronic data collection to give immediate feedback. They developed two instruments that focus on measuring the outcomes discomfort, distress, disability, and dependence, and the experience of delays, communication, care, and teamwork. The howRu instrument is 40 words in length, making it more readable and shorter than other instruments, and it proved effective when applied either electronically or by telephone, which deals with one of the potential obstacles for a questionnaire for use in routine practice by patients with visual impairment. ²⁸ It addresses the participants' requirement for a short, practical instrument and covers several of the issues outlined by the participants. It would be interesting to conduct a clinical pilot to compare the howRu and howRwe instruments with our POEM to establish patients' views on using these instruments in the management of patients with glaucoma, to see what effect their use has on practice, how easily such instruments can be incorporated into the wider deployment of electronic health records, what benefits are accrued, and whether these outweigh the cost of overcoming obstacles, including administration and analysis costs, difficulties filling in forms, functional illiteracy, lack of motivation for clinicians and patients, ⁴¹ and the difficulties of measuring the effects of visual impairment on HRQoL in patients who often are asymptomatic. ⁴²  

A further step should involve validating the findings of the combined POEM against existing measures that focus either on outcome or experience. Experience surveys are carried out typically on an annual basis, and further work on more frequent surveying and continuous quality improvement would be of interest. The outcome and experience domains were condensed to make the POEM a short instrument fit for routine use. Pilot studies should confirm that it still focusses on those issues of most importance to patients. ^11,43  

Participants indicated that HRQoL (PROs) and experience of care (PREMs) were important outcomes. This corroborates the conclusions of the International Society of Quality of Life Research, which suggested that PROMs have a range of uses in clinical practice, for example detecting physical and psychosocial problems that otherwise might be overlooked, monitoring disease and treatment effects, improving patient-healthcare provider communication, and altering referral patterns. ⁴⁴ The future of outcomes measurement in glaucoma care may combine a range of outcomes of importance to physicians and patients. Attending to the perspectives and viewpoints of the Glaucoma Think Tank participants suggests that such a system may incorporate both elements of HRQoL, and experience into an integrative measure of patient-reported outcome and experience, namely, a POEM. Combining a POEM with clinical measures of disease severity, and performance measures of real-life functioning has potential to provide a more complete picture of the impact of glaucoma on issues of importance to patients (Fig. 2). 

Figure 2. 

 

Outline of current outcome measures for glaucoma and proposed POEM's place.

Our study has developed the POEM in a robust manner, but further research is required to tests its practicality and validity in day-to-day care and research settings. While a proliferation of outcome measures in glaucoma already has occurred, it is to be hoped that the POEM may be developed further and used widely without the development of multiple, similar tools. If the POEM is adopted widely, analysis of variation of patient-reported outcome and experience data within and between centers may provide evidence to improve the quality of patient care. 

Limitations

Our study has several limitations. Only those instruments available in the public domain, and the EQ5D and howRu were considered, and it is possible that some proprietary measures may perform better against our participants' domains than those instruments tested. The participants were voluntary attendees of a National Glaucoma Think Tank and may not be representative of a wider body of patients with glaucoma. The study considered the theoretical use of PROMs and this may not have identified real life practical issues, which would be uncovered by a formal pilot study. Time constraints did not allow for detailed feedback from patients on how PRO data should be used and interpreted. Our study focused on the views of patients, but for PROMS to be of use input from clinicians and consideration of the uses they would make of the data are necessary additions to any developments. ⁴⁵ Consideration of how subsequent data may be used was beyond the scope of this project, with no discussion of the potential to use such data to define payment patterns, treatment thresholds, referral patterns, benchmark providers, and inform choice. ⁴⁶ For example, Burge et al. indicated that patients regard the relative performance of providers in improving their health as highly important in choosing where to receive surgery. ⁴⁷ Our study only explored participants' ideas about subjective questionnaires. It could be argued that measures of performance are less subjective, monitor real-life disability, are less susceptible to cultural and language differences, cognitive impairment, and illiteracy, and may be easier to apply. ⁴⁸ Although typically such measures have been time-consuming and labor-intensive, making them useless for clinical practice, recent studies show that abbreviated tests, such as the Assessment of Ability Related to Vision test (AARV), can provide useful information in much reduced times, ⁴⁹ and it would have been of interest to obtain the participants' views on such instruments. 

Finally, although we matched up the domains the study participants mentioned with those seen in current instruments, further psychometric testing of questionnaires often reveals redundancy of domains and clustering about different factors, for example after Rasch analysis the IVI was reduced from 32 to 28 questions and from five domains to 3. Analysis of the domains highlighted in our study may conclude that some of the items describe the same underlying concept. 

Conclusions

Using PROMs routinely in the management of glaucoma presents significant challenges. It is unclear how, when, to whom, and under what circumstances a routine patient-reported measure should be administered, although it should be quick and easy to deliver. Outcome and experience measures were important to participants in our study, and current instruments to measure outcome and experience appear to cover many of the domains identified by our participants. Nevertheless, no instrument covered all the domains and it is unclear if any existing instrument has practical value in routine practice. Further research is required to test the validity and reliability of the proposed items, and the feasibility of introducing a patient outcome and experience measure in glaucoma clinical care. 

Acknowledgments

Maria Prior, Health Services Research Unit, University of Aberdeen, provided access to the item pool generated for the development of the Aberdeen Glaucoma Questionnaire. 

Footnotes

 Supported in part by the Department of Health's NIHR Biomedical Research Centre for Ophthalmology at Moorfields Eye Hospital and UCL Institute of Ophthalmology. The views expressed in the publication are those of the authors and not necessarily those of the Department of Health.

Footnotes

 Disclosure: J.E.A. Somner, None; F. Sii, None; R.R. Bourne, None; V. Cross, None; J.M. Burr, None; P. Shah, None