Identifying the Critical Success Factors in the Coverage of Low Vision Services Using the Classification Analysis and Regression Tree Methodology

Abstract

Purpose.: To identify the critical success factors (CSF) associated with coverage of low vision services.

Methods.: Data were collected from a survey distributed to Vision 2020 contacts, government, and non-government organizations (NGOs) in 195 countries. The Classification and Regression Tree Analysis (CART) was used to identify the critical success factors of low vision service coverage. Independent variables were sourced from the survey: policies, epidemiology, provision of services, equipment and infrastructure, barriers to services, human resources, and monitoring and evaluation. Socioeconomic and demographic independent variables: health expenditure, population statistics, development status, and human resources in general, were sourced from the World Health Organization (WHO), World Bank, and the United Nations (UN).

Results.: The findings identified that having >50% of children obtaining devices when prescribed (χ² = 44; P < 0.000), multidisciplinary care (χ² = 14.54; P = 0.002), >3 rehabilitation workers per 10 million of population (χ² = 4.50; P = 0.034), higher percentage of population urbanized (χ² = 14.54; P = 0.002), a level of private investment (χ² = 14.55; P = 0.015), and being fully funded by government (χ² = 6.02; P = 0.014), are critical success factors associated with coverage of low vision services.

Conclusions.: This study identified the most important predictors for countries with better low vision coverage. The CART is a useful and suitable methodology in survey research and is a novel way to simplify a complex global public health issue in eye care.

Low vision impacts Quality of Life (QoL). Low vision services have shown to be effective in increasing the QoL in people with functional low vision. However, of the estimated 70 million of the 124 million people with low vision who require services, ¹ approximately 5–10% were estimated to have access to care. ^2,3  

The demands for low vision services will continue to grow due to the emerging global trends in ageing populations and changes in the epidemiology of vision impairment. ^4,5 Consequently, to address the current and future needs of low vision services, a global mapping survey of services was conducted. The aims of the survey were to map the low vision services provided in each country and identify the critical success factors in the coverage of low vision services. However, due to missing data, which is a common issue with postal surveys, Critical Success Factors (CSF) were more challenging to investigate. ^2,6 This is because missing data generates problems when using traditional regression modeling methods such as logistic regression. 

Initially logistic regression was used to identify the critical factors. Only two variables were detected, with only one variable showing statistical significance. These findings were affected by the amount of missing data; for instance, not every country had data or collected data on the main outcome—coverage—and not every question in the survey was answered, such as the numbers of low vision health professionals in the country. Thus many countries were automatically excluded. As logistic regression does not allow full exploration of the data, an alternative method was sought. Hence the Classification Analysis and Regression Tree methodology (CART) was used to overcome some the limitations associated with the missing data using logistic regression. 

The CART technique creates subgroups that have better predictive ability, equivalent sensitivity, and better specificity with common demographic and risk predictors. ⁷ It is also less restrictive with parametric assumptions. ⁸ Furthermore, variables in real-world settings seldom have straightforward linear relationships with important health outcomes; nonparametric techniques such as the CART are therefore better able to identify these complex relationships. ⁷  

The CART can perform well with many missing variables. ^9,10 The CART is also able to detect interactions and is not easily affected by multicollinearity between variables compared to other regression modeling techniques. ⁹  

The objective of this study is to therefore apply the CART to predict the most important factors associated with better coverage of low vision services that will assist the International Agency for the Prevention of Blindness (IAPB) Low Vision Working Group and Vision 2020 in improving the current models of service delivery, future planning, training, and priority setting. 

Methods

Survey Development, Distribution, and Follow Up

Details of the survey have been described previously. ² Briefly, a survey was carried out that consisted of 33 questions with the following themes: policies, epidemiology, provision of services (types of care, location of services, and sources of funding), equipment and infrastructure, barriers to services, human resources, and monitoring and evaluation. Survey participants were required to provide a national perspective on low vision services in their countries. 

Participants were identified from official organizations: government, World Health Organization (WHO), Vision 2020, and the International Agency for the Prevention of Blindness (IAPB), as well as non-official sources such as private and non-government voluntary organizations. There were no lists of potential contacts to complete the survey, thus a snowballing technique was used. ¹¹ National Vision 2020 and National Prevention of Blindness Committees (PBL), where available, were first approached. If no contacts were elicited, IAPB, the WHO, and government departments (Ministries of Health and Education) were contacted thereafter. If further contacts were required, then both international and local NGOs were approached for potential contacts. Surveys were distributed in 192 World Health Organization (WHO) member states and three additional countries, namely Hong Kong, Taiwan, and Wales. 

The survey was translated into seven languages and distributed to participants in 195 countries throughout 2006 to 2008. An average of five reminders was sent over a follow-up period of 13 months. For countries in which no data were reported, a secondary data search was conducted. This consisted of searching on the World Wide Web, gray literature, and unpublished resources such as conference proceedings and reports for further information on the presence of low vision services. The survey adhered to the guidelines of the Declaration of Helsinki. Informed consent was obtained from the participants. Ethics approval was received from the Royal Victorian Eye and Ear Hospital Human Research and Ethics Committee. 

Dependent Variable

Coverage of services was the dependent variable. It is defined as the proportion of people with low vision using low vision services. A country with better coverage is one that has >10% of those requiring services who access low vision services, while poor coverage is defined as those countries with ≤10%. The 10% cutoff point was used as the criterion for service coverage because previous estimates indicated that very few countries had >10% low vision service coverage (2001 WHO workshop). ³ Nonetheless, other cutoff points such as 20%, 30%, and 50% were explored with the CART methodology. However, these produced insignificant outputs as only a very small number of countries had >10% coverage. For instance, only eight countries reported >50% coverage. 

Independent Variables

The independent variables were factors judged to be critical in the delivery of low vision care, hence the term ‘critical factors.’ 

The factors were categorized into two groups. 

•  
  Service-related (Table 1), are from the items of the survey that had been sourced from the social determinants of health, ¹² appraisal of past and current models of low vision service delivery, and barriers facing service provision. ¹³
•  
  Socioeconomic and demographic variables (Table 2), were secondary data drawn from the WHO, ^1,14,15 Vision 2020, ¹⁶ United Nations (UN), ¹⁷ and the World Bank. ¹⁸ These variables were included to represent the broader health care system in which low vision services operate and assessed to determine whether they influence the coverage of low vision services.

Table 1.

 

Service-Related Factors in the Coverage of Low Vision Services

Service-related

National policies on low vision

Presence of a national eye care plan that includes low vision

National referral guidelines for low vision services

Standards of practice guidelines for low vision services

Type of low vision care—monodisciplinary care compared with the different types of multidisciplinary care:

Type 1—Clinical, education, rehabilitation and social welfare services

Type 2—Clinical and education and social welfare and any one of form of rehabilitation service*

Funding type

Fully funded by government

National health insurance

National health insurance (with universal access)

Subsidized by government

Out-of-pocket

Private insurance

Supported by LNGO

Supported by INGO

Barriers to access

Lack of integration between low vision services with other eye care services, education, and rehabilitation

Waiting times between urban and rural areas

Lack of funding

Lack of human resources

Lack of referral networks

Lack of awareness on low vision

Lack of infrastructure

Lack of a national policy on low vision

Low vision human resources per 10 million of population

Ophthalmologists

Optometrists

MLOP

Specialist teachers

Community based workers

Rehabilitation officers

Human resources skills mix e.g. within HR mix 1 and HR mix 2

HR mix 1:

MLOP † and (Education and rehabilitation)‡ vs.

Ophthalmologist and MLOP and (Education and Rehabilitation)‡

HR mix 2:

MLOP and (Education and Rehabilitation)‡ vs.

Ophthalmologist and MLOP

Clients—adults and children

Proportion of children and adults obtaining devices when prescribed

Proportion of children and adults receiving services

Location of low vision services

Location type 1:

Only hospitals vs.

Hospitals and have one of these rehabilitation locations§

Location type 2:

Public hospitals and NGO rehabilitation agencies or community based services vs.

Public hospitals and government rehabilitation agencies vs.

Private hospitals and clinics and NGO rehabilitation

*  Forms of rehabilitation: orientation and mobility, sports and recreation, vocational training and technology training. ¹²

†  MLOP, MLOP and Optometrists.

‡  Education and rehabilitation, CBR and Rehabilitation workers and teachers.

§  Rehabilitation locations: government rehabilitation agency, NGO rehabilitation agency, and community based services.

To reduce the risk of including too many critical factors into the model, resulting in data dredging, ¹⁹ the authors consulted low vision experts and reviewed the literature. Only the most relevant socioeconomic and demographic-related variables were included into the CART model. ²⁰ For instance, in cases where two variables were related to the same topic, only one would be included in the model, e.g., information and communication technologies policy and information and communication technologies access. The latter was selected, as ‘access’ was an integral part of service coverage. 

CART Analysis

CART analysis was conducted using commercial analysis software (SPSS v. 17.0; SPSS, Chicago, IL). The outcome variable—coverage—had either a value of 0 (poor coverage ≤10%) or 1 (better coverage >10%). CART constructed a tree that separated the data in the most appropriate or ‘best’ way by finding binary splits on variables and the best splitting point at each stage. The decision rule was determined by the Gini criterion, a measure of variability within the new subgroups. ²¹ The χ² Automatic Interaction Detection (CHAID) ‘growing’ method was applied to identify the independent variables that exhibited the strongest interaction with the outcome, coverage. ²²  

Other growing methods were also tested to identify which produced the best model fit (e.g., Exhaustive CHAID—a modification of CHAID that examines all possible splits for each predictor; QUEST [Quick, Unbiased, Efficient Statistical Tree])—a method that is fast and avoids other methods' bias in favor of predictors with many categories; and CRT [Classification and Regression Trees]—which split the data into segments that are as homogeneous as possible with respect to the dependent variable coverage). Homogeneous, also known as a ‘pure’ node, is a terminal node in which all cases have the same value for the dependent variable. ²² However, none of the other methods were able to achieve a higher predictive model performance percentage than CHAID. 

The following criteria were specified in the CHAID growing method to produce the CART model (Fig. 1) ²² : A ≤0.05 significance level for splitting nodes and merging categories and the likelihood ratio was calculated as the χ² statistic for determining node splitting and category merging. This method was the most suitable given the sample size (n = 131). The tree-growing criterion to control the number of levels in the tree and the minimum number of cases for parent (the root) and child nodes (branches) was limited to three levels to control the maximum number of levels of growth beneath the parent node. ²² The minimum number of cases, which controls the minimum numbers of cases for nodes, was 10 cases for parent nodes and five cases for child nodes. 

Figure 1.

 

Critical success factors in the coverage of low vision services.

In terms of handling missing values, the CHAID growing method first generated categories using valid values, and then decided whether to merge the missing category with its most similar (valid) category or keep it as a separate category. ²²  

Finally, a classification table is displayed. This table indicates the overall predictive performance of the model (i.e., percentage of countries correctly classified with respect to each category of the dependent variable coverage) as well as the sensitivity and the specificity. Sensitivity is defined in this study as the proportion of countries with poor coverage (<10%) that are correctly classified as such. Specificity measures the proportion of countries with good coverage (>10%) that are identified not to have poor coverage. 

Validation was carried out on the CART output (Fig. 1) to assess the generalizability (how well the findings of the CART can be applied to real-world settings) of the CART structure. ²² Two validation methods were available: cross-validation and split-sample validation. ²² The cross-validation method, with a 10 sample fold was chosen. This was due to the sample size, as there were insufficient data for a separate test sample. ²²  

Results

Information on coverage was available from 131 countries: 16 countries have no services (zero coverage), 85/131 (64.9%) countries have poor (≤10%) coverage, 30/131 (22.9%) countries have better (>10%) coverage. ²  

Thirty-six service-related (Table 1) and 17 socioeconomic and demographic critical factors were considered (Table 2). At first, all 53 critical factors were included in the CART. However, conflicting information was found in the following service-related critical factors: national referral guidelines for low vision services and standards of practice guidelines for low vision service. These two variables were found to have a negative impact on coverage of low vision services, where the authors had expected the reverse to be true. There may be other factors affecting these two variables and coverage that were not measured. It was determined that these two factors may not be directly linked to coverage and were thus removed from the CART analysis, leaving 51 critical factors to be included in the CART model. These two variables are an example of data dredging. 

Table 2.

 

Socioeconomic and Demographic Critical Factors in the Coverage of Low Vision Services

Socioeconomic and demographic-related

Health expenditure

Per capita total government expenditure on health

Private expenditure on health as percentage of total expenditure on health

Out-of-pocket spending on health as percentage of private expenditure on health

External resources on health as percentage of total expenditure on health

Social security expenditure on health as percentage of general government expenditure on health

Population statistics

Population older than 60 years (%)

Population younger than 15 years (%)

Population living in urban areas (%)

Population living on <$1 a day (%)

Human resources in general

Physicians per 10,000 population

Nursing and midwifery personnel per 10,000 population

Community health workers per 10,000 population

Other health care workers density per 10,000 population

Development status

Annual economic growth (GDP)

Country income level (low, middle, high)

Information and communication technologies access per 1000 population

Human poverty Index (HPI)

The critical success factors (adjusted for all service-related, socioeconomic, and demographic variables) are exemplified in the form of a decision tree by the CART (Fig. 1): the first box at the very top of the CART tree displays the outcome—coverage. To avoid repetition, the results of one side of the outcome are shown; that is, no or poor (≤ 10%) coverage. The boxes in gray are the important predictors to the outcome. Level 1 is the most important predictor followed by levels 2 and 3. The first predictor at level 1 splits the tree roots (parent node) into three branches (child nodes) and so on. Each of these branches then becomes a parent node and splits into further branches until the CART tree stops growing and the node terminates (e.g., node 13). Each node displays: the node number, the total number of countries, and the percentage of countries with poor (≤10%) coverage. 

Six critical success factors were found by the CART to be significant predictors of coverage. The most important factor identified was the proportion of children obtaining devices when prescribed (χ² = 44.0; P < 0.0001), followed by level 2 predicators, namely percentage of population urbanized (χ² = 14.5; P = 0.002), monodisciplinary versus multidisciplinary care (χ² = 4.7; P = 0.03), and the number of rehabilitation workers per 10 million of population (χ² = 4.5; P = 0.034). The third level predictors included private expenditure on health as percentage of total expenditure on health (χ² = 14.6; P = 0.015) and fully funded by government (χ² = 6.0; P = 0.014). 

In the first branch, where there are a higher percentage of children obtaining devices but less urbanization (most people in many countries live in rural areas), the proportion of countries with poor coverage is 78.9%. Furthermore, if countries have either lower (≤35.5%) or higher (>43.6%) private expenditure on health, then the proportion of countries with poor coverage is both 100%. This denotes that countries with the above characteristics will have poor coverage of low vision services. Examples of countries in these nodes included Guyana, the Gambia, Croatia, India, Guatemala, and the Dominican Republic. On the other hand, in more urbanized countries, if services were fully funded by government, the likelihood of that country having poor coverage is reduced to 0%. Countries classified under this node were UK (Wales), The Netherlands, Taiwan, Sweden, and Norway. 

Where there are a lower proportion of children with access to low vision devices and only monodisciplinary care, the proportion of countries with poor coverage is 68.8%. Sri Lanka, Laos, South Africa, Bahrain, and Argentina are examples of countries in this category. 

Nodes 7, 3, and 9 have been displayed by the CART as missing nodes. Unlike logistic regression, where variables with missing data are excluded from the analysis, the CART model groups missing data with the predictor or category it is most similar or highly related to and displays them as surrogate nodes. Therefore, in this instance, the CART tree indicates that countries in node 7 all have poor coverage and these countries are related to the proportion of children obtaining devices when prescribed and the predictor monodisciplinary versus multidisciplinary care. Examples of countries under node 7 are Tunisia, Namibia, Liberia, Gabon, and East Timor. 

Node 8 shows that the proportion of countries with poor coverage is related to countries having ≤3 rehabilitation workers per 10 million and the proportion of children obtaining devices when prescribed. Benin, Armenia, Cambodia, Ghana, and Cameroon were some of the countries here, while the Czech Republic, Germany, Cook Islands, and Haiti were a few examples of countries under node 9 where the proportion of countries with poor coverage is related to an unknown number of rehabilitation workers. 

Finally, cross validation analysis indicated that the predictive performance of this CART model was 90.1% with a sensitivity of 93.1% and specificity of 80.0%. This implies that very few countries have been incorrectly classified in their respective categories of coverage. 

Discussion

This study identified the six most important critical success factors (CSF) associated with coverage of low vision services using a statistical technique that overcomes the issue of missing data common in postal surveys. 

Six CSF were identified and a theme/topic can be identified from each level of the CART diagram when viewed horizontally: level one is an indication of an effective and comprehensive service; level two is about service provision, i.e., distribution of services, type of care, and human resources; and level three is about funding, that is, public and private sources (Fig. 1). 

Level one shows the most important predictor — proportion of children obtaining devices when prescribed. Historically, services in many countries were established by charities and philanthropists intended for children and schools for the blind. ²³ The focus in many countries today is still on children and education. ¹³ It could be speculated that if services are predominately available for children, then adults may not necessarily have access to appropriate and available services. On another point, this predictor also highlights the importance of obtaining low vision devices. There are several factors which can impact on people's access to devices. These include cost, lack of awareness, devices being too difficult to use, and not enough training opportunities. ²  

At level two, each of the three predictors is interrelated and of equal importance. The percentage of population urbanized implies the importance of the availability of services in both urban and rural areas. In less urbanized countries, the likelihood of having poor coverage is high. Not surprisingly, the survey showed that rural dwellers were one of the groups of people less likely to access low vision services. ²  

With the factor monodisciplinary versus multidisciplinary, recent studies have demonstrated multidisciplinary low vision care to be effective. ^{24 –26} This also shows that services needs to be able to address multiple needs of the low vision client. For example, Stelmack et al. ²⁵ conducted a multi-center randomized clinical trial that found a combination of the following interventions: clinical low vision care, provision and education in the use of low vision devices, and rehabilitation in the activities of daily living, showed significant improvements in all aspects of visual function for clients with macular diseases. These findings concur with that of Lamoureux et al., ²⁶ who also found that a multidisciplinary low vision rehabilitation program resulted in significant improvements in the overall quality of life of the individual. The number of rehabilitation workers points toward the resources available to provide multidisciplinary care particularly at the primary care level. Primary low vision care in turn could also address another challenge facing the low vision population which is waiting times and distribution of services between rural and urban regions, this is related to the predictor percentage of population urbanized. 

At level three, the percentage of private expenditure on health includes non-government organizations (NGO) funding and, as it was shown to be on the same level as being fully funded by government, it signifies that private funding is also needed along with being funded by government. The CART output indicated that countries with higher private funding (compared to government expenditure) can lead to poor coverage. Interestingly, having little private funding also seems to lead to poor coverage, suggesting that NGO support is still essential. Indeed studies on other areas of health have suggested that relying on complete government interventions to achieve universal health coverage has often proved unsuccessful. ^{27 –29} It is clear that both public and private funding sources are required. 

The issues above emphasize the importance of achieving financial sustainability in the provision of low vision services. Practical and realistic strategies are required. Although government funding are given high importance, the feasibility of this is often dependent on the socioeconomic status of the country. ³⁰ Also, governments are increasingly faced with demands from other competing health priorities and often have limited resources. ³⁰  

An example of data dredging was demonstrated in the early stages of the analysis. The results revealed that having national referral guidelines for low vision services and standards of practice guidelines for low vision services had a negative impact on coverage. The other implication of this, aside from data dredging, is that it may not be enough to only have guidelines and policies in place. Similarly, signing the Vision 2020 declaration alone may not result in action without a functioning Vision 2020 national committee to implement the strategies. Other components are required to follow through with the policy. For instance, sufficient numbers of human resources and funding are also needed to deliver services. This example also emphasizes that there should always be efforts made to minimize the risks of data dredging in future studies using the CART methodology. 

The application of the CART proved to be advantageous for this type of research, which included explaining complex non-linear relationships and maximized the use of the data available. Furthermore, past studies ^7,19,31 have reported overall predictive performances ranging from 51–77%, making the predictive performance score of 90.1% in this research an excellent value supporting this technique. 

The two major limitations of using this method is that the approach is not well known, and its application to ophthalmology and to eye care delivery is relatively novel. The literature search revealed that only two studies ^32,33 in the area of clinical ophthalmology have been published, but none in the area of delivery of eye care services. Possible reasons for the infrequent use of CART may be because traditionally health service outcomes are often dichotomous, thus traditional regression modeling methods such as logistic regression have been sufficient to determine the average effect of an independent variable on a dependent variable. ³⁴  

Notwithstanding, there can be wide-ranging implications for the application of the CART in the monitoring and evaluation of low vision service delivery under various settings and contexts. For instance, the CART methodology can be applied to identify the critical success factors specific to a country and the information used to develop national low vision frameworks that will guide referral systems and standards of practice. The CART may also be used to identify groups of people most at risk of not accessing low vision services and/or to investigate predictors of barriers that deter people from using services. 

Conclusions

This study identified the most important predictors related to better low vision coverage. These are related to providing comprehensive care, distribution and provision of multidisciplinary services, sufficient human resources, and sustainable funding. 

The CART statistical technique provided a better analysis of survey-related research than the more traditional methods such as logistic regression. The CART is a novel way to explore the relationships between the critical factors with the outcome of low vision coverage. 

Footnotes

 Supported by the Vision Cooperative Research Centre (Vision CRC); CERA receives Operational Infrastructure Support from the Victorian Government, Australia.

Footnotes

 Disclosure: P.P.-C. Chiang, None; J. Xie, None; J.E. Keeffe, None