HEALTH INDICATORS: Conceptual and operational considerations (Section 4)

Back to Main Menu

4.1 INTRODUCTION

Section 1 detailed the attributes of a good indicator, emphasizing that indicators should be measureable, feasible, valid, timely, replicable, sustainable, relevant, and comprehensive. In addition, indicators should be stratified by person, place, and time as desirable. This section describes some practical strategies for evaluating the quality of health indicators. It should be emphasized that sources of high quality data contribute to the development of high quality indicators. In addition to the quality of the sources and the data, the performance of the indicator should measure what it is intended to measure.

4.2 DEFINING AND CHARACTERIZING THE HEALTH INDICATOR BEING EVALUATED

In order to evaluate an indicator, its purpose and attributes must be well-defined. This involves more than simply knowing and adequately describing its numerator and denominator (assuming it is structured in that way). Some publications (RIPSA, UNAIDS, and the WHO Global Reference List of 100 Core Indicators) are useful references for this discussion. (1-3).

Examples:

Model of a technical file for defining and characterizing health indicators from the PAHO Strategic Plan (Annex) and selected indicators from the Global Reference List of 100 Core Indicators (Excel spreadsheet).

4.3 KEY STAKEHOLDERS IN THE PROCESS OF EVALUATING HEALTH INDICATORS

The evaluation of health indicators should, insofar as possible, include the key stakeholders involved in the production, analysis, and interpretation of data and information. These persons should be familiar with the processes involved in monitoring local, regional, and national trends and conditions. Notably, most health data and information are generated at the local level by local health workers who are more knowledgeable about the characteristics, strengths, and limitations of the data and the derived information. Accordingly, whenever possible, it is desirable for local-level personnel to participate in the first phase of the evaluation process. Data producers, managers, and users should promote a culture that values information and is conducive to data collection and its management. Ongoing training initiatives on data collection, management, evaluation, and analysis are very important to improve national capacities, especially at the local level.

As mentioned earlier, the quality of indicators depends, to a large extent, on the quality of the data and its sources. All major stakeholders, including data producers and managers of information systems, should be encouraged to play the role of users and critics so that they can be knowledgeable of the strengths and weaknesses of the system. Health information systems that cannot provide the underpinnings for decision-making in health contribute to a waste of scarce resources and the paucity of reliable information in the health sector. An efficient health information system generates products that are of increasing value to improvements in health care. The continuous need for quality health information is a strong motivator for strengthening and using national health information systems as well as for providing feedback about any limitation inherent in these systems.

Since the health sector is influenced by a wide range of factors, many of which fall outside the health care delivery sector, collaboration with the non-health sectors, such as in other government entities, universities, and research centers is important. Some of the core interests of the non-health sectors require defining, developing, analyzing, and using health indicators. As such, inter-sectoral collaboration will improve and optimize the quality and relevance of health indicators as well as promote evidence-informed decisions across all sectors.

4.4 STEPS FOR EVALUATING THE QUALITY OF HEALTH INDICATORS

Some authors have proposed guidelines for evaluating health data and indicators (3-4). However, there are some fundamental considerations that can be applied when evaluating health indicators and these are outlined in the following steps:

Step 1. Examine the integrity of the complete and valid data on which the indicator is based

1. Is the indicator based on data representative of the target population? Examine in detail the population the data is supposed to describe. Avoid undue generalizations (extrapolations). Be alert to possible selection bias due to nonresponse, demand and indication biases; ascertain whether some facilities generate more reports than others (e.g., public versus private facilities).
2. Are the variables used to calculate the indicator complete, adequate, and sufficient? Calculate and tabulate the characteristics of the variables used to develop the indicator. Include proportions of nonresponse (if possible), invalid responses, and other losses. Identify problems in coverage of the relevant variables, taking into account low representativeness; possible selection bias in an indicator; and calculations based on non-representative data.
3. Is the indicator based on valid data from the target population? Were the variables used to calculate the indicator measured correctly and was a minimum standard applied? Analyze in detail how the attributes of the variables that produced the indicator are defined, calculated, and compiled. This includes a review of case definitions, the competency of the personnel responsible for data collection, and quality of the instruments (diagnostic tests, measuring equipment, etc.) used to collect the data. Identify problems of validity in the relevant variables, and account for measurement bias in indicators based on problematic calculations.

Steps 2 to 5 are designed to evaluate the indicator's observed and expected values in different situations, according to the characteristics of person, place, and time. This evaluation will answer the following three questions:

1. Could the discrepancies that are discovered be the result of random fluctuations of small numbers? An insufficient number of observations makes it impossible to precisely estimate an indicator.
2. Could the observed discrepancies be the result of biases (systematic errors in indicator measurements) that compromise the quality of the indicator?
3. Could the observed discrepancies be valid? Discrepancies between expected and observed values should be examined carefully so as not to overlook actual variabilities attributable to local changes.

Step 2. Examine the consistency of the estimated indicator with regard to personal attributes

Is the indicator consistent based on personal characteristics? Analyze consistency, considering the personal variables (sex, age, etc.) of the data source, category by category, as relevant to the indicator in question. Observe the values of the indicator according to those variables, and analyze whether they are plausible. Are the results consistent with expectations for the given population subgroups?

For example, if the indicator is mortality from cardiovascular disease, the observed distribution of the indicator by sex and the age should, at a minimum, reflect the greater risk associated with certain groups (e.g., older men). Confirmation that the higher levels of these indicators are consistent with groups expected to be at greater risk for the disease bolsters confidence in the quality of the indicator.

Step 3. Examine the consistency of the estimated indicator with regard to place attributes

Is the indicator spatially consistent? If possible, analyze its spatial distribution (by municipality, state, urban versus rural residence, etc.). Most indicators have an expected spatial pattern that reflects the known distribution of important risk factors (e.g., poverty, young versus older populations, more or less urbanized areas, etc.). Examine the consistency of the indicator's pattern with regard to expectations, and identify signs suggesting that the quality is unreliable.

Table 4 presents average values of selected indicators for sub-regions of the Americas. These values can be used as a reference to assess the consistency of indicators in the countries. At the end of this section, there is a link to PAHO's list of core indicators and their trends.

* LAY = Latest available year
Source: PAHO. Core indicators 2016. Health Situation in the Americas.

Step 4. Examine the consistency of the estimated indicator with regard to temporal attributes

Is the indicator consistent over time? If possible, analyze its trends over time (years, months, weeks, etc.). Some indicators have a known seasonal cyclic pattern of variation, or they indicate historical trends that can serve as a reference for analysis. Moreover, most indicators show gradual fluctuations in temporal trends, such as slightly increasing or decreasing but no large increases, except in special situations. Major temporal fluctuations can indicate:

1. True fluctuations due to epidemics (dramatic events that alter an indicator's course). An example would be the unusual increase of microcephaly cases associated with the Zika virus epidemic in cities of northeastern Brazil.
2. Random fluctuations due to the small number of cases occurring in places with small populations (denominator) or due to a small number of events (as in the case of infrequent diseases). In these situations, the addition or subtraction of a few cases (the numerator) can produce a large increase or reduction in the resulting rate. Consequently, greater attention should be given to the absolute number of cases than to the rates since, in such situations, rates can lead to false interpretations. Situations of this type arise frequently but are easy to detect. All that is needed is an examination of the ratio of change in the rate in relation to the size of the reference population. To avoid this statistical phenomenon, the data over longer periods (e.g., three-year periods) can be combined; geographic areas (such as municipalities or similar entities close to each other) can also be combined. These adjustments make the indicator stable enough to be meaningful.
3. Fluctuations due to (non-random) error: Systematic errors in the measurement of the denominator and/or numerator at a given point in time can generate large variations in an indicator. Common examples of this phenomenon are changes in the definition of cases with the introduction of new diagnostic techniques in surveillance systems; under- or over-counting of cases beginning at a particular point in time; and problems with the methods used to estimate the population size between censuses (denominators). As indicated above, communication and partnership with the producers of the data on which the indicator was based can be helpful to provide clarification and/or retrospective correction of the phenomenon being observed.

Step 5. Examine the plausibility of the magnitude of the indicator in relation to other data sources

Compare the indicator's magnitude with existing information and evidence from other data sources. Is the result of the measurement of the indicator plausible considering what is already known about the subject? Is it plausible considering estimates made by other methods (indirect methods, research, or other data sources)? Is it plausible considering the current conditions of the population for which it was estimated? Is it plausible considering the risk factors present in the population? Lastly, is it plausible considering the values of the same indicator as estimated for other countries, states, or municipalities with better or worse conditions?

For example, findings of low maternal mortality ratios in countries where women's health care during pregnancy, childbirth, and puerperium is precarious and where the quality of national surveillance systems is not high, should cast doubt on the quality of the indicator. Comparing it with countries that have better health care can help clarify the perceived disparity.

4.5 EVALUATION OF MORTALITY DATA

AbouZahr et al. (4), proposed the following ten steps for evaluating mortality data:

1. Prepare basic tabulations of deaths by age, sex, ethnic origin, and cause of death
2. Review crude death rates
3. Review age- and sex-specific mortality
4. Review the age distribution of deaths
5. Review infant mortality rates
6. Review the distribution of the leading causes of death
7. Review the age pattern of the leading causes of death
8. Review the leading causes of death
9. Review the ratio of deaths from non-communicable diseases to deaths from communicable diseases
10. Review deaths for ill-defined causes

RELATED LINKS

• Pan American Health Organization. Core Indicators 2016. Health Situation in the Americas
• Pan American Health Organization. Health Information Platform for the Americas (time series of core indicators, other indicators, and health information)

REFERENCES

1. Inter-agency Health Information Network (RIPSA). Indicadores y datos básicos para la salud en Brasil. Available at: http://www.ripsa.org.br/vhl/indicadores-e-dados-basicos-para-a-saude-no-brasil-idb/ficha-de-qualificacao-do-indicador/

2. Joint United Nations Programme on HIV/AIDS (UNAIDS). Indicator Standards: operational guidelines for selecting indicators for the HIV response. Available at: http://www.unaids.org/sites/default/files/sub_landing/files/4_3_MERG_Indicator_Standards.pdf

3. World Health Organization (WHO). Global Reference List of 100 Core Health Indicators, 2014. Available at: https://www.who.int/healthinfo/indicators/2015/en/ [consulted March 2017).

4. AbouZahr, C. et al. Mortality statistics: a tool to enhance understanding and improve quality. Pacific Health Dialog. 2012;18(1):247-70. Available at: http://www.getinthepicture.org/sites/default/files/resources/Mortality%20statistics%20a%20tool%20to%20improve%20understanding%20and%20quality.pdf [consulted March

Back to Main Menu