Article Index

4.4 STEPS FOR EVALUATING THE QUALITY OF HEALTH INDICATORS

Some authors have proposed guidelines for evaluating health data and indicators (3-4). However, there are some fundamental considerations that can be applied when evaluating health indicators and these are outlined in the following steps:

Step 1. Examine the integrity of the complete and valid data on which the indicator is based

  1. Is the indicator based on data representative of the target population? Examine in detail the population the data is supposed to describe. Avoid undue generalizations (extrapolations). Be alert to possible selection bias due to nonresponse, demand and indication biases; ascertain whether some facilities generate more reports than others (e.g., public versus private facilities).
  2. Are the variables used to calculate the indicator complete, adequate, and sufficient? Calculate and tabulate the characteristics of the variables used to develop the indicator. Include proportions of nonresponse (if possible), invalid responses, and other losses. Identify problems in coverage of the relevant variables, taking into account low representativeness; possible selection bias in an indicator; and calculations based on non-representative data.
  3. Is the indicator based on valid data from the target population? Were the variables used to calculate the indicator measured correctly and was a minimum standard applied? Analyze in detail how the attributes of the variables that produced the indicator are defined, calculated, and compiled. This includes a review of case definitions, the competency of the personnel responsible for data collection, and quality of the instruments (diagnostic tests, measuring equipment, etc.) used to collect the data. Identify problems of validity in the relevant variables, and account for measurement bias in indicators based on problematic calculations.

Steps 2 to 5 are designed to evaluate the indicator's observed and expected values in different situations, according to the characteristics of person, place, and time. This evaluation will answer the following three questions:

  1. Could the discrepancies that are discovered be the result of random fluctuations of small numbers? An insufficient number of observations makes it impossible to precisely estimate an indicator.
  2. Could the observed discrepancies be the result of biases (systematic errors in indicator measurements) that compromise the quality of the indicator?
  3. Could the observed discrepancies be valid? Discrepancies between expected and observed values should be examined carefully so as not to overlook actual variabilities attributable to local changes.

Step 2. Examine the consistency of the estimated indicator with regard to personal attributes

Is the indicator consistent based on personal characteristics? Analyze consistency, considering the personal variables (sex, age, etc.) of the data source, category by category, as relevant to the indicator in question. Observe the values of the indicator according to those variables, and analyze whether they are plausible. Are the results consistent with expectations for the given population subgroups?

For example, if the indicator is mortality from cardiovascular disease, the observed distribution of the indicator by sex and the age should, at a minimum, reflect the greater risk associated with certain groups (e.g., older men). Confirmation that the higher levels of these indicators are consistent with groups expected to be at greater risk for the disease bolsters confidence in the quality of the indicator.

Step 3. Examine the consistency of the estimated indicator with regard to place attributes

Is the indicator spatially consistent? If possible, analyze its spatial distribution (by municipality, state, urban versus rural residence, etc.). Most indicators have an expected spatial pattern that reflects the known distribution of important risk factors (e.g., poverty, young versus older populations, more or less urbanized areas, etc.). Examine the consistency of the indicator's pattern with regard to expectations, and identify signs suggesting that the quality is unreliable.

Table 4 presents average values of selected indicators for sub-regions of the Americas. These values can be used as a reference to assess the consistency of indicators in the countries. At the end of this section, there is a link to PAHO's list of core indicators and their trends.

Table 4. Selected indicators of the Region of the Americas and its subregions
INDICATORS YEAR THE AMERICAS NORTH AMERICA LATIN AMERICA CENTRAL AMERICAN ISTHMUS LATIN CARIBBEAN ANDEAN AREA SOUTHERN CONE NON-LATIN CARIBBEAN
Life expectancy at birth (years) 2016 77.0 79.7 75.5 74.4 73.4 74.4 77.8 73.8
Maternal mortality rate reported/100 000 lb LAY* 46.8 12.1 60.8 80.2 104.4 77.3 35.2 88.8
Maternal mortality rate estimated/100 000 lb 2015 51 13 66 87 188 87 54 105
Infant mortality rate reported/100 000 lb LAY* 13.0 5.9 15.9 17.5 32.8 18.7 10.3 17.2
Neonatal mortality rate reported/100 000 lb LAY* 8.2 4.0 10.0 9.8 19.1 12.0 7.1 15.8
Under-five mortality rate reported/100 000 lb LAY* 15.9 6.9 19.6 22.2 48.9 22.9 11.9 18.9
General mortality rate/1 000 pop 2014 5.6 4.8 6.0 6.7 5.8 6.3 5.5 7.2
Mortality rate due to external causes/100 000 pop 2014 63.5 53.3 68.8 81.1 61.4 87.7 46.6 70.3
HIV incidence rate/100 000 pop 2015 12.9 13.1 12.3 12.4 18.0 18.2 15.3 56.8
Tuberculosis incidencie rate/100 000 pop 2014 22.1 3.0 33.2 28.4 58.8 45.3 20.8 15.9
Stunting in children aged < 5 years (%) 2012 10.1 2.1 13.2 30.2 13.1 16.5 7.1 6.9
Overweight in children aged < 5 years (%) 2012 7.2 6.0 7.7 5.4 7.7 6.6 10.0 6.6
Overweight in adults 2014 61.0 67.0 57.6 54.0 51.2 57.6 60.8 59.4

* LAY = Latest available year
Source: PAHO. Core indicators 2016. Health Situation in the Americas.

Step 4. Examine the consistency of the estimated indicator with regard to temporal attributes

Is the indicator consistent over time? If possible, analyze its trends over time (years, months, weeks, etc.). Some indicators have a known seasonal cyclic pattern of variation, or they indicate historical trends that can serve as a reference for analysis. Moreover, most indicators show gradual fluctuations in temporal trends, such as slightly increasing or decreasing but no large increases, except in special situations. Major temporal fluctuations can indicate:

  1. True fluctuations due to epidemics (dramatic events that alter an indicator's course). An example would be the unusual increase of microcephaly cases associated with the Zika virus epidemic in cities of northeastern Brazil.
  2. Random fluctuations due to the small number of cases occurring in places with small populations (denominator) or due to a small number of events (as in the case of infrequent diseases). In these situations, the addition or subtraction of a few cases (the numerator) can produce a large increase or reduction in the resulting rate. Consequently, greater attention should be given to the absolute number of cases than to the rates since, in such situations, rates can lead to false interpretations. Situations of this type arise frequently but are easy to detect. All that is needed is an examination of the ratio of change in the rate in relation to the size of the reference population. To avoid this statistical phenomenon, the data over longer periods (e.g., three-year periods) can be combined; geographic areas (such as municipalities or similar entities close to each other) can also be combined. These adjustments make the indicator stable enough to be meaningful.
  3. Fluctuations due to (non-random) error: Systematic errors in the measurement of the denominator and/or numerator at a given point in time can generate large variations in an indicator. Common examples of this phenomenon are changes in the definition of cases with the introduction of new diagnostic techniques in surveillance systems; under- or over-counting of cases beginning at a particular point in time; and problems with the methods used to estimate the population size between censuses (denominators). As indicated above, communication and partnership with the producers of the data on which the indicator was based can be helpful to provide clarification and/or retrospective correction of the phenomenon being observed.

Step 5. Examine the plausibility of the magnitude of the indicator in relation to other data sources

Compare the indicator's magnitude with existing information and evidence from other data sources. Is the result of the measurement of the indicator plausible considering what is already known about the subject? Is it plausible considering estimates made by other methods (indirect methods, research, or other data sources)? Is it plausible considering the current conditions of the population for which it was estimated? Is it plausible considering the risk factors present in the population? Lastly, is it plausible considering the values of the same indicator as estimated for other countries, states, or municipalities with better or worse conditions?

For example, findings of low maternal mortality ratios in countries where women's health care during pregnancy, childbirth, and puerperium is precarious and where the quality of national surveillance systems is not high, should cast doubt on the quality of the indicator. Comparing it with countries that have better health care can help clarify the perceived disparity.