Article Index



Criteria, process, stakeholder involvement, and practical strategies to evaluate the quality of health indicators.


To discuss strategies for evaluating the quality of health indicators.


  • How a health indicator is developed and validated
  • Principal criteria for the quality of health indicators
  • Steps involved in evaluating a health indicator
  • The role of stakeholders in evaluating indicators

Back to Main Menu


Section 1 detailed the attributes of a good indicator, emphasizing that indicators should be measureable, feasible, valid, timely, replicable, sustainable, relevant, and comprehensive. In addition, indicators should be stratified by person, place, and time as desirable. This section describes some practical strategies for evaluating the quality of health indicators. It should be emphasized that sources of high quality data contribute to the development of high quality indicators. In addition to the quality of the sources and the data, the performance of the indicator should measure what it is intended to measure.


In order to evaluate an indicator, its purpose and attributes must be well-defined. This involves more than simply knowing and adequately describing its numerator and denominator (assuming it is structured in that way). Some publications (RIPSA, UNAIDS, and the WHO Global Reference List of 100 Core Indicators) are useful references for this discussion. (1-3).

  1. The name and definition of the indicator
  2. The purpose and rationale for the indicator (identification of what is to be measured)
  3. The method of measurement or calculation of the indicator
  4. The data sources (institutions and information systems involved) for the indicator
  5. The data collection methodology
  6. The frequency of data collection
  7. The levels of data disaggregation required
  8. The guidelines for the indicator's interpretation and use
  9. The indicator's strengths, limitations, and challenges
  10. Additional sources of information
  11. Explanatory remarks


Model of a technical file for defining and characterizing health indicators from the PAHO Strategic Plan (Annex) and selected indicators from the Global Reference List of 100 Core Indicators (Excel spreadsheet).


The evaluation of health indicators should, insofar as possible, include the key stakeholders involved in the production, analysis, and interpretation of data and information. These persons should be familiar with the processes involved in monitoring local, regional, and national trends and conditions. Notably, most health data and information are generated at the local level by local health workers who are more knowledgeable about the characteristics, strengths, and limitations of the data and the derived information. Accordingly, whenever possible, it is desirable for local-level personnel to participate in the first phase of the evaluation process. Data producers, managers, and users should promote a culture that values information and is conducive to data collection and its management. Ongoing training initiatives on data collection, management, evaluation, and analysis are very important to improve national capacities, especially at the local level.

As mentioned earlier, the quality of indicators depends, to a large extent, on the quality of the data and its sources. All major stakeholders, including data producers and managers of information systems, should be encouraged to play the role of users and critics so that they can be knowledgeable of the strengths and weaknesses of the system. Health information systems that cannot provide the underpinnings for decision-making in health contribute to a waste of scarce resources and the paucity of reliable information in the health sector. An efficient health information system generates products that are of increasing value to improvements in health care. The continuous need for quality health information is a strong motivator for strengthening and using national health information systems as well as for providing feedback about any limitation inherent in these systems.

Since the health sector is influenced by a wide range of factors, many of which fall outside the health care delivery sector, collaboration with the non-health sectors, such as in other government entities, universities, and research centers is important. Some of the core interests of the non-health sectors require defining, developing, analyzing, and using health indicators. As such, inter-sectoral collaboration will improve and optimize the quality and relevance of health indicators as well as promote evidence-informed decisions across all sectors.


Some authors have proposed guidelines for evaluating health data and indicators (3-4). However, there are some fundamental considerations that can be applied when evaluating health indicators and these are outlined in the following steps:

Step 1. Examine the integrity of the complete and valid data on which the indicator is based

  1. Is the indicator based on data representative of the target population? Examine in detail the population the data is supposed to describe. Avoid undue generalizations (extrapolations). Be alert to possible selection bias due to nonresponse, demand and indication biases; ascertain whether some facilities generate more reports than others (e.g., public versus private facilities).
  2. Are the variables used to calculate the indicator complete, adequate, and sufficient? Calculate and tabulate the characteristics of the variables used to develop the indicator. Include proportions of nonresponse (if possible), invalid responses, and other losses. Identify problems in coverage of the relevant variables, taking into account low representativeness; possible selection bias in an indicator; and calculations based on non-representative data.
  3. Is the indicator based on valid data from the target population? Were the variables used to calculate the indicator measured correctly and was a minimum standard applied? Analyze in detail how the attributes of the variables that produced the indicator are defined, calculated, and compiled. This includes a review of case definitions, the competency of the personnel responsible for data collection, and quality of the instruments (diagnostic tests, measuring equipment, etc.) used to collect the data. Identify problems of validity in the relevant variables, and account for measurement bias in indicators based on problematic calculations.

Steps 2 to 5 are designed to evaluate the indicator's observed and expected values in different situations, according to the characteristics of person, place, and time. This evaluation will answer the following three questions:

  1. Could the discrepancies that are discovered be the result of random fluctuations of small numbers? An insufficient number of observations makes it impossible to precisely estimate an indicator.
  2. Could the observed discrepancies be the result of biases (systematic errors in indicator measurements) that compromise the quality of the indicator?
  3. Could the observed discrepancies be valid? Discrepancies between expected and observed values should be examined carefully so as not to overlook actual variabilities attributable to local changes.

Step 2. Examine the consistency of the estimated indicator with regard to personal attributes

Is the indicator consistent based on personal characteristics? Analyze consistency, considering the personal variables (sex, age, etc.) of the data source, category by category, as relevant to the indicator in question. Observe the values of the indicator according to those variables, and analyze whether they are plausible. Are the results consistent with expectations for the given population subgroups?

For example, if the indicator is mortality from cardiovascular disease, the observed distribution of the indicator by sex and the age should, at a minimum, reflect the greater risk associated with certain groups (e.g., older men). Confirmation that the higher levels of these indicators are consistent with groups expected to be at greater risk for the disease bolsters confidence in the quality of the indicator.

Step 3. Examine the consistency of the estimated indicator with regard to place attributes

Is the indicator spatially consistent? If possible, analyze its spatial distribution (by municipality, state, urban versus rural residence, etc.). Most indicators have an expected spatial pattern that reflects the known distribution of important risk factors (e.g., poverty, young versus older populations, more or less urbanized areas, etc.). Examine the consistency of the indicator's pattern with regard to expectations, and identify signs suggesting that the quality is unreliable.

Table 4 presents average values of selected indicators for sub-regions of the Americas. These values can be used as a reference to assess the consistency of indicators in the countries. At the end of this section, there is a link to PAHO's list of core indicators and their trends.

Table 4. Selected indicators of the Region of the Americas and its subregions
Life expectancy at birth (years) 2016 77.0 79.7 75.5 74.4 73.4 74.4 77.8 73.8
Maternal mortality rate reported/100 000 lb LAY* 46.8 12.1 60.8 80.2 104.4 77.3 35.2 88.8
Maternal mortality rate estimated/100 000 lb 2015 51 13 66 87 188 87 54 105
Infant mortality rate reported/100 000 lb LAY* 13.0 5.9 15.9 17.5 32.8 18.7 10.3 17.2
Neonatal mortality rate reported/100 000 lb LAY* 8.2 4.0 10.0 9.8 19.1 12.0 7.1 15.8
Under-five mortality rate reported/100 000 lb LAY* 15.9 6.9 19.6 22.2 48.9 22.9 11.9 18.9
General mortality rate/1 000 pop 2014 5.6 4.8 6.0 6.7 5.8 6.3 5.5 7.2
Mortality rate due to external causes/100 000 pop 2014 63.5 53.3 68.8 81.1 61.4 87.7 46.6 70.3
HIV incidence rate/100 000 pop 2015 12.9 13.1 12.3 12.4 18.0 18.2 15.3 56.8
Tuberculosis incidencie rate/100 000 pop 2014 22.1 3.0 33.2 28.4 58.8 45.3 20.8 15.9
Stunting in children aged < 5 years (%) 2012 10.1 2.1 13.2 30.2 13.1 16.5 7.1 6.9
Overweight in children aged < 5 years (%) 2012 7.2 6.0 7.7 5.4 7.7 6.6 10.0 6.6
Overweight in adults 2014 61.0 67.0 57.6 54.0 51.2 57.6 60.8 59.4

* LAY = Latest available year
Source: PAHO. Core indicators 2016. Health Situation in the Americas.

Step 4. Examine the consistency of the estimated indicator with regard to temporal attributes

Is the indicator consistent over time? If possible, analyze its trends over time (years, months, weeks, etc.). Some indicators have a known seasonal cyclic pattern of variation, or they indicate historical trends that can serve as a reference for analysis. Moreover, most indicators show gradual fluctuations in temporal trends, such as slightly increasing or decreasing but no large increases, except in special situations. Major temporal fluctuations can indicate:

  1. True fluctuations due to epidemics (dramatic events that alter an indicator's course). An example would be the unusual increase of microcephaly cases associated with the Zika virus epidemic in cities of northeastern Brazil.
  2. Random fluctuations due to the small number of cases occurring in places with small populations (denominator) or due to a small number of events (as in the case of infrequent diseases). In these situations, the addition or subtraction of a few cases (the numerator) can produce a large increase or reduction in the resulting rate. Consequently, greater attention should be given to the absolute number of cases than to the rates since, in such situations, rates can lead to false interpretations. Situations of this type arise frequently but are easy to detect. All that is needed is an examination of the ratio of change in the rate in relation to the size of the reference population. To avoid this statistical phenomenon, the data over longer periods (e.g., three-year periods) can be combined; geographic areas (such as municipalities or similar entities close to each other) can also be combined. These adjustments make the indicator stable enough to be meaningful.
  3. Fluctuations due to (non-random) error: Systematic errors in the measurement of the denominator and/or numerator at a given point in time can generate large variations in an indicator. Common examples of this phenomenon are changes in the definition of cases with the introduction of new diagnostic techniques in surveillance systems; under- or over-counting of cases beginning at a particular point in time; and problems with the methods used to estimate the population size between censuses (denominators). As indicated above, communication and partnership with the producers of the data on which the indicator was based can be helpful to provide clarification and/or retrospective correction of the phenomenon being observed.

Step 5. Examine the plausibility of the magnitude of the indicator in relation to other data sources

Compare the indicator's magnitude with existing information and evidence from other data sources. Is the result of the measurement of the indicator plausible considering what is already known about the subject? Is it plausible considering estimates made by other methods (indirect methods, research, or other data sources)? Is it plausible considering the current conditions of the population for which it was estimated? Is it plausible considering the risk factors present in the population? Lastly, is it plausible considering the values of the same indicator as estimated for other countries, states, or municipalities with better or worse conditions?

For example, findings of low maternal mortality ratios in countries where women's health care during pregnancy, childbirth, and puerperium is precarious and where the quality of national surveillance systems is not high, should cast doubt on the quality of the indicator. Comparing it with countries that have better health care can help clarify the perceived disparity.


AbouZahr et al. (4), proposed the following ten steps for evaluating mortality data:

  1. Prepare basic tabulations of deaths by age, sex, ethnic origin, and cause of death
  2. Review crude death rates
  3. Review age- and sex-specific mortality
  4. Review the age distribution of deaths
  5. Review infant mortality rates
  6. Review the distribution of the leading causes of death
  7. Review the age pattern of the leading causes of death
  8. Review the leading causes of death
  9. Review the ratio of deaths from non-communicable diseases to deaths from communicable diseases
  10. Review deaths for ill-defined causes



1. Inter-agency Health Information Network (RIPSA). Indicadores y datos básicos para la salud en Brasil. Available at:

2. Joint United Nations Programme on HIV/AIDS (UNAIDS). Indicator Standards: operational guidelines for selecting indicators for the HIV response. Available at:

3. World Health Organization (WHO). Global Reference List of 100 Core Health Indicators, 2014. Available at: [consulted March 2017).

4. AbouZahr, C. et al. Mortality statistics: a tool to enhance understanding and improve quality. Pacific Health Dialog. 2012;18(1):247-70. Available at: [consulted March

Back to Main Menu