HEALTH INDICATORS: Conceptual and operational considerations (Section 5)

Back to Main Menu

5.1 METHODS FOR INDICATOR ESTIMATION

An estimate is an approximate value calculated on the basis of the incomplete evidence or available data.

Almost all calculations of health indicators are based on estimates irrespective of whether the data was collected by direct measurements or by indirect measurement techniques; in both cases there is a margin of error. Where a direct measurement is involved, inaccuracies can stem from random errors that are inherent in the sampling processes or from systematic errors due to the procedures used to select the population, collect the data, and its subsequent analysis. In the case of estimates based on indirect measurements techniques that use mathematical or statistical models, inaccuracy can also occur because of inherent errors in the methodology owing to the assumptions and limitations of the model. Assumptions in such models are difficult to assess. This is particularly so in small populations where valid and replicable data and information are limited and, especially in the absence of sufficiently long and reliable times series.

This section uses the term estimate to refer to estimation by indirect methods based on techniques that employ mathematical and statistical models or other demographic techniques to adjust or correct direct data. Thus, estimates of indicators by indirect methods are in contrast to the direct calculation of indicators, which are based exclusively on data and information from primary or secondary sources of information.

5.2 USES AND LIMITATIONS OF ESTIMATING INDICATORS

In most countries, information systems and other sources of health data have improved considerably. Nevertheless, because of data gaps and measurement challenges, there is a need to calculate estimates of health indicators using different mathematical, statistical, and other methodologies. There are several reasons for using estimates for population health indicators. Following are examples of such situations:

• Total absence of information systems and other data sources to calculate core indicators of life events and other essential information for health management purposes.
• Absence of overall population data, or of population counts in the periods between censuses or in years since the most recent census (even in cases where periodic censuses are conducted).
• Gaps in health data, due to significant problems of validity and coverage at certain points in time or in particular geographic areas, as a consequence of limited technical capacity, changing political priorities, or lack of financial sustainability of health information systems, among other possible factors.
• Situations in which there are adequate data and health indicators, but they are derived from studies with probabilistic samples (partial observation of a whole), for which sampling variation needs to be incorporated through a process of estimation (statistical inference).
• The need for indicators that are of interest to international organizations in comparing and monitoring countries, as well as in producing estimates for major world regions, including countries with very different quality and coverage of health information (1).

The methodology that a country uses to make estimates through indirect methods to facilitate crossnational compatibility with the global indicators that are calculated by international organizations, should be viewed with caution. This issue has been widely debated (1, 2).

There is a consensus that direct data should, whenever possible, be assessed and evaluated continuously. Routine use of direct data can create opportunities for improving the data sources. The indiscriminate use of estimated indicators can undermine the authenticity of data and information originating directly from national health information systems. A possible consequence could be the possible reduction in the allocation of resources to improve health information systems, particularly in countries with scarce health resources.

Many indirect estimation methods (for demographic or other data) are subject to inaccuracies, especially in certain situations, such as when national data are rarely available or are incomplete. However, it is precisely in such situations that the calculation of estimates for health indicators becomes necessary. In order to overcome the problem of unavailability of data, imputed data are sometimes used to generate the data required for indirect estimation. The inherent limits of imputation are underestimated. These include lack of representativeness of a country's diversity, possible presence of undetected random error, possible presence of major systematic errors, etc. (2). Such errors can greatly compromise the accuracy of indirect estimates and may not necessarily be an improvement over limitations in the quality of direct data. Another relevant issue is the limited ability of most indirect techniques to accurately capture significant changes in the indicators being calculated. An example is the steep decline in fertility rates in Brazil and the inability of the government's population projection techniques to adequately explain the phenomenon, e.g., through live birth estimates.

Finally, indirect estimation processes have grown increasingly complex in recent years; this brings with it a diminished capacity for communication and replicability. In this context, the Guidelines for Accurate and Transparent Health Estimates Reporting (GATHER) (3) represent a commendable effort to formulate some guidelines.

Partnerships at the local, national, and global levels should be encouraged to strengthen national health information systems and building capacity for the production, analysis, and use of data and health indicators. The efforts of international organizations (WHO and other United Nations entities), scientific institutions, and governments to support the improvement of health information systems and analytical capacity deserve recognition.

The need for valid global, national, and subnational health indicators (regardless of their origin) is of fundamental importance since these indicators shape priorities for health-related investments; facilitate the assessment of progress and effectiveness of interventions; and are necessary for organizing strategic international cooperation. Accordingly, to address the need for credible health indicators, the best available evidence at a given time must be used, even when a degree of inaccuracy is inevitable. (4)

Examples of such situations are:

• Where data quality does not meet minimal standards or where no country-level information is available.
• In order to verify the reliability of the events being studied, as in the case of underreporting of mortality-particularly infant and maternal mortality.
• The need, at the global or regional level, to use standardized information to calculate indicators. Discrepancies in the quality of data and information, and differences between health systems' protocols concerning population representativeness, case definition, and data collection and analysis in different places (countries), and at different points in time, can greatly compromise the ability of indicators to provide comparisons between countries and regions.

The main sources for statistical estimates are: the United Nations Population Division and the United States Census Bureau (for population estimates); the World Bank (for estimates of socioeconomic and maternal mortality indicators); WHO (for mortality figures, mortality tables, and maternal mortality rates); UNICEF, UNFPA and CELADE (for mortality figures and tables); and academic institutions, using a variety of other estimates.

PAHO, as an international organization, uses population estimates provided by the United Nations Population Division rather than from the national censuses from its Member Countries. This approach ensures comparability with data on maternal and child mortality that come from the United Nations Inter-agency Group for Child Mortality Estimation (IGME). This agency was created in 2004 to harmonize estimates within the United Nations system; improve child mortality estimation methods; report on progress toward achieving the Millennium Development Goals and now the Sustainable Development Goals; and strengthen countries' capacity to conduct timely, properly evaluated calculations on infant mortality. The IGME is headed by UNICEF and the World Health Organization (WHO), with participation by the World Bank and the United Nations Population Division (part of the United Nations Department of Economic and Social Affairs).

5.3 ESTIMATION OF MATERNAL AND INFANT MORTALITY INDICATORS

Given the need to establish a baseline for measuring progress toward Millennium Development Goal 5 (MDG-5, and now SDG-3), and the lack of reliable data on global trends in maternal mortality, it was necessary to estimate the number of maternal deaths, as well as the maternal mortality ratio (MMR).

Many countries have made important advances in detecting and recording maternal deaths and live births; they therefore have reliable, though still imperfect, data. At the same time, measuring maternal mortality continues to pose a major challenge. In 2013, according to data reported to PAHO from Member Countries, the absolute number of maternal deaths for Latin America and the Caribbean was around 6,000 per year. The actual number is probably greater, since some countries with relatively high absolute numbers of maternal deaths (Bolivia, Guyana, Haiti, and Trinidad and Tobago) did not report data. However, for the same period, the Maternal Mortality Estimation Inter-agency Group (MMEIG), which includes WHO, UNICEF, UNFPA, and the World Bank, estimated approximately 9,300 maternal deaths, while the Institute for Health Metrics and Evaluation (IHME) estimated 7,600. These three different figures create considerable consternation among the reporting countries. Although there are some similarities among the methodologies used by various groups for estimating trends in maternal mortality, the causes of the major differences merit explanation.

Due to the importance of these indicators, two methodologies that can be used to measure the accuracy of the maternal mortality ratio and of infant mortality rates calculated with country-level sources will be discussed.

5.3.1 METHODOLOGY USED BY THE MATERNAL MORTALITY ESTIMATION INTER-AGENCY GROUP (MMEIG)

The United Nations MMEIG divides countries into three groups, A, B, and C. However, the countries in the Region of the Americas are in groups A and B. Group A is composed of countries with good vital registration data. Using the MMEIG methodology, the number of maternal deaths reported by a country is multiplied by a correction factor of 1.5 to correct for misclassifications except in cases where the country corrects its own information with national data from a published study on the proportion of underreported and poorly classified cases. The 1.5 correction factor stems from two studies by Lewis London: Confidential Enquiry into Maternal and Child Health (2004 and 2007).

The countries in Group B lack complete vital registration data but they use other types of data sources. For these countries, the MMEIG methodology estimates the maternal mortality ratio using a model with three predictive factors as measures of exposure to risk. These factors are:

• Per capita gross domestic product
• Proportion of live births attended by skilled personnel
• Overall fertility rate (live births per woman in the 15 to 49 year age group).

The proportion obtained is used to estimate the total number of deaths of women of childbearing age which is then divided by the total number of births to estimate the maternal mortality ratio. These two data items are drawn from United Nations Statistical Division (UNSD).

5.3.2 THE METHO DOLOGY USED BY THE INSTITUTE FOR HEALTH METRICS AND EVALUATION (IHME)

The model used by the Institute for Health Metrics and Evaluation (IHME) does not take account of variations in the quality of information from the countries, and thus the methodology is applied to all of the countries without distinction. The predictive variables used are:

• Per capita gross domestic product
• Educational level of women, differentiated by age
• Neonatal mortality rate
• Total fertility rate
• Prevalence of HIV/AIDS (this variable represents a difference from the model used by the MMEIG, which does not consider this variable; thus, an estimate of mortality due to this cause is first performed, and the estimate is then corrected).

The IHME corrects problems of under-enumeration and poor quality records by multiplying by a correction factor of 1.4.

5.3.3 ESTIMATES OF INFANT MORTALITY IN THE REGION OF THE AMERICAS

Assessment of achievements with regard to Millennium Development Goal 4 (MDG-4) was based on analysis of under-5 mortality. However, given the differences in mortality risk and in the cause-based structure of mortality during the first years of life, analysis that permits such disaggregation is essential for analyzing the impact of specific interventions and planning future actions.

The available information comes from different sources and methodologies, whose differences need to be assessed when interpreting the data. PAHO consolidates and presents data based on the countries' mortality reports. Annual birth figures are derived from estimates made by the United Nations Population Division and by the United States Census Bureau. Infant mortality and under-5 mortality rates are based on these data sources.

At the global level, estimates are provided by the U.N. Inter-agency Group for Child Mortality Estimation (IGME), as well as by the IHME. The methodological approaches of these two sources differ with regard to the basic data, their processing, and the adjustment processes employed. The most important discrepancies in the results are attributable to changes in mortality in the countries, the corrections or adjustments applied, and the models used to obtain the estimates in response to problems in coverage of vital statistics.

As with other data sources, the usefulness of mortality statistics, as well as the accuracy of the data, depends largely on their quality, which is associated primarily with above the degree of coverage.

The mid-term evaluation of the Regional Plan for Neonatal Health included an analysis of the coverage and accuracy of information on neonatal deaths obtained from vital statistics systems. The evaluation was done using information from the databases available to the PAHO Health Information and Analysis team. The databases included information on neonatal, infant, and child deaths for 47 countries in the Region between 1995 and 2010. These databases generated indicators of neonatal, infant, and child mortality, which were compared with direct estimates based on DHS/RHS and WHO (WHOSIS) surveys, as well as with indirect estimates of the IGME, IHME, and UNICEF (MICS surveys). Similarly, the PAHO databases were used to obtain the distribution of neonatal, infant, and child deaths by causes. In this case, direct estimation was complemented by the compilation of measures produced by the Child Health Epidemiology Reference Group (CHERG).

Based on the analyses, coverage of total deaths was determined to be good in 21 countries, satisfactory in six, and fair to deficient in 12. Thus, the average level of coverage of total deaths is high (median 94%). The consistency of the estimates is generally comparable with the data provided by UNSD for years close to the years analyzed, both with regard to the countries as a whole (median 93.5%), and to the majority of countries individually.

There is an inverse relationship between the percentage of coverage of deaths and the relative difference between the rates obtained by direct and indirect methodologies (the higher the former, the lower the latter). With regard to infant and child mortality, the correlations between the percentage of coverage and the relative difference between direct and indirectly estimated rates was greater than the indirect estimates calculated by the IHME. In the case of neonatal mortality, the association was higher in terms of the relationship between percentage of coverage and the relative difference between the direct rates and those calculated by the IGME.

It is also evident that the causes cited on declarations of death in the Region's countries are accurate to an acceptable degree, with the frequencies of ill-defined causes being less than 10%. Thus, although the quality of mortality information in the Region needs to be improved, levels of both coverage and precision are generally adequate. The better such measurements are, the greater will be the benefit of information based on direct, versus indirect, sources.

REFERENCES

1. Boerma T, Mathers CD. The World Health Organization and global health estimates: improving collaboration and capacity. BioMed Central - Medicine. 2015. DOI 10.1186/s12916-015-0286-7 https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-015-0286-7

2. Frias PG, Szwarcwald CL, Pis L. Estimação da mortalidade infantil no contexto de descentralização do Sistema Único de Saúde (SUS). Rev. Bras. Saúde Matern. Infant. 2011;11(4): 463-70.

3. Stevens, GA, et al. Guidelines for Accurate and Transparent Health Estimates Reporting (GATHER) PLoS Medicine, June 28, 2016 2016. Available at: http://journals.plos.org/plosmedicine/article/file?id=10.1371/journal.pmed.1002056&-type=printable [consulted 9 August 2017].

4. Unite for Sight. Global Health Estimates. Available at: http://www.uniteforsight.org/public-health-management/global-health-estimates [consulted 5 September 2016].

Back to Main Menu