5.2 USES AND LIMITATIONS OF ESTIMATING INDICATORS
In most countries, information systems and other sources of health data have improved considerably. Nevertheless, because of data gaps and measurement challenges, there is a need to calculate estimates of health indicators using different mathematical, statistical, and other methodologies. There are several reasons for using estimates for population health indicators. Following are examples of such situations:
- Total absence of information systems and other data sources to calculate core indicators of life events and other essential information for health management purposes.
- Absence of overall population data, or of population counts in the periods between censuses or in years since the most recent census (even in cases where periodic censuses are conducted).
- Gaps in health data, due to significant problems of validity and coverage at certain points in time or in particular geographic areas, as a consequence of limited technical capacity, changing political priorities, or lack of financial sustainability of health information systems, among other possible factors.
- Situations in which there are adequate data and health indicators, but they are derived from studies with probabilistic samples (partial observation of a whole), for which sampling variation needs to be incorporated through a process of estimation (statistical inference).
- The need for indicators that are of interest to international organizations in comparing and monitoring countries, as well as in producing estimates for major world regions, including countries with very different quality and coverage of health information (1).
The methodology that a country uses to make estimates through indirect methods to facilitate crossnational compatibility with the global indicators that are calculated by international organizations, should be viewed with caution. This issue has been widely debated (1, 2).
There is a consensus that direct data should, whenever possible, be assessed and evaluated continuously. Routine use of direct data can create opportunities for improving the data sources. The indiscriminate use of estimated indicators can undermine the authenticity of data and information originating directly from national health information systems. A possible consequence could be the possible reduction in the allocation of resources to improve health information systems, particularly in countries with scarce health resources.
Many indirect estimation methods (for demographic or other data) are subject to inaccuracies, especially in certain situations, such as when national data are rarely available or are incomplete. However, it is precisely in such situations that the calculation of estimates for health indicators becomes necessary. In order to overcome the problem of unavailability of data, imputed data are sometimes used to generate the data required for indirect estimation. The inherent limits of imputation are underestimated. These include lack of representativeness of a country's diversity, possible presence of undetected random error, possible presence of major systematic errors, etc. (2). Such errors can greatly compromise the accuracy of indirect estimates and may not necessarily be an improvement over limitations in the quality of direct data. Another relevant issue is the limited ability of most indirect techniques to accurately capture significant changes in the indicators being calculated. An example is the steep decline in fertility rates in Brazil and the inability of the government's population projection techniques to adequately explain the phenomenon, e.g., through live birth estimates.
Finally, indirect estimation processes have grown increasingly complex in recent years; this brings with it a diminished capacity for communication and replicability. In this context, the Guidelines for Accurate and Transparent Health Estimates Reporting (GATHER) (3) represent a commendable effort to formulate some guidelines.
Partnerships at the local, national, and global levels should be encouraged to strengthen national health information systems and building capacity for the production, analysis, and use of data and health indicators. The efforts of international organizations (WHO and other United Nations entities), scientific institutions, and governments to support the improvement of health information systems and analytical capacity deserve recognition.
The need for valid global, national, and subnational health indicators (regardless of their origin) is of fundamental importance since these indicators shape priorities for health-related investments; facilitate the assessment of progress and effectiveness of interventions; and are necessary for organizing strategic international cooperation. Accordingly, to address the need for credible health indicators, the best available evidence at a given time must be used, even when a degree of inaccuracy is inevitable. (4)
Examples of such situations are:
- Where data quality does not meet minimal standards or where no country-level information is available.
- In order to verify the reliability of the events being studied, as in the case of underreporting of mortality-particularly infant and maternal mortality.
- The need, at the global or regional level, to use standardized information to calculate indicators. Discrepancies in the quality of data and information, and differences between health systems' protocols concerning population representativeness, case definition, and data collection and analysis in different places (countries), and at different points in time, can greatly compromise the ability of indicators to provide comparisons between countries and regions.
The main sources for statistical estimates are: the United Nations Population Division and the United States Census Bureau (for population estimates); the World Bank (for estimates of socioeconomic and maternal mortality indicators); WHO (for mortality figures, mortality tables, and maternal mortality rates); UNICEF, UNFPA and CELADE (for mortality figures and tables); and academic institutions, using a variety of other estimates.
PAHO, as an international organization, uses population estimates provided by the United Nations Population Division rather than from the national censuses from its Member Countries. This approach ensures comparability with data on maternal and child mortality that come from the United Nations Inter-agency Group for Child Mortality Estimation (IGME). This agency was created in 2004 to harmonize estimates within the United Nations system; improve child mortality estimation methods; report on progress toward achieving the Millennium Development Goals and now the Sustainable Development Goals; and strengthen countries' capacity to conduct timely, properly evaluated calculations on infant mortality. The IGME is headed by UNICEF and the World Health Organization (WHO), with participation by the World Bank and the United Nations Population Division (part of the United Nations Department of Economic and Social Affairs).