HEALTH INDICATORS: Conceptual and operational considerations (Section 3)

When a decision is made to use indicators to monitor a given aspect of a population's health, there are two clear options:

1. To identify existing data sources which, though not formulated for the purpose, can facilitate the development of reliable indicator(s).
2. To devise specific data collection mechanisms for developing the indicator or set of indicators when there are no adequate data sources.

These two options define what is called secondary data from secondary sources and primary data from primary sources. The choice of whether to create new data sources for a given objective involving monitoring of a population's health or to use available data sources should be based on the advantages and constraints associated with these two options. It is therefore important to weigh the quality of the existing data sources against the effort required to create and maintain new data sources.

Back to Main Menu

3.1 DATA SOURCES

3.1.1 PRIMARY DATA SOURCES

The objective of most countries' disease reporting systems for tuberculosis is to support surveillance and control of this disease. The indicators generated from that information system are examples of indicators produced using primary data. Similarly, a survey of schoolchildren designed to estimate prevalence rates of behavioral risks in a sample population is considered as a primary source when it is used for this purpose.

3.1.2 SECONDARY DATA SOURCES

Secondary source data are data that was originally collected for other purposes. The data from these existing sources are considered secondary data. Although these sources were not created for the purpose at hand, they facilitate the development of the required indicators. Data from a census, research, information system, etc., are secondary source data.

3.1.3 ADVANTAGES AND DISADVANTAGES OF PRIMARY AND SECONDARY DATA SOURCES

The advantages and disadvantages of using primary and secondary data sources in preparing and monitoring health indicators are shown in Figure 3.

3.2 ATTRIBUTES TO CONSIDER IN SELECTING SECONDARY DATA SOURCES

If the secondary sources of the data required to develop the indicators to be monitored, the characteristics of the data must be taken into consideration prior to their selection. The relevant attributes for selecting secondary data sources to generate population-based health indicators are described below. However, these attributes can and should be evaluated in the context of the purpose for which the data will be used. In addition, the selection of secondary source data should also consider the advantages and disadvantages outlined in section 3.1. The attributes are:

• POPULATION REPRESENTATIVENESS: Representativeness is an attribute that involves the absence of selection bias with respect to the population that the indicator is intended to represent. Non-representative samples (such as convenience samples or samples based on sentinel units), samples with high rates of non-response, or samples that reflect underreporting in information systems, are examples of factors that can compromise the representativeness of a data source. For example, a country's live birth information system is a universal system, because it is supposed to include all children born alive in all types of facilities or birthing sites. However, it is known that births in conditions of greater vulnerability (poorer regions, rural areas, areas with lack of housing, indigenous ethnicity, among other factors) might not be reported to the system. In such a case, there is a bias in the representativeness of those population groups. Similarly, research on victims of violence based on sentinel unit samples (reference health services serving such victims), might not be representative of the population. One reason is that this type of sampling systematically excludes victims with less severe injuries or with fatal injuries but were not treated in a health facility.
• PERIODICITY: Data can be compiled continuously in systems such as civil registries, cancer registries, and surveillance systems for reportable diseases. Data can also be compiled periodically, which is to say at regular intervals (for example, 10-year population census, triennial survey of schoolchildren), or without predefined periodicity; and at a particular point in time (for example, health surveys on specific subjects, academic research projects, etc.). Although specific health-related studies are recognized as useful sources of important information for developing specific indicators, their usefulness for monitoring long-term indicators is limited. Nevertheless, a combination of various specific research studies can serve to indicate trends, even if the studies do not provide for ideal methodological comparability. One example is research on the prevalence of smoking conducted with different methodologies and target populations. Such research can, nevertheless, provide the general direction of a trend, with the caveat that the relevant limitations must be taken into account.
• VALIDITY: This refers to the ability of the source to measure what is intended to be measured (absence of distortions, bias, or systematic errors). The most relevant biases are those related to selection of the study population and the quality of the information compiled. The data source should include the variables needed to develop the indicator. An example is a live birth information system that includes data on congenital malformations (including microcephaly). In general, observations at birth without supplementary examination and monitoring of the children tend to underestimate the prevalence of congenital malformation. Although the system may be quite valid as a database for a series of other indicators, it is not valid to estimate the prevalence of congenital malformations in children.
• TIMELINESS: The timeliness of the source involves the availability and reliability of the data at the time it is needed to construct the indicators. Thus, timely produced indicators provide better opportunities for making health-related decisions.
• STRATIFICATION: Many health-related problems require indicators that are stratified according to population subgroups or by areas of particular interest. Multiple analytical interpretations can be derived from the level of disaggregation available in the selected data source. These considerations can significantly expand or limit the use of the indicator for decision-making.
• SUSTAINABILITY: This attribute represents the source's potential to remain relevant and be of the quality needed to generate information over time. This depends not only on the periodicity of the data collection, but on the availability of the financial resources needed to sustain that particular source of data; the presence of a legal framework; political will, among other factors. Surveys conducted by telephone tend to be more sustainable because they require fewer resources. However, telephone surveys have limitations not found in surveys based on personal interviews and biometrical measurements.
• PRECISION: Some well-designed probabilistic samples that ensure representativeness have some degree of imprecision-a factor to be considered in any sample-based indicator. Imprecision may arise, for example, when calculating confidence intervals that inform the user (usually with 95% confidence) of the plausible value of an indicator as applied to the population from which the sample was taken. Indicators developed from census sources, such as, population censuses, universal data sources, and vital statistics information systems, etc., are free of imprecision.
• ACCESS TO DATA: This refers to ensuring the availability of data to the public through national data repositories and other means.

3.3 MAIN DATA SOURCES

The main data sources include demographic censuses, vital statistics information systems, disease reporting systems, cancer registries, population-based research, and other sample-based research—which may be local (subnational) and/or periodic (regular)—and the various information systems created by the health and other sectors for administrative purposes.

Below is a description of the most frequently used data sources for the development of health indicators:

3.3.1 DEMOGRAPHIC CENSUSES

In most countries, the demographic census is the most commonly used data source on the characteristics of a population. A census is of paramount importance for preparing indicators and planning health interventions. Other demographic data sources are household censuses, civil registries, and national estimates for variables of interest. Demographic data are necessary for calculating many health-related indicators.

The data in a national census include: (a) total population, by sex, age, and ethnic origin; (b) increase in population; (c) rural and urban distribution of the population; and (d) dependency ratio. In addition, demographic censuses become secondary data sources when used to establish the denominators of many health indicators (rates, proportions, ratios): mortality; incidence and prevalence of diseases, accidents and violent acts; prevalence of risk factors for accidents and violence, as well as sequelae of these events; and ratio of hospital beds to population.

The growing need for information has made censuses an essential tool for countries' information systems. They serve a political, administrative, technical, and scientific purpose. Such data are compiled for the entire population, through personal interviews. In most countries, censuses represent periodic data that are collected every ten years, with publication of the information they yield approximately two years later.

One limitation in the use of census data is the possibility of inaccuracies in population estimates for the inter-censal years. These estimates tend to lose accuracy the further they are from the year of the census. In addition, the calculations are subject to changes as new demographic information is generated. There are various methods for making these estimates, each based on its own assumptions, but all include the basic demographic factors of fertility, mortality, and migration (1).

With major changes in the factors affecting a country's demographic structure, existing methodologies can become problematic, particularly with respect to population projections. There is the difficulty, for example, of making adequate population projections based on the 10-year demographic censuses given the steep decline in fertility rates that occurred between 1980 and 2010. Population migrations resulting from conflicts or economic factors can also affect population projections. In addition, these factors can affect estimates of health indicators since population data from demographic censuses and the corresponding projections are used as the denominator. For these reasons, if indicators are to be monitored over time, they should be recalculated retrospectively as new population projections are made.

3.3.2 HEALTH INFORMATION SYSTEMS

National health information systems provide data for health-related events; these systems also provide some census-related data. National health information systems can have sub-systems that address specific health events to include mortality, births, notifiable diseases, cancer and other disease registries.

• Mortality information systems: All of the countries in the Region of the Americas are required to report all deaths. In some countries, this requires the completion a form known as the "declaration of death," followed by registration of the event in the civil registry system where a death certificate (a legal document), is issued. WHO has established an international form of medical certificate of cause of death. It contains a minimum set of variables that should be included in a death certificate. These include the underlying, intermediate, and immediate causes of death. Most countries use the International Classification of Diseases (ICD) (2) to code the causes of death. This permits comparisons between countries and at different points in time. In some countries, especially in remote areas, the reporting of deaths is incomplete, thereby compromising the representativeness of mortality statistics. The proportion of under-reporting of deaths and the proportion of deaths due to ill-defined causes are indicators of the quality of mortality statistics (3-5).
• Birth registration systems: These are a source for census data and for the collection of vital statistics. In all of the countries in the Region of the Americas, it is compulsory for every live birth to be reported. In most of these countries, births occur in health facilities where a form known as the "declaration of live birth," is issued. Subsequently, the birth must be recorded in the civil registry where a birth certificate, a legal document, is issued. Declarations of births generate data for formulating indicators that are highly useful in monitoring maternal and child health during the prenatal, delivery, and perinatal periods, while also providing information on a population's fertility profile. Standardizing definitions, forms, and variables of interest facilitate comparisons between countries and over time. The main limitation in working with birth indicators is the fact that coverage may be incomplete, particularly in remote areas within certain countries thereby compromising the representativeness of the resulting statistics. The proportion of unregistered births and of incomplete information on important variables are indicators of the quality of birth registration (6).

• Public health surveillance systems: According to WHO, "public health surveillance is the continuous, systematic collection, analysis, and interpretation of health-related data needed for the planning, implementation, and evaluation of public health practice." (7). These systems are useful sources of data for developing morbidity indicators regarding the prevention and control of communicable diseases, noncommunicable diseases, accidents, and violence. Surveillance systems for communicable diseases play a key role in providing early warning of possible threats to public health and make it possible to monitor measures and programs for prevention and control. Effective national surveillance and response systems are therefore essential for national, regional, and global health security.

Most of the Region's countries have national surveillance systems established by their ministries of health. These differ, however, from country to country with regard to the number of diseases under surveillance, the type of information compiled, the use of electronic or hardcopy records, and the systems' coverage. Surveillance of noncommunicable diseases can be conducted by health care services (with information from health institutions) or using population-based health surveys. Additionally, surveillance can occur through disease-specific programs such as for tuberculosis and vaccine-preventable diseases. In some countries, the surveillance of certain diseases may not be integrated into the national surveillance system. Notably, surveillance case definitions can change over time in response to changes in the characteristics of an epidemic, as in the case of HIV/AIDS and H1N1. Another surveillance tool is the International Health Regulations 2005 (IHR) which calls for the monitoring of diseases that have the potential to cross borders and threaten people worldwide.

In recent years, surveillance systems have been used to monitor a broad range of health conditions, risk factors, and other public health issues. Some countries have implemented surveillance registries for noncommunicable diseases (cervical cancer, acute myocardial infarction, violence, diabetes, etc.) or for health problems caused by toxic substances in the environment. These registries compile information on all cases of these diseases through the health care network or through sentinel surveillance units. Such systems are a source of data on morbidity indicators that are useful in disease prevention and control efforts (8).

The collection, analysis and interpretation of surveillance data and indicators on immunization were essential in formulating strategies to control and eliminate vaccine-preventable diseases in the Region. Notably, PAHO is the first WHO region certified as polio-free and the first to have interrupted the endemic transmission of measles and rubella.

• Population-based cancer registries: These are the gold standard for providing information on this disease. Cancer registries compile, classify, analyze and report information on all cancers occurring in a geographically defined population. The information is collected from a multiplicity of sources, including hospitals, clinical analysis laboratories, and departments of vital statistics. Routine calculation of rates (per 100,000 inhabitants) produced by population-based cancer registries provides information that can assist public health officials to better understand the disease, its treatment, and to evaluate cancer prevention and control programs. This information provides a solid basis for planning and implementation of programs to reduce the cancer burden as well as provide input for research.

3.3.3 ROUTINE DATA FROM HEALTH FACILITIES

As part of their normal activities, health facilities generate highly useful data on the delivery of services (hospitalizations, consultations, surgeries, insertion of prostheses, etc.), coverage of health interventions, resources (number of physicians, nurses, and beds, number of vaccines given, health sector expenditures, etc.), and disease patterns. For a number of indicators-such as access to health services and the cesareansection rate-data from health facilities are the sole source. Moreover, health facilities are important sources of subnational data, by province, region, or district that have a direct bearing on management decisions.

Within the Region, particularly in the Caribbean, the countries have shown an increasing interest in implementing national information systems on hospitalizations, ambulatory and emergency services, and in the use of electronic medical records. Some countries, such as Belize and Dominica, have made major investments in these areas.

Since these data sources tend be created for administrative purposes rather than for monitoring of health events per se, the epidemiological data they generate may have biases that affect their coverage, quality, and usefulness. In some of the countries, these systems are fragmented, and information exchange between the public and private sectors is difficult, leading to late, incomplete, and/or incorrect reports. Despite these challenges, there is reason for optimism, given the growing introduction of new information technologies (electronic medical records, etc.) with the potential to substantially improve the quality and use of data for decision-making (9).

Data on human resources in health can be obtained through professional associations. However, the data can be distorted if the information on deaths, migration, type of training, and current occupation, etc., are not periodically updated.

3.3.4 POPULATION SURVEYS

Population surveys have become an important source of information on health status and the social determinants of health. They can provide data for several purposes such as risk factors; access to (and use of) services; medication availability and use; morbidity, mental health; violence and injuries; disability; drug abuse; reproductive health; work conditions; lifestyles, etc.

These surveys can estimate prevalence rates for diseases, risk factors, behavior, prior conditions, situations of vulnerability, knowledge, attitudes and habits regarding various health-related practices, use of health services, etc. The target population can be a general population defined by its geographic area, or a specific population defined by some attribute of interest such as adolescents, schoolchildren, women of childbearing age, males in specific age groups, persons deprived of liberty, etc.

An example of a population survey is the Global Tobacco Surveillance System (GTSS). This is an exhaustive survey that monitors tobacco use and the policies enunciated in the WHO Framework Convention on Tobacco Control (WHO/FCTC), though that system is not designed to facilitate comparisons with other factors. An example of a multi-thematic survey is the WHO STEPwise approach to surveillance, which has the characteristics of a general survey. This survey encompasses a multiplicity of issues for monitoring within-country trends. It permits the compilation of biological and physical measures, though it provides only limited possibilities for close monitoring of the entire range of issues, a constraint that particularly affects the ability to monitor policy implementation (10).

Survey data can be gathered through various types of communication, including personal interviews (household surveys), telephone interviews, and questionnaires completed by the subjects themselves. Population surveys make it possible to monitor the population over time using probabilistic samples—a method that involves lower cost and simpler fieldwork—or using a more complex and costly structure. Surveys conducted in schools have been used globally and are simple and relatively inexpensive way of surveying a population. This is in contrast to household surveys, which involve a more complex structure and process.

Household surveys can be used as data sources to estimate indicators regarding determinants of health, health status, and trends in health expenditures at the national and individual levels.

Constraints in using surveys as data sources include the following:
a) people provide information based on their own evaluation of their health and illnesses; b) in terms of response rates, the success of a survey depends on its planning and on interviewees' willingness to provide information; and c) representativeness and disaggregation of data can considerably increase the cost of a survey (10-11). Additionally, sustainability due to a scarcity of resources or political will can be an obstacle.

Many countries in the Region have conducted household surveys on different areas of interest. Some countries establish national health surveys as a key source of information on various health-related issues. Nevertheless, it is important that countries continue their efforts to strengthen national capacities, create sustainable surveillance systems, and provide adequate budgetary allocations to ensure the production of reliable, systematic, standardized, and timely national health information.

Figure 4 shows a selection of national and international surveys that have been conducted in the Region of the Americas to provide information on non-communicable diseases and risk factors.

Source: WHO, World Health Organization, Centers for Disease Control and Prevention (CDC); FCTC, WHO Framework Convention on Tobacco Control; United States Agency for International Development (USAID); United Nations Office on Drugs and Crime (UNODC); Organization of American States (OAS); Inter-American Drug Abuse Control Commission (CICAD); United Nations Children's Fund (UNICEF); human immunodeficiency virus (HIV).

3.4 MAPPING OF DATA SOURCES

A mapping of data sources consists of an inventory of sources available at the level of the country, state, province, or other geographic/administrative unit (12). It involves identifying and describing all the data sources available in the country in order to determine the possible sources that can be used to develop the health indicators. Mapping can help identify gaps and the need for new and/or complementary data sources to develop appropriate health indicators. If there is a paucity of data in a particular area or program component, it is important to collect the data for subsequent analysis. The mapping of data sources involves:

• Constructing a list (by type) of all available data sources used in formulating health-related indicators in the country together with the time period covered by the data set. Where relevant, preparing the same list with additional sources at the state, province, and community levels.
• Constructing a table with information on each source's level of disaggregation, insofar as this is pertinent for the analysis of health indicators.

RELATED LINKS

• List of national statistics institutes in Latin America, available on the web page of the Latin American and Caribbean Demographic Centre (CELADE)
• International Statistical Classification of Diseases and Related Health Problems. Tenth revision. 2008 edition. Volume 1.
• Strengthening civil registration and vital statistics for births, deaths, and causes of deaths - Resources kit
• World Health Organization. International Health Regulations.
• Mortality statistics: a tool to improve understanding and quality
• Improving the quality and use of birth, death and cause-of-death information: guidance for a standards-based review of country practices
• Global Youth Tobacco Survey and CDC
• Global school-based student health survey
• STEPwise Method (Survey of Risk Factors)
• Global Adult Tobacco Survey and CDC
• Demographic Health Survey
• Estudio sobre la medición de los niveles de vida
• Survey on drug abuse
• Multiple Indicator Cluster Survey
• Behavioral Risk Factor Surveillance System

REFERENCES

1. Laurenti R, Mello Jorge MHP, Lebrão ML, Gotlieb S. Estatísticas de Saúde 2nd edition. Chapter 4. Registro dos eventos vitais. E.P.U. São Paulo: Editora Pedagógica e Universitária Ltda; 2005.

2. World Health Organization/Pan American Health Organization. International Classification of Diseases and Related Health Problems. Tenth revision. 2015 edition. 3 vol. Washington, DC: WHO/PAHO; 2016.

3. World Health Organization (WHO). Health Metric Network. Framework standards for country health information systems. 2nd ed. Geneva: WHO; 2007.

4. World Health Organization (WHO). Handbook on Health Inequality Monitoring with a special focus on low- and middle-income countries. Geneva: WHO; 2013. Available at: http://who.int/gho/health_equity/handbook/en/ [consulted 9 August 2017].

5. World Health Organization (WHO). Mortality statistics: a tool to improve understanding and quality. Geneva: WHO; 2013. Available at: www.who.int/healthinfo/tool_cod_2010.pdf [consulted 7 June 2017].

6. World Health Organization (WHO). Improving the quality and use of birth, death and cause-of-death information: guidance for a standards-based review of country practices. Geneva: WHO; 2010. Available at: https://www.who.int/healthinfo/tool_cod_2010.pdf [consulted 7 June 2017].

7. World Health Organization (WHO). Global status report on noncomunicable diseases 2010. Available at: https://www.who.int/nmh/publications/ncd_report_full_en.pdf

8. McQueen D; Puska P. Global Behavioral Risk Factor Surveillance. New York: Kluwer Academic/Plenum Publishers; 2003.

9. Centers for Disease Prevention and Control. Principles of Epidemiology in Public Health Practice, 3rd edition; 2006. Available at: https://www.cdc.gov/ophss/csels/dsepd/ss1978/ss1978.pdf

10. United Nations Statitics Division. Household Sample Surveys in Developing and Transition Countries, 2005. Available at: https://unstats.un.org/unsd/hhsurveys/

11. World Health Organization. Health facility and community data toolkit. Available at: https://www.who.int/healthinfo/facility_information_systems/en/ [consulted 27 June 2017].

Back to Main Menu